More thoughts on Lua 5.2

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

More thoughts on Lua 5.2

Duck-2
I like what I have heard so far, in particular the implementation of a 
library like 'struct' (pack and unpack arbitrary binary data) in the core. 
(Don't forget to fix the buffer overflow I reported a few days ago :-)

I would like to reiterate my suggestion of introducing _three_ sorts of 
official Lua library, let's call them core, auxiliary and optional.

Core and auxiliary would be restricted to ANSI-C only, which isn't _quite_ 
the case at the moment. 

Optional libraries would contain officially-supported additional library 
code which might not be supported or portable to all platforms, and which 
IMO should start off with at least the following:

1. Dynamic library loading. (This is the current anomaly to the 'ANSI-C 
only' rule. With the introduction of 'optional' libraries, it gets a more 
logical home in the code tree.)

2. The C part of LuaSocket. (Leave out the Lua parts, so it's smaller, and 
can easily be compiled in if desired. UDP and TCP socket support is almost 
_sine qua non_ for a scripting language these days.)

3. An I/O library compatible with 'io' but using the OS's own low-level 
routines, not FILE *. (Inability to work with files > 2GB is a great 
annoyance when using Lua on Linux or Windows. Point it at such a file and 
it's lost -- seems a rather depressing singularity to me.)

4. A library like lfs. (But written to be properly and portably 
OS-agnostic, not as a Unix library of which large parts work on Windows, 
but some parts just return a 0 or nil placeholder answer.)

A few months ago I would have added...

5. Some sort of support for pre-emptive multiprogramming, probably a 
simple, message-passing solution such as LuaTask

...but I accept that's something which might not belong in an 
officially-supported library set.

Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Evan DeMond
On Thu, Feb 21, 2008 at 3:44 PM, Duck <[hidden email]> wrote:
I would like to reiterate my suggestion of introducing _three_ sorts of
official Lua library, let's call them core, auxiliary and optional.

Core and auxiliary would be restricted to ANSI-C only, which isn't _quite_
the case at the moment.

Optional libraries would contain officially-supported additional library
code which might not be supported or portable to all platforms, and which
IMO should start off with at least the following:

[...]

Just curious, I guess - what's the rationale for making these part of an official distribution?

Evan
Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Miles Bader-2
In reply to this post by Duck-2
Duck <[hidden email]> writes:
> 3. An I/O library compatible with 'io' but using the OS's own low-level 
> routines, not FILE *. (Inability to work with files > 2GB is a great 
> annoyance when using Lua on Linux or Windows. Point it at such a file and 
> it's lost -- seems a rather depressing singularity to me.)

Er, shouldn't the C library be able to handle large files using FILE?

-Miles

-- 
What the fuck do white people have to be blue about!?  Banana Republic ran
out of Khakis?  The Espresso Machine is jammed?  Hootie and The Blowfish are
breaking up??!  Shit, white people oughtta understand, their job is to GIVE
people the blues, not to get them!    -- George Carlin

Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

David Given
Miles Bader wrote:
Duck <[hidden email]> writes:
3. An I/O library compatible with 'io' but using the OS's own low-level routines, not FILE *. (Inability to work with files > 2GB is a great annoyance when using Lua on Linux or Windows. Point it at such a file and it's lost -- seems a rather depressing singularity to me.)

Er, shouldn't the C library be able to handle large files using FILE?

Yes, but you typically have to use a slightly different set of APIs --- fseeko() instead of fseek(), for example, in order to get the 64-bit file offset.

Also, 64 bit file offsets can't be expressed inside a standard Lua number (which only have 32 bits of integer precision). I'm not sure how you'd work around this.

--
David Given
[hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Duck-2
In reply to this post by Duck-2

Er, shouldn't the C library be able to handle
large files using FILE?

Depends. fseek() takes a long (i.e. a signed long) as the file offset, so if long is 32-bits then you can't get past 2GB-1.

Anyway, a default build of Lua on my 32-bit Linux has a 2GB-1 size limit on files it can process. (My QEMU virtual machine disk images are often bigger than that.)


Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Luiz Henrique de Figueiredo
In reply to this post by David Given
> Also, 64 bit file offsets can't be expressed inside a standard Lua 
> number (which only have 32 bits of integer precision).

IEEE doubles can store 52 bits of integer precision (but it does not
help in this case).

Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

steve donovan
52-bit integers represent a _very_ big file ;)

steve d.

On Fri, Feb 22, 2008 at 2:00 PM, Luiz Henrique de Figueiredo
<[hidden email]> wrote:
> > Also, 64 bit file offsets can't be expressed inside a standard Lua
>  > number (which only have 32 bits of integer precision).
>
>  IEEE doubles can store 52 bits of integer precision (but it does not
>  help in this case).
>

Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

David Given
In reply to this post by Luiz Henrique de Figueiredo
Luiz Henrique de Figueiredo wrote:
Also, 64 bit file offsets can't be expressed inside a standard Lua number (which only have 32 bits of integer precision).

IEEE doubles can store 52 bits of integer precision (but it does not
help in this case).

They certainly do --- apparently it's not my day today.

However, since 52 bit offsets are capable of handling files up to two exabytes long (including a bit for the sign; and assuming I haven't made a trivial mistake in the maths, which is certainly possible right now), I think the OP could probably solve their immediate problem by patching io to use fseeko and ftello and to pass offsets around as doubles. Call it... medium file support.

2EB of enough to store 20 years of continuous DV. I don't really know why I worked that out.

--
David Given
[hidden email]



Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Miles Bader
In reply to this post by David Given
David Given <[hidden email]> writes:
> Yes, but you typically have to use a slightly different set of APIs --- 
> fseeko() instead of fseek(), for example, in order to get the 64-bit
> file offset.

A few #ifdefs to support compiling Lua with this support seems
worthwhile to me (is there anything besides just using off_t/fseeko?),
but a whole new I/O library seems a bit over the top...

-Miles

-- 
Congratulation, n. The civility of envy.

Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

David Given
In reply to this post by David Given
David Given wrote:
[...]
However, since 52 bit offsets are capable of handling files up to two exabytes long (including a bit for the sign; and assuming I haven't made a trivial mistake in the maths, which is certainly possible right now),

You bet!

s/exa/peta/

--
David Given
[hidden email]


Reply | Threaded
Open this post in threaded view
|

RE: More thoughts on Lua 5.2

Al Pfalzgraf-2
In reply to this post by Miles Bader







In recent work it would have been helpful if the Lua number accomodated a 64 bit integer.
 
Would a change of LUA_INTEGER to fpos_t be unacceptable?
 
 
 
 


Helping your favorite cause is as easy as instant messaging. You IM, we give. Learn more.
Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Asko Kauppi

The LNUM patch is useful for this as well. It allows one to easily (with one to two -D flags) set Lua to use any combo of:

  LNUM_DOUBLE (default)         double precision FP (52 bit mantissa)
  LNUM_FLOAT                    single precision FP (23 bit mantissa)
LNUM_LDOUBLE extra precision FP (80+ bit mantissa, varies)

  LNUM_INT32                    -2^31..2^31-1 integer optimized
  LNUM_INT64                    -2^63..2^63-1 integer optimized
  (none)                        		all numbers stored as FP

LNUM_COMPLEX Complex (a+bi) number mode, scalars may be integer optimized

You'd use -DLNUM_INT64 to get 64-bit integer resolution. That's all. The patched Lua remains completely compatible to modules etc. - lua_Integer does become 64 bits wide. Numbers are stored as either 64- bit ints or doubles internally, but to your Lua code and C API it wouldn't be known.

-asko


Al Pfalzgraf kirjoitti 22.2.2008 kello 16:45:


In recent work it would have been helpful if the Lua number accomodated a 64 bit integer.

Would a change of LUA_INTEGER to fpos_t be unacceptable?


Helping your favorite cause is as easy as instant messaging. You IM, we give. Learn more.


Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Mark Hamburg-4
In reply to this post by Evan DeMond
on 2/21/08 12:51 PM, Evan DeMond at [hidden email] wrote:

> Just curious, I guess - what's the rationale for making these part of an
> official distribution?

Standardizing the API and the semantics would be the big reason.

Mark



Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Duck-2
In reply to this post by Duck-2
>>Optional libraries would contain officially-supported 
>>additional library code which might not be supported 
>>or portable to all platforms, and which IMO should start 
>>off with at least the following:

>>[OS-native file I/O + lfs + core of luasocket]

>Just curious, I guess - what's the rationale for making 
>these part of an official distribution?

It seems pretty important for a modern scripting language to have utility 
libraries allowing it to interact decently with the file system and with 
sockets. (PiL, for instance, goes as far as assuming that yu have 
LuaSocket when giving a real-world example of how to use coroutines.)

To me, this seems fundamental enough that officially-sanctioned libraries, 
built and tested routinely as part of Lua development, and shipped along 
with the Lua distro, are surely called for?

To me, there's a big difference between 'oh, you can fudge Lua to support 
64-bit file sizes by search-and-replacing all fseeks with fseekos and 
ftells with ftellos,' and 'choosing the correct file size bitness for 
your OS is, by design, an automatic part of the build process.'

Same sort of issue a recent poster had with the matter of building a 
floating-point-free Lua. Everyone gave theoretical details of how easy it 
is to do by tweaking luaconf -- except he'd tried this and found that it 
gave only a floating-point-reduced Lua. There can be a big difference 
between official and theoretical features in a language distro...

Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Rob Kendrick
In reply to this post by Duck-2
On Fri, 2008-02-22 at 22:44 +1100, Duck wrote:
> >Er, shouldn't the C library be able to handle
> >large files using FILE?
> 
> Depends. fseek() takes a long (i.e. a signed long) as the file offset, so 
> if long is 32-bits then you can't get past 2GB-1.
> 
> Anyway, a default build of Lua on my 32-bit Linux has a 2GB-1 size limit 
> on files it can process. (My QEMU virtual machine disk images are often 
> bigger than that.)

Well, you can: you use fseek(..., SEEK_CUR); iteratively and keep track
of the pointer yourself.

B.


Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Duck-2
In reply to this post by Duck-2
>Well, you can: you use fseek(..., SEEK_CUR); 
>iteratively and keep track of the pointer yourself.

Hmmm. Can you? I can't figure out how to be able to do this using a 
standard build of Lua:

> f,e,n = io.fopen('my3gbfile.raw')
> = f,e,n
> nil     my3gbfile.raw: File too large    27

Now, AFAIK you can open up and manually move around in 64-bit-sized files 
using 32-bit libraries (and offsets) if you use O_LARGEFILE when calling 
open(), except that we aren't using OS-level I/O, were using ANSI-C-level 
FILE *s, so we're calling fopen instead().

So, you could recompile everything with _FILE_OFFSET_BITS=64, but...

...is this supported? Was Lua built with it in mind? Can I expect it to 
work or do I need to reinvent my own tests and regression tests every time 
I update my Lua build? What other limits on my ability to access regular 
files in my filing system might exist in Lua's standard library code? What 
things won't work in the real world? 

(Path length is another problem for the pure ANSI-CD-based io library, 
seeing as it doesn't even admit of paths. Even lfs is somewhat hamstrung 
in this regard, with built-in hard wired string length limits such as 
256. Sure, you can re#define some of them, but to what? And why? And who 
has tested that before?)

My point was not so much that I wanted a way to access files bigger than 
2GB (or to make use of functions relating to the hierarchical file system 
which almost all Lua platforms support, or to use sockets), but that I was 
suggesting it might be great to have all these things available _as 
optional libraries that are provided out of PUC-Rio as part of the "this 
fits together as a whole" distro that is Lua_.

I trust LuaSocket and use it happily. But I'd be even happier if it were 
currently at LuaSocket 5.1.3, delivered as part of Lua 5.1.3. Same for 
lfs, which is a separate project, with a different outlook to Lua itself 
on what portability means. (lfs works on Windows, but it isn't really 
portable between Linux and Windows. Indeed, lfs is UNIX/Linux specific, 
but can be compiled "in part" so that a useful subset of it happens to 
work on Windows.)

Several respondents have explained to me how I can solve my >2GB 
"problem", mostly by hacking the Lua sources. But original suggestion was 
of a few additional libs which would help to create a (more useable and 
attractive) Lua distro in which that commonly-needed or expected hacks of 
that sort would simply not be necessary.


Reply | Threaded
Open this post in threaded view
|

Re: More thoughts on Lua 5.2

Fabio Mascarenhas
On Mon, Feb 25, 2008 at 2:05 AM, Duck <[hidden email]> wrote:
>
> Now, AFAIK you can open up and manually move around in 64-bit-sized files
> using 32-bit libraries (and offsets) if you use O_LARGEFILE when calling
> open(), except that we aren't using OS-level I/O, were using ANSI-C-level
> FILE *s, so we're calling fopen instead().
>
> So, you could recompile everything with _FILE_OFFSET_BITS=64, but...
>
> ...is this supported? Was Lua built with it in mind? Can I expect it to
> work or do I need to reinvent my own tests and regression tests every time
> I update my Lua build? What other limits on my ability to access regular
> files in my filing system might exist in Lua's standard library code? What
> things won't work in the real world?

Reimplementing the IO library to use POSIX functions is not hard, and
requires no messing with Lua internals. It s even easier if you write
the bulk of it in Lua itself; I have a reimplementation of the IO
library using POSIX non-blocking IO that is less than 500 lines of Lua
code (using Alien to call the POSIX functions). It would be trivial to
make it support large files after I add "long long" support to Alien.
lfs.lock and lfs.unlock won't work with this IO lib, but they are also
trivial to reimplement...

> (Path length is another problem for the pure ANSI-CD-based io library,
> seeing as it doesn't even admit of paths. Even lfs is somewhat hamstrung
> in this regard, with built-in hard wired string length limits such as
> 256. Sure, you can re#define some of them, but to what? And why? And who
> has tested that before?)

The path length limitation to lfs.currentdir() was removed in the last version.

> I trust LuaSocket and use it happily. But I'd be even happier if it were
> currently at LuaSocket 5.1.3, delivered as part of Lua 5.1.3. Same for
> lfs, which is a separate project, with a different outlook to Lua itself
> on what portability means. (lfs works on Windows, but it isn't really
> portable between Linux and Windows. Indeed, lfs is UNIX/Linux specific,
> but can be compiled "in part" so that a useful subset of it happens to
> work on Windows.)

Have you checked LuaRocks? It works really well (apart from some
quirky defaults) and the number of available packages keeps growing,
and it is doubles as a great build tool to make binary packages for
Windows. The Lua team can keep the Lua core as is for the embedders
and the "Lua as platform" folks can happilly install LuaRocks and
bloat out Lua installations with non-portable packages. :-)

> Several respondents have explained to me how I can solve my >2GB
> "problem", mostly by hacking the Lua sources. But original suggestion was
> of a few additional libs which would help to create a (more useable and
> attractive) Lua distro in which that commonly-needed or expected hacks of
> that sort would simply not be necessary.

No patching to the Lua core is necessary; pathing the Lua libraries is
only necessary if you want to reuse their code, but nothing stops you
from replacing them wholesale, as long as you keep the API for scripts
the same (messing with other libraries internals is frowned upon, lfs
being an exception that has already caused problems in Windows). So
libraries can fix a lot of the shortcomings, and I think having a good
way to deliver them obviates the need to have them bundled with the
core.

--
Fabio Mascarenhas