Lua 5.3: file:seek() and files larger than 2 GB

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua 5.3: file:seek() and files larger than 2 GB

Dmitry V. Zaitsev
*file:seek("end")* returns -1 if the file is more than 2,147,483,647 bytes.
*file:seek("set", offset)* return "not an integer in proper range" if *offset* > 2,147,483,647

This happens regardless of the 32/64 bit compiler. Maybe there is some *define* when compiling?
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

William Ahern
On Fri, May 15, 2015 at 10:58:45PM +0300, Dmitry V. Zaitsev wrote:
> *file:seek("end")* returns -1 if the file is more than 2,147,483,647 bytes.
> *file:seek("set", offset)* return "not an integer in proper range" if
> *offset* > 2,147,483,647
>
> This happens regardless of the 32/64 bit compiler. Maybe there is some
> *define* when compiling?

Odd that you get the same behavior when compiling for 64-bit. What platform
are you using?

Try building Lua with -D_FILE_OFFSET_BITS=64.


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

Luiz Henrique de Figueiredo
> Try building Lua with -D_FILE_OFFSET_BITS=64.

That should have been automatically defined for POSIX platforms.
How did you build Lua 5.3?

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

Dmitry V. Zaitsev
I'm use Win7 x64, GCC version 4.9.2 (x86_64-win32-seh-rev1, Built by MinGW-W64 project)

but _FILE_OFFSET_BITS defined only in lprefix.h and no longer used anywhere. Аs I understand, the problem in this:

/* liolib.c */

#elif defined(LUA_USE_WINDOWS) && !defined(_CRTIMP_TYPEINFO) \
   && defined(_MSC_VER) && (_MSC_VER >= 1400) /* }{ */

/* Windows (but not DDK) and Visual C++ 2005 or higher */
#define l_fseek(f,o,w) _fseeki64(f,o,w)
#define l_ftell(f) _ftelli64(f)
#define l_seeknum __int64

#else /* }{ */

/* ISO C definitions */
#define l_fseek(f,o,w) fseek(f,o,w)
#define l_ftell(f) ftell(f)
#define l_seeknum long

#endif /* } */ 

If I define _MSC_VER, it is utilized the 64-bit functions, MinGW can use them, but not by default.

2015-05-16 0:20 GMT+03:00 Luiz Henrique de Figueiredo <[hidden email]>:
> Try building Lua with -D_FILE_OFFSET_BITS=64.

That should have been automatically defined for POSIX platforms.
How did you build Lua 5.3?


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

Luiz Henrique de Figueiredo
> _FILE_OFFSET_BITS defined only in lprefix.h and no longer used anywhere.

Defining _FILE_OFFSET_BITS affects the inclusion of stdio.h, which is done
in liolib.c. That is the reason for lprefix.h: to define symbols *before*
including standard headers.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

Dmitry V. Zaitsev
I tried to build Lua with and without -D_FILE_OFFSET_BITS=64it did not help. Functions file:read() and file:write() work fine in this example:

local f = io.open("very_big_file.bin", "w+b")

print("1)", f:seek("end", -4))
f:write(string.pack("L", 12345678))

print("2)", f:seek("end", -4))
print("3)", string.unpack("L", f:read(4)))

print("4)", f:seek("set"))

f:close()

Output:

1)      -1
2)      -1
3)      12345678        5
4)      0

But file:seek() returns and takes the numbers of no more than 2^31-1. I do not see any other way except to override l_seeknum to 64-bit type.


2015-05-16 1:11 GMT+03:00 Luiz Henrique de Figueiredo <[hidden email]>:
> _FILE_OFFSET_BITS defined only in lprefix.h and no longer used anywhere.

Defining _FILE_OFFSET_BITS affects the inclusion of stdio.h, which is done
in liolib.c. That is the reason for lprefix.h: to define symbols *before*
including standard headers.


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

Roberto Ierusalimschy
> I tried to build Lua with and without -D_FILE_OFFSET_BITS=64, it did not
> help. Functions file:read() and file:write() work fine in this example:

FILE_OFFSET_BITS is a POSIX thing, I do not think it affects Windows.


> If I define _MSC_VER, it is utilized the 64-bit functions, MinGW can use
> them, but not by default.

Isn't this a solution?

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

Dmitry V. Zaitsev
> If I define _MSC_VER, it is utilized the 64-bit functions, MinGW can use
> them, but not by default.

Isn't this a solution?
 
use of this define globally (such as make MYCFLAGS="-D_MSC_VER=1400" mingw) leads to a large number of errors at link time. and just have to do a patch, it is best to do so:

liolib.c:
 #if !defined(l_fseek) /* { */
  
- #if defined(LUA_USE_POSIX) /* { */
+ #if defined(LUA_USE_POSIX) || defined(__MINGW32__) /* { */
  
  #include <sys/types.h>
  
  #define l_fseek(f,o,w) fseeko(f,o,w)
  #define l_ftell(f) ftello(f)
  #define l_seeknum off_t

tested in MinGW 4.9.2 in Ubuntu and Windows (both x86_64 and i686 version). 
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.3: file:seek() and files larger than 2 GB

Roberto Ierusalimschy
> > > If I define _MSC_VER, it is utilized the 64-bit functions, MinGW can use
> > > them, but not by default.
> >
> > Isn't this a solution?
> >
>
> use of this define globally (such as make MYCFLAGS="-D_MSC_VER=1400" mingw)
> leads to a large number of errors at link time. and just have to do a
> patch, it is best to do so:
>
> [...]

With a little more work, you can do the patch in 'luaconf.h', which is
an "official" place to do patchesj by only adding these lines in its
"Local configuration" section:

  #if defined(liolib_c) && defined(__MINGW32__)
  #include <sys/types.h>

  #define l_fseek(f,o,w) fseeko(f,o,w)
  #define l_ftell(f) ftello(f)
  #define l_seeknum off_t
  #endif

-- Roberto