luac and loadstring hang

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

luac and loadstring hang

Shmuel Zeigerman
The string below, when fed to loadstring, causes the interpreter to
hang. (Same behavior if this string is saved to file and then fed to
luac). Lua 5.1.4, Windows XP SP2.

-- test begin --
local str =
   "\27\76\117\97\81\0\1\4\4\4\8\0\7\0\0\0\64\49\46\108\117\97" ..
   "\0\0\0\0\0\0\0\0\0\0\0\2\2\3\0\0\0\1\0\0\14\0\0\0\30\0\128" ..
   "\0\1\0\0\0\3\0\0\0\0\0\0\23\64\0\0\0\0\3\0\0\0\1\0\0\0\2\0" ..
   "\0\0\2\0\0\0\1\0\0\0\2\0\0\0\102\0\1\0\0\0\2\0\0\0\0\0\0\0"

loadstring(str)

--
Shmuel
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Luiz Henrique de Figueiredo
> The string below, when fed to loadstring, causes the interpreter to
> hang. (Same behavior if this string is saved to file and then fed to
> luac). Lua 5.1.4, Windows XP SP2.
>
> -- test begin --
> local str =
>    "\27\76\117\97\81\0\1\4\4\4\8\0\7\0\0\0\64\49\46\108\117\97" ..
>    "\0\0\0\0\0\0\0\0\0\0\0\2\2\3\0\0\0\1\0\0\14\0\0\0\30\0\128" ..
>    "\0\1\0\0\0\3\0\0\0\0\0\0\23\64\0\0\0\0\3\0\0\0\1\0\0\0\2\0" ..
>    "\0\0\2\0\0\0\1\0\0\0\2\0\0\0\102\0\1\0\0\0\2\0\0\0\0\0\0\0"
>
> loadstring(str)

changing the last line to print(loadstring(str)) gives me this:

nil binary string: bad constant in precompiled chunk

but in an older machine it did hang. (Both are Linux machines.)
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Shmuel Zeigerman
The test string was obtained the following way:

File 1.lua
   local f = 23/4
   f = f*f

was compiled by luac.exe, then the 41-st character was deleted.

--
Shmuel
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Luiz Henrique de Figueiredo
I ran some instrumentation and it seems that the program is spending
a lot of time at one line in LoadConstants (lundump.c):

static void LoadConstants(LoadState* S, Proto* f)
{
 int i,n;
 n=LoadInt(S);                                  << n = 50331648
 f->k=luaM_newvector(S->L,n,TValue);            << no problems here
 f->sizek=n;
 for (i=0; i<n; i++) setnilvalue(&f->k[i]);     << takes a while here
 ...

In my old Pentium II with 256Mb of memory running Linux the process is
killed by the OS after a while. Roberto has suggested that it's probably
a paging problem. In newer machines with larger memory the program runs
correctly, aborting with a sensible error message (bad constant).
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Roberto Ierusalimschy
> In my old Pentium II with 256Mb of memory running Linux the process is
> killed by the OS after a while. Roberto has suggested that it's probably
> a paging problem.

Actually "optimistic memory allocation".

-- Roberto
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Luiz Henrique de Figueiredo
> > In my old Pentium II with 256Mb of memory running Linux the process is
> > killed by the OS after a while. Roberto has suggested that it's probably
> > a paging problem.
>
> Actually "optimistic memory allocation".

Right:

  "We learned that malloc should return NULL in case of out-of-memory.
  However, it is not the case in Linux. By default, Linux uses
  optimistic memory allocation strategy. Under this strategy,
  Linux assumes there always exists free memory. The memory region
  returns by malloc is not actually allocated until the process
  touches the memory region. This means the memory region returns
  by malloc may not be available. In case of out-of-memory, the
  OOM Killer in Linux will pick up one or more process to kill."
  http://ahlamnote.blogspot.com/2007/04/optimistic-memory-allocation-strategy.html

Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Rob Kendrick
On Mon, 16 Nov 2009 13:47:32 -0200
Luiz Henrique de Figueiredo <[hidden email]> wrote:

> > Actually "optimistic memory allocation".  
>
> Right:
>
>   "We learned that malloc should return NULL in case of out-of-memory.
>   However, it is not the case in Linux. By default, Linux uses
>   optimistic memory allocation strategy. Under this strategy,
>   Linux assumes there always exists free memory..."

And any Firefox or OpenOffice users should be thankful, given how much
memory these travesties allocate and then never touch :)

You can turn off this over-commit by uttering the correct rune
into /proc somewhere, if memory serves.

B.
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Shmuel Zeigerman
In reply to this post by Luiz Henrique de Figueiredo
Luiz Henrique de Figueiredo wrote:
> In my old Pentium II with 256Mb of memory running Linux the process is
> killed by the OS after a while. Roberto has suggested that it's probably
> a paging problem. In newer machines with larger memory the program runs
> correctly, aborting with a sensible error message (bad constant).

Yes, my machine is old and also has 256 MB of memory. Now that I was
testing more thoroughly, I had the test finished with an error message
on both Windows and Ubuntu, it just took a long time to finish (Windows
= 85 sec., Ubuntu = 13 sec.)

--
Shmuel
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Enrico Colombini
In reply to this post by Luiz Henrique de Figueiredo
Luiz Henrique de Figueiredo wrote:
>   "We learned that malloc should return NULL in case of out-of-memory.
>   However, it is not the case in Linux. By default, Linux uses
>   optimistic memory allocation strategy. Under this strategy,
>   Linux assumes there always exists free memory. The memory region
>   returns by malloc is not actually allocated until the process
>   touches the memory region. This means the memory region returns
>   by malloc may not be available. In case of out-of-memory, the
>   OOM Killer in Linux will pick up one or more process to kill."
>   http://ahlamnote.blogspot.com/2007/04/optimistic-memory-allocation-strategy.html

The number of things one learns just by reading this list is amazing!
Today I've already been surprised twice :-)

   Enrico
Reply | Threaded
Open this post in threaded view
|

Re: luac and loadstring hang

Florian Weimer
In reply to this post by Rob Kendrick
* Rob Kendrick:

> You can turn off this over-commit by uttering the correct rune
> into /proc somewhere, if memory serves.

vm.overcommit_memory = 2 is the magic incantation.

Some garbage collectors don't work as a result, though, because they
allocate the heap as one block at program start.  SBCL is pretty
obnoxious in this regard because it scales its heap so that you can
only start one or two instances in parallel, no matter how much
additional swap you add.

I tend to use it on most machines, though, because it makes OOM
failures somewhat more predictable.