LuaTask and memory allocation

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

LuaTask and memory allocation

Duck-2
Using LuaTask 1.6.0 statically linked into
Lua 5.1.2. (A few other packages are built-in,
namely the LuaSocket core, lfs, lrandom, bitlib
and struct.) This is on Linux.

Starting Lua, mem use (as shown by top) is about
1m resident, 2.2m allocated (virtual). Seems quite 
modest to me.

Every time I call task.create() (even if the tasks
do nothing but sleep immediately) mem usage goes
up by about 70k resident, but a whopping 8m 
of virtual memory. So with 250 tasks I am _using_ 
just 17m but have 2g allocated! Perhaps unsurprisingly,
at 257 tasks Lua segfaults...

Why 2m for Lua but an additional 8m of virtual memory
for each created task, esp. when the task is only
_using_ a modest 70k?

I was rather hoping to handle a couple of hundred 
proxied sockets inside a simple Lua program... whilst
there is plenty of physical memory to support this I
am, absurdly, running out of address space :-)

What's going on?


Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Rici Lake-2

On 19-Jun-07, at 4:21 PM, Paul Ducklin wrote:

Using LuaTask 1.6.0 statically linked into
Lua 5.1.2. (A few other packages are built-in,
namely the LuaSocket core, lfs, lrandom, bitlib
and struct.) This is on Linux.

Starting Lua, mem use (as shown by top) is about
1m resident, 2.2m allocated (virtual). Seems quite
modest to me.

Every time I call task.create() (even if the tasks
do nothing but sleep immediately) mem usage goes
up by about 70k resident, but a whopping 8m
of virtual memory. So with 250 tasks I am _using_
just 17m but have 2g allocated! Perhaps unsurprisingly,
at 257 tasks Lua segfaults...

Why 2m for Lua but an additional 8m of virtual memory
for each created task, esp. when the task is only
_using_ a modest 70k?

I was rather hoping to handle a couple of hundred
proxied sockets inside a simple Lua program... whilst
there is plenty of physical memory to support this I
am, absurdly, running out of address space :-)

What's going on?

Each task requires a C stack; the default stack allocation
on linux is quite large. There might be a way to reduce it;
I'm not al tanto with LuaTask. Other operating systems may
(by default) make a smaller C stack, but since C does not
check for stack overflow, that can be dangerous.

You might want to try using plain old Lua coroutines,
which do not require a C stack but which do have some
limitations as a result.


Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Taj Khattra
In reply to this post by Duck-2
Every time I call task.create() (even if the tasks
do nothing but sleep immediately) mem usage goes
up by about 70k resident, but a whopping 8m
of virtual memory.

8MB might be RLIMIT_STACK - see
http://lua-users.org/lists/lua-l/2006-11/msg00533.html

Reply | Threaded
Open this post in threaded view
|

How can Lua be compiled as unicode

Yangfan-2
In reply to this post by Duck-2
Hi,all    
 
  I'm working on a project compiled as unicode charset, and I embed Lua 5.1 interpreter. As you can imagine, I get a widechar string always, and will be passed to Lua functions, or I fetch a result from Lua function with the char* type, I have to convert it to wchar_t*. So inconvenient!
  So I hope to find a solution that can help compiling Lua as unicode charset. Any reply would be appreciated.
 

        Regards, 

              YangFan
              [hidden email]
               2007-06-20   


Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Asko Kauppi
In reply to this post by Rici Lake-2

Lua Lanes does the following to limit OS side stack usage.

I've tested what seems to be the minimum stack size on each OS (they do overestimate, "just in case"). Of course, one might think of a C binding that really needs huuuuuge stack size, but I doubt it in action.

Lua itself uses mostly the heap so this should be safe.


#ifndef _THREAD_STACK_SIZE
  #if (defined PLATFORM_WIN32) || (defined PLATFORM_WINCE)
    #define _THREAD_STACK_SIZE 0
      // TBD: does it work with less?

  #elif (defined PLATFORM_OSX)
    #define _THREAD_STACK_SIZE (524288/2)   // 262144
      // OS X: "make test" works on 65536 and even below
// "make perftest" works on >= 4*65536 == 262144 (not 3*65536)

  #elif (defined PLATFORM_LINUX) && (defined __i386)
    #define _THREAD_STACK_SIZE (2097152/16)  // 131072
// Linux x86 (Ubuntu 7.04): "make perftest" works on /16 (not on /32)

  #elif (defined PLATFORM_BSD) && (defined __i386)
    #define _THREAD_STACK_SIZE (1048576/8)  // 131072
// FreeBSD 6.2 SMP i386: ("gmake perftest" works on /8 (not on /16)
  #endif
#endif

...

#if (defined _THREAD_STACK_SIZE) && (_THREAD_STACK_SIZE > 0)
    PT_CALL( pthread_attr_setstacksize( a, _THREAD_STACK_SIZE ) );
#endif



Rici Lake kirjoitti 20.6.2007 kello 0:31:


On 19-Jun-07, at 4:21 PM, Paul Ducklin wrote:

Using LuaTask 1.6.0 statically linked into
Lua 5.1.2. (A few other packages are built-in,
namely the LuaSocket core, lfs, lrandom, bitlib
and struct.) This is on Linux.

Starting Lua, mem use (as shown by top) is about
1m resident, 2.2m allocated (virtual). Seems quite
modest to me.

Every time I call task.create() (even if the tasks
do nothing but sleep immediately) mem usage goes
up by about 70k resident, but a whopping 8m
of virtual memory. So with 250 tasks I am _using_
just 17m but have 2g allocated! Perhaps unsurprisingly,
at 257 tasks Lua segfaults...

Why 2m for Lua but an additional 8m of virtual memory
for each created task, esp. when the task is only
_using_ a modest 70k?

I was rather hoping to handle a couple of hundred
proxied sockets inside a simple Lua program... whilst
there is plenty of physical memory to support this I
am, absurdly, running out of address space :-)

What's going on?

Each task requires a C stack; the default stack allocation
on linux is quite large. There might be a way to reduce it;
I'm not al tanto with LuaTask. Other operating systems may
(by default) make a smaller C stack, but since C does not
check for stack overflow, that can be dangerous.

You might want to try using plain old Lua coroutines,
which do not require a C stack but which do have some
limitations as a result.



Reply | Threaded
Open this post in threaded view
|

PLEASE (Re: How can Lua be compiled as unicode)

Asko Kauppi
In reply to this post by Yangfan-2

Once again for the whole list (normally, I put these just to the individual authors):

Please DO NOT start a new thread by pressing 'reply'. Press 'new message'.

Emails have a reference field that ties responses to the original thread, even though the title and the message contents are completely remade. This will cause your message to be 'lost' among some other list, if the reader has an email client showing discussions by thread.

So it serves both your, readers, and for archiving reasons everyone's interests not to mix and match message threads.

Thanks.  Again.  :|


The message below (and all replies to it) are under the "LuaTask and memory allocation" thread.


YangFan kirjoitti 20.6.2007 kello 4:18:

Hi,all    

I'm working on a project compiled as unicode charset, and I embed Lua 5.1 interpreter. As you can imagine, I get a widechar string always, and will be passed to Lua functions, or I fetch a result from Lua function with the char* type, I have to convert it to wchar_t*. So inconvenient! So I hope to find a solution that can help compiling Lua as unicode charset. Any reply would be appreciated.


        Regards,

              YangFan
              [hidden email]
               2007-06-20   



Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Daniel Quintela
In reply to this post by Asko Kauppi
Asko Kauppi escribió:

Lua Lanes does the following to limit OS side stack usage.

I've tested what seems to be the minimum stack size on each OS (they do overestimate, "just in case"). Of course, one might think of a C binding that really needs huuuuuge stack size, but I doubt it in action.

Lua itself uses mostly the heap so this should be safe.


#ifndef _THREAD_STACK_SIZE
  #if (defined PLATFORM_WIN32) || (defined PLATFORM_WINCE)
    #define _THREAD_STACK_SIZE 0
      // TBD: does it work with less?

  #elif (defined PLATFORM_OSX)
    #define _THREAD_STACK_SIZE (524288/2)   // 262144
      // OS X: "make test" works on 65536 and even below
// "make perftest" works on >= 4*65536 == 262144 (not 3*65536)

  #elif (defined PLATFORM_LINUX) && (defined __i386)
    #define _THREAD_STACK_SIZE (2097152/16)  // 131072
// Linux x86 (Ubuntu 7.04): "make perftest" works on /16 (not on /32)

  #elif (defined PLATFORM_BSD) && (defined __i386)
    #define _THREAD_STACK_SIZE (1048576/8)  // 131072
      // FreeBSD 6.2 SMP i386: ("gmake perftest" works on /8 (not on /16)
  #endif
#endif

...

#if (defined _THREAD_STACK_SIZE) && (_THREAD_STACK_SIZE > 0)
    PT_CALL( pthread_attr_setstacksize( a, _THREAD_STACK_SIZE ) );
#endif



Rici Lake kirjoitti 20.6.2007 kello 0:31:


On 19-Jun-07, at 4:21 PM, Paul Ducklin wrote:

Using LuaTask 1.6.0 statically linked into
Lua 5.1.2. (A few other packages are built-in,
namely the LuaSocket core, lfs, lrandom, bitlib
and struct.) This is on Linux.

Starting Lua, mem use (as shown by top) is about
1m resident, 2.2m allocated (virtual). Seems quite
modest to me.

Every time I call task.create() (even if the tasks
do nothing but sleep immediately) mem usage goes
up by about 70k resident, but a whopping 8m
of virtual memory. So with 250 tasks I am _using_
just 17m but have 2g allocated! Perhaps unsurprisingly,
at 257 tasks Lua segfaults...

Why 2m for Lua but an additional 8m of virtual memory
for each created task, esp. when the task is only
_using_ a modest 70k?

I was rather hoping to handle a couple of hundred
proxied sockets inside a simple Lua program... whilst
there is plenty of physical memory to support this I
am, absurdly, running out of address space :-)

What's going on?

Each task requires a C stack; the default stack allocation
on linux is quite large. There might be a way to reduce it;
I'm not al tanto with LuaTask. Other operating systems may
(by default) make a smaller C stack, but since C does not
check for stack overflow, that can be dangerous.

You might want to try using plain old Lua coroutines,
which do not require a C stack but which do have some
limitations as a result.


Adding:

pthread_attr_setstacksize( &attr, 2097152/16 ); // Thanks Asko !!!

before pthread_create in syncos.c line 62, my test runs using 42m of virtual memory on Ubuntu 7.04

My test is a main task with 255 subtasks blocked on task.receive( - 1).

But... with more than 255 subtasks... SEG FAULT

I will try to found the bug asap...

Regards,
Daniel


Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Asko Kauppi

You can actually limit the stack size also with some environment variable (without touching code), maybe the RLIMIT_STACK that Taj Khattra mentioned.


Daniel Quintela kirjoitti 20.6.2007 kello 15:40:

Asko Kauppi escribió:

Lua Lanes does the following to limit OS side stack usage.

I've tested what seems to be the minimum stack size on each OS (they do overestimate, "just in case"). Of course, one might think of a C binding that really needs huuuuuge stack size, but I doubt it in action.

Lua itself uses mostly the heap so this should be safe.


#ifndef _THREAD_STACK_SIZE
  #if (defined PLATFORM_WIN32) || (defined PLATFORM_WINCE)
    #define _THREAD_STACK_SIZE 0
      // TBD: does it work with less?

  #elif (defined PLATFORM_OSX)
    #define _THREAD_STACK_SIZE (524288/2)   // 262144
      // OS X: "make test" works on 65536 and even below
// "make perftest" works on >= 4*65536 == 262144 (not 3*65536)

  #elif (defined PLATFORM_LINUX) && (defined __i386)
    #define _THREAD_STACK_SIZE (2097152/16)  // 131072
// Linux x86 (Ubuntu 7.04): "make perftest" works on /16 (not on /32)

  #elif (defined PLATFORM_BSD) && (defined __i386)
    #define _THREAD_STACK_SIZE (1048576/8)  // 131072
// FreeBSD 6.2 SMP i386: ("gmake perftest" works on /8 (not on /16)
  #endif
#endif

...

#if (defined _THREAD_STACK_SIZE) && (_THREAD_STACK_SIZE > 0)
    PT_CALL( pthread_attr_setstacksize( a, _THREAD_STACK_SIZE ) );
#endif



Rici Lake kirjoitti 20.6.2007 kello 0:31:


On 19-Jun-07, at 4:21 PM, Paul Ducklin wrote:

Using LuaTask 1.6.0 statically linked into
Lua 5.1.2. (A few other packages are built-in,
namely the LuaSocket core, lfs, lrandom, bitlib
and struct.) This is on Linux.

Starting Lua, mem use (as shown by top) is about
1m resident, 2.2m allocated (virtual). Seems quite
modest to me.

Every time I call task.create() (even if the tasks
do nothing but sleep immediately) mem usage goes
up by about 70k resident, but a whopping 8m
of virtual memory. So with 250 tasks I am _using_
just 17m but have 2g allocated! Perhaps unsurprisingly,
at 257 tasks Lua segfaults...

Why 2m for Lua but an additional 8m of virtual memory
for each created task, esp. when the task is only
_using_ a modest 70k?

I was rather hoping to handle a couple of hundred
proxied sockets inside a simple Lua program... whilst
there is plenty of physical memory to support this I
am, absurdly, running out of address space :-)

What's going on?

Each task requires a C stack; the default stack allocation
on linux is quite large. There might be a way to reduce it;
I'm not al tanto with LuaTask. Other operating systems may
(by default) make a smaller C stack, but since C does not
check for stack overflow, that can be dangerous.

You might want to try using plain old Lua coroutines,
which do not require a C stack but which do have some
limitations as a result.


Adding:

pthread_attr_setstacksize( &attr, 2097152/16 ); // Thanks Asko !!!

before pthread_create in syncos.c line 62, my test runs using 42m of virtual memory on Ubuntu 7.04

My test is a main task with 255 subtasks blocked on task.receive( - 1).

But... with more than 255 subtasks... SEG FAULT

I will try to found the bug asap...

Regards,
Daniel




Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Duck-2
In reply to this post by Duck-2

[Agglommerating multiple posts long after the event ;-]

I was rather hoping to handle a couple of hundred
proxied sockets inside a simple Lua program... whilst
there is plenty of physical memory to support this I
am, absurdly, running out of address space :-)

What's going on?

Each task requires a C stack; the default stack allocation
on linux is quite large.

You might want to try using plain old Lua coroutines,
which do not require a C stack but which do have some
limitations as a result.

The good news is that my failure to do this the "easy" way using LuaTasks (which are actually very cool -- kind of like fork()s from a separation point of view, but without the fork()ing overhead) means that I will almost certainly knuckle down and do it the "proper" way using coroutines.

It's my own fault for not pursuing the elegant solution in the first place :-)

Adding:

pthread_attr_setstacksize( &attr, 2097152/16 ); // Thanks Asko !!!

before pthread_create in syncos.c line 62, my test runs using 42m of virtual memory on Ubuntu 7.04

But... with more than 255 subtasks... SEG FAULT

Since this implies a LuaTask 1.6.1 some time in the not-too-distant future, how about adding an optional third param to task.create() allowing a stack size to be specified?

(You could add a new error code, say -5, for "stack size out of range" in case pthreads or _beginthreadex() doesn't like the size, e.g. it's too small, or just re-use error -4 in this case.)

For an OS implementation in which the stack size can't (easily) be set, or isn't needed, simply ignore the parameter quietly...

Just a thought. Would allow the stack size to be tuned for individual tasks, which would be in keeping with Lua's "don't be huge" attitude! OSes seem to vary quite a bit -- 64k on FreeBSD IIRC, 1m on Windows, 8m in Linux pthreads -- so some way of "saying what you mean" for each subtask would be entirely suitable IMO.


(BTW: about the segfault, which I blithely said was "unsurprising"...I assumed it was happening at around 250 because 256*8m = 2g, but your tests show it's not down to that. Anyway, Linux slices VM between user and kernel at 3g, doesn't it? Not the default 2g of Windows, which I am more familiar with. Oh, unless you run Exchange, which is such a beast that Windows offers you the special "/3G" bootup option to start kernel-land at 0xC0000000 to make room for all those pesky calendar items :-)


Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Asko Kauppi

I don't believe it's that simple..

pthread nor _beginthreadex() will hardly disqualify a stack size for being too low. It's only later, when running one's program that awkward events would take place. Most likely, C libraries would be the ones taking most stack; Lua stack usage (recursion levels, and C automatic variables) should be pretty dependaple. Remember that the _Lua_ stack is kept in heap, and not affected.

So, my solution in Lua Lanes was to pick "suitable" stack sizes for all major OS'es. The script, nor the end user, has no way or need to modify them. Without recompiling, that is. Giving that freedom would in theory be "nice" but I think in practise it really wouldn't help much. Current defaults are about "twice as much as seems to be needed", to have some overhead.

Better ideas of course considered. :)

-asko


Duck kirjoitti 27.6.2007 kello 8:00:

Since this implies a LuaTask 1.6.1 some time in the not-too-distant future, how about adding an optional third param to task.create() allowing a stack size to be specified?


(You could add a new error code, say -5, for "stack size out of range" in case pthreads or _beginthreadex() doesn't like the size, e.g. it's too small, or just re-use error -4 in this case.)


Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Glauber Prado
In reply to this post by Duck-2
Ola eu acho que esse é o pior tipo de apresentação possivel então eu vou continuar :), essa lista não possui um [lua] no Re: pra eu poder filtrar no cliente de email?

Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Glauber Prado
Glauber Prado wrote:
Ola eu acho que esse é o pior tipo de apresentação possivel então eu vou continuar :), essa lista não possui um [lua] no Re: pra eu poder filtrar no cliente de email?

Sorry for crossposting.

Reply | Threaded
Open this post in threaded view
|

Re: LuaTask and memory allocation

Duck-2
In reply to this post by Duck-2

I don't believe it's that simple..

Surely it _is_ that simple! I'm not suggesting that everyone will want to set a thread stack size, or even know how to chose one if they feel the urge. So they can just stick with the OS's default. (I was suggesting an _optional_ third parameter to task.create().)

At least in pthreads on Linux, the default is very large -- much larger than is usually necessary, let alone desirable. Received wisdom on the net suggests that it is _very_ much larger than appropriate (and encourages the development of software which blindly ignores its stack requirements and thus isn't as safely portable as might be liked, since one ought to be trying to compute a sensible stack size and stick to it so far as is possible, etc. etc.)

pthread nor _beginthreadex() will hardly disqualify a stack
size for being too low.

The MSDN blurb for _beginthreadex() doesn't say, but the pthreads docs say this...

"pthread_attr_setstacksize(): Change the size of the stack created for the thread. The value defines the minimum stack size, in bytes. If the value exceeds the system's maximum stack size, or is smaller than PTHREAD_STACK_MIN, pthread_attr_setstacksize will fail and return EINVAL."

So, my solution in Lua Lanes was to pick "suitable" stack sizes
for all major OS'es.

Which pretty much proves my point -- since you admit that the _default_ defaults are wrong :-) Otherwise you could surely have let the OS pick "suitable" sizes...

Giving that freedom would in theory be "nice" but I think in practise
it really wouldn't help much. Current defaults are about "twice as much
as seems to be needed"

Thus halving the maximum number of concurrent threads available. Why not, say, 25% more than seems to be needed? Surely if you waste as much space again as you think is necessary, you don't really have any sort of idea how much is necessary. Why not three times as much, to be doubly additionally safe :-)

I still think it would be handy to be able to start tasks which you know have small stack requirements with small stack sizes -- for example, if you have a bunch of worker tasks which analyse the first N bytes of a file to try to guess its type, without recursive code, like libmagic but smarter; or a bunch of subtasks to relay fixed-size chunks of data to and from a proxy socket in another process. Giving them 8m each seems a little excessive...

Of course on Windows my suggestion is mostly uncontroversial since stacks grow from their initial size automatically if needed. Max stack size is set in the EXE header of the parent thread, and defaults IIRC to all VM. You can lock in a different maximum stack size for individual threads, but [a] only IIRC on versions after Windows 2000 and [b] not via the _beginthreadex() wrapper to CreateThread(), which LuaTask rightly uses.