[ANN] LuaTask 1.6.1

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[ANN] LuaTask 1.6.1

Daniel Quintela
Hi everybody:

The 1.6.1 release of LuaTask is ready to download:
http://luaforge.net/frs/?group_id=65

* If is defined LUATASK_PTHREAD_STACK_SIZE we use thread_attr_setstacksize.
* BUG: OsLockMutex misplaced on taskthread.
* BUG: Volatile attribute missing for aTask.

Best regards,
Daniel




Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Duck-2

The 1.6.1 release of LuaTask is ready to download

Is this intended to fix the "more than 255 tasks segfault?"

It seems to fix it when there are N tasks each just blocking indefinitely. (This code would reliably segfault under 1.6.0 for N>= 255.)

I have code which creates N tasks which sort M random itmes P times. As long as I'm careful not to set M so high that I blow out the C stack of the tasks, I can use almost any combination of M, N and P as long as N <= 255. So sorting 10,000 items 50 times in 255 concurrent tasks is fine, if somewhat slow :-)

With N >= 256, the code always segfaults, even with, say, 10 items sorted 10 times, or even 1 items sorted 1 time.

FWIW I set LUATASK_PTHREAD_STACK_SIZE to 2097152 (2m) in my makefile.

Of course if this version isn't intended to address the "255" issue then ignore this email -- though note that the "just blocking" case above always segfaulted at 256 with 1.6.0 yet doesn't now. Though this could be down to the fact that the binary is rebuilt and thus differently laid out...

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Daniel Quintela
Duck escribió:
Is this intended to fix the "more than 255 tasks segfault?"

It seems to fix it when there are N tasks each just blocking indefinitely. (This code would reliably segfault under 1.6.0 for N>= 255.)

I have code which creates N tasks which sort M random itmes P times. As long as I'm careful not to set M so high that I blow out the C stack of the tasks, I can use almost any combination of M, N and P as long as N <= 255. So sorting 10,000 items 50 times in 255 concurrent tasks is fine, if somewhat slow :-)

With N >= 256, the code always segfaults, even with, say, 10 items sorted 10 times, or even 1 items sorted 1 time.

FWIW I set LUATASK_PTHREAD_STACK_SIZE to 2097152 (2m) in my makefile.

Of course if this version isn't intended to address the "255" issue then ignore this email -- though note that the "just blocking" case above always segfaulted at 256 with 1.6.0 yet doesn't now. Though this could be down to the fact that the binary is rebuilt and thus differently laid out...

Please: change TASK_SLOTS_STEP to 384 (ltask.c:49) and check again with N between 255 and 380. Tell me if the segfault is gone.

Thanks,
Daniel



Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Duck-2
In reply to this post by Daniel Quintela

Please: change TASK_SLOTS_STEP to 384 (ltask.c:49) and check
again with N between 255 and 380.  Tell me if the segfault is
gone.

The segfaults seem to start at N == TASK_SLOTS_STEP.

With TASK_SLOTS_STEP at 256, it works until N == 255, then segfaults. I set it to 384 and the segfaults start when N == 384. For N at 381, 382, 383 it all work fine.

I tried setting TASK_SLOTS_STEP to 443, and the behaviour was what you might expect: it worked up to 442 tasks and segfaulted from 443 and up.

Hope this helps.


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Daniel Quintela
Duck escribió:

The segfaults seem to start at N == TASK_SLOTS_STEP.

With TASK_SLOTS_STEP at 256, it works until N == 255, then segfaults. I set it to 384 and the segfaults start when N == 384. For N at 381, 382, 383 it all work fine.

I tried setting TASK_SLOTS_STEP to 443, and the behaviour was what you might expect: it worked up to 442 tasks and segfaulted from 443 and up.

Hope this helps.


Thanks,

Yes... this helps.

LuaTask allocates TASK_ENTRY memory in TASK_SLOTS_STEP * sizeof(TASK_ENTRY) chunks.
Problem arises when it reallocates memory to grow one step more.

The size of TASK_ENTRY structure is 52 bytes on Linux.
As a workaround, you can set TASK_SLOTS_STEP larger enough in order to avoid reallocation.

It's time to work on 1.6.2 ... :-(

Regards,
Daniel


Reply | Threaded
Open this post in threaded view
|

NT Service and Lua Version

Rafael - SosCpdTerra
	Hello there

	Im kind of translating old lua 4 Lua-Rpc module from Russell L.
Smith to a new version of lua, to build a system service on MS NT based
systems to read and write serial ports from a rpc based daemon. Like serial
devices sharing. The system core is written in C++, and I have experienced
some problems finding a way to embed lua 5.1.2 in the core. Maybe I'm not
looking on the right place, and if so, can you guys help me pointing some
place where is an example of specific implementation of Lua 5.1.2 in c++?

	Thanks in advance.


	Rafael Menezes


Reply | Threaded
Open this post in threaded view
|

RES: C++ Typecast - so far newbie question

Rafael - SosCpdTerra
Did someone know a better way to typecast a const char * (that is about
lua_tostring(L, n)) into a char * then vector or c_str()? I'm fighting with
that...

About the Lua 5.1.2... well, I must say that im really stupid. For future
posts from someone interested, here comes:

/*File whatever.cpp*/
extern "C"
{
   #include <lua.h>
   #include <lauxlib.h>
   #include <lualib.h>
}
#include <iostream>

"whatever_type" "whatever_your_function_name_have"(lua_State *L) 
    {
        int argc = lua_gettop(L); /*the function arguments came from here.
First time, i just cant figure out that*/
        
        return "whatever_type";
    }
    
int main(int argc, char* argv[])
    {
        int ret;
        const char *filename;
        const char *returndeal;
        filename = argv[1];
        lua_State *L;
        L = lua_open();
        luaL_openlibs(L); //Load all resources from the lua env. To see what
is loading, go to lualib.h
        lua_register(L, "serial", serial);
        //This is how to pass a table to lua. The table name will be...
        lua_newtable(L);
        lua_pushnumber( L, 1 );
        lua_pushstring( L, returndeal);
        lua_rawset( L, -3 );
        lua_setglobal(L, "arg" );
        //... arg!!!
        int result = 0;
        ret = luaL_loadfile(L,filename); /*here you load the file. You can
do that in a different manner to pick up the args directly fom the prompt
command line*/
        if (ret == 0)
            result = lua_pcall( L, 0, LUA_MULTRET, 0 ); //This line will
execute the script
        else
            std::cout << "bad" << std::endl;/*and this one will tell you,
iff appears, that you must study the lua script manual again.*/

        lua_close(L);
        return 0;
    }
/*Eof whatever.cpp*/

There are many ways to build lua embedded in your application. To do so in a
very easy (and static) way, do as this follow instructions (for Lua 5.1.2):

Building lualib

/*LuaLib.c - remember to set include directory to your lua/src directory*/
#define lualib_c
#include <unistd.h> 
#define LUA_TMPNAMBUFSIZE       32 
#define lua_tmpnam(b,e)         { e = (tmpnam(b) == NULL); } 


#include "lauxlib.c"
#include "lbaselib.c"
#include "ldblib.c"
#include "liolib.c"
#include "linit.c"
#include "lmathlib.c"
#include "loadlib.c"
#include "loslib.c"
#include "lstrlib.c"
#include "ltablib.c"

/*Eof Lualib.c*/

Build this one, whatever name you like, as a static library. So with this
other file:

/*LuaCore - remember to set include directory to your lua/src directory */

#define luacore_c

#include "lapi.c"
#include "lcode.c"
#include "ldebug.c"
#include "ldo.c"
#include "ldump.c"
#include "lfunc.c"
#include "lgc.c"
#include "llex.c"
#include "lmem.c"
#include "lobject.c"
#include "lopcodes.c"
#include "lparser.c"
#include "lstate.c"
#include "lstring.c"
#include "ltable.c"
#include "ltm.c"
#include "lundump.c"
#include "lvm.c"
#include "lzio.c"

/*Eof LuaCore.c*/

Include the 2 libs (remember to set include directory to your lua/src
directory) in the project where you will build the very first example in
this email and Voila!!! You get yourself this thing done!!


Reply | Threaded
Open this post in threaded view
|

Re: RES: C++ Typecast - so far newbie question

Rici Lake-2

On 2-Jul-07, at 2:40 PM, SosCpdTerra wrote:

Did someone know a better way to typecast a const char * (that is about
lua_tostring(L, n)) into a char * then vector or c_str()? I'm fighting with
that...

Strings returned by lua_tostring *must not be modified*.

So don't cast them to char* or evil things will happen.


Reply | Threaded
Open this post in threaded view
|

RES: RES: C++ Typecast - so far newbie question

Rafael - SosCpdTerra
What are you talking about? I cannot pass this return value to a variable
and typecast into whatever I want? I guess this is the evil!!

Like:

I have this
void figureout(char *what, int ever)

and I must his 2 args came from lua. Im trying to do this:

int serial(lua_State *L) //my registered function
    {
        int argc = lua_gettop(L);
        const char var1 = lua_tostring(L, 1);
        int var2 = int(lua_tonumber(L, 2));
        figureout(var1, var2);
        return 0;
    }

There is another way? I can't see that way, and that is my problem.


-----Mensagem original-----
De: [hidden email]
[[hidden email]] Em nome de Rici Lake
Enviada em: segunda-feira, 2 de julho de 2007 17:58
Para: Lua list
Assunto: Re: RES: C++ Typecast - so far newbie question


On 2-Jul-07, at 2:40 PM, SosCpdTerra wrote:

> Did someone know a better way to typecast a const char * (that is about
> lua_tostring(L, n)) into a char * then vector or c_str()? I'm fighting 
> with
> that...

Strings returned by lua_tostring *must not be modified*.

So don't cast them to char* or evil things will happen.



__________ NOD32 2371 (20070702) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com



Reply | Threaded
Open this post in threaded view
|

RE: RES: C++ Typecast - so far newbie question

Patrick Donnelly-2
In reply to this post by Rafael - SosCpdTerra
> On 2-Jul-07, at 2:40 PM, SosCpdTerra wrote:
>
> > Did someone know a better way to typecast a const char * (that is about
> > lua_tostring(L, n)) into a char * then vector or c_str()? I'm fighting
> > with
> > that...
>
> Strings returned by lua_tostring *must not be modified*.
>
> So don't cast them to char* or evil things will happen.

I'm curious about the technical reasons/consequences for this. Naturally lua strings are meant to be immutable. I'm betting this has something to do with their hash value changing, and thus their index to the global environment (or whatever table they're in) being lost because the hash changes? I assume this would cause not only losing your string in the table but some segmentation faults if one attempts to access them? Would the garbage collector lose sight of these strings, or collect it almost immediately?

-Patrick Donnelly

"One of the lessons of history is that nothing is often a good thing to do and always a clever thing to say."

-Will Durant
Reply | Threaded
Open this post in threaded view
|

Re: RES: C++ Typecast - so far newbie question

Rici Lake-2

On 2-Jul-07, at 4:08 PM, Patrick Donnelly wrote:

> On 2-Jul-07, at 2:40 PM, SosCpdTerra wrote:
>
> > Did someone know a better way to typecast a const char * (that is about > > lua_tostring(L, n)) into a char * then vector or c_str()? I'm fighting
> > with
> > that...
>
> Strings returned by lua_tostring *must not be modified*.
>
> So don't cast them to char* or evil things will happen.

I'm curious about the technical reasons/consequences for this. Naturally lua strings are meant to be immutable. I'm betting this has something to do with their hash value changing, and thus their index to the global environment (or whatever table they're in) being lost because the hash changes? I assume this would cause not only losing your string in the table but some segmentation faults if one attempts to access them? Would the garbage collector lose sight of these strings, or collect it almost immediately?

Lua interns strings; there is only one instance of each string. So if you changed, say, "print" to "prbnt", you'd end up changing every usage of "print" in the lua environment, including constants in programs and keys in hash tables.

After that, it gets really confusing. But it won't segfault afaik.


Reply | Threaded
Open this post in threaded view
|

Re: RES: RES: C++ Typecast - so far newbie question

Rici Lake-2
In reply to this post by Rafael - SosCpdTerra

On 2-Jul-07, at 3:05 PM, SosCpdTerra wrote:

What are you talking about? I cannot pass this return value to a variable
and typecast into whatever I want? I guess this is the evil!!

If you want a mutable copy of the string, you need to make a copy of the string.


Like:

I have this
void figureout(char *what, int ever)

and I must his 2 args came from lua. Im trying to do this:

int serial(lua_State *L) //my registered function
    {
        int argc = lua_gettop(L);
        const char var1 = lua_tostring(L, 1);
        int var2 = int(lua_tonumber(L, 2));
        figureout(var1, var2);
        return 0;
    }

There is another way? I can't see that way, and that is my problem.

Declare figureout properly:

void figureout(const char *what, int ever)

Functions which take string arguments and do not modify them should not be declared as taking char *.


Reply | Threaded
Open this post in threaded view
|

RES: RES: RES: C++ Typecast - so far newbie question

Rafael - SosCpdTerra
Hell no... that works!!! Thanks a lot Rici. Fixed with figureout(const char
*what, int ever).

Again, thanks.

Rafael

-----Mensagem original-----
De: [hidden email]
[[hidden email]] Em nome de Rici Lake
Enviada em: segunda-feira, 2 de julho de 2007 18:24
Para: Lua list
Assunto: Re: RES: RES: C++ Typecast - so far newbie question


On 2-Jul-07, at 3:05 PM, SosCpdTerra wrote:

> What are you talking about? I cannot pass this return value to a 
> variable
> and typecast into whatever I want? I guess this is the evil!!

If you want a mutable copy of the string, you need to make a copy of 
the string.

>
> Like:
>
> I have this
> void figureout(char *what, int ever)
>
> and I must his 2 args came from lua. Im trying to do this:
>
> int serial(lua_State *L) //my registered function
>     {
>         int argc = lua_gettop(L);
>         const char var1 = lua_tostring(L, 1);
>         int var2 = int(lua_tonumber(L, 2));
>         figureout(var1, var2);
>         return 0;
>     }
>
> There is another way? I can't see that way, and that is my problem.

Declare figureout properly:

void figureout(const char *what, int ever)

Functions which take string arguments and do not modify them should not 
be declared as taking char *.



__________ NOD32 2371 (20070702) Information __________

This message was checked by NOD32 antivirus system.
http://www.eset.com



Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Duck-2
In reply to this post by Daniel Quintela

[snipped info about fiddling with TASK_SLOTS_STEP]

LuaTask allocates TASK_ENTRY memory in TASK_SLOTS_STEP *
sizeof(TASK_ENTRY) chunks. Problem arises when it
reallocates memory to grow one step more.

Actually, the segfault doesn't happen directly at the point the realloc() happens. (After looking at the code I realised I should have said this :-) It's only later that it crashes, but apparently only after such a realloc() has taken place.

In my code, I call task.create() more than TASK_SLOTS_STEP times. All my tasks then block on task.receive(-1) until the master task has finished creating them all. The master task then messages each one in turn. Then each task sleeps for a couple of (run-time specified) seconds. Then it sort()s the table it was passed as an argument a few times, messages the master task and ends. When the master task has received messages from all the subtasks, it records success and exits.

From what I can see from debugging print()s in the code, the segfault only
happens when the first subtask finishes, like this (Sn means task n finished its sleep and is about to start sorting, and Fn means it finished its sorting and is about to exit):

:~/lua-try$ lua master.lua 255 10 1000 2
255 tasks sorting 10 items 1000 times (initial delay 2s).
Creating tasks...done.
Starting tasks...done.
Waiting...
S2 S3 S4 F3 F2 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 F6 F5 F8 F7 F9 F11 F10 F12 F13 F14 F4 F15 S16 S17 F17 S18 S19 S20 S21 F21 S22 S23 S24 F24 S25
[blah, blah, snip]
F225 F218 F108 F137 F118 F141 F150 F139 F232 F181
***All reported finished

~/lua-try$ lua master.lua 256 10 10 2 256 tasks sorting 10 items 10 times (initial delay 2s).
Creating tasks...done.
Starting tasks...done.
Waiting...
S2 F2 Segmentation fault

~/lua-try$ lua master.lua 258 1000 10 2
258 tasks sorting 1000 items 10 times (initial delay 2s).
Creating tasks...done.
Starting tasks...done.
Waiting...
S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 S16 S17 S18 S19 S20 F3 Segmentation fault

~/lua-try$ lua master.lua 300 10 1000 2
300 tasks sorting 10 items 1000 times (initial delay 2s).
Creating tasks...done.
Starting tasks...done.
Waiting...
S2 S3 S4 S5 S6 S7 S8 S9 S10 S11 S12 S13 S14 S15 F8 *** glibc detected *** dlua: double free or corruption (out): 0x08072330 ***
[trace of corrupted heap snipped]

The size of TASK_ENTRY structure is 52 bytes on Linux.
As a workaround, you can set TASK_SLOTS_STEP larger
enough in order to avoid reallocation

And to prevent the realloc(), I've just changed this in ltask.c, so that the TASK_SLOT_STEPping never happens and the problem is apparently prevented:

[in int_taskcreate]
  for( i = 0; i < countTask; i++)
    if( !( aTask[i].running))
      break;
  if( i == countTask) {
    TASK_ENTRY *te;
    long j;

    te = ( TASK_ENTRY *) realloc( aTask, sizeof( TASK_ENTRY) * ( countTask + TASK_SLOTS_STEP));
    if( te == NULL) {
      OsUnlockMutex( tlMutex);
      return( -1);
    }
    aTask = te;


[changed to]:

    for( i = 0; i < countTask; i++)
      if( !( aTask[i].running))
        break;
    if( i == countTask) {
        OsUnlockMutex( tlMutex);
        return( -1);
    }
    [delete to next closing squiggly bracket]

Apologies for the rather long post.


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Thomas Lauer-3
In reply to this post by Daniel Quintela
Daniel Quintela <[hidden email]> wrote:

First, this is a great library! I am using it for a few days now and it
works rather well.

Here's a small hint for people who need lots of threads (under Win32):
the number of creatable threads is limited by the stack space they
reserve (not commit!). Available address space is around 2 GB; if each
thread takes up the default of 1 MB that means somewhere around ~1700
threads (there are other overheads to take into account).

To get more threads, use 'link -edit /stack:...' on lua.exe with a value
smaller than 1 MB. (Obviously, a small stack may produce other
problems.)

> It's time to work on 1.6.2 ... :-(

Here are two things. I've compiled for the native Win32 implementation
and found a buglet. Look in syncos.c, line 55:
    return( th == NULL ? -1 : 0);

I think this should be:
    return( *th == NULL ? -1 : 0);

Second, I observe a similar phenomenon as Duck. I've set TASK_SLOTS_STEP
to all sorts of values (I've tried different values between 10 and 512)
and my test scripts sometimes (but alas not always) crash shortly
*after* a realloc has taken place. However, they *never* crash at other
times.

Let's say TASK_SLOTS_STEP==64. Then I see creation of, say, thread 128
or 192 or even 1024. Next, I see a realloc. Then the creation for thread
129 (or 193 or 1025) goes ahead and succeeds. But very shortly
afterwards, the whole thing crashes.

I have inspected the code and found no obvious problem. But something
fishy must be going on here. So for the time being using a big value for
TASK_SLOTS_STEP seems like a good idea;-)

This is Win2K, SP4/rollup.

-- 
cheers  thomasl

web : http://thomaslauer.com/start


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Duck-2
>Obviously, a small stack may produce other
>problems.

IIRC Windows will extend the stack of any thread 
(there is nothing special about threads on Windows; 
they're effectively the native task scheduling unit)
up to the max stack size set in the EXE header, so
starting with a smaller size for all threads shouldn't
be a huge problem. The default max stack size is
IIRC 'all available VM'.

You can use the hackette I posted this a.m. to avoid
the realloc and thus to get a detectable error from
task.create before any trouble can happen. 

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Thomas Lauer-3
Paul Ducklin <[hidden email]> wrote:
> >Obviously, a small stack may produce other
> >problems.
> 
> IIRC Windows will extend the stack of any thread 
> (there is nothing special about threads on Windows; 
> they're effectively the native task scheduling unit)
> up to the max stack size set in the EXE header, so
> starting with a smaller size for all threads shouldn't
> be a huge problem. The default max stack size is
> IIRC 'all available VM'.

There are two stack size values to specify when linking an exe:
'reserve' and 'commit'. I was only talking about minimising reserved
memory as the amount of committed memory is not that important in this
context anyway.

It is quite feasible to have 1500 threads with most of them using no or
almost no committed memory because they sleep. But all these threads
still need their share of reserved memory. And that's what I was talking
about.

-- 
cheers  thomasl

web : http://thomaslauer.com/start


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Kacper Wysocki
In reply to this post by Duck-2
On 7/3/07, Duck <[hidden email]> wrote:
[snipped info about fiddling with TASK_SLOTS_STEP]
[snipped info about yer program]
And to prevent the realloc(), I've just changed this in ltask.c, so that
the TASK_SLOT_STEPping never happens and the problem is apparently
prevented:

[in int_taskcreate]
   for( i = 0; i < countTask; i++)
     if( !( aTask[i].running))
       break;
   if( i == countTask) {
     TASK_ENTRY *te;
     long j;

     te = ( TASK_ENTRY *) realloc( aTask, sizeof( TASK_ENTRY) * ( countTask + TASK_SLOTS_STEP));
     if( te == NULL) {
       OsUnlockMutex( tlMutex);
       return( -1);
     }
     aTask = te;

A race with the aTask pointer update might be your problem :-/

If I understand the design of LuaTasks right, there may be several
threads calling into functions in ltask.c at once, and they are only
synchronized with the tlMutex global. You do well to serialize updates
to the aTask pointer, however there are several functions which do not
lock on access (reg_tasklist, reg_taskreceive, reg_getqhandle), which
lead to a race. Let me illustrate:

Let's say the realloc executes, and returns a new pointer ('te' in the
above code snipplet). realloc() needn't return a new pointer, but
often it will do just that. The old pointer - the global aTask, is
free() and invalid right after the realloc() call, yet any other
thread that has the old pointer might end up dereferencing aTask
before you manage to assign it the new pointer.

That should explain your random crashes, and why they don't always happen.

Solution? Lock all accesses - reads and writes - through any globals
in a multithreaded environment.

-Kacper

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaTask 1.6.1

Daniel Quintela
Kacper Wysocki escribió:

A race with the aTask pointer update might be your problem :-/

If I understand the design of LuaTasks right, there may be several
threads calling into functions in ltask.c at once, and they are only
synchronized with the tlMutex global. You do well to serialize updates
to the aTask pointer, however there are several functions which do not
lock on access (reg_tasklist, reg_taskreceive, reg_getqhandle), which
lead to a race. Let me illustrate:

Let's say the realloc executes, and returns a new pointer ('te' in the
above code snipplet). realloc() needn't return a new pointer, but
often it will do just that. The old pointer - the global aTask, is
free() and invalid right after the realloc() call, yet any other
thread that has the old pointer might end up dereferencing aTask
before you manage to assign it the new pointer.

That should explain your random crashes, and why they don't always happen.

Solution? Lock all accesses - reads and writes - through any globals
in a multithreaded environment.

-Kacper


Yes... the problem is aTask...
Inside "taskthread" there is a pointer to his own TASK_ENTRY structure... well... after the realloc: it points to The Limbo.
I'm changing the allocation/reallocation scheme.

Regards,
Daniel


Reply | Threaded
Open this post in threaded view
|

Luasys Win32

Rafael - SosCpdTerra

Did someone have some more win32 specific instructions about build Luasys?
Some place to have an example?

Thanks in advance.

Rafael


12