LibEvent binding to Lua? Mix w/ HelperThreads?

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

LibEvent binding to Lua? Mix w/ HelperThreads?

Thomas Harning Jr.
Just wondering, how does a LibEvent binding into Lua sound?  Right now
it seems to be a good option/addition to the HelperThreads library for
my MoonLog application (syslog/metalog replacement).

Basically you set up file descriptor events that LibEvent will watch
for (such as socket readiness, file has data available, etc... using
the 'best' method available, be it EPoll, kqueue, /dev/poll, poll,...)
and will dispatch event handlers for you... However you must run a
dispatch loop of sorts (either theirs or roll-your-own).

Does anyone have any idea of a good interface to Lua?  For
HelperThreads, are there any ideas on how to best merge these
together... since basically there has to be a merged interface... or a
combined dispatch loop (alternating between HelperThreads waiting and
LibEvent waiting).

... Another idea I thought of at this later hour is to have
HelperThreads be the core... and setup a task/thread as the LibEvent
dispatch loop... and whenever there's events, either signal_task and
wait, or queue up the data in a threadsafe queue and
signal_task(0).  When the update handler gets called, the queue gets
transferred somehow (either by returning an object w/ a CClosure that
can access the latest queue objects and then free the data via __gc
and/or :free())

--
Thomas Harning Jr.
Fortune:
        Against his wishes, a math teacher's classroom was remodeled.
Ever since, he's been talking about the good old dais.  His students
planted a small orchard in his honor; the trees all have square roots.

signature.asc (207 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Diego Nehab-3
Hi,

> Just wondering, how does a LibEvent binding into Lua sound?  Right now
> it seems to be a good option/addition to the HelperThreads library for
> my MoonLog application (syslog/metalog replacement).

It would be nice to come up with something that worked both on Windowses
and Unixes. Anyone has experience with libevent on Windows? Does it work
only for sockets?

Also, it would be good if whateve solution we come up with worked with
coroutine-based dispatchers where threads are not available.

Whatever happens, I want to make LuaSocket as easy as possible to
integrate with dispatch loops.

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Javier Guerra Giraldez
In reply to this post by Thomas Harning Jr.
On Friday 31 March 2006 1:52 am, Thomas Harning Jr. wrote:
> Just wondering, how does a LibEvent binding into Lua sound?  Right now
> it seems to be a good option/addition to the HelperThreads library for
> my MoonLog application (syslog/metalog replacement).

it wouldn't be hard to do a thin binding library; the main hurdle i see is
that any other library you'd want to produce those events would have to give
you a file descriptor (fd) to wait on.

that's not hard for 'real' file based I/O and sockets; but what about
databases?  these could block for significant time, and i don't know of
anyone that gives you an fd telling "call me when this fd is writeable and i
won't block"

my goal with HelperThreads is to make it easy to write libraries binding any
kind of blocking C libraries; since several of them doesn't provide any
non-blocking interface; and those that do might use any of a number of ways
to signal completion.... the most general mechanism is to spawn a thread and
let it block.

> ... Another idea I thought of at this later hour is to have
> HelperThreads be the core... and setup a task/thread as the LibEvent
> dispatch loop... and whenever there's events, either signal_task and
> wait, or queue up the data in a threadsafe queue and
> signal_task(0).  When the update handler gets called, the queue gets
> transferred somehow (either by returning an object w/ a CClosure that
> can access the latest queue objects and then free the data via __gc
> and/or :free())

maybe the nicest way to do it would be to add either a Lua object to be
returned by the update handler.  just set up a hidden Lua table indexed by a
lightuserdata that holds the pointer given to event_set().  when the event is
signalled, just get the associated content and return it.

the same mechanism could be used to associate a Lua function to an event, and
call it when the event is ready.

also, it wouldn't be too hard to do it both as an independent library, and
provide a 'task-able' event_loop() if Helper-Threads is present.

--
Javier

attachment0 (207 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Javier Guerra Giraldez
In reply to this post by Diego Nehab-3
On Friday 31 March 2006 2:00 am, Diego Nehab wrote:
> Hi,
>
> > Just wondering, how does a LibEvent binding into Lua sound?  Right now
> > it seems to be a good option/addition to the HelperThreads library for
> > my MoonLog application (syslog/metalog replacement).
>
> It would be nice to come up with something that worked both on Windowses
> and Unixes. Anyone has experience with libevent on Windows? Does it work
> only for sockets?

i'd also like if somebody with windows experience would help porting
HelperThreads... i guess your pt.c layer would be a good start, but i don't
have a windows development system to test it.

> Also, it would be good if whateve solution we come up with worked with
> coroutine-based dispatchers where threads are not available.

i've been thinking on this, in fact, trying to do it all at once was a major
stumbling block for my two or three initial versions of HTT.

but, once the main (thread based) API is somewhat stable, and (hopefully)
accepted; it wouldn't be too hard to add a polling API, where the (*work)()
callback is called repeatedly until it's done, maybe even setting some fd and
telling HTT "don't bother calling me until this fd is writeable/readable"

when that's done, a simple compile flag might disable the threading API and
making the whole thing usable on non-threading environments.

with all this on place, any library writer would be able to choose between
ease to write (threads), or wider compatibility (poll).  i guess that most
libraries for 'big' (linux,windows) systems would tend to use threads, and
those that hope to be usable on 'small' (embedded) systems would try to do it
with polling.

--
Javier

attachment0 (207 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Mildred Ki'Lya
In reply to this post by Thomas Harning Jr.

Hi,

I just have a question about HelperThreads ... Can we use it to make a
multithread lua application ? Can we use it to have multiple lua
threads (that can share lua variables) at the same time or not ?

Thanks

--
Mildred       <xmpp:[hidden email]> <http://mildred632.free.fr/>
Clef GPG :    <hkp://pgp.mit.edu> ou <http://mildred632.free.fr/gpg_key>
Fingerprint : 197C A7E6 645B 4299 6D37 684B 6F9D A8D6 [9A7D 2E2B]
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Javier Guerra Giraldez
On Friday 31 March 2006 7:43 am, Mildred wrote:
> I just have a question about HelperThreads ... Can we use it to make a
> multithread lua application ? Can we use it to have multiple lua
> threads (that can share lua variables) at the same time or not ?

no, for that you can use LuaThreads.

HelperThreads has much lower performance impact precisely because of the
constraint that a background task MUST NOT touch or access the Lua state in
any way.

i have some (currently vague) plans to do a Rings-like library with HTT,
resulting in something very similar to LuaTasks, but that doesn't let you
share lua variables between states.

i'd love to see a solution for this... but until there's a per-object locking
in the Lua core, LuaThreads will be the only way.

--
Javier

attachment0 (207 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Doug Currie
In reply to this post by Diego Nehab-3
Friday, March 31, 2006, 2:00:58 AM, Diego Nehab wrote:

>> Just wondering, how does a LibEvent binding into Lua sound?  Right now
>> it seems to be a good option/addition to the HelperThreads library for
>> my MoonLog application (syslog/metalog replacement).

> It would be nice to come up with something that worked both on Windowses
> and Unixes. Anyone has experience with libevent on Windows? Does it work
> only for sockets?

Looking at the source in the libevent CVS it appears that only sockets
are supported since select() is used for dispatching. This makes win32
libevent useless for files (and timers, queues, etc.). This is what
drove me to roll my own using IOCP...

http://lua-users.org/lists/lua-l/2006-01/msg00613.html

e

--
Doug Currie
Londonderry, NH

Reply | Threaded
Open this post in threaded view
|

RE: LibEvent binding to Lua? Mix w/ HelperThreads?

Vijay Aswadhati-2
In reply to this post by Javier Guerra Giraldez

On Friday, March 31, 2006 3:59 AM Javier Guerra wrote:


> i'd also like if somebody with windows experience would help
> porting HelperThreads... i guess your pt.c layer would be a good
> start, but i don't have a windows development system to test it.

I am stuck in a critical path in my current venture that has my ....
under fire; I would have loved to port HTT to Windows and start
moving my codebase to Lua.

So if by August 15th HTT still doesn't have a Windows port then sign
me up for the porting task.

Cheers
Vijay Aswadhati



Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Diego Nehab-3
In reply to this post by Doug Currie
Hi,

> Looking at the source in the libevent CVS it appears that only sockets
> are supported since select() is used for dispatching. This makes win32
> libevent useless for files (and timers, queues, etc.). This is what
> drove me to roll my own using IOCP...
>
> http://lua-users.org/lists/lua-l/2006-01/msg00613.html

Yeah, this is what I noticed from the sources too. Is your IOCP library
compatible to libevent?

[]s,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Doug Currie
Friday, March 31, 2006, 12:18:46 PM, Diego Nehab wrote:

>> Looking at the source in the libevent CVS it appears that only sockets
>> are supported since select() is used for dispatching. This makes win32
>> libevent useless for files (and timers, queues, etc.). This is what
>> drove me to roll my own using IOCP...
>>
>> http://lua-users.org/lists/lua-l/2006-01/msg00613.html

> Yeah, this is what I noticed from the sources too. Is your IOCP library
> compatible to libevent?

No. Libevent API is dependent on (struct timeval) and (fd) and more;
it expects (fd)s to work a la UNIX. That was too big an impedance
mismatch for me to bother accommodating.

It may be possible to make a lua wrapper that creates a common API
between libevent and a Windows IOCP library, but the abstraction would
have to be carefully designed to avoid these UNIXisms.

e

--
Doug Currie
Londonderry, NH

Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Diego Nehab-3
Hi,

>> Yeah, this is what I noticed from the sources too. Is your IOCP library
>> compatible to libevent?
>
> No. Libevent API is dependent on (struct timeval) and (fd) and more;
> it expects (fd)s to work a la UNIX. That was too big an impedance
> mismatch for me to bother accommodating.
>
> It may be possible to make a lua wrapper that creates a common API
> between libevent and a Windows IOCP library, but the abstraction would
> have to be carefully designed to avoid these UNIXisms.

I believe we need some C support. I was thinking about creating a
layer in C that implements a libevent-like dispatch loop but that
uses opaque descriptors. Those could be fds on UNIX or HANDLEs on
Windows.

We would also have to adapt Lua's file I/O functions and LuaSocket
functions to integrate well with this abstraction layer.

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Daniel Quintela
Hi,

Wim Couwenberg and D Burgess comments about Events lib and Win32:
http://lua-users.org/lists/lua-l/2004-06/msg00636.html

Regards,
Daniel

Diego Nehab escribió:

> Hi,
>
>>> Yeah, this is what I noticed from the sources too. Is your IOCP library
>>> compatible to libevent?
>>
>>
>> No. Libevent API is dependent on (struct timeval) and (fd) and more;
>> it expects (fd)s to work a la UNIX. That was too big an impedance
>> mismatch for me to bother accommodating.
>>
>> It may be possible to make a lua wrapper that creates a common API
>> between libevent and a Windows IOCP library, but the abstraction would
>> have to be carefully designed to avoid these UNIXisms.
>
>
> I believe we need some C support. I was thinking about creating a
> layer in C that implements a libevent-like dispatch loop but that
> uses opaque descriptors. Those could be fds on UNIX or HANDLEs on
> Windows.
>
> We would also have to adapt Lua's file I/O functions and LuaSocket
> functions to integrate well with this abstraction layer.
>
> Regards,
> Diego.


Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Javier Guerra Giraldez
In reply to this post by Diego Nehab-3
On Friday 31 March 2006 5:38 pm, Diego Nehab wrote:
> I believe we need some C support. I was thinking about creating a
> layer in C that implements a libevent-like dispatch loop but that
> uses opaque descriptors. Those could be fds on UNIX or HANDLEs on
> Windows.

that's what i've done, but using pthreads instead of events

> We would also have to adapt Lua's file I/O functions and LuaSocket
> functions to integrate well with this abstraction layer.

done that too, but the network lib is just the bare minimum for TCP.

the whole point of HTT is making easy to write non-blocking libraries and
provide a consistent Lua interface

--
Javier

attachment0 (207 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Wim Couwenberg-2
In reply to this post by Diego Nehab-3
> I believe we need some C support. I was thinking about creating a
> layer in C that implements a libevent-like dispatch loop but that
> uses opaque descriptors. Those could be fds on UNIX or HANDLEs on
> Windows.

This support should also incorporate the spawn discussion of a while
back.  I.e. properly support "inheritance" via fcntl/FD_CLOEXEC vs.
SetHandleInformation etc. over implementations.  Also, read/write/etc.
actions on fd/Handle should behave the same, e.g. consistent
blocking/EOF/error behaviour for fifo's and named pipes, etc.  This,
combined with consistent AIO handling, was the reason that I did not
finish such a lib that I started some time ago.  It would still be
very welcome though...

--
Wim


Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Diego Nehab-3
Hi,

> This support should also incorporate the spawn discussion of a while
> back.  I.e. properly support "inheritance" via fcntl/FD_CLOEXEC vs.
> SetHandleInformation etc. over implementations.  Also, read/write/etc.
> actions on fd/Handle should behave the same, e.g. consistent
> blocking/EOF/error behaviour for fifo's and named pipes, etc.  This,
> combined with consistent AIO handling, was the reason that I did not
> finish such a lib that I started some time ago.  It would still be
> very welcome though...

When I wrote LuaThread, I ported a subset of the pthreads API to Win32.
Would it make sense to try to replicate the libevent API, but based on
HANDLEs, or should we write something completely new?

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Doug Currie
Monday, April 3, 2006, 1:12:21 AM, Diego Nehab wrote:

> Would it make sense to try to replicate the libevent API, but based on
> HANDLEs, or should we write something completely new?

Lua uses C Library (FILE *) handles. There is a means of converting
these to HANDLEs on Windows, e.g., see
http://www.codeproject.com/file/handles.asp?df=100&forumid=2073&exp=0&select=429628
but for use with IOCP the HANDLEs have to be opened with
FILE_FLAG_OVERLAPPED; see
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createiocompletionport.asp
> You must specify the FILE_FLAG_OVERLAPPED flag when using the
> CreateFile function to obtain the handle.
and the Win32 CRT doesn't open files this way.

> We would also have to adapt Lua's file I/O functions and LuaSocket
> functions to integrate well with this abstraction layer.

It seems to me that trying to simultaneously provide Lua stdlib
semantics along with AIO semantics on both Windows and Linux will be a
huge task.

Javier's HelperThreads is an interesting compromise; it retains Lua
stdlib and inter-operation by providing a separate scheduler and i/o
functions. It finesses non-blocking i/o by using threads.

IOCP on windows and libevent on Linux could provide better
performance. But it is not clear to me what you mean by "integrating
well" with Lua's and LuaSocket's I/O functions, since they were not
written for AIO.

Regards,

e

--
Doug Currie
Londonderry, NH

Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Diego Nehab-3
Hi,

> It seems to me that trying to simultaneously provide Lua stdlib
> semantics along with AIO semantics on both Windows and Linux will be a
> huge task.
>
> IOCP on windows and libevent on Linux could provide better
> performance. But it is not clear to me what you mean by "integrating
> well" with Lua's and LuaSocket's I/O functions, since they were not
> written for AIO.

I am in fact talking about replacing the way Lua does I/O on Windows and
on Unix, to make it work well with libevent on Unix and with IOCP on
Windows. If that means I would have to eliminate FILE* usage from the
Lua stdlib, I am fine with that. Probably no need to even touch Lua's C
code. It can all be done with a plugin.

As for LuaSocket, I can rewrite as much as I have to.

Why do you think this is too complicated?

As for mixing with HelperThreads, it is completely orthogonal to what I
am suggesting. Although LuaSocket is not completely thread-safe, it can
be made so with just a little work.

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Doug Currie
Monday, April 3, 2006, 2:06:30 PM, Diego Nehab wrote:

> I am in fact talking about replacing the way Lua does I/O on Windows and
> on Unix, to make it work well with libevent on Unix and with IOCP on
> Windows. If that means I would have to eliminate FILE* usage from the
> Lua stdlib, I am fine with that. Probably no need to even touch Lua's C
> code. It can all be done with a plugin.

Yes, that is the approach I have used for my IOCP library.

> Why do you think this is too complicated?

I was reading too much into your "adapt Lua's file I/O functions and
LuaSocket functions to integrate well with this abstraction layer"
comment. Giving Lua the benefit of AIO in its existing I/O libraries
would mean integrating a scheduler into Lua. You don't seem to be
advocating that, but rather providing a library to support building
applications with AIO. That is much easier. The only drawback is that
a programmer can inadvertently use a non-AIO function, and block the
whole system. Perhaps that can be avoided with an AIO sandbox.

> As for mixing with HelperThreads, it is completely orthogonal to what I
> am suggesting. Although LuaSocket is not completely thread-safe, it can
> be made so with just a little work.

I see that HelperThreads can, e.g., take a Lua file handle (FILE *)
and use it in a non-blocking way using threads. This is something a
Windows IOCP library couldn't do.

I note that the libevent API is based on (fd and timeval) callbacks,
whereas IOCP is based on polling (GetQueuedCompletionStatus). I.e.,
the dispatcher is built outside library in my IOCP approach, whereas
it is inside the library in libevent. Which approach do you prefer?

e

--
Doug Currie
Londonderry, NH

Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Diego Nehab-3
Hi,

>> As for mixing with HelperThreads, it is completely orthogonal to what I
>> am suggesting. Although LuaSocket is not completely thread-safe, it can
>> be made so with just a little work.
>
> I see that HelperThreads can, e.g., take a Lua file handle (FILE *)
> and use it in a non-blocking way using threads. This is something a
> Windows IOCP library couldn't do.

Right, but if a thread blocks on a call that chooses never to return,
the thread is dead, right?

> I note that the libevent API is based on (fd and timeval) callbacks,
> whereas IOCP is based on polling (GetQueuedCompletionStatus). I.e.,
> the dispatcher is built outside library in my IOCP approach, whereas
> it is inside the library in libevent. Which approach do you prefer?

Can't you use WaitForMultipleObjects with an associated condition or
something like that instead of polling?

Regards,
Diego.
Reply | Threaded
Open this post in threaded view
|

Re: LibEvent binding to Lua? Mix w/ HelperThreads?

Adrian Sietsma
 From a Lua pov., what I would like to be able to do in an async io world is
either this:

s,err = wait_read(object,size,timeout) -- yields to the scheduler until data
available or timeout
...
wait_event(object,timeout) -- yields to the scheduler until event or timeout


or (contract approach)
token=read(object,size)
...
...
s, err = get_data(token)-- return any data so far
...
...
wait_for_completion(token,timeout) -- yields
s,err = get_data(token)

The first approach can be implemented using either select or overlapped io
(I have implemented it using LuaSocket);  from the Lua side, the thread just
"sleeps" until data is there.

My current project uses an event iterator that returns triggered events
(socket/timer/user) when called (again using LuaSocket); and a coroutine
dispatcher that associates event with coroutines, and resumes the
appropriate coro for the event. This approach could easily be integrated
with overlapped io.

Anyway, as I see it there are 3 basic actions:
1/ Initiate a request
2/ Check (poll) the status/results of a request.
3/ Wait for a request to be fulfilled.

3 requires some sort of sheduler, although that can be based on
3a/ Wait for the next outstanding request to be fulfilled.

Any higher-level interface can be built from these primitives.

I see no alternative to different implementations for *nix and windows, but
it would be nice to have an abstraction/interface built around the Lua usage
requirements, not the os primitives.

Adrian
12