helper threads layer

[Javier Guerra Giraldez]

hi everybody

finally, i think i have it working! it's a library of helper threads, meant to
make it easy to write nonblocking libraries in C.

to be a bit more specific: in C, you split funcions in three:

static int xxx_prepare (lua_State *L, void **udata)
static int xxx_work (void *udata)
static int xxx_finish (lua_State *L, void *udata)

and register it with a macro add_helperfunc (L, &ops)   (where ops is a struct
with those three C functions).  after that, you get a Lua function in the
stack.

from Lua, you call that function, it calls the xxx_prepare() funcion with any
given parameters and returns a 'task'.

before that, you would have created a couple (or more) FIFO queues with
helper.newqueue(), and one or more threads with helper.newthread().  each
thread has one input and one output queue (several threads can share a
queue).

when you add the task to the queue (with queue:addtask (task)), a thread will
pick it up, and execute xxx_work() with the same udata built by the
xxx_prepare().  after xxx_work() returns, the thread pushes the task in the
output queue.

back in Lua, you can do a queue:wait(), it blocks until there's a task in the
queue (put there by a thread).  then you call helper.finish(task) to execute
xxx_finish().  xxx_finish() should dispose any memory used by the task (in
udata), and return any value to the Lua caller.

now, we could write non-blocking versions of most IO libraries, and build Lua
wrappers that use this primitives to get a coroutine scheduler.

i'll post it later in LuaForge; any thoughts about the architecture?

--
Javier

[Vijay Aswadhati-2]

On Thursday, March 09, 2006 9:16 AM, Javier Guerra wrote:

> hi everybody

> finally, i think i have it working! it's a library of helper
> threads, meant to make it easy to write nonblocking libraries in
> C.

This may be the beginning of something new and exciting!!

> to be a bit more specific: in C, you split funcions in three:
>
> static int xxx_prepare (lua_State *L, void **udata)
> static int xxx_work (void *udata)
> static int xxx_finish (lua_State *L, void *udata)

> and register it with a macro add_helperfunc (L, &ops)   (where ops
> is a struct with those three C functions).  after that, you get a
> Lua function in the stack.

> from Lua, you call that function, it calls the xxx_prepare()
> funcion with any given parameters and returns a 'task'.

> before that, you would have created a couple (or more) FIFO queues
> with helper.newqueue(), and one or more threads with
> helper.newthread().  each thread has one input and one output
> queue (several threads can share a queue).

> when you add the task to the queue (with queue:addtask (task)), a
> thread will pick it up, and execute xxx_work() with the same udata
> built by the xxx_prepare().  after xxx_work() returns, the thread
> pushes the task in the output queue.

> back in Lua, you can do a queue:wait(), it blocks until there's a
> task in the queue (put there by a thread).  then you call
> helper.finish(task) to execute xxx_finish().  xxx_finish() should
> dispose any memory used by the task (in udata), and return any
> value to the Lua caller.

> now, we could write non-blocking versions of most IO libraries,
> and build Lua wrappers that use this primitives to get a coroutine
> scheduler.

> i'll post it later in LuaForge; any thoughts about the
> architecture?

A few concrete examples would be of great help. I am particularly
interested in how one would go about creating a singleton timer
instance using this package and how the main dispatch loop would
look like in the "C" or "C++" world.

I have a C++ network layer that maintains its own send and receive
queues that is pumped by threads managed within this layer. It
dispatches remote messages received to various message handlers
which are now in C++; I would like to add the capability to add lua
based message handlers.

But I am having difficulty in conceptualizing the main loop that
pumps the network layer, the timer facility and the Lua engine. That
is my problem of course, but any code to illustrate how a main loop
with threads in the picture would probably help me in rearchitecting
my current project to use Lua.

On a frivolous note, I would probably name this package as
'concurrency' instead of 'helper'.



--
Javier

[Javier Guerra Giraldez]

On Thursday 09 March 2006 1:16 pm, Vijay Aswadhati wrote:
> This may be the beginning of something new and exciting!!

i hope so! either because it's accepted, or because somebody does something
much better after seeing this as a proof of concept.

> A few concrete examples would be of great help. I am particularly
> interested in how one would go about creating a singleton timer

this is precisely what i did as a testbed for the library.  here's the full C
code:

#include <sys/time.h>
#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>

#include "lua.h"
#include "lauxlib.h"

#include "helper.h"

typedef struct timer_udata {
        struct timeval tv;
        int ret;
} timer_udata;

static int timer_prepare (lua_State *L, void **udata) {
        lua_Number t = luaL_checknumber (L, 1);
        timer_udata *td = (timer_udata *)malloc (sizeof (timer_udata));
        if (!td)
                luaL_error (L, "can't alloc udata");
       
        td->tv.tv_sec = (int) t;
        td->tv.tv_usec = (t - td->tv.tv_sec) * 1000000;
       
        *udata = td;
       
        return 0;
}

static int timer_work (void *udata) {
        timer_udata *td = (timer_udata *)udata;
        fd_set fd_a, fd_b, fd_c;
        FD_ZERO (&fd_a);
        FD_ZERO (&fd_b);
        FD_ZERO (&fd_c);
       
        printf ("before select\n");
        td->ret = select (0, &fd_a, &fd_b, &fd_c, &td->tv);
        printf ("after select\n");
       
        return 0;
}

static int timer_finish (lua_State *L, void *udata) {
        timer_udata *td = (timer_udata *)udata;
       
        if (td->ret < 0)
                luaL_error (L, strerror (td->ret));
       
        free (td);
       
        return 0;
}

static task_ops timer_ops = {
        timer_prepare,
        timer_work,
        timer_finish
};

int luaopen_timer (lua_State *L);
int luaopen_timer (lua_State *L) {

        helper_init ();
        add_helperfunc (L, &timer_ops);
        return 1;
}


as you can see, timer_prepare() gets the Lua parameters (a single number),
allocates a private structure and fills it with a timeval (with sec and usec
fields).  timer_work() runs in the background, so it can't touch the Lua
space; it gets the timeval struct and does a select() to wait some time.
timer_finish() verifies the return value of the select(), and frees the
udata.  if there was any final result (like on a read funcion), it would be
pushed in the Lua stack.

note that since it defines just a single function, it's returned in the stack
by luaopen_timer(), instead of building a package.

this is how i tested it from Lua:

require "helper"
timer = require "timer"

q1=helper.newqueue()
q2=helper.newqueue()
print ("queues:", q1, q2)

th=helper.newthread (q1, q2)
print ("thread:", th)

tsk=timer(10)
print ("task:", tsk)

q1:addtask (tsk)
t2=q2:wait()
print ("t2:", t2)

helper.finish (t2)
print ("end")

----------------------------

th is the helper thread.  it uses q1 and q2 as input and output queues,
respectively.  as soon as it's created, it tries to get a task from q1; but
since it's empty, it's blocked.

the timer() function doesn't do the 'work' immediately, it just creates and
returns a 'task' (a lightuserdata).  it also calls the timer_prepare()
function.

when the task is added to q1, the thread gets it and executes the timer_work()
function.  the Lua code continues to run in the 'main' thread.

the q2:wait() function blocks until there's a task in q2.  when timer_work()
ends, the task is pushed to q2 and the Lua code unblocks.  t2 gets the task,
it should be equal to tsk (that's why i used lightuserdata).  the
helper.finish() call gets any result (none in this example)

> instance using this package and how the main dispatch loop would
> look like in the "C" or "C++" world.

my idea is that the main dispatch loop would be Lua code, not C.

i haven't written one yet, but this would be the general idea:

wrap all functions that return a task, like this:

local _read = read
function read (file)
    return helper.finish (yield (_read (file)))
end

and in the dispatcher:

while true do
    local tsk_done = out_q:wait()
    local co = coros [tsk_done]
    local tsk_blk = coroutine.resume (co, tsk_done)
    coros [tsk_blk] = co
    in_q:addtask (tsk_blk)
end


> On a frivolous note, I would probably name this package as
> 'concurrency' instead of 'helper'.

i'm notoriously bad at picking names for my code... any explanation why you
think 'concurrency' is better than 'helper' ?

--
Javier

[Vijay Aswadhati-2]

On Thursday, March 09, 2006 1:07 PM, Javier Guerra wrote:


> i hope so! either because it's accepted, or because somebody does
> something much better after seeing this as a proof of concept.

Thank you for taking the time to explain with an example. I think it
is reminiscent of the Windows QueueUserWorkItem() API. The missing
part provided by your package is the use of queues to make it safe
to communicate the results to the Lua universe. Do these queues
provide a 'peek' and 'wait for a duration' functions?

I am interested in durable tasks that run outside the Lua universe
and emit application events that need to be transported into the Lua
universe safely.

>From the example you have provided and what I have understood so far
it seems like the package is meant for one-shot (i.e ephemeral)
tasks which just run to completion. Is that a fair observation? Or
am I not looking hard enough?


> i'm notoriously bad at picking names for my code... any
> explanation why you think 'concurrency' is better than 'helper'?

Glad to know that I am not alone. My suggestion to name it
'conurrency' package may be inappropriate as well if my
understanding of the package is correct. Still it looks promising so
here are my thoughts:

- Java has a 'concurrency' package that provides several different
ways to 'execute' something concurrently and how these executables
get scheduled. My first read felt like there were similarities in
scope.

- In a mult-cpu machine the task may indeed be executing
concurrently.

- It has the potential to be 'the concurrency' package ;-)

If all of the above seem shallow arguments then I use my parachute
to remind you that I did preface it as 'on a frivolous note...'

Cheers
Vijay Aswadhati

[Javier Guerra Giraldez]

On Thursday 09 March 2006 5:39 pm, Vijay Aswadhati wrote:
> Thank you for taking the time to explain with an example. I think it
> is reminiscent of the Windows QueueUserWorkItem() API. The missing
> part provided by your package is the use of queues to make it safe
> to communicate the results to the Lua universe. Do these queues
> provide a 'peek' and 'wait for a duration' functions?

not yet...

'peek' would be easy, but 'wait for a duration' not so much.

BTW, i've only write it in Linux; i guess the structure would work in windows,
but it'll need some kind of API compatibility layer (hopefully just a few
macros)

> I am interested in durable tasks that run outside the Lua universe
> and emit application events that need to be transported into the Lua
> universe safely.
>
>From the example you have provided and what I have understood so far
> it seems like the package is meant for one-shot (i.e ephemeral)
> tasks which just run to completion. Is that a fair observation? Or
> am I not looking hard enough?

each task runs only once, but threads stay running until explicitly killed.  
the idea is that you would spawn several threads, even on the same
input/output queues.  another possibility is to have several input queues and
a single output queue, or anything like this.  that makes it more desirable
to have 'peek' and 'wait for a duration' functions.

i think it would be possible to do some control inversion to get the
'application events' pattern.

> If all of the above seem shallow arguments then I use my parachute
> to remind you that I did preface it as 'on a frivolous note...'

i'm not proud of the 'helper' name; but i remember seeing this kind of tricks
referred as "helper threads".

i still have some linking quirks to clean up, and i'll post it.  i hope to
have a better name by then.

--
Javier

[Adrian Sietsma]

Javier Guerra wrote:
> hi everybody
>
> finally, i think i have it working! it's a library of helper threads, meant to
> make it easy to write nonblocking libraries in C.
>

I have implemented a pure lua version of an event iterator based on Mike
Pall / Diego Nehab model of 2004.
It is working, but i'm still tinkering with parameter order etc.

I am using it as a prototype for co-operative multiplexing; it should end up
in c when the design is frozen.

The companion thread despatcher (coplex) should be functional in a few days.

I personally am hooked on the idea of 2 seperate layers - one to provide
events / event notifications, and ont to despatch event handlers.

How do we go about stimulating discussion / idea swapping in this area ?

This list ? A new wiki page ?

Adrian (draft event spec attached)

Event

The event namespace offers a generic mechanism for event handling. This is primarily used whenever you need to watch for multiple sources of events (e.g. multiple sockets) and cannot afford to block waiting on any single one to become ready.

The event handling mechanism in this module was designed and implemented by Mike Pall and Diego Nehab for Luasocket as a better alternative to the abominable socket.select() API.

Modified by A. Sietsma to match prototype Lua implementation.
modified/deleted functions are shown thus
new/modified functions are shown thus
—Adrian

The main features of this API are:

• A unified event model for various kinds of event types and event sources (such as sockets or timers).
• Triggered events can be fetched with a passive iterator. Any kind of dispatching mechanism can be adapted to work on top of this API.
• A common internal API for the OS-specific backends.

To obtain the event namespace, run:

-- Loads the event module and everything it requires.
local event = require("event")

Concepts

An event is any occurence which may happen at some future time (or may have already happened).
An event has the following properties:

• The event id is a unique number that is assigned to an event when it is added to an event container.
• The event type specifies the conditions under which an event is triggered.
• The event source specifies a system resource or a virtual object that is able to trigger an event.
• Zero, one or more associated objects can be associated with an event. These can be any Lua object and the application is free to use them for any purpose.
• A context can be associated with an event. This can be any Lua object and the application is free to use them for any purpose.
• A timeout can be specified for any event. The event will be returned after timeout seconds, if not already triggered.

An event container allows for adding, deleting and triggering events and provides an iterator to fetch triggered events.

Functions provided by the Event namespace

• new  Creates and initializes a new event container.
• time  Returns the system time.

Event container (meta) methods

• :add  Adds an event to the container.
• :once  Add an immediate one-shot event.
• :del  Deletes an event from the container.
• :set  Modifies a previously added event.
• :trigger  Trigger a virtual event.
• :stop  Stops any running iterator (on this container).
• :clear  Stops any running iterator, and deletes all events from container.
• __call  The event container can be called and returns an iterator. Calling the iterator returns triggered events from the container.

Here are the steps you have to do in order to use the event handling mechanism:

• Create a new event container.
• Add all events you are interested in to the container.
• Create an iterator object from the container and use it in a loop to fetch the triggered events.
• Act on the triggered events in the body of the loop. E.g. read some data, run a callback function or resume a thread.
• You can add new events to the container or delete events from it anytime during processing.

Here is a simple example:

-- Create a new event container with 1 associated object per event.
local ev = event.newset(1)-- Create a new event container.

-- Add some events to the container.
ev:add("t", 2.5,"One-shot timer expired")
local tid = ev:add("T", 5.0, "Periodic timer triggered")
local vid = ev:add("v", nil, "Virtual event triggered")


-- Process all triggered events in an infinite loop.
for id, typ, src, str in ev(true) do 
for id, why, src, str in ev(true) do 
  -- Print the current time and the associated object.
  print(socket.gettime(), str)
  if id == tid then
    ev:trigger(vid)
  elseif id == vid then
    ev:add("t", 1.0, "One second later ...")
  end
end

Strings are used as associated objects for the sake of keeping this example simple. The body of the loop does nothing spectacular either. Better examples that show various event dispatching strategies will follow in the next sections.

Event types

The following table lists all supported event types:

event type	event source	trigger condition	trigger return	POSIX	Windows
"v", "V"	Virtual event	has been triggered	"triggered"	X	X
"t", "T"	Timer	has expired	"timeout"	X	X
"r", "R"	POSIX file	is readable	"readable"	X
"w", "W"	POSIX file	is writable	"writable"	X
"r", "R"	Lua Socket	is readable	"readable"	X	X
"w", "W"	Lua Socket	is writable	"writable"	X	X
"x", "X"	POSIX file	has an exception pending	X
"s", "S"	POSIX signal	has been caught	X
"h", "H"	Windows handle	state is signaled		X
"m", "M"	Windows message	is in the input queue		X

Event types are passed to the API as one-character strings. Lowercase letters indicate temporary events; these are deleted from the event container once the event has been triggered. Uppercase letters indicate persistent events; you have to explicitly delete them from the container.

There are other event sources imaginable. E.g. virtual events that work across threads or processes or virtual events with a counter. Extending the list with events triggered by other system resources is difficult, because you need a single call in the backend that checks for all events at once. Otherwise you are doomed to use polling. Ugh. More input is welcome. — Mike

Event sources

Virtual events

A virtual event is a simple condition that can be either in the untriggered state or in the triggered state.

The event source must be set to nil when adding a virtual event with ev:add.

The event source will be treated as the initial triggered state when adding a virtual event with ev:add. eg. a source of true will cause the event to be immediately triggered.

Virtual events can only be triggered with ev:trigger.

One-shot virtual events are deleted when they are returned from the iterator; persistent virtual events can be triggered multiple times and have to be deleted manually, unless they have a timeout. A virtual event is not retriggerable until it is returned by the iterator. It will be returned only once by the iterator, no matter how many times it has been triggered in-between.

Virtual events are tested prior to the internal select() call, and again if the select() times out; they will not interrupt the select.

POSIX and Windows notes removed : I have no idea how much of it is still true—Adrian

Timers

A Timer is specified by a number that gives the time interval after which the timer expires. The time unit is one second; fractional values are allowed. The time interval is relative to the time the event was added. Temporary timers trigger an event only once; Persistent timers periodically trigger the event until you delete them from the event container.

Note: Timers never expire before the requested time, but may expire shortly after the requested time depending on system timer resolution and event queueing delays. In general you cannot rely on 100% accurate timing in a non-realtime operating system anyway.

POSIX and Windows events section hidden (not supported).

Socket/File handles

A socket (and file on unix) is identified by a file descriptor. More specifically, it must return a file descriptor via getfd for the LuaSocket select() function. This is restricted to LuaSocket objects on Windows, but can be a disk file, a pipe,a socket or any other file resource your OS provides. An associated event will be triggered when a file is readable or writable. , writable or has an exception pending, depending on the event type.

A file descriptor obtained from an external Lua module can be passed in as an object that has a getfd() function.

Note: You have to make sure that the container only holds valid file descriptors at all times. You have to delete all untriggered events and all persistent events before closing the file descriptor.

Note: The results of adding the same file descriptor with the same event type to more than one event container are undefined.

Functions provided by the Event namespace

ev = event.new([numobj],)

Creates and initializes a new event container.

The numobj parameter gives the maximum number of objects that can be associated with each event. The default is zero.

The function returns the newly created event container. See the section on Event container methods below.

Associated objects An associated object can be used to pass context information along with events. They are It is passed in when the event is added to the container and will be returned by the iterator when the event has been triggered. Associated objects are typically used by a dispatcher to tie an event to a callback, a state or a thread.

The event container needs to create numobj tables to store the associated object. Adding, deleting or iterating over events incurs additional overhead depending on the number of associated objects.

Note: Usually only a single event container is needed in a single process. However different event containers are independent of each other and can be used e.g. in different native threads in a shared Lua universe. It is not safe to access the same event container from different native threads without proper locking.

Note: The effects of adding the same system event source to different event containers are unpredictable. In general the set of event sources should not overlap between any two event containers in a single process.

Example:

-- Create a new event container
 with 1 associated object per event.
local ev = event.newset(1)

Event container methods

id = ev:add(type [, source [, obj*]])

id = ev:add(type [, source [,context [, timeout]]])

Adds an event to the event container with the given event type, event source , timeout, and context object. and associated objects Depending on the event type a previous event with the same type and/or the same source may be replaced.

For details about the valid event types and sources see the sections on Event types and Event sources. See below for the timeout parameter.

Returns the event id for the newly created event as a LIGHTUSERDATA object negative number

Timeout

A timeout can be specified when adding an event. The event will be returned with a reason of "timeout" after timeout seconds, if not triggered before then. A non-numeric timeout value means no timeout. Note: For persistent events, the timeout is re-applied when the event is returned (triggered).

The timeout parameter is ignored for timer events.

The following examples show different ways to use associated objects. Each one of them needs an appropriate dispatcher that knows what to do with the associated object(s) returned from the iterator. I.e. call it or do some other processing.

-- Create a one-shot timer with a callback function.
ev:add("t", 10.0, function() print("Timer expired!") end)

-- Create a persistent timer with a table that holds a complex object.
local obj = { notify = "acceptor", rehash = true }
obj.id = ev:add("T", 60.0, obj)

-- Typical event mode socket reader.
local function reader(ev, obj)
  local s, err, part, id = obj.sock:receive()
  while s do
    obj:receiver(obj.part .. s)
    obj.part = ""
    s, err, part, id = obj.sock:receive()
  end

  if err == "event" then
    if part then obj.part = obj.part .. part end
    ev:set(id, reader, obj) -- Set associated objects for the created event.
    return
  end

  obj:onerror(err)          -- Finalize the object and pass the error.
  sock:close()
end

-- (Some initialization omitted for brevity)
obj.part = ""               -- Initialize partial receive buffer.
obj.sock:seteventmode(ev)   -- Put the socket into event mode.
ev:once(reader, obj)        -- Run the reader once to setup the transfer.

POSIX and Windows events examples hidden—Adrian

id = ev:set(id [, obj*]) id = ev:set(id [, obj, [timeout]])

Sets or overrides the associated objects and timeout for an existing event.

The event id specifies the event that is to be associated with the given objects. The timeout parameter (if non-nil) specifies the new timeout.

Returns the modified event id as a convenience. Returns nil if the event id is invalid or has been deleted.

Example:

-- Add a persistent timer with a callback function.
local tid = ev:add("T", 2.5, function() print("Timer expired!") end)

-- Later, somewhere else, maybe in a callback: Modify the callback.
ev:set(tid, function() print("Oh no, the timer expired again!") end)

id = ev:trigger(id)

Trigger a virtual event. The event will be delivered synchronously by the iterator.

The event id specifies the event to trigger.

Returns the id of the triggered event as a convenience. Returns nil if the event id is invalid, has been deleted or does not refer to a virtual event.

The C API only requires the specification of the event id. Neither the event container object nor the Lua state is required because event ids are unique across all event containers of a single process.

Example:

-- Add a persistent virtual event.
local vid = ev:add("V", nil, function() print("Triggered!") end)

-- Later, somewhere else, maybe in a callback: Trigger it (once).
ev:trigger(vid)             -- Event will be returned in dispatcher loop.

-- Add a virtual event with a lifetime of 1 minute
local vid = ev:add("v", nil, function() print("Triggered!") end,60.0)

id = ev:once([obj*])

Generate an immediate one-shot event. This is implemented by adding a virtual event and triggering it. The event will be delivered synchronously by the iterator.

The event is added with the specified associated objects.

Returns the id of the added event.

Example:

-- Arrange for calling two functions from the dispatcher loop.
ev:once(function() print("Hello world!") end)
ev:once(function() print("The current time is:", os.date()) end)

id = ev:del(id)

Deletes an event from the event container.

The event to be deleted is specified by the event id.

Returns the id of the deleted event as a convenience. Returns nil if the event id is invalid or has already been deleted.

Example:

-- Add a persistent timer that prints a counter every second.
local cnt = 0
local tid = ev:add("T", 1.0, function() cnt = cnt + 1; print(cnt) end)

-- Later, somewhere else, maybe in a callback: Delete the timer to stop it.
ev:del("t", tid)

ev:stop()

Stops any running iterator.

This method can be used anywhere inside the iteration loop, e.g. in callback functions. It forces any running iterator to return nil on the next invocation which terminates the loop. Note that any pending triggered events are not deleted. Creating a new iterator and restarting the loop will just continue returning them.

ev:clear()

Deletes all events from an event container.

This method can be used from anywhere, e.g. inside the iteration loop or in callback functions. Pending triggered events are deleted, too. A surrounding iteration loop will abort since the iterator returns nil when the container is empty.

The container can be reused and new events can be added immediately after calling ev:clear(). A surrounding iteration loop will still abort if non-blocking behaviour is selected since a new event collection cycle has to be started.

iterator, ev, nil = ev([block])

id, trigger, source, context = iterator()
id, type, source [, obj*] = iterator(ev, id)

The event container object can be called and returns an iterator. Calling the iterator returns triggered events from the container.

The block parameter specifies the blocking behaviour of the iterator.
true = block. false = never block, number = block for n seconds (fractional). nil (default) = block only if the container holds at least one timer event.

Three values are returned by calling the event container object: The iterator function, the event container object and nil. The iterator function returns the event id, the reason it was triggered ( one of "read", "write", "triggered", "timeout"), the event source and the context object. Note that any event type may be returned with a reason of "timeout". The iterator function returns the event id, the event type, the event source and any associated objects.

Each call to the iterator function returns a single event that has been triggered. The iterator returns nil if a) the container is empty or b) non-blocking behaviour was selected and no triggered events from the current event collection cycle are remaining.

The iterator is destructive because it removes triggered events and may also delete events from the container as a result. Do not use multiple iterators over the same container in parallel. Aborting the loop and creating a new iterator is allowed though. It is safe to call other methods even while using an iterator over the same event container.

Iterators are typically used in for loops and that takes care of managing the iterator object.

-- Careful: Using 'type' as a variable name will override the global
-- function of the same name. Using 'etype' is a better choice.

-- Process all triggered events in an infinite loop.
for id, etype, src, obj in ev(true) do
for id, trigger, src, obj in ev(true) do
  ...
end

-- One-shot loop through all events that have been triggered (non-blocking).
-- ( Do not wrap this up into another loop unless the other loop blocks
-- for some time. Otherwise you get polling behaviour.)
for id, etype, src, obj1 , obj2 in ev(false) do  
for id, trigger, src, context in ev(false) do  
  ...
end


-- Process all triggered events, waiting for up to 1 second for select.
for id, trigger, etype, src, obj in ev(1.0) do
  ...
end

Dispatching strategies section omitted—Adrian

---

Last modified by Adrian Sietsma on
Thu Mar 09 05:00:00 CEST 2006

[Javier Guerra Giraldez]

On Thursday 09 March 2006 7:00 pm, Adrian Sietsma wrote:
> Javier Guerra wrote:
> > hi everybody
> >
> > finally, i think i have it working! it's a library of helper threads,
> > meant to make it easy to write nonblocking libraries in C.
>
> I have implemented a pure lua version of an event iterator based on Mike
> Pall / Diego Nehab model of 2004.
> It is working, but i'm still tinkering with parameter order etc.

amazing; i read the model proposal and found it very compelling, but too much
extensive.  and (for me) the weak point is how to create new libraries that
would work nicely with the event framework.

so, i proposed a different approach: make it easy to write nonblocking
libraries with a regular API, so that a pure Lua dispatcher would be much
easier to write.

mostly, i wanted to keep with the primitives/mechanisms separation.  i think
the dispatcher should be an application specific mechanism, but to make it
easy, it was needed a new primitive.  i think it's called a 'future' in other
languages.

> I personally am hooked on the idea of 2 seperate layers - one to provide
> events / event notifications, and ont to despatch event handlers.

i first tinkered with LuaThreads, but it seems that everything is far easier
and there's less resource contention if the background threads can't touch
the Lua State.

i'm trying to keep the dispatcher design out of my library, so that you could
choose between any number of threading styles: one-per-tasktype,
one-per-user, worker-pool, etc.

> How do we go about stimulating discussion / idea swapping in this area ?
> This list ? A new wiki page ?

as long as nobody objects, i'd suggest staying on this list...


--
Javier

[Jim Mellander]

Hi everyone:

I'm considering porting (part of) a perl application to lua to run on an
embedded Linux platform.  One missing component I don't see easily
available is the ability to send out arbitrarily constructed packets via
a raw socket interface.  Has anyone created such a binding?

--
Jim Mellander
Incident Response Manager
Computer Protection Program
Lawrence Berkeley National Laboratory
[phone]

Your fortune for today is:

You will be the last person to buy a Chrysler.

[D Burgess-4]

Not that I am aware of, but it should be a relatively simple matter
to adapt Luasocket to do this. Small amount of code change
is required.

David B.

On 3/10/06, Jim Mellander <[hidden email]> wrote:

> Hi everyone:
>
> I'm considering porting (part of) a perl application to lua to run on an
> embedded Linux platform.  One missing component I don't see easily
> available is the ability to send out arbitrarily constructed packets via
> a raw socket interface.  Has anyone created such a binding?
>
> --
> Jim Mellander
> Incident Response Manager
> Computer Protection Program
> Lawrence Berkeley National Laboratory
> [phone]
>
> Your fortune for today is:
>
> You will be the last person to buy a Chrysler.
>
>

[Diego Nehab-3]

Hi,

> I'm considering porting (part of) a perl application to lua to run on an
> embedded Linux platform.  One missing component I don't see easily available
> is the ability to send out arbitrarily constructed packets via a raw socket
> interface.  Has anyone created such a binding?

I can send you one that implements Unix domain sockets. The example
version I distributed with LuaSocket 2.0 is broken (my bad), but it
should be easy to fix and I will do that soon. From that, you can figure
out how to change it to raw sockets.

Regards,
Diego.

[Jim Mellander]

Diego Nehab wrote:

> Hi,
>
>> I'm considering porting (part of) a perl application to lua to run on
>> an embedded Linux platform.  One missing component I don't see easily
>> available is the ability to send out arbitrarily constructed packets
>> via a raw socket interface.  Has anyone created such a binding?
>
>
> I can send you one that implements Unix domain sockets. The example
> version I distributed with LuaSocket 2.0 is broken (my bad), but it
> should be easy to fix and I will do that soon. From that, you can figure
> out how to change it to raw sockets.
>
> Regards,
> Diego.
>

Thanks, I'll take you up on that offer.

The application I'm interested in is a small version of "PacketFence"
www.packetfence.org, on a commodity router, for deployment internally in
our network for network control.  In particular, I intend to use ARP
spoofing to isolate internal miscreant systems.  I could write the app
in C, but I expect lua will provide a quick proof of concept, and likely
adequate performance for production, at a savings in programming time.
The original app requires perl, mysql, etc. - way more than I can put on
the box, and way more functionality than I want anyway.....




--
Jim Mellander
Incident Response Manager
Computer Protection Program
Lawrence Berkeley National Laboratory
[phone]

Your fortune for today is:

Men still remember the first kiss after women have forgotten the last.

[Mark Hamburg-4]

Here is a short overview of the system we're using in Lightroom. It isn't as
simple as what Javier has proposed but it shares some similarities. It also
has evolved over time and has not had the opportunity to undergo as many
complexity-reduction passes as it probably deserves.

Concept 1: Multiple universes

We run multiple Lua universes that can generally function independently
without regard to a mutex.

Concept 2: Universe-to-universe copying

We implemented copying for a subset of Lua data structures between
universes. It is up to the caller to insure that neither universe is
attempting to execute other code at the time.

Concept 3: Transit universe

We implemented a primary transit universe which is wrapped with a mutex and
that is intended for communication between universes. (But see below.)

Concept 4: Transit queues

We implemented queues with values in the transit universe and appropriate
conditions and mutexes for reading and writing. Transit queues are capable
of being referenced from Lua and are reference counted so that they can be
referenced from multiple universes safely. (Note: In retrospect, we could
have just implemented each queue with its own universe.)

Concept 5: Spawning universes

We implemented a call which takes a Lua function without upvalues and for
which a default environment is acceptable and provided a way to create a new
Lua universe, load that function, and start it running with a set of
parameters carried over again via the intra-universe copying mechanism. In
particular, we can pass it queues to wait on for input and to write to with
results.

These required small bits of native code to implement, but the result is a
system in which most of the structure can be wired up in Lua.

Code running in other universes should work on limited amounts of data since
we want to limit the amount of data to copy between universes, but there are
plenty of tasks where we can take a small description, go churn away for a
while, and come back with a reasonably small answer.

Mark

[Asko Kauppi]

Not reading the whole message thread (sorry), but LuaX code has a  
working inter-thread messaging code (no dependencies) which was  
needed to get SDL_Mixer callbacks running (to Lua). They are called  
for pre/postprocessing of any sample blocks being played, but the  
call comes from a separate thread.

It consists of a data queue (FIFO), adding stuff in C side, and  
removal (by polling) in the Lua side.

Not sure this helps, but at least it shows there is a need for such a  
feature. Sometimes. :)

btw, the efficiency of running a music, and letting Lua code show its  
waveform on the screen is rather Good! :D

-asko


Javier Guerra kirjoitti 10.3.2006 kello 2.26:

> On Thursday 09 March 2006 7:00 pm, Adrian Sietsma wrote:
>> Javier Guerra wrote:
>>> hi everybody
>>>
>>> finally, i think i have it working! it's a library of helper  
>>> threads,
>>> meant to make it easy to write nonblocking libraries in C.
>>
>> I have implemented a pure lua version of an event iterator based  
>> on Mike
>> Pall / Diego Nehab model of 2004.
>> It is working, but i'm still tinkering with parameter order etc.
>
> amazing; i read the model proposal and found it very compelling,  
> but too much
> extensive.  and (for me) the weak point is how to create new  
> libraries that
> would work nicely with the event framework.
>
> so, i proposed a different approach: make it easy to write nonblocking
> libraries with a regular API, so that a pure Lua dispatcher would  
> be much
> easier to write.
>
> mostly, i wanted to keep with the primitives/mechanisms  
> separation.  i think
> the dispatcher should be an application specific mechanism, but to  
> make it
> easy, it was needed a new primitive.  i think it's called a  
> 'future' in other
> languages.
>
>> I personally am hooked on the idea of 2 seperate layers - one to  
>> provide
>> events / event notifications, and ont to despatch event handlers.
>
> i first tinkered with LuaThreads, but it seems that everything is  
> far easier
> and there's less resource contention if the background threads  
> can't touch
> the Lua State.
>
> i'm trying to keep the dispatcher design out of my library, so that  
> you could
> choose between any number of threading styles: one-per-tasktype,
> one-per-user, worker-pool, etc.
>
>> How do we go about stimulating discussion / idea swapping in this  
>> area ?
>> This list ? A new wiki page ?
>
> as long as nobody objects, i'd suggest staying on this list...
>
>
> --
> Javier

[Javier Guerra Giraldez]

On Friday 10 March 2006 12:40 am, Mark Hamburg wrote:
> Here is a short overview of the system we're using in Lightroom. It isn't
> as simple as what Javier has proposed but it shares some similarities. It
> also has evolved over time and has not had the opportunity to undergo as
> many complexity-reduction passes as it probably deserves.

i saw the description in the Adobe presentation, it was one of the case points
i considered while designing my package.  it seems your system manages the
much more general case about running Lua code in parallel. I still think the
best way to get that should be something like LuaThreads, but with a much
finer mutex handling.  unfortunately, that would mean a very deep analysis of
the tables and stack handling core.

my package is simpler because it limits what can be done in 'background' code:
it shouldn't touch the Lua State.  that's enough to write C glue code to
blocking libraries without freezing the Lua code.


--
Javier

[Zachary P. Landau-4]

On 3/9/06, Diego Nehab <[hidden email]> wrote:

> Hi,
>
> > I'm considering porting (part of) a perl application to lua to run on an
> > embedded Linux platform.  One missing component I don't see easily available
> > is the ability to send out arbitrarily constructed packets via a raw socket
> > interface.  Has anyone created such a binding?
>
> I can send you one that implements Unix domain sockets. The example
> version I distributed with LuaSocket 2.0 is broken (my bad), but it
> should be easy to fix and I will do that soon. From that, you can figure
> out how to change it to raw sockets.

Diego,

I also have a use for being able to manipulate raw sockets with LuaSocket.
If you don't mind, could you send me the updated version of the code as well?

--
Zachary P. Landau <[hidden email]>

[Adrian Sietsma]

I am interested in using daniel manura's cnumber patch
<http://lua-users.org/wiki/CnumberPatch>
  for simple interlocked/mutexed data interchange between lua universes.

On Windows, I can use the Interlocked... family of functions for lightweight
locking... anyone see a major flaw ?

Adrian

[Adrian Sietsma]

Adrian Sietsma wrote:
> I am interested in using daniel manura's cnumber patch
> <http://lua-users.org/wiki/CnumberPatch>
>  for simple interlocked/mutexed data interchange between lua universes.
>
> On Windows, I can use the Interlocked... family of functions for
> lightweight locking... anyone see a major flaw ?
>
Scratch that idea now.

foo = foo + 1

back to cfunction set & get methods (sigh)

Adrian