Properly binding C libraries using callbacks

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Properly binding C libraries using callbacks

Rob Hoelz-2
Hello list,

I'm currently writing a binding for linenoise
(https://github.com/antirez/linenoise), and I'm trying to make sure and
do things the "right way" when it comes to binding its completion
callbacks.

The signature for the callback setting function is this:

void linenoiseSetCompletionCallback(void (*callback)(const char *, linenoiseCompletions *));

and this is what my wrapper looks like:

static int completionFuncRef;
static lua_State *completionState;

static void completionCallbackWrapper(const char *line, linenoiseCompletions *completions)
{
    lua_State *L = completionState;
    int status;

    lua_rawgeti(L, LUA_REGISTRYINDEX, completionFuncRef);
    lua_pushlightuserdata(L, completions);
    lua_pushstring(L, line);

    /* XXX error handling */
    status = lua_pcall(L, 2, 0, 0);
}

static int l_setcompletion(lua_State *L)
{
    luaL_checktype(L, 1, LUA_TFUNCTION);

    lua_pushvalue(L, 1);
    completionFuncRef = luaL_ref(L, LUA_REGISTRYINDEX);
    linenoiseSetCompletionCallback(completionCallbackWrapper);
    completionState = L;

    return 0;
}

Obviously, this implementation has some holes; for instance, this example will crash:

local ln = require 'linenoise'

local co = coroutine.create(function()
  ln.setcompletion(function(completions, line)
    ln.addcompletion(completions, 'foo')
    ln.addcompletion(completions, 'bar')
    ln.addcompletion(completions, 'baz')
  end)
end)

coroutine.resume(co)

co = nil

collectgarbage 'collect'

local line = ln.linenoise '> '
print(line)

Can someone give me some pointers on the proper way to bind a library that uses callbacks, especially one that doesn't
allow you to thread your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that
storing things in global variables in the binding are a faux pas.

Thanks,
Rob

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

Patrick Rapin
>I'm currently writing a binding for linenoise

Your work interests me. I also tried to replace the Readline library
with Linenoise for the completion of my LuaDura application.
After some time, I gave up, not because of crashes, but because it
seemed that the project lacks of maturity, and I got lost in the 42
different forks inside GitHub...
Does Antirez still maintain his library? Based on the number of pull
requests, one might have some doubts.
So what fork are you using?

Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

Rob Hoelz-2
On Thu, 20 Oct 2011 23:22:35 +0200
Patrick Rapin <[hidden email]> wrote:

> >I'm currently writing a binding for linenoise
>
> Your work interests me. I also tried to replace the Readline library
> with Linenoise for the completion of my LuaDura application.
> After some time, I gave up, not because of crashes, but because it
> seemed that the project lacks of maturity, and I got lost in the 42
> different forks inside GitHub...
> Does Antirez still maintain his library? Based on the number of pull
> requests, one might have some doubts.
> So what fork are you using?
>
I'm just using antirez's version.  It's not very mature, but it's
simple and does the bare minimum that I need it to.  I'm also working
on an editline binding (https://github.com/hoelzro/luael, but I haven't
touched it in some time.  I intend to work on it in the next few weeks.

signature.asc (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

Doug Currie
In reply to this post by Rob Hoelz-2

On Oct 20, 2011, at 5:07 PM, Rob Hoelz wrote:

> Can someone give me some pointers on the proper way to bind a library that uses callbacks, especially one that doesn't
> allow you to thread your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that
> storing things in global variables in the binding are a faux pas.

There are several approaches.

1. Put the onus on the library user to make sure the thread that creates the callback remains in existence at least as long as the callback may be executed, i.e., punt

2. When a callback is created, put the thread in a table maintained by the library for the purpose of preventing the thread from being garbage collected; this table can be in the library's environment (or be an upvalue of the library's functions) or may be in the registry

3. When the library is loaded, capture the thread into which it is loaded (which is more likely to the the main thread) and use that thread for callbacks; optionally add #2 to keep this thread from being collected

4. When the library is loaded, create a new thread that is used exclusively to process callbacks; use #2 to prevent this thread from being collected

5. In Lua 5.2 use LUA_RIDX_MAINTHREAD to get a handle on the main thread and use it for callbacks

Of these options, #4 is the safest in Lua 5.1, but uses extra memory.

e


Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

Ignacio Burgueño
In reply to this post by Rob Hoelz-2
On Thu, Oct 20, 2011 at 7:07 PM, Rob Hoelz <[hidden email]> wrote:
Hello list,

...

Can someone give me some pointers on the proper way to bind a library that uses callbacks, especially one that doesn't
allow you to thread your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that
storing things in global variables in the binding are a faux pas.

Thanks,
Rob


Maybe the code here can shed some light?


Regards,
Ignacio
Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

Sam Roberts
In reply to this post by Rob Hoelz-2
On Thu, Oct 20, 2011 at 2:07 PM, Rob Hoelz <[hidden email]> wrote:

In:

> static int l_setcompletion(lua_State *L)
> {
>    luaL_checktype(L, 1, LUA_TFUNCTION);
>
>    lua_pushvalue(L, 1);
>    completionFuncRef = luaL_ref(L, LUA_REGISTRYINDEX);
>    linenoiseSetCompletionCallback(completionCallbackWrapper);

Don't do this:

>    completionState = L;

As soon as  l_setcompletion() returns, L should be treated as
undefined (because it could become so, as you demonstrated).

In ln.linenoise() -- you don't show the implementation but I guess it
looks like below --  you should do this:

void lua_linenoise(lua_State* L) {
   completionState = L; /* L is guaranteed to be defined and valid
until lua_linenoise() returns, so use it */

   char* s = linenoise_do_your_thing(); /* this is the whatever
function will call completionCallbackWrapper() */
   lua_pushstring(L, s);
   return 1;
}

That way the completionCallbackWrapper will use the correct lua_State.

However, it calls lua_pcall(), and that lua code can call
ln.linenoise()... is linenoise reentrant? Can you call back into it
from it's callback? Probably not, and you might want to protect
against that happening:

void lua_linenoise(lua_State* L) {
   if(completionState) {
      luaL_error(L, "linenoise is not reentrant, naughty!");
      }
   completionState = L; /* L is guaranteed to be defined and valid
until lua_linenoise() returns, so use it */
   char* s = linenoise_do_your_thing(); /* this is the whatever
function will call completionCallbackWrapper() */
   completionState = NULL;
   lua_pushstring(L, s);
   return 1;
}

If it is reentrant, and you want to support that, and the
completionCallbackWrapper() can be called multiple times before
ln.linenoise() returns (all things I don't know), then after calling
lua_pcall() in the completionCallbackWrapper, you have to again set
the completionState back to the L that it saved, because that's the
valid one.

> Can someone give me some pointers on the proper way to bind a library that uses callbacks, especially one that doesn't
> allow you to thread your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that
> storing things in global variables in the binding are a faux pas.

linenoise itself uses global state, it looks like, since
linenoiseSetCompletionCallback()  doesn't take
any kind of context as an argument. In that case, have the
completionState be global doesn't make
the binding any worse than the library it binds to.

Cheers,
Sam

Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

Tim Caswell
I have a similar problem in my luvit project.  I'm wrapping an
inherently async library that provides non-blocking I/O.  I ended up
creating a helper that calls my lua callbacks from the main thread.

https://github.com/creationix/luvit/blob/master/src/utils.c#L7-37

As far as getting a reference to the lua state, the API I'm using
always has a place for an opaque data pointer and I put in some struct
that contains L and some other data (like a lua_ref integer) to get at
the real data.

On Thu, Oct 20, 2011 at 3:26 PM, Sam Roberts <[hidden email]> wrote:

> On Thu, Oct 20, 2011 at 2:07 PM, Rob Hoelz <[hidden email]> wrote:
>
> In:
>
>> static int l_setcompletion(lua_State *L)
>> {
>>    luaL_checktype(L, 1, LUA_TFUNCTION);
>>
>>    lua_pushvalue(L, 1);
>>    completionFuncRef = luaL_ref(L, LUA_REGISTRYINDEX);
>>    linenoiseSetCompletionCallback(completionCallbackWrapper);
>
> Don't do this:
>
>>    completionState = L;
>
> As soon as  l_setcompletion() returns, L should be treated as
> undefined (because it could become so, as you demonstrated).
>
> In ln.linenoise() -- you don't show the implementation but I guess it
> looks like below --  you should do this:
>
> void lua_linenoise(lua_State* L) {
>   completionState = L; /* L is guaranteed to be defined and valid
> until lua_linenoise() returns, so use it */
>
>   char* s = linenoise_do_your_thing(); /* this is the whatever
> function will call completionCallbackWrapper() */
>   lua_pushstring(L, s);
>   return 1;
> }
>
> That way the completionCallbackWrapper will use the correct lua_State.
>
> However, it calls lua_pcall(), and that lua code can call
> ln.linenoise()... is linenoise reentrant? Can you call back into it
> from it's callback? Probably not, and you might want to protect
> against that happening:
>
> void lua_linenoise(lua_State* L) {
>   if(completionState) {
>      luaL_error(L, "linenoise is not reentrant, naughty!");
>      }
>   completionState = L; /* L is guaranteed to be defined and valid
> until lua_linenoise() returns, so use it */
>   char* s = linenoise_do_your_thing(); /* this is the whatever
> function will call completionCallbackWrapper() */
>   completionState = NULL;
>   lua_pushstring(L, s);
>   return 1;
> }
>
> If it is reentrant, and you want to support that, and the
> completionCallbackWrapper() can be called multiple times before
> ln.linenoise() returns (all things I don't know), then after calling
> lua_pcall() in the completionCallbackWrapper, you have to again set
> the completionState back to the L that it saved, because that's the
> valid one.
>
>> Can someone give me some pointers on the proper way to bind a library that uses callbacks, especially one that doesn't
>> allow you to thread your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that
>> storing things in global variables in the binding are a faux pas.
>
> linenoise itself uses global state, it looks like, since
> linenoiseSetCompletionCallback()  doesn't take
> any kind of context as an argument. In that case, have the
> completionState be global doesn't make
> the binding any worse than the library it binds to.
>
> Cheers,
> Sam
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

Pierre Chapuis
In reply to this post by Patrick Rapin
On Thu, 20 Oct 2011 23:22:35 +0200, Patrick Rapin wrote:

> Does Antirez still maintain his library? Based on the number of pull
> requests, one might have some doubts.

He does since it's used in Redis. But he probably won't implement
things he doesn't need in Redis for now.

The next thing he'll do is probably that:
https://github.com/antirez/linenoise/issues/23

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

RE: Properly binding C libraries using callbacks

AllanPfalzgraf
In reply to this post by Rob Hoelz-2
This is just another approach for your consideration.
I built a module that used a callback function for asynchronous events and while various methods to call a Lua function from the C library's callback function worked to a degree, none of them were solidly dependable.
The final solution was to use a UDP socket on 127.0.0.1.  The data on the C library side was converted to a text string for ease of handling on the Lua side.  A pleasant surprise was that the socket provides some queuing of events for those times with the reader could not keep up with the writer.  So far, this has been most reliable solution to the problem.

Lua code: http://codepad.org/K8XT6dX1
C code:    http://codepad.org/MvOYdUD7

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of [hidden email]
Sent: Thursday, October 20, 2011 5:48 PM
To: [hidden email]
Subject: lua-l Digest, Vol 15, Issue 105

Send lua-l mailing list submissions to
        [hidden email]

To subscribe or unsubscribe via the World Wide Web, visit
        http://vlists.pepperfish.net/cgi-bin/mailman/listinfo/lua-l-lists.lua.org

or, via email, send a message with subject or body 'help' to
        [hidden email]

You can reach the person managing the list at
        [hidden email]

When replying, please edit your Subject line so it is more specific than "Re: Contents of lua-l digest..."


Today's Topics:

   1. Re: modules, require, magic (Hisham)
   2. Lua 5.2 OP_LOADNIL instruction (liam mail)
   3. Properly binding C libraries using callbacks (Rob Hoelz)
   4. Re: Lua-friendly OSes (Ben Kelly)
   5. Re: Properly binding C libraries using callbacks (Patrick Rapin)
   6. Re: Properly binding C libraries using callbacks (Rob Hoelz)
   7. Re: Properly binding C libraries using callbacks (Doug Currie)
   8. Re: Properly binding C libraries using callbacks
      (Ignacio Burgue?o)
   9. Re: Lua-friendly OSes (Marc Balmer)


----------------------------------------------------------------------

Message: 1
Date: Thu, 20 Oct 2011 18:37:40 -0200
From: Hisham <[hidden email]>
Subject: Re: modules, require, magic
To: Lua mailing list <[hidden email]>
Message-ID:
        <[hidden email]>
Content-Type: text/plain; charset=UTF-8

On Thu, Oct 20, 2011 at 3:03 PM, Roberto Ierusalimschy <[hidden email]> wrote:
>> > Moreover,
>> > a modified 'load' function forces everyone to live with its magic.
>>
>> Not necessarily; Fabio's original loader did this but loader5
>> restricted the magic to the users of module().
>
> Maybe I am missing something, but I got the impression that, with
> loader5 (as posted to the list), I get an empty environment if I do
> not call 'module'.

Indeed; and at a first glance I don't seem to be able to fix this without resorting to the debug library. In any case, at this point I assume that "_ENV = module('name')" would be the preferred direction to go for a possible Lua 5.2 version of module().

> Â local function loader(name, modpath) Â  Â local mod, env = {}, {} Â  
> env.module = function (name)  ...   -- not called   ...
> Â  local ret = assert(load(source, modpath, 'bt', env))(name) Â  ...
> Â end
>
> Moreover, it seems that loader5 has some other drawbacks. (It needs to
> read the whole module as a string; all accesses to globals go all the
> time through function calls.)

With the module() + package.seebase variation suggested at [1] I think the only remaining drawback would be that all accesses to globals go all the time through function calls when using package.seebase.

Myself, I could live with that (resorting to the ugly "local print=print" idiom if performance was really needed); I'd rather take this hit than the other issues of package.seeall (in which globals already go through a metamethod anyway, no?), or having to localize every needed global from the base library.

Having said that, if the standard library remained the same as the one in Lua 5.1, I'd probably keep using package.seeall instead of rolling my own package.seebase, for the sake of sticking to standard idioms.

[1] http://article.gmane.org/gmane.comp.lang.lua.general/84444

-- Hisham
http://hisham.hm/ - http://luarocks.org/



------------------------------

Message: 2
Date: Thu, 20 Oct 2011 21:51:36 +0100
From: liam mail <[hidden email]>
Subject: Lua 5.2 OP_LOADNIL instruction
To: Lua mailing list <[hidden email]>
Message-ID:
        <CAOMMs2RHxEKecyD2uX+o1PNLGuRPpBhxVc7KXJ=[hidden email]>
Content-Type: text/plain; charset=ISO-8859-1

I was wondering why Lua 5.2 does not preform the same optimisation for bytecode which 5.1 did when a there is not an operation before an OP_LOADNIL, for 5.1 the generator would remove the load nil instructions. For example a simple file which just contains a local which will be set to nil:
$ cat ./op_loadnil.lua
local a

For 5.1 produces the following:
$ luac -l ./op_loadnil.lua
main <./op_loadnil.lua:0,0> (1 instruction, 4 bytes at 0x100101050)
0+ params, 2 slots, 0 upvalues, 1 local, 0 constants, 0 functions
        1 [1] RETURN   0 1

Yet for 5.2 it leaves in the instruction:
$ ./luac -l ./op_loadnil.lua
main <./op_loadnil.lua:0,0> (2 instructions at 0x100100ee0)
0+ params, 2 slots, 1 upvalue, 1 local, 0 constants, 0 functions
        1 [1] LOADNIL   0 0
        2 [1] RETURN   0 1

Would someone please shine some light on this and any implications that may follow it.

Thanks.
Liam



------------------------------

Message: 3
Date: Thu, 20 Oct 2011 16:07:35 -0500
From: Rob Hoelz <[hidden email]>
Subject: Properly binding C libraries using callbacks
To: Lua mailing list <[hidden email]>
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="us-ascii"

Hello list,

I'm currently writing a binding for linenoise (https://github.com/antirez/linenoise), and I'm trying to make sure and do things the "right way" when it comes to binding its completion callbacks.

The signature for the callback setting function is this:

void linenoiseSetCompletionCallback(void (*callback)(const char *, linenoiseCompletions *));

and this is what my wrapper looks like:

static int completionFuncRef;
static lua_State *completionState;

static void completionCallbackWrapper(const char *line, linenoiseCompletions *completions) {
    lua_State *L = completionState;
    int status;

    lua_rawgeti(L, LUA_REGISTRYINDEX, completionFuncRef);
    lua_pushlightuserdata(L, completions);
    lua_pushstring(L, line);

    /* XXX error handling */
    status = lua_pcall(L, 2, 0, 0);
}

static int l_setcompletion(lua_State *L) {
    luaL_checktype(L, 1, LUA_TFUNCTION);

    lua_pushvalue(L, 1);
    completionFuncRef = luaL_ref(L, LUA_REGISTRYINDEX);
    linenoiseSetCompletionCallback(completionCallbackWrapper);
    completionState = L;

    return 0;
}

Obviously, this implementation has some holes; for instance, this example will crash:

local ln = require 'linenoise'

local co = coroutine.create(function()
  ln.setcompletion(function(completions, line)
    ln.addcompletion(completions, 'foo')
    ln.addcompletion(completions, 'bar')
    ln.addcompletion(completions, 'baz')
  end)
end)

coroutine.resume(co)

co = nil

collectgarbage 'collect'

local line = ln.linenoise '> '
print(line)

Can someone give me some pointers on the proper way to bind a library that uses callbacks, especially one that doesn't allow you to thread your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that storing things in global variables in the binding are a faux pas.

Thanks,
Rob
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://vlists.pepperfish.net/cgi-bin/mailman/private/lua-l-lists.lua.org/attachments/20111020/5242a47a/signature-0001.pgp

------------------------------

Message: 4
Date: Thu, 20 Oct 2011 17:11:15 -0400
From: Ben Kelly <[hidden email]>
Subject: Re: Lua-friendly OSes
To: [hidden email]
Message-ID: <20111020171115.45f0e039@thoth>
Content-Type: text/plain; charset=US-ASCII

On Thu, 20 Oct 2011 20:27:59 +0100
Matthew Wild <[hidden email]> wrote:

> > On which systems has the Lua ecosystem (libraries, etc) developed
> > best?
>
> Windows (thanks to Lua for Windows), Debian/Ubuntu (thanks to the
> quality and quantity of Lua packages in the repository, including
> LuaRocks).
>
> Platforms I've encountered less Lua joy on are the RPM-based Linux
> distros, including RHEL, CentOS, etc. simply because the packages are
> broken or inadequate, often requiring compiling things from source
> (entirely possible, just less than convenient).
>

My own experience parallels this.
- Debian and its derivatives (Ubuntu, Mint, etc) reliably have lua and
  a lot of commonly-used modules like luasocket and copas in the
  repositories, once you figure out their weird package naming scheme.
- Windows would be a pain in the ass, except that for most apps you can
  just say "install Lua for Windows" and you have everything you need;
  it only gets annoying when you need to use a binary library that's not
  included in L4W, or a more recent version than L4W includes.
- OpenSUSE has a weird mix where it has lua, luafilesystem, luaexpat,
  and a bunch of lua binding generators, but is missing other common
  libraries like luasocket.
- Other RPM-based distros like Fedora I've had similar experiences to
  SUSE on, although I haven't used any recently.

        Ben Kelly



------------------------------

Message: 5
Date: Thu, 20 Oct 2011 23:22:35 +0200
From: Patrick Rapin <[hidden email]>
Subject: Re: Properly binding C libraries using callbacks
To: Lua mailing list <[hidden email]>
Message-ID:
        <CAB5CACmkGp0HN6Hp3xkmxfsjdsic=[hidden email]>
Content-Type: text/plain; charset=ISO-8859-1

>I'm currently writing a binding for linenoise

Your work interests me. I also tried to replace the Readline library with Linenoise for the completion of my LuaDura application.
After some time, I gave up, not because of crashes, but because it seemed that the project lacks of maturity, and I got lost in the 42 different forks inside GitHub...
Does Antirez still maintain his library? Based on the number of pull requests, one might have some doubts.
So what fork are you using?



------------------------------

Message: 6
Date: Thu, 20 Oct 2011 16:32:15 -0500
From: Rob Hoelz <[hidden email]>
Subject: Re: Properly binding C libraries using callbacks
To: [hidden email]
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="us-ascii"

On Thu, 20 Oct 2011 23:22:35 +0200
Patrick Rapin <[hidden email]> wrote:

> >I'm currently writing a binding for linenoise
>
> Your work interests me. I also tried to replace the Readline library
> with Linenoise for the completion of my LuaDura application.
> After some time, I gave up, not because of crashes, but because it
> seemed that the project lacks of maturity, and I got lost in the 42
> different forks inside GitHub...
> Does Antirez still maintain his library? Based on the number of pull
> requests, one might have some doubts.
> So what fork are you using?
>

I'm just using antirez's version.  It's not very mature, but it's simple and does the bare minimum that I need it to.  I'm also working on an editline binding (https://github.com/hoelzro/luael, but I haven't touched it in some time.  I intend to work on it in the next few weeks.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
Url : http://vlists.pepperfish.net/cgi-bin/mailman/private/lua-l-lists.lua.org/attachments/20111020/94d41172/signature-0001.pgp

------------------------------

Message: 7
Date: Thu, 20 Oct 2011 18:00:24 -0400
From: Doug Currie <[hidden email]>
Subject: Re: Properly binding C libraries using callbacks
To: Lua mailing list <[hidden email]>
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=us-ascii


On Oct 20, 2011, at 5:07 PM, Rob Hoelz wrote:

> Can someone give me some pointers on the proper way to bind a library
> that uses callbacks, especially one that doesn't allow you to thread
> your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that storing things in global variables in the binding are a faux pas.

There are several approaches.

1. Put the onus on the library user to make sure the thread that creates the callback remains in existence at least as long as the callback may be executed, i.e., punt

2. When a callback is created, put the thread in a table maintained by the library for the purpose of preventing the thread from being garbage collected; this table can be in the library's environment (or be an upvalue of the library's functions) or may be in the registry

3. When the library is loaded, capture the thread into which it is loaded (which is more likely to the the main thread) and use that thread for callbacks; optionally add #2 to keep this thread from being collected

4. When the library is loaded, create a new thread that is used exclusively to process callbacks; use #2 to prevent this thread from being collected

5. In Lua 5.2 use LUA_RIDX_MAINTHREAD to get a handle on the main thread and use it for callbacks

Of these options, #4 is the safest in Lua 5.1, but uses extra memory.

e




------------------------------

Message: 8
Date: Thu, 20 Oct 2011 20:01:39 -0200
From: Ignacio Burgue?o <[hidden email]>
Subject: Re: Properly binding C libraries using callbacks
To: Lua mailing list <[hidden email]>
Message-ID:
        <CAHPMtKTA+ZJRoYg-+-6FY9kViQv9egL4y=0hP=[hidden email]>
Content-Type: text/plain; charset="utf-8"

T24gVGh1LCBPY3QgMjAsIDIwMTEgYXQgNzowNyBQTSwgUm9iIEhvZWx6IDxyb2JAaG9lbHoucm8+
IHdyb3RlOgoKPiBIZWxsbyBsaXN0LAo+Cj4gLi4uCj4KPiBDYW4gc29tZW9uZSBnaXZlIG1l
IHdyb3RlOgoKPiBIZWxsbyBsaXN0LAo+IHNv
bWUgcG9pbnRlcnMgb24gdGhlIHByb3BlciB3YXkgdG8gYmluZCBhIGxpYnJhcnkgdGhhdAo+IHVz
ZXMgY2FsbGJhY2tzLCBlc3BlY2lhbGx5IG9uZSB0aGF0IGRvZXNuJ3QKPiBhbGxvdyB5b3UgdG8g
dGhyZWFkIHlvdXIgb3duIHVzZXJkYXRhIHRocm91Z2ggdGhlIGNhbGxiYWNrcz8gIENvcm91dGlu
ZXMKPiBtYWtlIGl0IHRyaWNreSwgYW5kIG15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdAo+IHN0
ZXMKPiBtYWtlIGl0IHRyaWNreSwgYW5kIG15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdAo+b3Jp
bmcgdGhpbmdzIGluIGdsb2JhbCB2YXJpYWJsZXMgaW4gdGhlIGJpbmRpbmcgYXJlIGEgZmF1eCBw
YXMuCj4KPiBUaGFua3MsCj4gUm9iCj4KCgpNYXliZSB0aGUgY29kZSBoZXJlIGNhbiBzaGVkIHNv
bWUgbGlnaHQ/CgpodHRwczovL2dpdGh1Yi5jb20vcnJ0aG9tYXMvbHVhLXJsY29tcGxldGVyCgpS
ZWdhcmRzLApJZ25hY2lvCi0tLS0tLS0tLS0tLS0tIG5leHQgcGFydCAtLS0tLS0tLS0tLS0tLQpB
biBIVE1MIGF0dGFjaG1lbnQgd2FzIHNjcnViYmVkLi4uClVSTDogaHR0cDovL3ZsaXN0cy5wZXBw
ZXJmaXNoLm5ldC9jZ2ktYmluL21haWxtYW4vcHJpdmF0ZS9sdWEtbC1saXN0cy5sdWEub3JnL2F0
dGFjaG1lbnRzLzIwMTExMDIwL2JmYzdmNTc5L2F0dGFjaG1lbnQtMDAwMS5odG1sCg==

------------------------------

Message: 9
Date: Fri, 21 Oct 2011 00:16:16 +0200
From: Marc Balmer <[hidden email]>
Subject: Re: Lua-friendly OSes
To: [hidden email]
Message-ID: <[hidden email]>
Content-Type: text/plain; charset=UTF-8

Am 20.10.11 19:47, schrieb Pascal Du Maurrier:
> Hi everyone,
>
> I thought it would be beneficial to start a thread where Lua
> programmers can share their impressions and experiences about getting
> Lua to work on the various operating systems. And especially how they
> interact with  the operating system itself.
>
> Which are the most Lua-friendly operating systems?

NetBSD (-current) has Lua in the base install and there is an ongoing project to use Lua in the kernel.  That does not mean, however, that NetBSD is the most Lua friendly operating system.  As Lua is highly portable, probably any OS is "most Lua firendly".

> Which OSes let Lua access their APIs, and to what extent?

NetBSD has it's own interface to SQLite (which is also in the base install of the OS) and a module to access GPIO (General Purpose Input
Output) pins using Lua.

> On which systems has the Lua ecosystem (libraries, etc) developed best?
>
> Please share your insights about programming on the various OSes,
> possibly with a mention of their pros and cons vis-a-vis Lua.
>




------------------------------

_______________________________________________
lua-l mailing list
[hidden email]
http://vlists.pepperfish.net/cgi-bin/mailman/listinfo/lua-l-lists.lua.org


End of lua-l Digest, Vol 15, Issue 105
**************************************

Reply | Threaded
Open this post in threaded view
|

RE: Properly binding C libraries using callbacks

Thijs Schreijer
In reply to this post by Tim Caswell
>
> I have a similar problem in my luvit project.  I'm wrapping an
> inherently async library that provides non-blocking I/O.  I ended up
> creating a helper that calls my lua callbacks from the main thread.
>
> https://github.com/creationix/luvit/blob/master/src/utils.c#L7-37
>
> As far as getting a reference to the lua state, the API I'm using
> always has a place for an opaque data pointer and I put in some struct
> that contains L and some other data (like a lua_ref integer) to get at
> the real data.
>

I'm running into the callback issue as well, gave it some thought and came up with this; (warning; I only read about the c api, never used it, so might be rubbish...)


Async callback helper lib;

when the helper is loaded it registers a 'poll' function and places a value in the lua register. The key is the string, the value is a c pointer to an 'enqueue' function for reception of the async data.
so; [myhelperlib] = <address pointer to enqueue function>

the 'enqueue' function takes 2 arguments; 1) pointer to data, 2) pointer to a library specific 'decode function' and it should be protected/threadsafe. The helper will store the both pointers in a queue waiting for lua to collect it.

Any library that has async callbacks to deliver data to the luastate, looks for the pointer to the 'enqueue' function in the register (when its 'openlib' function is called, not on the callback thread of course) and calls that function whenever it has data to deliver. It provides a lump of data (pointer 1) and a function (pointer 2). The function is a 'decode' function that is capable of decoding the 'lump of data' into lua data.

Whenever the 'poll' function (of helper) is called (from lua), the helper gets the first item from its queue, and calls the 'decode' function with 2 arguments; 1) the lump of data to decode, 2) the luastate. The 'decode' function is now called on the main luathread, so it can safely push the decoded return values onto the stack and return. (if there is nothing on the queue the 'poll' simply returns nil.)

As a notification mechanism; Whenever the helper receives data in its queue, it sends a UDP packet with only the number of items in the queue to 'localhost:someport' (possibly repeat this every x millisecs as long as there is data).

By convention the first argument returned would be a lua-function (a callback) which will be called with the other arguments as parameters.

End result;
 - uniform way of dealing with (async) callback data
 - helper is a standalone library, that can be used by multiple
   libraries at once
 - lua can call 'poll' for updates if its actively running
 - lua can use a socket select statement to wait for incoming
   data, without a busy wait (udp works on windows and posix.
   On posix pipes or filehandles may be used as well).


Could this work? Any caveats?

Thijs



Reply | Threaded
Open this post in threaded view
|

Re: Properly binding C libraries using callbacks

jiang yu
In reply to this post by Rob Hoelz-2
First of all, completionCallbackWrapper should be
static void completionCallbackWrapper(lua_State*L,int ref,const char *line, linenoiseCompletions *completions)
and you should bind { lua_State, ref, completionCallbackWrapper } as a function pass to linenoise.
Second, at the end of coroutine, you should call ln.clearcompletion()
and before 'co = nil' you should call ln.clearcompletion(co)
I suppose there is no automate way to handle this.

2011/10/21 Rob Hoelz <[hidden email]>
Hello list,

I'm currently writing a binding for linenoise
(https://github.com/antirez/linenoise), and I'm trying to make sure and
do things the "right way" when it comes to binding its completion
callbacks.

The signature for the callback setting function is this:

void linenoiseSetCompletionCallback(void (*callback)(const char *, linenoiseCompletions *));

and this is what my wrapper looks like:

static int completionFuncRef;
static lua_State *completionState;

static void completionCallbackWrapper(const char *line, linenoiseCompletions *completions)
{
   lua_State *L = completionState;
   int status;

   lua_rawgeti(L, LUA_REGISTRYINDEX, completionFuncRef);
   lua_pushlightuserdata(L, completions);
   lua_pushstring(L, line);

   /* XXX error handling */
   status = lua_pcall(L, 2, 0, 0);
}

static int l_setcompletion(lua_State *L)
{
   luaL_checktype(L, 1, LUA_TFUNCTION);

   lua_pushvalue(L, 1);
   completionFuncRef = luaL_ref(L, LUA_REGISTRYINDEX);
   linenoiseSetCompletionCallback(completionCallbackWrapper);
   completionState = L;

   return 0;
}

Obviously, this implementation has some holes; for instance, this example will crash:

local ln = require 'linenoise'

local co = coroutine.create(function()
 ln.setcompletion(function(completions, line)
   ln.addcompletion(completions, 'foo')
   ln.addcompletion(completions, 'bar')
   ln.addcompletion(completions, 'baz')
 end)
end)

coroutine.resume(co)

co = nil

collectgarbage 'collect'

local line = ln.linenoise '> '
print(line)

Can someone give me some pointers on the proper way to bind a library that uses callbacks, especially one that doesn't
allow you to thread your own userdata through the callbacks?  Coroutines make it tricky, and my understanding is that
storing things in global variables in the binding are a faux pas.

Thanks,
Rob