Guide for Lua bindings

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Guide for Lua bindings

Gregg Reynolds-2
A humble suggestion for Roberto (or anybody else): a thorough guide to Lua as extension and extending Lua.

I just finally paid good $$ for PIL 4, and am glad I did. However, it's main focus, naturally, is Lua programming. The material on the C interface is good as far as it goes, but it leaves me hungry for more.

I would pay good Bitcoin for a Guide to writing Lua bindings that covers the hairy bits - like dealing with coroutines, strategies for organizing code, etc. - in a little more detail, with examples, maybe covering multiple languages.

There's plenty of info along these lines on the web, but it sure would be nice to have an authoritative Guide.

My 2 cents,

Gregg
Reply | Threaded
Open this post in threaded view
|

Re: Guide for Lua bindings

Luiz Henrique de Figueiredo
> There's plenty of info along these lines on the web, but it sure would be
> nice to have an authoritative Guide.

Indeed. But I think this would require getting into details of what you
want to bind Lua to and would take the writer too far into non Lua land.

I suggest you start reading Lua bindings for C libraries. The source of
the Lua standard libraries is a good starting point:
        https://www.lua.org/source/5.3/

See also my libraries at
        http://webserver2.tecgraf.puc-rio.br/~lhf/ftp/lua/

Reply | Threaded
Open this post in threaded view
|

Re: Guide for Lua bindings

Dirk Laurie-2
In reply to this post by Gregg Reynolds-2
2017-12-22 2:54 GMT+02:00 Gregg Reynolds <[hidden email]>:

> A humble suggestion for Roberto (or anybody else): a thorough guide to Lua
> as extension and extending Lua.
>
> I just finally paid good $$ for PIL 4, and am glad I did. However, it's main
> focus, naturally, is Lua programming. The material on the C interface is
> good as far as it goes, but it leaves me hungry for more.
>
> I would pay good Bitcoin for a Guide to writing Lua bindings that covers the
> hairy bits - like dealing with coroutines, strategies for organizing code,
> etc. - in a little more detail, with examples, maybe covering multiple
> languages.
>
> There's plenty of info along these lines on the web, but it sure would be
> nice to have an authoritative Guide.

There are at least four approaches:

1. Use a tool that generates bindings for you directly from the headerfiles.
2. Use LuaJIT which has an FFI built in.
3. Do it neatly by hand but stay so close to the the original that the C documentation can be used as is. Most of the better contributions to LuaRocks do this.
4. Hide all the details under a level of abstraction so that the user need not even know which library is being bound. This approach sits underneath Love2D.

Each of these has advantages that in certain situations makes it more approprate tham the others.

I think it is a good thing, and typical of the Zen of Lua [1], that the Lua team does not enshrine any of the above.

[1] Less is more.

Reply | Threaded
Open this post in threaded view
|

RE: Guide for Lua bindings

Kalafut, Bennett

A fifth approach is to use third-party bindings for standard Lua (no stuck-in-5.1 LuaJIT needed) like Sol2.


From: Dirk Laurie [mailto:[hidden email]]
Sent: Thursday, December 21, 2017 8:34 PM
To: Lua mailing list
Subject: Re: Guide for Lua bindings

 

2017-12-22 2:54 GMT+02:00 Gregg Reynolds <[hidden email]>:
> A humble suggestion for Roberto (or anybody else): a thorough guide to Lua
> as extension and extending Lua.
>
> I just finally paid good $$ for PIL 4, and am glad I did. However, it's main
> focus, naturally, is Lua programming. The material on the C interface is
> good as far as it goes, but it leaves me hungry for more.
>
> I would pay good Bitcoin for a Guide to writing Lua bindings that covers the
> hairy bits - like dealing with coroutines, strategies for organizing code,
> etc. - in a little more detail, with examples, maybe covering multiple
> languages.
>
> There's plenty of info along these lines on the web, but it sure would be
> nice to have an authoritative Guide.

There are at least four approaches:

1. Use a tool that generates bindings for you directly from the headerfiles.

2. Use LuaJIT which has an FFI built in.

3. Do it neatly by hand but stay so close to the the original that the C documentation can be used as is. Most of the better contributions to LuaRocks do this.

4. Hide all the details under a level of abstraction so that the user need not even know which library is being bound. This approach sits underneath Love2D.

 

Each of these has advantages that in certain situations makes it more approprate tham the others.

 

I think it is a good thing, and typical of the Zen of Lua [1], that the Lua team does not enshrine any of the above.

[1] Less is more.

 

Reply | Threaded
Open this post in threaded view
|

Re: Guide for Lua bindings

ThePhD
I like the fifth approach. I'm a bit biased because I'm the one who wrote sol2, but higher-level abstractions are almost always infinitely more useful.

But, the main objective of this thread (I believe) is moreso something that teaches you about the ins and outs of using the Lua C API, and any quirks that come with it. In my opinion, the manual is your best friend here. When you're programming against any API, you need hard-guarantees about what individual function invocations do. The state it leaves things in, how it interprets arguments (if it's non-obvious), and other such things. Any higher-level tutorial likely won't serve you as well as just knowing the manual: plus, it's much easier to hold a developer accountable to their documentation, (well-written, short) examples and manual than to a general best-practices guide.

The only thing I found missing in the manual is some of the behavior of the default Lua's Coroutines. For example, when you do

runner = coroutine.wrap( ... ) -- or
runner = coroutine.create( ... )

The C library creates a new lua value of type LUA_TTHREAD, pushes the function you pass to invoke into that Lua Thread's stack, and then leaves it as is. The first time you invoke runner(), it will pull that function off the stack. Then, subsequent invocations will use {implementation defined magic} to keep track of that function and run it on the coroutine's thread even though it's disappeared of the Lua Thread's execution stack after the first time you run it (that is, it's no longer accessible to you in an obvious way).

What's fun is wrapping your head around that and then also learning that lua_resume (https://www.lua.org/manual/5.3/manual.html#lua_resume) does not work like lua_call or lua_pcall despite having an incredibly similar signature and usage notes: the latter 2 functions pull and call your argument from the top of the stack and generally don't mess with things below that (sans direct user invervention).

lua_resume reserves the right to own the thread's entire stack, and thusly will actually clear things off the stack below the nargs + function you push onto the stack to make lua_resume work (in fact, it's questionable whether you need the function there either, but at present sol2 pushes it onto the stack anyhow). This means that any abstraction you make around `coroutine` isn't as simple as just duplicating whatever `function` abstraction you have around `lua_pcall`: you instead have to take into account that the entire stack can (and generally, is) owned by the entire execution thread, and it's liable to clear the entire thing by the time `yield()` is called.

Just some quirks I picked up, but honestly they're not really all that important in the grand scheme of things. Coroutines and how they work were the most difficult to understand, not because I mistaked them for operating system threads, but because I didn't quite grok that in this case "Coroutine" and "Thread" just mean "Separate Execution Space". Using that terminology, it makes a lot more sense.


On Fri, Dec 22, 2017 at 12:21 PM, Kalafut, Bennett <[hidden email]> wrote:

A fifth approach is to use third-party bindings for standard Lua (no stuck-in-5.1 LuaJIT needed) like Sol2.


From: Dirk Laurie [mailto:[hidden email]]
Sent: Thursday, December 21, 2017 8:34 PM
To: Lua mailing list
Subject: Re: Guide for Lua bindings

 

2017-12-22 2:54 GMT+02:00 Gregg Reynolds <[hidden email]>:
> A humble suggestion for Roberto (or anybody else): a thorough guide to Lua
> as extension and extending Lua.
>
> I just finally paid good $$ for PIL 4, and am glad I did. However, it's main
> focus, naturally, is Lua programming. The material on the C interface is
> good as far as it goes, but it leaves me hungry for more.
>
> I would pay good Bitcoin for a Guide to writing Lua bindings that covers the
> hairy bits - like dealing with coroutines, strategies for organizing code,
> etc. - in a little more detail, with examples, maybe covering multiple
> languages.
>
> There's plenty of info along these lines on the web, but it sure would be
> nice to have an authoritative Guide.

There are at least four approaches:

1. Use a tool that generates bindings for you directly from the headerfiles.

2. Use LuaJIT which has an FFI built in.

3. Do it neatly by hand but stay so close to the the original that the C documentation can be used as is. Most of the better contributions to LuaRocks do this.

4. Hide all the details under a level of abstraction so that the user need not even know which library is being bound. This approach sits underneath Love2D.

 

Each of these has advantages that in certain situations makes it more approprate tham the others.

 

I think it is a good thing, and typical of the Zen of Lua [1], that the Lua team does not enshrine any of the above.

[1] Less is more.

 


Reply | Threaded
Open this post in threaded view
|

Re: Guide for Lua bindings

Gregg Reynolds-2


On Dec 22, 2017 1:39 PM, "ThePhD" <[hidden email]> wrote:
I like the fifth approach. I'm a bit biased because I'm the one who wrote sol2, but higher-level abstractions are almost always infinitely more useful.

But, the main objective of this thread (I believe) is moreso something that teaches you about the ins and outs of using the Lua C API, and any quirks that come with it. In my opinion, the manual is your best friend here

Which manual? The ref man or PIL? Both are great but neither does much hand holding. OTH the Lua implementation itself has great examples. That's prolly your best best friend.
...
 Any higher-level tutorial likely won't serve you as well as just knowing the manual.

In the Humanities this is known as the Hermeneutical Circle. You cannot understand the whole until you understand the parts. But you cannot understand the parts until you understand the whole. That's what guides are for. A ref manual is great for experts, not so much for noobs.

The only thing I found missing in the manual is some of the behavior of the default Lua's Coroutines. 

Yeah.  Having just started reading PIL 4 the use of coroutines with c is very interesting but not so easy to grok.

Thanks,
G
Reply | Threaded
Open this post in threaded view
|

Re: Guide for Lua bindings

ThePhD
The reference manual, really. I can see the use of a guide, however... in fact, there was mention of a Lua Documentation project on the mailing list earlier. I wonder if that would be the best place to -- for each API function -- put out a short, small, complete example demonstrating a use for it, much like the way cppreference has short examples that show off the type or function being explained.

That's beyond the scope of my expertise, however, and smarter people will have to tackle that beast. I already get into quite the tussle writing documentation for sol2, and even the amount of examples and code I've put in my docs is still not enough to help "noobs" get going quickly when they want to do something less trivial (as evidenced by the staggering, jaw-dropping amount of `Helpdesk`-labeled issues in the repository and the questions that come through the Gitter).

On Fri, Dec 22, 2017 at 3:06 PM, Gregg Reynolds <[hidden email]> wrote:


On Dec 22, 2017 1:39 PM, "ThePhD" <[hidden email]> wrote:
I like the fifth approach. I'm a bit biased because I'm the one who wrote sol2, but higher-level abstractions are almost always infinitely more useful.

But, the main objective of this thread (I believe) is moreso something that teaches you about the ins and outs of using the Lua C API, and any quirks that come with it. In my opinion, the manual is your best friend here

Which manual? The ref man or PIL? Both are great but neither does much hand holding. OTH the Lua implementation itself has great examples. That's prolly your best best friend.
...
 Any higher-level tutorial likely won't serve you as well as just knowing the manual.

In the Humanities this is known as the Hermeneutical Circle. You cannot understand the whole until you understand the parts. But you cannot understand the parts until you understand the whole. That's what guides are for. A ref manual is great for experts, not so much for noobs.

The only thing I found missing in the manual is some of the behavior of the default Lua's Coroutines. 

Yeah.  Having just started reading PIL 4 the use of coroutines with c is very interesting but not so easy to grok.

Thanks,
G

Reply | Threaded
Open this post in threaded view
|

Re: Guide for Lua bindings

Gregg Reynolds-2


On Dec 22, 2017 3:30 PM, "ThePhD" <[hidden email]> wrote:

..
I
 already get into quite the tussle writing documentation for sol2, 

Heh. Writing good code is easy. Writing good docs, not so much.