Lua vs Python C bindings

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua vs Python C bindings

Dibyendu Majumdar
Recently I started created a Python binding for one of my projects. I
already had Ravi/Lua bindings hand-coded in C++/C.

With regards to the C api of Python vs Lua there is no contest. The
Python C api is much more verbose and hence writing a binding in that
is a big task.

But what surprised me is a tool called Cython.

It accepts an extended Python syntax, where you can statically declare
C and C++ types, and then use them in the Python code. Then you run
the Cython compiler and it generates C or C++ code and that's it.
Basically you write Python like high level language most of the time,
and everything else is taken care of. Ref counting, type checking, all
taken care of.

Amazingly it works with C++ too as my project is in C++. No need to
write C wrappers.

So it seems that writing a C/C++ binding for Python is actually less
effort than writing one for Lua.

It seems like creating a Cython clone for Lua might be a good idea.
Maybe someone is already doing this?

Regards
Dibyendu

v
Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

v
On Sun, 2019-06-23 at 00:39 +0100, Dibyendu Majumdar wrote:
> Recently I started created a Python binding for one of my projects. I
> already had Ravi/Lua bindings hand-coded in C++/C.

What I know is tool tolua. Its fork is used in cocos2d-x framework to
make Lua bindings, that's where I know it from.
However, it seems to don't maintained anymore, and last version is for
Lua 5.1/5.2.

I'm also pretty sure that there are some other tools, but most of them
seem to be unmaintained as well.

> It seems like creating a Cython clone for Lua might be a good
> idea.

Indeed it is! Sounds like tedious work, though.

P.S.: I'm working on Lua/C++ bridge now, but I'm aiming at embedded Lua
usage case and not standalone, but you may run any Lua script with it
anyways, so making standalone version shouldn't be too hard.
However, it still require some modifications to code (and more
significant ones if you want advanced stuff like Lua-specific
constructions and metamethods). Simple example, adding method to Lua
look like

const Lua::MethodsTable<MyTestClass> MyTestClass::methods = {
{"TestMethod", Lua::CppMethodWrapper<MyTestClass, Lua::Number,
std::string>(&MyTestClass::TestMethod)}
};

While this looks kinda messy (so far), it allows you to avoid code
generation.

--
v <[hidden email]>


Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Coda Highland
In reply to this post by Dibyendu Majumdar


On Sat, Jun 22, 2019 at 6:39 PM Dibyendu Majumdar <[hidden email]> wrote:
Recently I started created a Python binding for one of my projects. I
already had Ravi/Lua bindings hand-coded in C++/C.

With regards to the C api of Python vs Lua there is no contest. The
Python C api is much more verbose and hence writing a binding in that
is a big task.

But what surprised me is a tool called Cython.

It accepts an extended Python syntax, where you can statically declare
C and C++ types, and then use them in the Python code. Then you run
the Cython compiler and it generates C or C++ code and that's it.
Basically you write Python like high level language most of the time,
and everything else is taken care of. Ref counting, type checking, all
taken care of.

Amazingly it works with C++ too as my project is in C++. No need to
write C wrappers.

So it seems that writing a C/C++ binding for Python is actually less
effort than writing one for Lua.

It seems like creating a Cython clone for Lua might be a good idea.
Maybe someone is already doing this?

Regards
Dibyendu


SWIG has a lot of different targets available. Lua is among them but I don't know what versions of Lua it supports.

/s/ AdamĀ 
Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Russell Haley
Sorry for the top post. A response from my request on the swig mailing list about swig 4 and lua 5.3.5:

"For the Plplot project, the combination of swig-4.0.0 and Lua 5.3.5
works fine for me on Linux where I built both because my Linux
distribution (Debian Testing) does not currently supply either one.
However, another PLplot developer has reported that Lua 5.3.5 supplied
by the MinGW-w64/MSYS2 Windows platform is just plain broken right now
(independent of swig). So at least recent lua version reliability can
vary with platform so it is a matter of "try and see".

Alan"

Russ

Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
  Original Message  
From: Coda Highland
Sent: Saturday, June 22, 2019 5:31 PM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Re: Lua vs Python C bindings



On Sat, Jun 22, 2019 at 6:39 PM Dibyendu Majumdar <[hidden email]> wrote:
Recently I started created a Python binding for one of my projects. I
already had Ravi/Lua bindings hand-coded in C++/C.

With regards to the C api of Python vs Lua there is no contest. The
Python C api is much more verbose and hence writing a binding in that
is a big task.

But what surprised me is a tool called Cython.

It accepts an extended Python syntax, where you can statically declare
C and C++ types, and then use them in the Python code. Then you run
the Cython compiler and it generates C or C++ code and that's it.
Basically you write Python like high level language most of the time,
and everything else is taken care of. Ref counting, type checking, all
taken care of.

Amazingly it works with C++ too as my project is in C++. No need to
write C wrappers.

So it seems that writing a C/C++ binding for Python is actually less
effort than writing one for Lua.

It seems like creating a Cython clone for Lua might be a good idea.
Maybe someone is already doing this?

Regards
Dibyendu


SWIG has a lot of different targets available. Lua is among them but I don't know what versions of Lua it supports.

/s/ Adam 

Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Hugo Musso Gualandi
In reply to this post by Dibyendu Majumdar
> It seems like creating a Cython clone for Lua might be a good idea.
> Maybe someone is already doing this?

Pallene/Titan[1] are being developed in a related space. They let you
write extension modules in a language that is close to to Lua, but much
faster. You can also define custom low-level struct-like data-types
(built on top of userdata). The compiler produces the required Lua
bindings and makes sure that garbage collection does the right thing.

The main differences between Pallene/Titan and Cython are that Cython is
a superset of Python, while Pallene/Titan is closer to a typed subset of
Lua. Additionally, the Pallene compiler can optimize some code that uses
Lua tables, as long as it has the appropriate type annotations. With
Cython only the parts of the program that use the "C types" are
optimized.

> So it seems that writing a C/C++ binding for Python is actually less
effort than writing one for Lua.

This is definitely one of our goals with the project as well :)


[1] https://github.com/pallene-lang/pallene

- Hugo

Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Dibyendu Majumdar
On Sun, 23 Jun 2019 at 03:08, Hugo Musso Gualandi
<[hidden email]> wrote:

>
> > It seems like creating a Cython clone for Lua might be a good idea.
> > Maybe someone is already doing this?
>
> Pallene/Titan[1] are being developed in a related space. They let you
> write extension modules in a language that is close to to Lua, but much
> faster. You can also define custom low-level struct-like data-types
> (built on top of userdata). The compiler produces the required Lua
> bindings and makes sure that garbage collection does the right thing.
>
> The main differences between Pallene/Titan and Cython are that Cython is
> a superset of Python, while Pallene/Titan is closer to a typed subset of
> Lua. Additionally, the Pallene compiler can optimize some code that uses
> Lua tables, as long as it has the appropriate type annotations. With
> Cython only the parts of the program that use the "C types" are
> optimized.
>
> > So it seems that writing a C/C++ binding for Python is actually less
> effort than writing one for Lua.
>
> This is definitely one of our goals with the project as well :)
>
>
> [1] https://github.com/pallene-lang/pallene
>

Cool.
Pallene does look a similar initiative, but creates a different
statically typed language from Lua, right?
I liked that Cython is a superset of Python. Another cool feature is
that it can wrap C++ code as well.

Of course I have been using it mainly to provide Python bindings to
existing C++ code rather than writing Python/Cython modules from
scratch.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Hugo Musso Gualandi
>
> Cool.
> Pallene does look a similar initiative, but creates a different
> statically typed language from Lua, right?

Pallene is still Lua, in the sense that it is a typed subset of it.
If you remove all the type annotations from a Pallene program it
should be executable as a Lua program.

We chose to give it a different name because, while it is easy to
convert Pallene code to Lua, the converse is not true. You should not
expect to be able to easily translate any idiomatic Lua program to
Pallene just by adding type annotations, as the Pallene type system is
too rigid for that. This was a conscious design decision because we
wanted programmers to be able to feel that if they write their code in
Pallene they should be able to expect that the compiler will generate
clean and efficient code for it. This is not possible if the type
system is designed for idiomatic Lua.

> I liked that Cython is a superset of Python.

It boils down to the level of granularity for mixing the static and
dynamic sublanguages. Ravi and Cython choose to have an expression-
level granularity, while in Pallene it is module-level.

While we don't have any plans to allow mixing Lua and Pallene fragments
inside a single source file, I don't think this is a fundamental design
decision that tells the two approaches apart. It is the sort of thing
that could conceivably change in the future.

> Another cool feature is
> that it can wrap C++ code as well.

The Lua-C API has to be the way it is because the PUC-Lua interpreter
uses only portable standards-compliant C-code. It can only call C
functions if they all have the same type signature: receiving a
lua_State pointer as a parameter, and returning an int.

This is not a problem if you have a compiled language, or are willing
to use a slow and non-portable implementation for calling C functions
inside the interpreter.

The LuaJIT FFI interface is an example of this. Inside JIT compiled
traces it simply inserts a call to the C function inside the generated
machine instructions. It is as fast as such a call can get. However,
outside compiled traces the interpreter must implement FFI calls with
clever non-portable assembly code that knows how to call an arbitrary C
function with potentially any number and type of parameters. This is
slower than when the C calls are compiled to machine code, which is the
case of the traditional Lua-C API and of LuaJIT's FFI inside compiled
traces. This means that if you are using the LuaJIT FFI you really want
to be sure that the code surrounding your FFI calls is free of "not yet
implemented" language features.

A fully-compiled language like Cython, Pallene, or Titan doesn't have
to worry about interpreting the C calls, which interpreters like
PUC-Lua, LuaJIT or Ravi have to. Titan already has some experimental
support for external C calls through a LuaJIT-like interface. Pallene
doesn't have this yet but it is on our long term plans.

-- Hugo

Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Dibyendu Majumdar
On Sun, 23 Jun 2019 at 15:48, Hugo Musso Gualandi
<[hidden email]> wrote:

>
> The LuaJIT FFI interface is an example of this. Inside JIT compiled
> traces it simply inserts a call to the C function inside the generated
> machine instructions. It is as fast as such a call can get. However,
> outside compiled traces the interpreter must implement FFI calls with
> clever non-portable assembly code that knows how to call an arbitrary C
> function with potentially any number and type of parameters. This is
> slower than when the C calls are compiled to machine code, which is the
> case of the traditional Lua-C API and of LuaJIT's FFI inside compiled
> traces. This means that if you are using the LuaJIT FFI you really want
> to be sure that the code surrounding your FFI calls is free of "not yet
> implemented" language features.
>
> A fully-compiled language like Cython, Pallene, or Titan doesn't have
> to worry about interpreting the C calls, which interpreters like
> PUC-Lua, LuaJIT or Ravi have to. Titan already has some experimental
> support for external C calls through a LuaJIT-like interface. Pallene
> doesn't have this yet but it is on our long term plans.
>

I think that the issue is not whether it is compiled or not. Lua
functions are values, so you don't know that they are actually
functions or not, what arguments they take, and what the return type
is. Thus, you don't know that the function being called is a C
function, until you see the value. You cannot assume that the next
time you see the value it will be the same function. In a language
where a function has a static identity it would be possible to create
specialized bytecodes that efficiently invoke the C function.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Hugo Musso Gualandi
> I think that the issue is not whether it is compiled or not. Lua
> functions are values, so you don't know that they are actually
> functions or not, what arguments they take, and what the return type
> is. Thus, you don't know that the function being called is a C
> function, until you see the value. You cannot assume that the next
> time you see the value it will be the same function.

These things make it harder to optimize the code around the C function
call but do not affect how hard it is for the interpreter to actually
invoke the C function.

> In a language
> where a function has a static identity it would be possible to create
> specialized bytecodes that efficiently invoke the C function.

The problem is that it is impossible to implement these bytecodes if
your interpreter is written in standards-compliant C. The C language
doesn't have a way to apply an arbitrary function (which can be of any
type) to a dynamically constructed list of arguments (which can be of
any length). You would need to create an infinite set of bytecodes, one
for each possible C function type.

This is why projects like libffi[1] exist, and why LuaJIT must use non-
portable assembly to implement this part of the FFI in its interpreter.

[1] https://sourceware.org/libffi/

  -- Hugo

Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Jim-2
In reply to this post by Dibyendu Majumdar
23.06.2019, 01:39, "Dibyendu Majumdar" <[hidden email]>:
> But what surprised me is a tool called Cython.
>
> It accepts an extended Python syntax, where you can statically declare
> C and C++ types, and then use them in the Python code. Then you run
> the Cython compiler and it generates C or C++ code and that's it.
> Basically you write Python like high level language most of the time,
> and everything else is taken care of. Ref counting, type checking, all
> taken care of.

Akin to Tcl's Critcl ?
https://wiki.tcl-lang.org/page/Critcl
https://github.com/andreas-kupries/critcl


Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Dibyendu Majumdar
In reply to this post by Hugo Musso Gualandi
On Sun, 23 Jun 2019 at 18:47, Hugo Musso Gualandi
<[hidden email]> wrote:

>
> > I think that the issue is not whether it is compiled or not. Lua
> > functions are values, so you don't know that they are actually
> > functions or not, what arguments they take, and what the return type
> > is. Thus, you don't know that the function being called is a C
> > function, until you see the value. You cannot assume that the next
> > time you see the value it will be the same function.
>
> These things make it harder to optimize the code around the C function
> call but do not affect how hard it is for the interpreter to actually
> invoke the C function.

Quite a bit hard. Basically if you don't have static typing / function
declarations then performance will degrade significantly.

>
> > In a language
> > where a function has a static identity it would be possible to create
> > specialized bytecodes that efficiently invoke the C function.
>
> The problem is that it is impossible to implement these bytecodes if
> your interpreter is written in standards-compliant C. The C language
> doesn't have a way to apply an arbitrary function (which can be of any
> type) to a dynamically constructed list of arguments (which can be of
> any length). You would need to create an infinite set of bytecodes, one
> for each possible C function type.
>

Agreed that one cannot have an infinite set of bytecodes. I had the
idea of handling common scenarios as a fast path - where arguments are
64-bit quantities (integer or floating point) and passed by register
(4 on win64, 6 on Linux64).
The main issue I found was not the number of bytecodes - the issue was
each argument has to be checked for correct type.

What I was trying to say is that static typing is the key factor in
performance. But it poses interoperability issues with Lua.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: Lua vs Python C bindings

Jim-2
In reply to this post by Dibyendu Majumdar
On Sun, Jun 23, 2019 at 12:39:02AM +0100, Dibyendu Majumdar wrote:
> With regards to the C api of Python vs Lua there is no contest. The
> Python C api is much more verbose and hence writing a binding in that
> is a big task.

you can tell a good C API by its ease of use in hand written code.
first of all it should be directly usable without requiring automated
binding generators.

> So it seems that writing a C/C++ binding for Python is actually less
> effort than writing one for Lua.

at least when using such helper tools.