Fengari: Why we rewrote Lua in JS

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Fengari: Why we rewrote Lua in JS

Benoit Giannangeli
Hi there,

Fengari is a project me and daurnimator have been working on the past months which aims to bring Lua to the browser.

Here is the introductory article for it: http://lua.space/webdev/why-we-rewrote-lua-in-js

--
Benoit Giannangeli
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Jakub Jirutka
Hi Benoit,

Fengari is really awesome project, thank you and daurnimator very much for it!

I have one question. How about Lua/C modules? Is it currently possible to compile e.g. LPeg into JS using Emscripten and use with Fengari?

Jakub

On 26. Jul 2017, at 14:34, Benoit Giannangeli <[hidden email]> wrote:

Hi there,

Fengari is a project me and daurnimator have been working on the past months which aims to bring Lua to the browser.

Here is the introductory article for it: http://lua.space/webdev/why-we-rewrote-lua-in-js

--
Benoit Giannangeli
 

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Charles Heywood
I'm not sure about using Emscripten, but there is LuLPeg which is mostly compatible with LPeg - enough that MoonScript can be compiled with it.

On Wed, Jul 26, 2017 at 1:10 PM Jakub Jirutka <[hidden email]> wrote:
Hi Benoit,

Fengari is really awesome project, thank you and daurnimator very much for it!

I have one question. How about Lua/C modules? Is it currently possible to compile e.g. LPeg into JS using Emscripten and use with Fengari?

Jakub

On 26. Jul 2017, at 14:34, Benoit Giannangeli <[hidden email]> wrote:

Hi there,

Fengari is a project me and daurnimator have been working on the past months which aims to bring Lua to the browser.

Here is the introductory article for it: http://lua.space/webdev/why-we-rewrote-lua-in-js

--
Benoit Giannangeli
 

--
--
Ryan <[hidden email]>
Software Developer / System Administrator
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Jakub Jirutka
I know about LuLPeg, but I’ve used LPeg just as an example. I’m interesting about general support of Lua/C modules. Is it currently somehow possible?

Jakub

On 27. Jul 2017, at 0:23, Charles Heywood <[hidden email]> wrote:

I'm not sure about using Emscripten, but there is LuLPeg which is mostly compatible with LPeg - enough that MoonScript can be compiled with it.

On Wed, Jul 26, 2017 at 1:10 PM Jakub Jirutka <[hidden email]> wrote:
Hi Benoit,

Fengari is really awesome project, thank you and daurnimator very much for it!

I have one question. How about Lua/C modules? Is it currently possible to compile e.g. LPeg into JS using Emscripten and use with Fengari?

Jakub

On 26. Jul 2017, at 14:34, Benoit Giannangeli <[hidden email]> wrote:

Hi there,

Fengari is a project me and daurnimator have been working on the past months which aims to bring Lua to the browser.

Here is the introductory article for it: http://lua.space/webdev/why-we-rewrote-lua-in-js

--
Benoit Giannangeli
 

--
--
Ryan <[hidden email]>
Software Developer / System Administrator

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Hisham
On 26 July 2017 at 19:42, Jakub Jirutka <[hidden email]> wrote:
> I know about LuLPeg, but I’ve used LPeg just as an example. I’m interesting
> about general support of Lua/C modules. Is it currently somehow possible?

The motivations one uses Lua/C modules are either for extra speed, or
for interfacing with existing libraries. For the first motivation,
going through Emscripten and then the Lua/C API and run the whole
thing on top of JS would probably be slower than writing a module
directly in Lua; the approach for critical speed would be to write
something in JavaScript instead and go through Fengari's Lua/JS API.
For the second motivation, I suppose most existing libraries one would
want to interact with in a browser would be existing JS libraries, and
not C libraries. For a lot of C libraries, it wouldn't make sense to
port them into the browser, but for those that can be compiled into JS
libraries via Emscripten, I suppose one can bind them to Lua via the
Lua/JS API, but the binding module needs to be rewritten (from C to
JS).

The linked article mentions that the approach of compiling Lua C code
into the browser via Emscripten makes you end up with two garbage
collectors. For this reason, simply building both a C library and its
Lua/C binding with Emscripten and expecting them to work with Fengari
doesn't sound feasible. But the approach I described above, in which
the C library is compiled into JS and a Lua/JS binding is
custom-written for it, does (unless I'm missing something—the Fengari
authors will surely correct me if that's the case).

-- Hisham

> Jakub
>
> On 27. Jul 2017, at 0:23, Charles Heywood <[hidden email]> wrote:
>
> I'm not sure about using Emscripten, but there is LuLPeg which is mostly
> compatible with LPeg - enough that MoonScript can be compiled with it.
>
> On Wed, Jul 26, 2017 at 1:10 PM Jakub Jirutka <[hidden email]> wrote:
>>
>> Hi Benoit,
>>
>> Fengari is really awesome project, thank you and daurnimator very much for
>> it!
>>
>> I have one question. How about Lua/C modules? Is it currently possible to
>> compile e.g. LPeg into JS using Emscripten and use with Fengari?
>>
>> Jakub
>>
>> On 26. Jul 2017, at 14:34, Benoit Giannangeli <[hidden email]> wrote:
>>
>> Hi there,
>>
>> Fengari is a project me and daurnimator have been working on the past
>> months which aims to bring Lua to the browser.
>>
>> Here is the introductory article for it:
>> http://lua.space/webdev/why-we-rewrote-lua-in-js
>>
>> --
>> Benoit Giannangeli
>>
>> https://github.com/giann
>>
>>
> --
> --
> Ryan <[hidden email]>
> Software Developer / System Administrator
> https://hashbang.sh
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Jakub Jirutka
There’s a third motivation for using Lua/C modules in browser – there’s an existing library implemented in C, no compatible JS variant and/or you don’t wanna rewrite it.

I’ve mentioned LPeg. Yes, there’s LuLPeg, but “mostly compatible with LPeg” is not very convincing for me.

I know it’s tough task, I’m just asking about possibilities we have.

To add some context, I’m thinking about running Lua in browser for demo purposes (e.g. CLI utility you can try in a browser before installing) more than writing web applications in Lua.

Thanks,
Jakub

> On 27. Jul 2017, at 1:07, Hisham <[hidden email]> wrote:
>
> On 26 July 2017 at 19:42, Jakub Jirutka <[hidden email]> wrote:
>> I know about LuLPeg, but I’ve used LPeg just as an example. I’m interesting
>> about general support of Lua/C modules. Is it currently somehow possible?
>
> The motivations one uses Lua/C modules are either for extra speed, or
> for interfacing with existing libraries. For the first motivation,
> going through Emscripten and then the Lua/C API and run the whole
> thing on top of JS would probably be slower than writing a module
> directly in Lua; the approach for critical speed would be to write
> something in JavaScript instead and go through Fengari's Lua/JS API.
> For the second motivation, I suppose most existing libraries one would
> want to interact with in a browser would be existing JS libraries, and
> not C libraries. For a lot of C libraries, it wouldn't make sense to
> port them into the browser, but for those that can be compiled into JS
> libraries via Emscripten, I suppose one can bind them to Lua via the
> Lua/JS API, but the binding module needs to be rewritten (from C to
> JS).
>
> The linked article mentions that the approach of compiling Lua C code
> into the browser via Emscripten makes you end up with two garbage
> collectors. For this reason, simply building both a C library and its
> Lua/C binding with Emscripten and expecting them to work with Fengari
> doesn't sound feasible. But the approach I described above, in which
> the C library is compiled into JS and a Lua/JS binding is
> custom-written for it, does (unless I'm missing something—the Fengari
> authors will surely correct me if that's the case).
>
> -- Hisham
>
>> Jakub
>>
>> On 27. Jul 2017, at 0:23, Charles Heywood <[hidden email]> wrote:
>>
>> I'm not sure about using Emscripten, but there is LuLPeg which is mostly
>> compatible with LPeg - enough that MoonScript can be compiled with it.
>>
>> On Wed, Jul 26, 2017 at 1:10 PM Jakub Jirutka <[hidden email]> wrote:
>>>
>>> Hi Benoit,
>>>
>>> Fengari is really awesome project, thank you and daurnimator very much for
>>> it!
>>>
>>> I have one question. How about Lua/C modules? Is it currently possible to
>>> compile e.g. LPeg into JS using Emscripten and use with Fengari?
>>>
>>> Jakub
>>>
>>> On 26. Jul 2017, at 14:34, Benoit Giannangeli <[hidden email]> wrote:
>>>
>>> Hi there,
>>>
>>> Fengari is a project me and daurnimator have been working on the past
>>> months which aims to bring Lua to the browser.
>>>
>>> Here is the introductory article for it:
>>> http://lua.space/webdev/why-we-rewrote-lua-in-js
>>>
>>> --
>>> Benoit Giannangeli
>>>
>>> https://github.com/giann
>>>
>>>
>> --
>> --
>> Ryan <[hidden email]>
>> Software Developer / System Administrator
>> https://hashbang.sh
>>
>>
>


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Daurnimator
In reply to this post by Hisham
On 27 July 2017 at 09:07, Hisham <[hidden email]> wrote:
> The linked article mentions that the approach of compiling Lua C code
> into the browser via Emscripten makes you end up with two garbage
> collectors. For this reason, simply building both a C library and its
> Lua/C binding with Emscripten and expecting them to work with Fengari
> doesn't sound feasible. But the approach I described above, in which
> the C library is compiled into JS and a Lua/JS binding is
> custom-written for it, does (unless I'm missing something—the Fengari
> authors will surely correct me if that's the case).

Actually it's feasible.

The core issue is that a garbage collector needs to be able to sweep
*all* references: when you e.g. compile lua via emscripten the JS GC
has no idea what lua objects reference what. However the lua C api
doesn't create out-of-band references: you create+manage lua
references yourself.

That doesn't mean that it's currently turn-key to get an
emscripten-compiled lua C library to work with fengari. Someone will
need to write a table of emscripten <-> fengari bridge functions (one
for each C-api function).

However, note that fengari doesn't support __gc (which most C bindings
use), and can't until JS supports finalisers.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Daurnimator
On 27 July 2017 at 11:09, Daurnimator <[hidden email]> wrote:
> However the lua C api
> doesn't create out-of-band references: you create+manage lua
> references yourself.

I should rephrase:

The Lua C API doesn't permit out-of-band references between lua
objects: any references have to be created and dereferenced via
calling C api functions, hence meaning that the lua GC (and in
fengari's case, the JS GC) can do full sweeps.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Russell Haley
In reply to this post by Benoit Giannangeli
On Wed, Jul 26, 2017 at 5:34 AM, Benoit Giannangeli <[hidden email]> wrote:
> Hi there,
>
> Fengari is a project me and daurnimator have been working on the past months
> which aims to bring Lua to the browser.
>
> Here is the introductory article for it:
> http://lua.space/webdev/why-we-rewrote-lua-in-js

Wow. Just awesome guys. I can't wait to play with this.

The link to an example fully functional co-routine is dead.
Text: "Here's a simple example (fully functional version here):"
Link: https://gist.github.com/giann/f231cce5f17bde18aceb8537855cd51c

Russ

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Soni L.
In reply to this post by Daurnimator


On 2017-07-26 10:09 PM, Daurnimator wrote:

> On 27 July 2017 at 09:07, Hisham <[hidden email]> wrote:
>> The linked article mentions that the approach of compiling Lua C code
>> into the browser via Emscripten makes you end up with two garbage
>> collectors. For this reason, simply building both a C library and its
>> Lua/C binding with Emscripten and expecting them to work with Fengari
>> doesn't sound feasible. But the approach I described above, in which
>> the C library is compiled into JS and a Lua/JS binding is
>> custom-written for it, does (unless I'm missing something—the Fengari
>> authors will surely correct me if that's the case).
> Actually it's feasible.
>
> The core issue is that a garbage collector needs to be able to sweep
> *all* references: when you e.g. compile lua via emscripten the JS GC
> has no idea what lua objects reference what. However the lua C api
> doesn't create out-of-band references: you create+manage lua
> references yourself.
>
> That doesn't mean that it's currently turn-key to get an
> emscripten-compiled lua C library to work with fengari. Someone will
> need to write a table of emscripten <-> fengari bridge functions (one
> for each C-api function).
>
> However, note that fengari doesn't support __gc (which most C bindings
> use), and can't until JS supports finalisers.
>

What stops Lua/Fengari from having its own GC? Just make it so pushing a
JS object/lightuserdata finds or creates a lookup table entry for
mapping between JS and the GC? And don't expose Lua GC objects? This
gives you the exact Lua/C behaviour without screwing up the JS GC.
(Heavy userdata may be trickier, tho. Maybe require use of typed arrays
or something?)

I don't see why you'd need synced GCs. (I do see why one would want
them, because it simplifies implementation. But it's not portable - e.g.
for weak tables. Even if you could do a synced GC with JS, would you
still be able to do Lua 5.2+ ephemeral tables?)

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Daurnimator
On 27 July 2017 at 14:39, Soni L. <[hidden email]> wrote:

> What stops Lua/Fengari from having its own GC? Just make it so pushing a JS
> object/lightuserdata finds or creates a lookup table entry for mapping
> between JS and the GC? And don't expose Lua GC objects? This gives you the
> exact Lua/C behaviour without screwing up the JS GC. (Heavy userdata may be
> trickier, tho. Maybe require use of typed arrays or something?)
>
> I don't see why you'd need synced GCs. (I do see why one would want them,
> because it simplifies implementation. But it's not portable - e.g. for weak
> tables. Even if you could do a synced GC with JS, would you still be able to
> do Lua 5.2+ ephemeral tables?)

Because you need to be able to do a full sweep, otherwise cycles
cannot be detected.
JS (and the DOM) expose no way to do a full sweep: so we can't write
our own GC and still be able to call arbitrary JS functions.

The only solution was to structure our code so that the JS GC also
behaved as the lua GC.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Fengari: Why we rewrote Lua in JS

Soni L.


On 2017-07-27 01:45 AM, Daurnimator wrote:

> On 27 July 2017 at 14:39, Soni L. <[hidden email]> wrote:
>> What stops Lua/Fengari from having its own GC? Just make it so pushing a JS
>> object/lightuserdata finds or creates a lookup table entry for mapping
>> between JS and the GC? And don't expose Lua GC objects? This gives you the
>> exact Lua/C behaviour without screwing up the JS GC. (Heavy userdata may be
>> trickier, tho. Maybe require use of typed arrays or something?)
>>
>> I don't see why you'd need synced GCs. (I do see why one would want them,
>> because it simplifies implementation. But it's not portable - e.g. for weak
>> tables. Even if you could do a synced GC with JS, would you still be able to
>> do Lua 5.2+ ephemeral tables?)
> Because you need to be able to do a full sweep, otherwise cycles
> cannot be detected.
> JS (and the DOM) expose no way to do a full sweep: so we can't write
> our own GC and still be able to call arbitrary JS functions.
>
> The only solution was to structure our code so that the JS GC also
> behaved as the lua GC.
>

If you control every object inside your state you can do a full sweep.

As I said, treat pushing arbitrary JS objects as equivalent to pushing
arbitrary lightuserdata. GCing those objects does no cleanup, leaving
the cleanup to JS. But tables can still get __gc, since you have your
own GC, with your own JS-based data structures.

If heavy userdata were just blobs of bytes, you'd also be able to add
__gc to those. And cleanup would be all about using a JS array and
inserting objects into it and removing objects from it when __gc is
called, and so on. You'd be able to call __gc, you'd be able to do
everything normally. It'd be a fully featured Lua.

To call arbitrary JS functions you do an FFI that manages those
heavyuserdata for you. JS objects get put in an array, __gc removes them
from the array, __index looks them up in the array, etc. If it's
possible to FFI C++, then it must also be possible to FFI JS.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Loading...