[ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Anton Soldatov
Dear all,

tl;dr After several years of in-house development and production
usage, IPONWEB releases LuaVela, an implementation of Lua 5.1 based on
LuaJIT 2.0. For now the project supports Linux x86-64 only. The
license is MIT.

Please check out https://github.com/iponweb/luavela, patches and bug
reports are welcome.

A longer version:

I'm glad to announce the first public release of LuaVela, an
implementation of Lua 5.1 based on LuaJIT 2.0. The project was started
as a fork of LuaJIT 2.0 in 2015. The primary goal was to fix the
notorious 2Gb memory limit on the x86-64 platform. Eventually the
project accumulated several features that we would like to share with
the community.

List of major features:

* Full support for 64-bit memory without any tricks or hacks in the
interpreter and JIT compiler;

* "Sealing": An ability to hide some data from the garbage collector.
In IPONWEB, we use this generation-like (or, better, Eden-like) trick
to mark data with the same lifetime as the application instance itself
reducing overall pressure on GC;

* Immutability: Data structures may be (recursively) marked immutable
in run-time. This implemented via an extension API, the syntax of the
language is unaffected;

* Coroutine timeouts: There are C-level extension APIs that allow to
control the life time of coroutines – once a coroutine runs for too
long, it is terminated by the virtual machine;

* Some new optimizations in the JIT compiler (but some of them are not
brand new if one compares with LuaJIT 2.1);

* New C- and Lua-level extension APIs;

* Platform-level sampling profiler;

* Memory usage profiler;

* Platform-level code coverage.

Apart from that:

* CMake is used as a build system for the project;

* 6 test suites are bundled with the project: Lua 5.1 test suite,
LuaJIT test suite, CERN MAD test suite (partially), lua-Harness test
suite and two suites written inside IPONWEB (for testing at Lua- and
C-level, respectively);

* Documentation bundle is included into the release, too. All the docs
are in the RST format and `make docs` will build you the HTML version
if you have Sphinx installed. This one is the last thing we prepared
before the release, so apologies for some mishmash there, we will sort
it out eventually.

Feel free to ask any questions, but here is a FAQ list based on what
I've been asked during my talks, etc.:

Q: What does "Vela" in LuaVela mean?

A: It is Vela constellation.

Q: What does "uJIT" mean in the docs and the core code base?

A: This is an internal name of the project in IPONWEB. Unfortunately,
we could not simply strip it off prior to the public release, so
apologies for possible confusion. LuaVela is the preferred public name
of the project, uJIT is something like an internal "code name".

Q: What is the level of compatibility with Lua 5.1?

A: The same as for LuaJIT 2.0. But we wish we could make the
implementation closer to the PUC-Rio implementation.

Q: Why not LuaJIT 2.1 + LJ_GC64?

A: As far as I know, this solution got stabilized by 2016-2017. Alas,
we needed something working earlier. Besides, some experiments we ran
in 2015 with LuaJIT 2.1 showed performance degradation for our cases.

Q: What about support for Lua 5.2+?

A: Unfortunately, we do not have resources to implement it on our own.
However, patches are welcome. We definitely do *not* plan to increase
the compatibility gap with the PUC-Rio implementation. E.g. dropping
support for C API or any initiative like this is not an option for the
project.

Q: What about support for other OSes and other platforms than Linux x86-64?

A: I doubt that 32-bit platforms will be supported. Regarding other
operating systems, we do not have resources to implement it on our
own. However, patches are welcome.

And last, but definitely not least. Many people have worked hard to
make this public release happen. I would like to acknowledge some of
them personally:

* Igor Ehrlich, the godfather of the project who started it and did an
unbelievable amount of work during the project's early development and
establishment phases. In 2016, he gave an excellent talk (sorry, in
Russian only) about our reasons to start an own implementation [1];
* Maxim Bolshov, who put much of his expertise into JIT-related things;
* Ilya Dailidyonok, who, apart from delivering various features, did a
fantastic work polishing the code base at all levels preparing it for
the public release.

P.S. For those who wish more context, here is a couple of my talks on
the matter:

* "Challenges Building Yet Another Lua Implementation", at Lua in
Moscow 2017 (in English) [2];
* "Rewriting LuaJIT: Why and How?", at Lua Workshop 2018. The talk is
in English, but I as far as I know there are slides only [3].

[1] https://www.youtube.com/watch?v=64c4ihDl6MM
[2] http://lua.moscow/conf/2017-03-LuaInMoscow/index.html#soldatov
[3] https://www.lua.org/wshop18/Soldatov.pdf

--
Med vennlig hilsen,
Anton Soldatov

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Javier Guerra Giraldez
On Thu, 8 Aug 2019 at 09:52, Anton Soldatov <[hidden email]> wrote:

>
> Dear all,
>
> tl;dr After several years of in-house development and production
> usage, IPONWEB releases LuaVela, an implementation of Lua 5.1 based on
> LuaJIT 2.0. For now the project supports Linux x86-64 only. The
> license is MIT.
>
> Please check out https://github.com/iponweb/luavela, patches and bug
> reports are welcome.


This is great news!

even if many of us might not use it directly, there's a _ton_ of hard
work there and lots of things to learn from.   the first thing I'd
like to check is how much work and overhead is there from the bigger
TValue struct.  the current 47-bit limit is annoying, and just saying
"this is a design issue, it's not going away" is just not enough.  now
we can compare with a real, tested, alternate implementation.


lots of big thanks to all those involved!

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Dibyendu Majumdar
In reply to this post by Anton Soldatov
Hi Anton,

On Thu, 8 Aug 2019 at 17:52, Anton Soldatov <[hidden email]> wrote:
> tl;dr After several years of in-house development and production
> usage, IPONWEB releases LuaVela, an implementation of Lua 5.1 based on
> LuaJIT 2.0. For now the project supports Linux x86-64 only. The
> license is MIT.
>

It is great that you have been able to share this.
Looks like you guys have put in a lot of work!

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Dibyendu Majumdar
> It is great that you have been able to share this.
> Looks like you guys have put in a lot of work!

Also fantastic effort documenting stuff.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

JeanHeyd Meneide
Dear Anton Soldatov,

     This looks wonderful! I'm glad you worked on it and put it out there. I know a lot of people who had problems with the 2GB memory limit.

     As a side note, there were some problems with LuaJIT and Mac OSX, where on 64-bit builds LuaJIT would twiddle bits in the 32-bit range. This required people to build their applications with -pagezero_size=(something much smaller than the default 2^32) and -image_base=(something much smaller than 2^32, which is the default). This ruined people who wanted to inject Lua extensions as part of a dylib extension to a main application which did not change their image_base/pagezero_size.

     Does 64-bit "clean" mean that, if this makes it to Mac OSX, this would no longer be a problem?

Sincerely,
ThePhD
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Anton Soldatov
Hi Javier,

> the first thing I'd like to check is how much
> work and overhead is there from
> the bigger TValue struct

The benchmarks from the LuaJIT's suite show that the wide TValue and
"fair" 64-bit pointers are somewhat slower than LuaJIT (although there
are cases when LuaVela is faster). However, as far as I know, we did
not experience any severe degradation in production (sometimes we even
won over FFI-based code, but this is whole another story), and
IPONWEB's business (real-time bidding) is all about fitting business
logic into hard timeouts (80-120ms).

Anyway, please do share your results!

tor. 8. aug. 2019 kl. 23:27 skrev JeanHeyd Meneide <[hidden email]>:

>
> Dear Anton Soldatov,
>
>      This looks wonderful! I'm glad you worked on it and put it out there. I know a lot of people who had problems with the 2GB memory limit.
>
>      As a side note, there were some problems with LuaJIT and Mac OSX, where on 64-bit builds LuaJIT would twiddle bits in the 32-bit range. This required people to build their applications with -pagezero_size=(something much smaller than the default 2^32) and -image_base=(something much smaller than 2^32, which is the default). This ruined people who wanted to inject Lua extensions as part of a dylib extension to a main application which did not change their image_base/pagezero_size.
>
>      Does 64-bit "clean" mean that, if this makes it to Mac OSX, this would no longer be a problem?
>
> Sincerely,
> ThePhD



--
Med vennlig hilsen,
Anton Soldatov

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Anton Soldatov
Hello JeanHeyd,

> Does 64-bit "clean" mean that, if this makes it to Mac OSX,
> this would no longer be a problem?

Not much an expert in programming for Mac OSX. But since LuaVela uses
"fair" 64-bit pointers and can allocate from any region in the address
space of the process, I guess the answer is yes. (Jumps within
dynamically generated machine code must still fit into 32 bits,
though)

By the way, confirming that currently the build for Mac OSX is broken,
but I'd like to re-enable it (with some very non-portable features
turned off) as I use MacBook myself. Last time I managed to compile
the code, but linkage failed. Anyway, I hope that enabling Mac OSX is
nearby.

tor. 8. aug. 2019 kl. 23:43 skrev Anton Soldatov <[hidden email]>:

>
> Hi Javier,
>
> > the first thing I'd like to check is how much
> > work and overhead is there from
> > the bigger TValue struct
>
> The benchmarks from the LuaJIT's suite show that the wide TValue and
> "fair" 64-bit pointers are somewhat slower than LuaJIT (although there
> are cases when LuaVela is faster). However, as far as I know, we did
> not experience any severe degradation in production (sometimes we even
> won over FFI-based code, but this is whole another story), and
> IPONWEB's business (real-time bidding) is all about fitting business
> logic into hard timeouts (80-120ms).
>
> Anyway, please do share your results!
>
> tor. 8. aug. 2019 kl. 23:27 skrev JeanHeyd Meneide <[hidden email]>:
> >
> > Dear Anton Soldatov,
> >
> >      This looks wonderful! I'm glad you worked on it and put it out there. I know a lot of people who had problems with the 2GB memory limit.
> >
> >      As a side note, there were some problems with LuaJIT and Mac OSX, where on 64-bit builds LuaJIT would twiddle bits in the 32-bit range. This required people to build their applications with -pagezero_size=(something much smaller than the default 2^32) and -image_base=(something much smaller than 2^32, which is the default). This ruined people who wanted to inject Lua extensions as part of a dylib extension to a main application which did not change their image_base/pagezero_size.
> >
> >      Does 64-bit "clean" mean that, if this makes it to Mac OSX, this would no longer be a problem?
> >
> > Sincerely,
> > ThePhD
>
>
>
> --
> Med vennlig hilsen,
> Anton Soldatov



--
Med vennlig hilsen,
Anton Soldatov

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Kaj Eijlers
This sounds really interesting, I tend to use lua a lot for brute force stuff that uses a ton of memory. 

Quick question re Coroutine timeouts, since I didn't generate the help files and don't know how else to find this - does it support a callback or other notification?
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Anton Soldatov
Hi Kaj,

Yes, callbacks are supported. Plus an extra status code LUAE_TIMEOUT is added to lua_resume.

пт, 9 авг. 2019 г. в 7:17, Kaj Eijlers <[hidden email]>:
This sounds really interesting, I tend to use lua a lot for brute force stuff that uses a ton of memory. 

Quick question re Coroutine timeouts, since I didn't generate the help files and don't know how else to find this - does it support a callback or other notification?
--
Med vennlig hilsen,
Anton Soldatov
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Ką Mykolas
Really nice to see You guys releasing it to the public after nice talk
in LuaWorkshop in Kaunas :)

Despite guys on Telegram complaining about it being mainly focused on
Linux servers, kudos and I hope community effort is incoming!

On Fri, Aug 9, 2019 at 11:44 AM Anton Soldatov <[hidden email]> wrote:

>
> Hi Kaj,
>
> Yes, callbacks are supported. Plus an extra status code LUAE_TIMEOUT is added to lua_resume.
>
> пт, 9 авг. 2019 г. в 7:17, Kaj Eijlers <[hidden email]>:
>>
>> This sounds really interesting, I tend to use lua a lot for brute force stuff that uses a ton of memory.
>>
>> Quick question re Coroutine timeouts, since I didn't generate the help files and don't know how else to find this - does it support a callback or other notification?
>
> --
> Med vennlig hilsen,
> Anton Soldatov

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

François Perrad
In reply to this post by Dibyendu Majumdar


Le jeu. 8 août 2019 à 21:15, Dibyendu Majumdar <[hidden email]> a écrit :
> It is great that you have been able to share this.
> Looks like you guys have put in a lot of work!

Also fantastic effort documenting stuff.


Yes, great documentation, but I found a missing section

## Using Luarocks with LuaVela

After configuring/building LuaVela with option -DCMAKE_INSTALL_PREFIX=SOMEWHERE
You could use the latest dev Luarocks on master branch, with the following options
./configure
     --prefix=SOMEWHERE
     --with-lua=SOMEWHERE
     --with-lua-include=SOMEWHERE/include/ujit
     --with-lua-interpreter=luavela
After, make & make install, you could run
    SOMEWHERE/bin/luarocks install somerock

That works with pure Lua modules, but fails with native modules.
Because, package.cpath contains SOMEWHERE/lib/x86_64-linux-gnu/lua/5.1/?.so
and luarocks install them in SOMEWHERE/lib/lua/5.1

## A nice to have

currently, we could see versions, like this:
$ luavela -v
LuaVela aka uJIT 0.24-dev0 -- Copyright (C) 2015-2019 IPONWEB Ltd. https://github.com/iponweb/luavela
$ luavela -e 'print(_VERSION)'
Lua 5.1
$ luavela -e 'print(jit.version)'
LuaJIT 2.0.5

It could be nice to have:
$ luavela -e 'print(ujit.version)'
LuaVela 0.24-dev0

François
 
Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

Anton Soldatov
Hi François,

Thanks, we’ll clean it up.

сб, 10 авг. 2019 г. в 12:18, François Perrad <[hidden email]>:


Le jeu. 8 août 2019 à 21:15, Dibyendu Majumdar <[hidden email]> a écrit :
> It is great that you have been able to share this.
> Looks like you guys have put in a lot of work!

Also fantastic effort documenting stuff.


Yes, great documentation, but I found a missing section

## Using Luarocks with LuaVela

After configuring/building LuaVela with option -DCMAKE_INSTALL_PREFIX=SOMEWHERE
You could use the latest dev Luarocks on master branch, with the following options
./configure
     --prefix=SOMEWHERE
     --with-lua=SOMEWHERE
     --with-lua-include=SOMEWHERE/include/ujit
     --with-lua-interpreter=luavela
After, make & make install, you could run
    SOMEWHERE/bin/luarocks install somerock

That works with pure Lua modules, but fails with native modules.
Because, package.cpath contains SOMEWHERE/lib/x86_64-linux-gnu/lua/5.1/?.so
and luarocks install them in SOMEWHERE/lib/lua/5.1

## A nice to have

currently, we could see versions, like this:
$ luavela -v
LuaVela aka uJIT 0.24-dev0 -- Copyright (C) 2015-2019 IPONWEB Ltd. https://github.com/iponweb/luavela
$ luavela -e 'print(_VERSION)'
Lua 5.1
$ luavela -e 'print(jit.version)'
LuaJIT 2.0.5

It could be nice to have:
$ luavela -e 'print(ujit.version)'
LuaVela 0.24-dev0

François
 
Regards
Dibyendu

--
Med vennlig hilsen,
Anton Soldatov
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaVela, an implementation of Lua 5.1 based on LuaJIT 2.0

François Perrad
In reply to this post by Anton Soldatov


Le jeu. 8 août 2019 à 18:52, Anton Soldatov <[hidden email]> a écrit :
Dear all,

tl;dr After several years of in-house development and production
usage, IPONWEB releases LuaVela, an implementation of Lua 5.1 based on
LuaJIT 2.0. For now the project supports Linux x86-64 only. The
license is MIT.

Please check out https://github.com/iponweb/luavela, patches and bug
reports are welcome.



François

--
Med vennlig hilsen,
Anton Soldatov