How to make a static Lua?

[Steve Litt]

Hi all,

After introduction of the systemd init system, a lot of people are
creating their own init systems. One guy did it in Ruby.

I'm thinking that Lua is much smaller than Ruby, and in my opinion it's
more understandable, and from what I read, it's faster. The reason it
would be good to be static is so that an initramfs wouldn't be
necessary (though it usually is for other reasons). But static
compilation isn't absolutely necessary: all the libraries could go in
an initramfs.

The version of Lua for this thing would need to have signal handling,
forking, and exec'ing. Are those features in place with standard Lua?

Is 5.3 available pretty much everywhere these days? If not, which Lua
is the most widely available?

Thanks,

SteveT

Steve Litt
April 2016 featured book: Rapid Learning for the 21st Century
http://www.troubleshooters.com/rl21

[Rena]

On Sat, Apr 16, 2016 at 9:09 PM, Steve Litt <[hidden email]> wrote:

Hi all,

After introduction of the systemd init system, a lot of people are
creating their own init systems. One guy did it in Ruby.

I'm thinking that Lua is much smaller than Ruby, and in my opinion it's
more understandable, and from what I read, it's faster. The reason it
would be good to be static is so that an initramfs wouldn't be
necessary (though it usually is for other reasons). But static
compilation isn't absolutely necessary: all the libraries could go in
an initramfs.

The version of Lua for this thing would need to have signal handling,
forking, and exec'ing. Are those features in place with standard Lua?

Is 5.3 available pretty much everywhere these days? If not, which Lua
is the most widely available?

Thanks,

SteveT

Steve Litt
April 2016 featured book: Rapid Learning for the 21st Century
http://www.troubleshooters.com/rl21

My thoughts:

* Lua as an init system sounds fantastic. In fact a *lot* of the various scripts and config files around a Linux system could be replaced with Lua scripts.

* BSD did some work on Lua in the kernel. Look into that.

* Stock Lua doesn't have signal handling, and that's something that's very difficult to implement correctly, but very easy to implement in a way that *seems* correct until it blows up in your face. Tread carefully.

* It doesn't provide fork or exec either, but those are simple enough. You're going to need to add some C functions anyway.

* I don't see any reason you couldn't use 5.3. However I think there are more libraries available for 5.1. Whether you'd want to use them in this context, I can't say.

--

[Sean Conner]

It was thus said that the Great Steve Litt once stated:
> Hi all,
>
> After introduction of the systemd init system, a lot of people are
> creating their own init systems. One guy did it in Ruby.

  I started down that road myself, using Lua.  And it's a nice design too,
but not all currently developed Unix daemons would work out of the box with
it (daemons that daemonize themselves [1] with not option to run in the
foreground for instance).

> I'm thinking that Lua is much smaller than Ruby, and in my opinion it's
> more understandable, and from what I read, it's faster. The reason it
> would be good to be static is so that an initramfs wouldn't be
> necessary (though it usually is for other reasons). But static
> compilation isn't absolutely necessary: all the libraries could go in
> an initramfs.
>
> The version of Lua for this thing would need to have signal handling,
> forking, and exec'ing. Are those features in place with standard Lua?

  No, they are not.  There are Lua modules that will do that though, for
instance:

        org.conman.signal [2]
        org.conman.process [3]

The fork() and exec() calls are easy enough to deal with in Lua, but
signals?  Not as straightforward to handle [4] and it gets messy rather
quickly.

> Is 5.3 available pretty much everywhere these days? If not, which Lua
> is the most widely available?

  I don't know, but my guess would be Lua 5.1, followed by Lua 5.3 (there
just isn't a compelling reason to upgrade to Lua 5.2 in my opinion).  I
should note that the modules I listed above can compile against Lua 5.1, Lua
5.2 and Lua 5.3).

  -spc

[1] If that doesn't mean anything to you, don't worry about it.

[2] https://github.com/spc476/lua-conmanorg/blob/master/src/signal.c

[3] https://github.com/spc476/lua-conmanorg/blob/master/src/process.c

[4] gopher://gopher.conman.org/0Gopher%3aSrc%3areapchild.c [5]

[5] If you can't fetch that, here's the relevant bit:

* This entire file exists *because* I can't properly handle SIGCHLD in the
* server process.  What I want to do is just get the reason for a handler
* process terminating, but not wanting to deal with EINTR, so I set the
* handler to restart system calls.  But while I can do that in Lua, the Lua
* signal handler doesn't get control until the Lua VM gets control, and if
* we're sitting in a system call, that might take some time.  So, we do this
* in C, which can get the job done.
*
* This code is straightforward---just export a single function to Lua that
* run called, catches SIGCHLD, and the signal handler will reap the
* children.

[Daurnimator]

On 17 April 2016 at 13:12, Sean Conner <[hidden email]> wrote:

> It was thus said that the Great Steve Litt once stated:
>> Hi all,
>>
>> After introduction of the systemd init system, a lot of people are
>> creating their own init systems. One guy did it in Ruby.
>
>   I started down that road myself, using Lua.  And it's a nice design too,
> but not all currently developed Unix daemons would work out of the box with
> it (daemons that daemonize themselves [1] with not option to run in the
> foreground for instance).
>
>> I'm thinking that Lua is much smaller than Ruby, and in my opinion it's
>> more understandable, and from what I read, it's faster. The reason it
>> would be good to be static is so that an initramfs wouldn't be
>> necessary (though it usually is for other reasons). But static
>> compilation isn't absolutely necessary: all the libraries could go in
>> an initramfs.
>>
>> The version of Lua for this thing would need to have signal handling,
>> forking, and exec'ing. Are those features in place with standard Lua?
>
>   No, they are not.  There are Lua modules that will do that though, for
> instance:
>
>         org.conman.signal       [2]
>         org.conman.process      [3]
>
> The fork() and exec() calls are easy enough to deal with in Lua, but
> signals?  Not as straightforward to handle [4] and it gets messy rather
> quickly.
>
>> Is 5.3 available pretty much everywhere these days? If not, which Lua
>> is the most widely available?
>
>   I don't know, but my guess would be Lua 5.1, followed by Lua 5.3 (there
> just isn't a compelling reason to upgrade to Lua 5.2 in my opinion).  I
> should note that the modules I listed above can compile against Lua 5.1, Lua
> 5.2 and Lua 5.3).
>
>   -spc
>
> [1]     If that doesn't mean anything to you, don't worry about it.
>
> [2]     https://github.com/spc476/lua-conmanorg/blob/master/src/signal.c
>
> [3]     https://github.com/spc476/lua-conmanorg/blob/master/src/process.c
>
> [4]     gopher://gopher.conman.org/0Gopher%3aSrc%3areapchild.c [5]
>
> [5]     If you can't fetch that, here's the relevant bit:
>
> * This entire file exists *because* I can't properly handle SIGCHLD in the
> * server process.  What I want to do is just get the reason for a handler
> * process terminating, but not wanting to deal with EINTR, so I set the
> * handler to restart system calls.  But while I can do that in Lua, the Lua
> * signal handler doesn't get control until the Lua VM gets control, and if
> * we're sitting in a system call, that might take some time.  So, we do this
> * in C, which can get the job done.
> *
> * This code is straightforward---just export a single function to Lua that
> * run called, catches SIGCHLD, and the signal handler will reap the
> * children.

What's wrong with using signalfd? (or on a BSD: kqueue with EVFILT_SIGNAL)
When your signalfd for SIGCHLD polls ready, call wait() with NOHANG
until it doesn't return a pid.

[Daurnimator]

On 17 April 2016 at 11:09, Steve Litt <[hidden email]> wrote:
> After introduction of the systemd init system, a lot of people are
> creating their own init systems. One guy did it in Ruby.

I actually like systemd.....

> The version of Lua for this thing would need to have signal handling,
> forking, and exec'ing. Are those features in place with standard Lua?

FWIW, to build this project I would use these libraies:
  - cqueues: http://25thandclement.com/~william/projects/cqueues.html
to manage your event loop, listen for signals and socket communication
  - lua-spawn https://github.com/daurnimator/lua-spawn to start programs
  - lunix http://25thandclement.com/~william/projects/lunix.html for
other misc things (e.g. pipe())

> Is 5.3 available pretty much everywhere these days? If not, which Lua
> is the most widely available?

5.3 is about as widely available as 5.1 is.
Though if you're making your own init system, you have no reason to be
limited by what people already have. Changing init system is a rather
big commitment.

[Sean Conner]

It was thus said that the Great Daurnimator once stated:

> On 17 April 2016 at 13:12, Sean Conner <[hidden email]> wrote:
> >>
> > * This entire file exists *because* I can't properly handle SIGCHLD in the
> > * server process.  What I want to do is just get the reason for a handler
> > * process terminating, but not wanting to deal with EINTR, so I set the
> > * handler to restart system calls.  But while I can do that in Lua, the Lua
> > * signal handler doesn't get control until the Lua VM gets control, and if
> > * we're sitting in a system call, that might take some time.  So, we do this
> > * in C, which can get the job done.
> > *
> > * This code is straightforward---just export a single function to Lua that
> > * run called, catches SIGCHLD, and the signal handler will reap the
> > * children.
>
> What's wrong with using signalfd? (or on a BSD: kqueue with EVFILT_SIGNAL)
> When your signalfd for SIGCHLD polls ready, call wait() with NOHANG
> until it doesn't return a pid.

  It isn't portable (why yes, I'm still using a Linux system without it).

  -spc (Not to mention that signals are probably the worst abstraction to
        come out of Unix ...)

[Daurnimator]

On 17 April 2016 at 16:32, Sean Conner <[hidden email]> wrote:

> It was thus said that the Great Daurnimator once stated:
>> On 17 April 2016 at 13:12, Sean Conner <[hidden email]> wrote:
>> >>
>> > * This entire file exists *because* I can't properly handle SIGCHLD in the
>> > * server process.  What I want to do is just get the reason for a handler
>> > * process terminating, but not wanting to deal with EINTR, so I set the
>> > * handler to restart system calls.  But while I can do that in Lua, the Lua
>> > * signal handler doesn't get control until the Lua VM gets control, and if
>> > * we're sitting in a system call, that might take some time.  So, we do this
>> > * in C, which can get the job done.
>> > *
>> > * This code is straightforward---just export a single function to Lua that
>> > * run called, catches SIGCHLD, and the signal handler will reap the
>> > * children.
>>
>> What's wrong with using signalfd? (or on a BSD: kqueue with EVFILT_SIGNAL)
>> When your signalfd for SIGCHLD polls ready, call wait() with NOHANG
>> until it doesn't return a pid.
>
>   It isn't portable (why yes, I'm still using a Linux system without it).

Really? signalfd was added in kernel 2.6.22. which was released in 2007.
The oldest linux kernel still maintained for security updates is 3.2.
If you're running a kernel older than that you're probably not looking
to run newer software on it (and hopefully don't have it connected to
the internet, or allow anyone even vaguely untrusted to touch it).

Otherwise, you can fall back to using sigtimedwait (with cqueues does
do for e.g. solaris).

[Marc Balmer]

> Am 17.04.2016 um 03:09 schrieb Steve Litt <[hidden email]>:
>
> Hi all,
>
> After introduction of the systemd init system, a lot of people are
> creating their own init systems. One guy did it in Ruby.
>
> I'm thinking that Lua is much smaller than Ruby, and in my opinion it's
> more understandable, and from what I read, it's faster. The reason it
> would be good to be static is so that an initramfs wouldn't be
> necessary (though it usually is for other reasons). But static
> compilation isn't absolutely necessary: all the libraries could go in
> an initramfs.
>
> The version of Lua for this thing would need to have signal handling,
> forking, and exec'ing. Are those features in place with standard Lua?

I have written a unix module to provide unix functionality to Lua programs.  We use it to write daemons in Lua.

See github.com/arcapos/luaunix if you are interested.

>
> Is 5.3 available pretty much everywhere these days? If not, which Lua
> is the most widely available?

Since such a systemd like program is run early in the boot process, I suggest you write a small core in C that statically links against Lua 5.3.

>
> Thanks,
>
> SteveT
>
> Steve Litt
> April 2016 featured book: Rapid Learning for the 21st Century
> http://www.troubleshooters.com/rl21
>

[Viacheslav Usov]

On Sun, Apr 17, 2016 at 3:09 AM, Steve Litt <[hidden email]> wrote:

> The reason it would be good to be static is so that an initramfs wouldn't be necessary (though it usually is for other reasons). But static compilation isn't absolutely necessary: all the libraries could go in an initramfs.

This reminds me of something that I have been thinking of for a while:

1. Have an executable built with Lua statically.

2. Load native Lua modules built for the same Lua version, that would then use the static Lua library in the host executable.

It is acceptable that both Lua and modules need to be built with a special define or defines.

Perhaps something like that has already been done. Anyone know?

Cheers,

V.

[Luiz Henrique de Figueiredo]

> This reminds me of something that I have been thinking of for a while:
>
> 1. Have an executable built with Lua statically.
>
> 2. Load native Lua modules built for the same Lua version, that would then
> use the static Lua library in the host executable.

This is exactly what the command line interpreter lua does.

> It is acceptable that both Lua and modules need to be built with a special
> define or defines.

For Linux, you need to use -Wl,-E when linking so that symbols in the
executable are seen for dynamic linking.

For Mac OS X, no magic is needed.

[Viacheslav Usov]

On Mon, Apr 18, 2016 at 4:19 PM, Luiz Henrique de Figueiredo <[hidden email]> wrote:

> This reminds me of something that I have been thinking of for a while:
>
> 1. Have an executable built with Lua statically.
>
> 2. Load native Lua modules built for the same Lua version, that would then
> use the static Lua library in the host executable.

This is exactly what the command line interpreter lua does.

> It is acceptable that both Lua and modules need to be built with a special
> define or defines.

For Linux, you need to use -Wl,-E when linking so that symbols in the
executable are seen for dynamic linking.

For Mac OS X, no magic is needed.

And for Windows?

Cheers,

V.

[steve donovan]

On Mon, Apr 18, 2016 at 4:44 PM, Viacheslav Usov <[hidden email]> wrote:
> And for Windows?

Sure, Windows executables can export symbols like DLLs can. With mingw
-Wl,-E makes this happen as well.

[Luiz Henrique de Figueiredo]

> And for Windows?

I don't know about Windows. Sorry. But I expect many here do.

[Viacheslav Usov]

On Mon, Apr 18, 2016 at 4:48 PM, steve donovan <[hidden email]> wrote:

On Mon, Apr 18, 2016 at 4:44 PM, Viacheslav Usov <[hidden email]> wrote:
> And for Windows?

Sure, Windows executables can export symbols like DLLs can.

Well, yes, but there is an important difference with Unix shared libraries. In Windows, an executables that wants to import function foo, has to specify that it is contained in DLL bar. In Unix, those two things are independent (which is how one can redirect/override imports in Unix) .

So if my Lua module was built with Lua 5.2 for Windows, it will have an import blob that says "load lua52.dll, find lua_verison, lua_tonumber, etc in its export section and use them". Having those same functions exported from my host.exe is useless, because the loader won't even look there.

With mingw -Wl,-E makes this happen as well.

Is that something that you know (rather than just think) will work?

Cheers,

V.

[Paul K-2]

> So if my Lua module was built with Lua 5.2 for Windows, it will have an import blob that says "load lua52.dll, find lua_verison, lua_tonumber, etc in its export section and use them". Having those same functions exported from my host.exe is useless, because the loader won't even look there.

Right, on Windows you'd need to have a proxy DLL that will forward
those requests to the exported functions in the main executable. See
http://lua-users.org/wiki/LuaProxyDllFour and this thread for the
related discussion:
http://lua-users.org/lists/lua-l/2013-11/msg00831.html

Paul.

[steve donovan]

On Mon, Apr 18, 2016 at 6:32 PM, Viacheslav Usov <[hidden email]> wrote:
>> With mingw -Wl,-E makes this happen as well.
>
> Is that something that you know (rather than just think) will work?

Of course, you _do_ have to rebuild all your DLLs to refer to the
symbols from the executable; that's indeed the big difference.  I know
this works, because if an application like SciTE which embeds Lua is
allowed to export its symbols, it can load modules, recompiled against
SciTE.exe.

The other approach is to build a fat executable with modules
statically compiled, like I do with the luabuild project.

steve d.

[Coda Highland]

On Mon, Apr 18, 2016 at 10:22 AM, steve donovan
<[hidden email]> wrote:

> On Mon, Apr 18, 2016 at 6:32 PM, Viacheslav Usov <[hidden email]> wrote:
>>> With mingw -Wl,-E makes this happen as well.
>>
>> Is that something that you know (rather than just think) will work?
>
> Of course, you _do_ have to rebuild all your DLLs to refer to the
> symbols from the executable; that's indeed the big difference.  I know
> this works, because if an application like SciTE which embeds Lua is
> allowed to export its symbols, it can load modules, recompiled against
> SciTE.exe.
>
> The other approach is to build a fat executable with modules
> statically compiled, like I do with the luabuild project.
>
> steve d.
>

A little bit of extra detail: .exe and .dll files on Windows are the
same format. You can link to an .exe just like you can a .dll, and you
can execute a .dll that exposes an entry point just like you an an
.exe (and in fact the OS does this in many places).

/s/ Adam

[Viacheslav Usov]

On Mon, Apr 18, 2016 at 6:54 PM, Paul K <[hidden email]> wrote:

> So if my Lua module was built with Lua 5.2 for Windows, it will have an import blob that says "load lua52.dll, find lua_verison, lua_tonumber, etc in its export section and use them". Having those same functions exported from my host.exe is useless, because the loader won't even look there.

Right, on Windows you'd need to have a proxy DLL that will forward
those requests to the exported functions in the main executable. See
http://lua-users.org/wiki/LuaProxyDllFour and this thread for the
related discussion:
http://lua-users.org/lists/lua-l/2013-11/msg00831.html

Ah, indeed. A cursory examination of the project indicates that it actually creates a function with a jump to the real thing. I wonder if this approach can be used instead:

https://blogs.msdn.microsoft.com/oldnewthing/20060719-24/?p=30473

I am actually pretty sure it can be, I just do no think I will have enough time within the next few days to play with this.

Cheers,

V.