Modify Lua Executable for a -51 option

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Modify Lua Executable for a -51 option

Russell Haley
Hi,

So the idea came to me that it would be easier to maintain one copy of lua that can run the 5.1 dll/so and the 5.x dll/so based on a command line switch. The thought is the lua executable is potentially rather stable in the 5.x series. Since 5.1 isn't changing any time soon, there would be very little maintenance to do?

I do plan on diffing lua.c at some point shortly, but any input would be grand?

Thanks,

Russ
Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Dibyendu Majumdar
On 28 April 2018 at 04:52, Russell Haley <[hidden email]> wrote:
> So the idea came to me that it would be easier to maintain one copy of lua
> that can run the 5.1 dll/so and the 5.x dll/so based on a command line
> switch. The thought is the lua executable is potentially rather stable in
> the 5.x series. Since 5.1 isn't changing any time soon, there would be very
> little maintenance to do?
>
> I do plan on diffing lua.c at some point shortly, but any input would be
> grand?
>


Look at https://github.com/corsix/twoface

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Russell Haley


On Sat, Apr 28, 2018 at 12:02 AM, Dibyendu Majumdar <[hidden email]> wrote:
On 28 April 2018 at 04:52, Russell Haley <[hidden email]> wrote:
> So the idea came to me that it would be easier to maintain one copy of lua
> that can run the 5.1 dll/so and the 5.x dll/so based on a command line
> switch. The thought is the lua executable is potentially rather stable in
> the 5.x series. Since 5.1 isn't changing any time soon, there would be very
> little maintenance to do?
>
> I do plan on diffing lua.c at some point shortly, but any input would be
> grand?
>


Look at https://github.com/corsix/twoface

Thanks! This looks interesting, but a little more involved then what I'm attempting. I want to call 5.1 or 5.x exclusively, whereas twoface seems to provide a custom ABI for running lua 5.1 within lua 5.2. I'm reading twoface.c to see if I can steal any ideas.

Cheers, 
Russ

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Luiz Henrique de Figueiredo
In reply to this post by Russell Haley
> So the idea came to me that it would be easier to maintain one copy of lua
> that can run the 5.1 dll/so and the 5.x dll/so based on a command line
> switch

Do you mean this?

#!/bin/sh
# usage: lua VERSION other args

V=$1
shift

echo exec /usr/local/bin/lua$V $@

Typical usage:  lua -51 foo.lua arg1 arg2 ... but all command-liine
switches are available.

This assumes the existence of /usr/local/bin/lua-51, lua-52, etc.

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Russell Haley


On Sat, Apr 28, 2018 at 11:23 AM, Luiz Henrique de Figueiredo <[hidden email]> wrote:
> So the idea came to me that it would be easier to maintain one copy of lua
> that can run the 5.1 dll/so and the 5.x dll/so based on a command line
> switch

Do you mean this?

#!/bin/sh
# usage: lua VERSION other args

V=$1
shift

echo exec /usr/local/bin/lua$V $@

Typical usage:  lua -51 foo.lua arg1 arg2 ... but all command-liine
switches are available.

This assumes the existence of /usr/local/bin/lua-51, lua-52, etc.

That's definitely the pragmatic approach to the problem, which is probably where I'll wind up. That said, the idea of wrapping lua in a batch script makes me sad and I don't want to spend money on a signing key for powershell (not to mention it's SLOW). 

The 'experiment' is a single executable to handle various configuration options and then modify luarocks to install a rock tree per directory. I'm using node.js at work and I liked being able to create a per directory or global rock tree. Both 'experimentalLua' and 'modluarocks' would be modified to look for configuration data. Said applications would first check the command line, then the local directory, then a global config. To handle 32 bit and 64 bit, the 32 bit executable would always be called first, which could then call the 64 bit executable if specified (I have no idea if that would work). 

At first I thought to write a wrapper. And then I thought, "Well, fuck, it's just a tiny C program. How complicated can it be"? (tee hee) The most portable and performant way to handle configuration is in C. In the end, modifying Lua itself seems like the best approach?

Thanks!
Russ


Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Dirk Laurie-2
2018-04-29 6:54 GMT+02:00 Russell Haley <[hidden email]>:

> the idea of wrapping lua in a batch script makes me sad

About emotions there is no disputing ...

A swig of bourbon, straight, is said to a wonderful cure for sadness.

> In the end, modifying Lua itself seems like the best approach?

Any question that reaches this answer is one step short. "Or maybe
just ditch the notion?"

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Viacheslav Usov
In reply to this post by Russell Haley
On Sun, Apr 29, 2018 at 6:54 AM, Russell Haley <[hidden email]> wrote:

> At first I thought to write a wrapper. And then I thought, "Well, fuck, it's just a tiny C program. How complicated can it be"? (tee hee) The most portable and performant way to handle configuration is in C. In the end, modifying Lua itself seems like the best approach?

If all you want is really a Lua interpreter that loads either lua51.dll or lua5x.dll based on a command line switch, then you could exploit the fact that lua.c defines everything except main() as static. You can therefore take both versions of lua.c, rename their main() functions to main51() and main5x() respectively. So now you can have two versions of lua.c compiled into one executable. Note you will also need two full sets of Lua's headers to make that work.

That only solves part of the problem, because you still need to load the appropriate DLL dynamically (easy) and make sure that all the API calls by main51() and main5x(), and the functions called by them,are routed to the correct DLL. A first step toward that goal would be not linking your executable against any luazx.dll, then all the API calls will result in linker errors, giving you a list of API functions used by both versions.

The next step is ugly, and perhaps there is a more elegant way (of which I will be happy to hear). For each API call in a given lua.c, modify its declaration in the corresponding lua.h. For example, the .1 version calls lua_newstate(), which is defined as

LUA_API lua_State *(lua_newstate) (lua_Alloc f, void *ud);

Modify that to read

static lua_State *(*lua_newstate) (lua_Alloc f, void *ud);

That creates a static pointer to function. Once you have done all that properly, your executable should link successfully, but it will crash when run, because all those pointers need to be set up with the addresses of the corresponding API functions resolved from the dynamically loaded DLL. I think you should be able to fill in the gaps at this point.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Peter Cawley
On Sun, Apr 29, 2018 at 10:07 AM, Viacheslav Usov <[hidden email]> wrote:
> That only solves part of the problem, because you still need to load the
> appropriate DLL dynamically (easy) and make sure that all the API calls by
> main51() and main5x(), and the functions called by them,are routed to the
> correct DLL. A first step toward that goal would be not linking your
> executable against any luazx.dll, then all the API calls will result in
> linker errors, giving you a list of API functions used by both versions.

Alternatively, you could load both DLLs and just then ignore one of
them. Compile the file defining main51, link it against lua51.dll to
produce a main51 static library. Compile the file defining main5x,
link it against lua5x.dll to produce a main5x static library. Compile
a stub main which calls either main51 or main5x, link it against both
static libraries to produce an executable. (I _think_ this'll work
with static libraries, though if not it'll certainly work if you
replace the word "static" with "dynamic" throughout).

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Viacheslav Usov
On Sun, Apr 29, 2018 at 1:02 PM, Peter Cawley <[hidden email]> wrote:

> Alternatively, you could load both DLLs and just then ignore one of them. Compile the file defining main51, link it against lua51.dll to produce a main51 static library. Compile the file defining main5x, link it against lua5x.dll to produce a main5x static library. Compile a stub main which calls either main51 or main5x, link it against both static libraries to produce an executable.

Problem is, it is only at the final linking step that  externals are resolved. With the approach above, there will be conflicting externals (same names) from lua51.dll and lua5x.dll. The intermediate static library step does not change that (at least not without some other hackery).

It would really be greatl if Lua had built-in means to support multiple versions in a single executable.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Peter Cawley
On Sun, Apr 29, 2018 at 2:32 PM, Viacheslav Usov <[hidden email]> wrote:
> Problem is, it is only at the final linking step that  externals are
> resolved. With the approach above, there will be conflicting externals (same
> names) from lua51.dll and lua5x.dll. The intermediate static library step
> does not change that (at least not without some other hackery).

Hmm. So put the main functions in their own (tiny) dynamic libraries,
rather than in static libraries. Or go down the hackery route (such as
delay-importing both Lua DLLs, and having a custom delay-import
binding routine).

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Viacheslav Usov
On Sun, Apr 29, 2018 at 5:05 PM, Peter Cawley <[hidden email]> wrote:

> Hmm. So put the main functions in their own (tiny) dynamic libraries,

Yup, this is probably the cleanest method overall.

> delay-importing both Lua DLLs, and having a custom delay-import binding routine

First you would have to make the linker ignore duplicate externals, which is possible with the /force option. Then, these forced imports would need to be compatible with the /delayload option. The documentation on all of that is sparse, so this sounds like a lot of fun to play with, but probably not something to depend on.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Axel Kittenberger
I don't quite see how these dynamic library shenanigans are better than a wrapper script. If a fork() that a shell wrapper would do, sounds like an too expensive syscall, simply write a c wrapper, that directly calls execvp() which keeps the process and just replaces the running binary with another one. I don't know and didn't bother to research, but I can't imagine that execvp() is much more expensive than dlopen() anyway.

However, eventually I don't see why lua -51 is any better in a script compared to say lua-51  or simply ${LUA} which is a variable to what lua you detected before, or been configured for.

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Viacheslav Usov
On Sun, Apr 29, 2018 at 7:21 PM, Axel Kittenberger <[hidden email]> wrote:

> fork() ... execvp() ... dlopen()

The previous discussion was about Windows. Creating a new process there is somewhat more expensive than a fork() in a typical Unix, and the second option is not available. That said, I do not think this is really about the last drop of performance, I think the original poster had some other reasoning.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Russell Haley
In reply to this post by Dirk Laurie-2


On Sun, Apr 29, 2018 at 1:46 AM, Dirk Laurie <[hidden email]> wrote:
2018-04-29 6:54 GMT+02:00 Russell Haley <[hidden email]>:

> the idea of wrapping lua in a batch script makes me sad

About emotions there is no disputing ...

A swig of bourbon, straight, is said to a wonderful cure for sadness.
I find I am quite allergic to bourbon. I break out in black eyes and hand-cuffs.  ;)

> In the end, modifying Lua itself seems like the best approach?

Any question that reaches this answer is one step short. "Or maybe
just ditch the notion?"
Quite right. I ended there a few times. I've meditated on conformance and expectations of users as well. I'm not considering modifying the way lua works other than where the configuration and executables come from. Lua.org can't really go there due to the nature of lua (The ethos? ISO C conformance...). 

I am also fond of chasing my tail if you haven't yet noticed. PIL often refers to the fact that the Lua interpreter is really a case study for learning the Lua API. 

Russ

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Russell Haley
In reply to this post by Axel Kittenberger


On Sun, Apr 29, 2018 at 10:21 AM, Axel Kittenberger <[hidden email]> wrote:
I don't quite see how these dynamic library shenanigans are better than a wrapper script. If a fork() that a shell wrapper would do, sounds like an too expensive syscall, simply write a c wrapper, that directly calls execvp() which keeps the process and just replaces the running binary with another one. I don't know and didn't bother to research, but I can't imagine that execvp() is much more expensive than dlopen() anyway.
Scripts add complexity. While I'm currently playing on Windows, I want this to be supportable on RTEMS. Ultimately the same C Lua uses is the most portable, easiest way to achieve my goal. By easiest, I mean 'put my big boy pants on' and improve my C.


However, eventually I don't see why lua -51 is any better in a script compared to say lua-51  or simply ${LUA} which is a variable to what lua you detected before, or been configured for.

I may be totally wrong. That's half the fun of the adventure. :)

Russ
Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Russell Haley
In reply to this post by Viacheslav Usov


On Sun, Apr 29, 2018 at 2:07 AM, Viacheslav Usov <[hidden email]> wrote:
On Sun, Apr 29, 2018 at 6:54 AM, Russell Haley <[hidden email]> wrote:

> At first I thought to write a wrapper. And then I thought, "Well, fuck, it's just a tiny C program. How complicated can it be"? (tee hee) The most portable and performant way to handle configuration is in C. In the end, modifying Lua itself seems like the best approach?

If all you want is really a Lua interpreter that loads either lua51.dll or lua5x.dll based on a command line switch, then you could exploit the fact that lua.c defines everything except main() as static. You can therefore take both versions of lua.c, rename their main() functions to main51() and main5x() respectively. So now you can have two versions of lua.c compiled into one executable. Note you will also need two full sets of Lua's headers to make that work.

That only solves part of the problem, because you still need to load the appropriate DLL dynamically (easy) and make sure that all the API calls by main51() and main5x(), and the functions called by them,are routed to the correct DLL. A first step toward that goal would be not linking your executable against any luazx.dll, then all the API calls will result in linker errors, giving you a list of API functions used by both versions.

The next step is ugly, and perhaps there is a more elegant way (of which I will be happy to hear). For each API call in a given lua.c, modify its declaration in the corresponding lua.h. For example, the .1 version calls lua_newstate(), which is defined as

LUA_API lua_State *(lua_newstate) (lua_Alloc f, void *ud);

Modify that to read

static lua_State *(*lua_newstate) (lua_Alloc f, void *ud);

That creates a static pointer to function. Once you have done all that properly, your executable should link successfully, but it will crash when run, because all those pointers need to be set up with the addresses of the corresponding API functions resolved from the dynamically loaded DLL. I think you should be able to fill in the gaps at this point.

Cheers,
V.

Thanks V. I'll study this idea. 

Russ
Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Dirk Laurie-2
In reply to this post by Russell Haley
2018-04-29 20:42 GMT+02:00 Russell Haley <[hidden email]>

>
> On Sun, Apr 29, 2018 at 1:46 AM, Dirk Laurie <[hidden email]> wrote:
>>
>> 2018-04-29 6:54 GMT+02:00 Russell Haley <[hidden email]>:
>>
>> > the idea of wrapping lua in a batch script makes me sad
>>
>> About emotions there is no disputing ...
>>
>> A swig of bourbon, straight, is said to a wonderful cure for sadness.
>
> I find I am quite allergic to bourbon. I break out in black eyes and
> hand-cuffs.  ;)
>>
>>
>> > In the end, modifying Lua itself seems like the best approach?
>>
>> Any question that reaches this answer is one step short. "Or maybe
>> just ditch the notion?"
>
> Quite right. I ended there a few times. I've meditated on conformance and
> expectations of users as well. I'm not considering modifying the way lua
> works other than where the configuration and executables come from. Lua.org
> can't really go there due to the nature of lua (The ethos? ISO C
> conformance...).
>
> I am also fond of chasing my tail if you haven't yet noticed. PIL often
> refers to the fact that the Lua interpreter is really a case study for
> learning the Lua API.

If by "modifying Lua itself" you merely mean "modifiying the Lua
interpreter" I am with you all the way. I do that myself.

Up the the manual called the interpreter supplied with Lua "a sample
host program".

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Russell Haley
In reply to this post by Peter Cawley


On Sun, Apr 29, 2018 at 4:02 AM, Peter Cawley <[hidden email]> wrote:
On Sun, Apr 29, 2018 at 10:07 AM, Viacheslav Usov <[hidden email]> wrote:
> That only solves part of the problem, because you still need to load the
> appropriate DLL dynamically (easy) and make sure that all the API calls by
> main51() and main5x(), and the functions called by them,are routed to the
> correct DLL. A first step toward that goal would be not linking your
> executable against any luazx.dll, then all the API calls will result in
> linker errors, giving you a list of API functions used by both versions.

Alternatively, you could load both DLLs and just then ignore one of
them. Compile the file defining main51, link it against lua51.dll to
produce a main51 static library. Compile the file defining main5x,
link it against lua5x.dll to produce a main5x static library. Compile
a stub main which calls either main51 or main5x, link it against both
static libraries to produce an executable. (I _think_ this'll work
with static libraries, though if not it'll certainly work if you
replace the word "static" with "dynamic" throughout).


I had a roughly similar thought too (yours is more complete than mine!). Thanks for the suggestion.

Russ

Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Russell Haley
In reply to this post by Dirk Laurie-2


On Sun, Apr 29, 2018 at 11:54 AM, Dirk Laurie <[hidden email]> wrote:
2018-04-29 20:42 GMT+02:00 Russell Haley <[hidden email]>
>
> On Sun, Apr 29, 2018 at 1:46 AM, Dirk Laurie <[hidden email]> wrote:
>>
>> 2018-04-29 6:54 GMT+02:00 Russell Haley <[hidden email]>:
>>
>> > the idea of wrapping lua in a batch script makes me sad
>>
>> About emotions there is no disputing ...
>>
>> A swig of bourbon, straight, is said to a wonderful cure for sadness.
>
> I find I am quite allergic to bourbon. I break out in black eyes and
> hand-cuffs.  ;)
>>
>>
>> > In the end, modifying Lua itself seems like the best approach?
>>
>> Any question that reaches this answer is one step short. "Or maybe
>> just ditch the notion?"
>
> Quite right. I ended there a few times. I've meditated on conformance and
> expectations of users as well. I'm not considering modifying the way lua
> works other than where the configuration and executables come from. Lua.org
> can't really go there due to the nature of lua (The ethos? ISO C
> conformance...).
>
> I am also fond of chasing my tail if you haven't yet noticed. PIL often
> refers to the fact that the Lua interpreter is really a case study for
> learning the Lua API.

If by "modifying Lua itself" you merely mean "modifiying the Lua
interpreter" I am with you all the way. I do that myself.
Yes, the interpreter only. With the inclusion of NIL_IN_TABLE, I find nothing in the language to complain about, though I've not written large applications in it.


Up the the manual called the interpreter supplied with Lua "a sample
host program".


Reply | Threaded
Open this post in threaded view
|

Re: Modify Lua Executable for a -51 option

Sean Conner
In reply to this post by Russell Haley
It was thus said that the Great Russell Haley once stated:

> On Sun, Apr 29, 2018 at 4:02 AM, Peter Cawley <[hidden email]> wrote:
>
> > On Sun, Apr 29, 2018 at 10:07 AM, Viacheslav Usov <[hidden email]>
> > wrote:
> > > That only solves part of the problem, because you still need to load the
> > > appropriate DLL dynamically (easy) and make sure that all the API calls
> > by
> > > main51() and main5x(), and the functions called by them,are routed to the
> > > correct DLL. A first step toward that goal would be not linking your
> > > executable against any luazx.dll, then all the API calls will result in
> > > linker errors, giving you a list of API functions used by both versions.
> >
> > Alternatively, you could load both DLLs and just then ignore one of
> > them. Compile the file defining main51, link it against lua51.dll to
> > produce a main51 static library. Compile the file defining main5x,
> > link it against lua5x.dll to produce a main5x static library. Compile
> > a stub main which calls either main51 or main5x, link it against both
> > static libraries to produce an executable. (I _think_ this'll work
> > with static libraries, though if not it'll certainly work if you
> > replace the word "static" with "dynamic" throughout).
> >
> >
> I had a roughly similar thought too (yours is more complete than mine!).
> Thanks for the suggestion.

  You are going to have a fun time [1] with Lua modules written in C.  You
may even have a fun time [1] with Lua modules written in Lua.  There are
probably easier ways to accomplish this without mucking with linkers.

  -spc

[1 And by "fun time" I mean an "interesting time" [2]

[2] In the Chinese curse [3] sense of "May you live in interesting times."

[3] Apocraphal, citation needed.


12