Lua without IO

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua without IO

Stojan Dimitrovski
Hey Lua people,

I've started work on a (not yet public) project that requires Lua to
run without any IO / syscall dependencies.

I've grabbed the 5.3.5 source and added preprocessor macros all over
the place to disable all of that code. It works! You can take a look
at the changes here: https://github.com/hf/lua-noio/

I was wondering, if I clean it up a bit, how do I submit it as an
actual patch to Lua itself? I also see some other places the IO
situation with Lua can be improved, so I'd also like to improve on
that as well.

Thanks,
Stojan
--
Stojan Dimitrovski
http://stojan.me

Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Jonathan Goble
On Mon, Feb 25, 2019 at 6:35 PM Stojan Dimitrovski <[hidden email]> wrote:
I was wondering, if I clean it up a bit, how do I submit it as an
actual patch to Lua itself? I also see some other places the IO
situation with Lua can be improved, so I'd also like to improve on
that as well.

Thanks,
Stojan

Lua does not accept patches. Although Lua is open source, it is not open developed. The Lua team (Roberto, Luiz, and one other person who never posts here) make all decisions and write all code for Lua themselves. They do take community feedback into consideration via this mailing list, but if they like an idea, they will write their own code to implement it rather than accepting someone else's patch.

Also, the 5.4 development cycle is underway, and massive changes have been made since 5.3.5. You can take a look at that work at https://github.com/lua/lua, which is an unofficial mirror of the Lua repository as seen by the Lua team. It is always slightly out-of-date as commits are not mirrored in real-time. Pull requests are not accepted for the reasons discussed in the previous paragraph.
Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Stojan Dimitrovski
Ah, OK I see. Well in that case let me present why I think Lua should
include minimal support for running in a no-syscall system.

Who'd need this?

This would allow embedding Lua inside very restricted environments.
I'm personally using it in an Intel SGX project.

How to implement it?

It's actually pretty easy to do in the Lua 5.3.5 codebase. What I've
done is section out all IO use with the LUA_NOIO macro. This does
restrict parts of the standard library a bit too restrictively and
would require reimplementation by the embedder for the use case. Aside
from the IO lib, almost all of the baselib needs to be reimplemented
-- modules and all that. So that should be probably thought about more
carefully. There are a few places that use locale.h, time.h and
similar, but they are already controllable through defining macros.
The lua.c and luac.c files are obviously unusable, so those just
shouldn't make it in the embedder's Makefile.

Improving on the existing implicit IO

So Lua has a few of these helper functions that actually do IO
implicitly. lua_writerror, and friends, are macros that write directly
with fprintf or fwrite to stdout or stderr. These can all be redefined
to customize, though a problem I see with them is that those macros
are less flexible from an embedder's point of view. As a first step
toward improving them, I'd just add the lua_State that is calling them
as an argument (even if it's unused in the standard Lua). What this
will allow is implementing virtual IO channels in restricted
environments. Basically, Lua should provide at least the lua_State
whenever it's doing any kind of operation.

One can easily make a case for a virtual IO system, but I think that's
not necessary presently, as this kind of work is fairly rare.




On Tue, Feb 26, 2019 at 1:10 AM Jonathan Goble <[hidden email]> wrote:

>
> On Mon, Feb 25, 2019 at 6:35 PM Stojan Dimitrovski <[hidden email]> wrote:
>>
>> I was wondering, if I clean it up a bit, how do I submit it as an
>> actual patch to Lua itself? I also see some other places the IO
>> situation with Lua can be improved, so I'd also like to improve on
>> that as well.
>>
>> Thanks,
>> Stojan
>
>
> Lua does not accept patches. Although Lua is open source, it is not open developed. The Lua team (Roberto, Luiz, and one other person who never posts here) make all decisions and write all code for Lua themselves. They do take community feedback into consideration via this mailing list, but if they like an idea, they will write their own code to implement it rather than accepting someone else's patch.
>
> Also, the 5.4 development cycle is underway, and massive changes have been made since 5.3.5. You can take a look at that work at https://github.com/lua/lua, which is an unofficial mirror of the Lua repository as seen by the Lua team. It is always slightly out-of-date as commits are not mirrored in real-time. Pull requests are not accepted for the reasons discussed in the previous paragraph.



--
Stojan Dimitrovski
http://stojan.me

Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Allan Bowe
Lua 5.2 (without the IO library) is implemented in the SAS language (SAS institute), used inside restricted enterprise data environments by tens of thousands of programmers globally, myself included.

Perhaps if there was further support for no-syscall, we’d get a much desired Lua upgrade!

/Allan

On Wed, 27 Feb 2019 at 00:45, Stojan Dimitrovski <[hidden email]> wrote:
Ah, OK I see. Well in that case let me present why I think Lua should
include minimal support for running in a no-syscall system.

Who'd need this?

This would allow embedding Lua inside very restricted environments.
I'm personally using it in an Intel SGX project.

How to implement it?

It's actually pretty easy to do in the Lua 5.3.5 codebase. What I've
done is section out all IO use with the LUA_NOIO macro. This does
restrict parts of the standard library a bit too restrictively and
would require reimplementation by the embedder for the use case. Aside
from the IO lib, almost all of the baselib needs to be reimplemented
-- modules and all that. So that should be probably thought about more
carefully. There are a few places that use locale.h, time.h and
similar, but they are already controllable through defining macros.
The lua.c and luac.c files are obviously unusable, so those just
shouldn't make it in the embedder's Makefile.

Improving on the existing implicit IO

So Lua has a few of these helper functions that actually do IO
implicitly. lua_writerror, and friends, are macros that write directly
with fprintf or fwrite to stdout or stderr. These can all be redefined
to customize, though a problem I see with them is that those macros
are less flexible from an embedder's point of view. As a first step
toward improving them, I'd just add the lua_State that is calling them
as an argument (even if it's unused in the standard Lua). What this
will allow is implementing virtual IO channels in restricted
environments. Basically, Lua should provide at least the lua_State
whenever it's doing any kind of operation.

One can easily make a case for a virtual IO system, but I think that's
not necessary presently, as this kind of work is fairly rare.




On Tue, Feb 26, 2019 at 1:10 AM Jonathan Goble <[hidden email]> wrote:
>
> On Mon, Feb 25, 2019 at 6:35 PM Stojan Dimitrovski <[hidden email]> wrote:
>>
>> I was wondering, if I clean it up a bit, how do I submit it as an
>> actual patch to Lua itself? I also see some other places the IO
>> situation with Lua can be improved, so I'd also like to improve on
>> that as well.
>>
>> Thanks,
>> Stojan
>
>
> Lua does not accept patches. Although Lua is open source, it is not open developed. The Lua team (Roberto, Luiz, and one other person who never posts here) make all decisions and write all code for Lua themselves. They do take community feedback into consideration via this mailing list, but if they like an idea, they will write their own code to implement it rather than accepting someone else's patch.
>
> Also, the 5.4 development cycle is underway, and massive changes have been made since 5.3.5. You can take a look at that work at https://github.com/lua/lua, which is an unofficial mirror of the Lua repository as seen by the Lua team. It is always slightly out-of-date as commits are not mirrored in real-time. Pull requests are not accepted for the reasons discussed in the previous paragraph.



--
Stojan Dimitrovski
http://stojan.me

Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Sean Conner
It was thus said that the Great Allan Bowe once stated:
> Lua 5.2 (without the IO library) is implemented in the SAS language (SAS
> institute), used inside restricted enterprise data environments by tens of
> thousands of programmers globally, myself included.

  I'm not seeing how this can be useful.  If there's no I/O, then how does
one get the results of running a program?  *Something* has to be doing the
I/O ...

  -spc (Or it turns into a very expensive way to heat coffee ... )


Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Allan Bowe
Well, not so great, as you eloquently demonstrate!

I meant “OS” (eg os.date(), os.clock()), apologies, it’s early here.  Crawling back into my hole.

/Allan

On Wed, 27 Feb 2019 at 08:47, Sean Conner <[hidden email]> wrote:
It was thus said that the Great Allan Bowe once stated:
> Lua 5.2 (without the IO library) is implemented in the SAS language (SAS
> institute), used inside restricted enterprise data environments by tens of
> thousands of programmers globally, myself included.

  I'm not seeing how this can be useful.  If there's no I/O, then how does
one get the results of running a program?  *Something* has to be doing the
I/O ...

  -spc (Or it turns into a very expensive way to heat coffee ... )


Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Viacheslav Usov
In reply to this post by Sean Conner
On Wed, Feb 27, 2019 at 8:47 AM Sean Conner <[hidden email]> wrote:

> I'm not seeing how this can be useful.  If there's no I/O, then how does one get the results of running a program?

Lua is an "embeddable scripting language". The utility of a Lua VM to the host may have nothing to do with its ability to do IO through its stock IO facilities.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Luiz Henrique de Figueiredo
In reply to this post by Stojan Dimitrovski
> I've started work on a (not yet public) project that requires Lua to
> run without any IO / syscall dependencies.

The Lua 5.3 core depends only on these standard C functions:

abort abs floor fmod frexp localeconv longjmp memcmp memcpy pow setjmp
snprintf strchr strcmp strcoll strcpy strlen strpbrk strspn strtod
time

Most of these can be written directly in C if needed. Some are hard:
frexp. Some are impossible: longjmp, setjmp. Some are controlled via
luaconf.h and l_mathop: floor, food frexp, localeconv, pow, snprintf.

In other words, changing the dependency of the Lua core on libc can be
mostly done by editing  luaconf.h, as expected.

Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Pedro Tammela-2
In reply to this post by Stojan Dimitrovski
> > I've started work on a (not yet public) project that requires Lua to
> > run without any IO / syscall dependencies.
>
> The Lua 5.3 core depends only on these standard C functions:
>
> abort abs floor fmod frexp localeconv longjmp memcmp memcpy pow setjmp
> snprintf strchr strcmp strcoll strcpy strlen strpbrk strspn strtod
> time
>
> Most of these can be written directly in C if needed. Some are hard:
> frexp. Some are impossible: longjmp, setjmp. Some are controlled via
> luaconf.h and l_mathop: floor, food frexp, localeconv, pow, snprintf.
>
> In other words, changing the dependency of the Lua core on libc can be
> mostly done by editing  luaconf.h, as expected.

+1.

We have been using this approach for Lua in Kernel for years. The best
part about this is that new patches apply cleanly or without minimal
changes (usually whitespace related). This way we keep up with
upstream as soon as it's released.

Reply | Threaded
Open this post in threaded view
|

Re: Lua without IO

Stojan Dimitrovski
In reply to this post by Luiz Henrique de Figueiredo
Just to clarify a few points from the discussion:

Lua without IO actually means without system-provided IO. In my
current implementation, much of that IO is abstracted away into
non-syscall ways of doing IO. (Basically reading and writing to
special places in memory.) The problem is that Lua depends on the IO
provided by the kernel, which is not a problem most of the time.
However, in some restricted environments like Intel SGX kernel-based
IO is not at all possible to do. I imagine it's quite similar in other
applications as well.

So the request is not to abstract away the kernel IO use of Lua, as
that would not really be beneficial in general, but just define a few
more macros (and make existing more useful) so that when someone needs
to use Lua in a syscall-less environment, they don't have to do it
manually (and possibly with errors).

I personally would not expect such a Lua compile-time configuration to
implement the OS, IO and some base library functions. Those would be
up to the embedder to do correctly -- something I'm doing in my own
project.

On Wed, Feb 27, 2019 at 9:40 PM Philippe Verdy <[hidden email]> wrote:

>
> frexp() is not so hard, even if it is much faster if you have access to the native assembly instructions. In pure C, this could be still implemented with some platform-testing function (using tests on known constants), but you have to use unsafe pointer conversion and be aware of the byte order (endianness).
> In practice you can still go faster if you know the CPU and FPU architecture and some of them are now used everywhere : x86, x64/amd64, and arm64 (all the others are older processors or processors for niche markets on embedded systems, many of them using now some arm32 or arm64; there's almost no more use for other x80, x88, 65xx, 68k, Sparc and Mips processors, but there remains now a growing niche with GPUs, mostly from nVdidia and ATI, and for some high-end very costly processors used in supercomputers, and for "virtual processors" running on top of a VM with its own "portable" ABI, including Java, Perl, Lua, or Javascript).
> abort() may seem complex but can be implemented on top of longjmp (and more complete implemention if adding support for signal and raise).
> If you compile Lua against a C++ compiler, you can still use the C++ exception handlers to implement setjmp/longjmp in an additional C++ compiled module (but of course you need a C++ compiler like GCC or clang) to declare the needed function with 'extern "C"...' linkage . I bet there are today a C++ compiler available on almost all platforms and they already come with the machinery library to link the few C++ modules with the other C modules or libraries needed by Lua (this is difficult only when you don't have access to the native instruction set and are bound only to a virtual processor, in which case you need to use the VM API, but most VMs have basic support for exception handling or error handling).
> The really complex part is if you need to make Lua exist within limits of a multithreading environment, and make manage these limits (notably in memory allocation, time slot, time precision, and permissions).
> If you don't have the standard C I/O the program in Lua will still perform I/O with another library (it could be a network API or an IPC mechanism with named channels, using some specific resource allocators, or a database or "registry" service).
> One thing that is still missing in Lua is a safe way to integrate it with true (Posix) threads instead of being limited to the "cooperative" model (by which all Lua threads used in a program exist within the same system thread, using the very complex setjmp/longjmp mechanism): we still cannot safely and simply declare to Lua to exist the cooperative model and allow threads to run concurrently, because Lua has no way to coordinate them (it would require the addition of synchronization and serialization, and a definition of event queues, and resource managers: in standard Lua there's a single queue and a single resource manager for everything, the rest is part of the internal integration library but inacessible to Lua programs themselves; this caues huge difficulties to implement efficient networking, notably for implementing services without excessive blocking and without huge variation in response time in a very busy service replying to thousands of queries per second or more, or for interfacing with external database services).
> And unlike Java or Javascript, Lua still has no model for implementing security domains (with efficient resource sharing and data exchanges), and still no clear definition of a basic set of datatypes built for portability (including with known and reproduceable limits): for that you need to add other Lua packages, the basic library is too much limited and too much oriented for basic POSIX C (not much more than the old K&R model but still not enough to even support the POSIX standard). This causes many Lua programs to be non-portable (not even between x86 and x64 architectures, or between ARM32 and ARM64, on the same type of OS like Linux, Windows, or major branches MacOS/iOS).
> So Lua is basically a language defined by its syntax, it still has no formal API (let's put aside the ABI or its implementing VM): it works, yes, but the results are not reproductible, so Lua-written software is difficult to test: you need to develop Lua programs as well as test programs at the same time to target each platform and check the dependencies. There's no test framework like in Java, Perl or Javascript, not even for basic conformance level: you cannot develop on a system and deploy to others easily: you need to test/debug against each platform and there's nothing in Lua that allows asserting that a Lua module is compatible with some known platform type, no versioning system; each Lua implementation is very tightly linked to its initial platform (including the current "standard" Lua), and this also applies to its documentation: you need to tweak also this documentation to specify the limits and target platforms on which it was tested.
>
> Lua is still now is in the same "infant" stage as what was "K&R" C in the 1980's (unfortunately it remained too long in that stage and caused a proliferation of ABI's and later huge complexity to write portable C programs; the same risk exists now for Lua). May be there will be a change later when there will be a version of the language sponsored by a good standard body (ISO, IEEE, ECMA, ANSI...) and its development will be open (and Lua will not just be "opensource" as of today). Or when its development will be open to more members (e.g. the Mozilla, Linux, GNU foundations) and with formal process (but it will introduce bureaucracy and costs and possibly tensions between members like those for Java or C++ today). The costs would be supported by large service providers and sellers offering services in Lua but they may influence the future design of the language.
>
> Le mer. 27 févr. 2019 à 12:47, Luiz Henrique de Figueiredo <[hidden email]> a écrit :
>>
>> > I've started work on a (not yet public) project that requires Lua to
>> > run without any IO / syscall dependencies.
>>
>> The Lua 5.3 core depends only on these standard C functions:
>>
>> abort abs floor fmod frexp localeconv longjmp memcmp memcpy pow setjmp
>> snprintf strchr strcmp strcoll strcpy strlen strpbrk strspn strtod
>> time
>>
>> Most of these can be written directly in C if needed. Some are hard:
>> frexp. Some are impossible: longjmp, setjmp. Some are controlled via
>> luaconf.h and l_mathop: floor, food frexp, localeconv, pow, snprintf.
>>
>> In other words, changing the dependency of the Lua core on libc can be
>> mostly done by editing  luaconf.h, as expected.
>>


--
Stojan Dimitrovski
http://stojan.me