Blocking calls and module composability

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Blocking calls and module composability

Archie Cobbs
Hello,

I'm new to Lua and (like many before) am grappling with understanding
what can and can't be done in the language. My background: been
programming in C since the late 80's and involved in open source since
the '90's. I may not be good, but I'm old :)

I have a couple of initial observations, plus an idea I'd like to
throw out there... apologies in advance for the long post. Hopefully
this has a logical flow to it.

OBSERVATIONS

I could be misunderstanding things below - if so please correct me.

First, Lua is designed for embedability and for that reason it adheres
to strict ANSI C. This is a laudable goal but it's also necessarily
pretty restrictive, especially for "plain" Lua (i.e., Lua without any
other assumptions beyond ANSI C).

The most profound restriction is the limitation to a single C call
stack (you can't assume pthreads), and so Lua must longjmp() every
time there is a coroutine context switch. In other words, the stack
frames of any C functions in the call stack of a coroutine have to
tossed when that coroutine is switched out.

For "plain" Lua, restriction to ANSI C creates these problems:

    Problem #1. Lua is single-threaded and can't take advantage of
multi-core machines
    Problem #2. If Lua invokes any blocking C function, then the
entire VM blocks (all coroutines). In other words, coroutine blocking
is serialized.
    Problem #3. C functions that yield must be written so they return
immediately and provide a resume() callback (relatively minor
inconvenience)

and these advantages:

    Advantage #1. Lua is extremely portable
    Advantage #2. Lua coroutines can pretend all code that runs
between two yield()'s executes atomically

Problem #1 speaks for itself and is easy to understand. You can
address this by creating multiple Lua contexts, but then you have to
handle the resulting isolation: there are no shared globals or
upvalues, and so all communication between contexts requires
serialization of some kind.

Within a single Lua context, it's not possible to address Problem #1
in plain Lua. But even beyond plain Lua, it's hard to imagine how you
could address this, while also preserving Advantage #2, without adding
a bunch of locking overhead or creating a weird hybrid.

In any case, let's ignore Problem #1 for now.

As for Problem #3, it's more of an inconvenience than a major problem.

Let's talk about Problem #2!

In my view, Problem #2 is the most serious problem. It eliminates a
huge class of applications, namely, any application supporting
simultaneous blocking operations. For example, a web server!  I'm
assuming there that a web server that would serialize every blocking
I/O call - which is all you can do in plain Lua - does not count as a
real web server :)

For many applications (e.g., web servers) this is too restrictive, and
so people become willing to compromise on Advantage #1 (i.e.,
sacrifice portability) in order to get a solution to Problem #2.

For the sake of argument, let's assume we are willing to make that trade-off...

I'm still trying to sort through all the ways people have tried to
address Problem #2.

But let's take the "luv" module (event library based on non-blocking
file descriptors) as an exmaple of a reasonable and popular solution.

OK so far so good...  BUT - there's still a larger problem with "luv"
or any other solution to Problem #2 I've run across so far: none of
them are COMPOSABLE with other, unrelated Lua modules.

Here's what I mean: take "luv" for example. It only works if all file
descriptors are set to non-blocking mode. Now suppose you pull some
random mysql module off the shelf that communicates over a network
socket to your SQL server. And suppose that mysql library uses
LuaSocket to create a network connection to the database.

Unless that mysql module is a priori designed to work with "luv", as
soon as you start using it, it will make some blocking call, and your
entire application will block - oops, here comes Problem #2 again.

Even if you could somehow access the socket that the mysql module uses
and set it to non-blocking mode, that still wouldn't work because the
mysql module wouldn't know what to do with the  new EWOULDBLOCK error
code it would get back.

In other words, there exist "solutions" to Problem #2, but if you use
them, then EVERY other module you use that makes blocking calls needs
to also be designed to work with that solution, or else you haven't
really solved the problem.

Another example: suppose you want your application to fetch an HTTPS
URL like "https://foobar.com/" without blocking the entire VM, using
some simple "request" object API. Then you have to find a module that
contains ALL THREE of (a) non-blocking I/O, (b) HTTP client support,
and (c) SSL support... or else you have to find three separate modules
for (a), (b), and (c) that were all written to work together
(unlikely).

In other words, because there is no standard solution to Problem #2,
you can't compose arbitrary modules and solve Problem #2 at the same
time.

In my opinion this problem is actually the biggest problem with the
Lua "ecosystem". As evidence look at the discussion going on in the
"batteries" thread right now.

OK end of observations..

PROPOSAL

I tried to think about whether there is a way to address Problem #2
directly and transparently, i.e., in a way that doesn't require other
modules to know anything about it. So that, using the above example,
if you started using some random mysql module that makes blocking
network calls, it wouldn't lock up the entire VM while it waits.

How could you do this? Here was my first idea:

    1. Give each coroutine it's own pthread and C stack.
    2. Disable preemption (i.e., context switch only on blocking call
or yield())
    3. Disable parallel execution to preserve Advantage #2 (e.g., lock
all pthreads to one CPU)

The key here is that the combination of #2 and #3 means only one
coroutine runs at a time, and it runs without being preempted until it
either (a) yields, or (b) blocks. In effect, a blocking call is
treated like a yield(), with the corresponding resume() occurring when
the file descriptor becomes readable (or whatever).

Can this be done? Here's how you can actually do this on Linux:

    1. You get separate C stacks by using a separate pthread for each
coroutine, or using setcontext() etc.
    2. You can disable preemption via pthread_attr_setschedpolicy(3)
with SCHED_FIFO
    3. You can lock all pthreads to one CPU using sched_setaffinity(3))

Unfortunately on Linux step #2 requires root permission, and moreover
makes your process "real-time" and therefore a CPU hog.

More generally, the pthreads specification provides no portable way to
simply turn off preemption. So that is a major problem with this idea.

Too bad. But let's complete the thought experiment...

This type of solution preserves Advantage #2 (and provides a simple
solution to Problem #3) with no additional locking. There is only one
small downside compared to "plain" Lua: blocking calls now behave as
if they could yield() internally. For example, this code could behave
differently:

    counter = 1;
    function mycoroutine(client)
        print(counter)
        print(counter)     // counter must be the same here
        client:receive()   // this is a blocking call
        print(counter)     // counter could have changed here!
    end

In plain Lua the counter could not have changed in the last print() statement.

But having blocking calls appear to yield() is a very natural change -
and in any case the same thing already is happening with "luv" and all
other solutions to Problem #2, which is proof that it won't be a big
problem.

However, compared to those existing solutions, this solution has the
key advantage that it doesn't break composability.

Now, back to the issue of how to implement it... is there any hope?

In other words, let's summarize the situation:

  1. Problem #2 is a major problem with Lua that everyone wants to solve
  2. The existing solutions work but they are not composable with
arbitrary other Lua modules
  3. We could make them composable if we could do the following:
      (a) Give coroutines (or at least those that need to block) their
own C stacks
      (b) Disable preemption
      (c) Disable parallel execution

This would have to be done in such a way that existing C modules could
be adapted with minimal change, AND so that the same C module code
could still work on "plain" Lua as before.

Since we can't do it with pthreads, what about GNU pth? It is
specifically designed for non-preemptive multitasking and gives you
3(a), 3(b), and 3(c) by design. Moreover, pth is very portable and
stable (I used it back in the 90's).

(If people barf on the GPL, it would be easy to write the same thing
from scratch under a looser license.)

What would be required of existing C modules to support this? => Any
blocking calls would have to be changed so they invoke pth's version.
For example, read becomes pth_read(), sleep() becomes pth_sleep(),
etc.

Does this mean C modules would end up being littered with #ifdef's and
incompatible with "plain" Lua?

No. Instead, this addition to the Lua API would define a Lua wrapper
funtion for each blocking function.

So the C module would change read() into lua_read(), sleep() into
lua_sleep(), etc., after including some appropriate Lua header.

But these wrapper functions could work either way: the same C module
would compile and work on either "plain" Lua or on this new, optional
"nonblock" version of Lua: on platforms supporting "nonblock" the
wrappers would redirect to non-blocking versions; on "plain" Lua
platforms, the wrappers would just fall back to the original versions
- so you get exactly what you did before.

To summarize this idea:

  1. Add new, optional "nonblock" support to Lua on platforms that can
handle it, based on cooperative threading (GNU pth or equivalent)
  2. There is a new header file #include "lua_nonblock.h" defining
wrappers for blocking C calls: lua_read(), lua_write(), lua_sleep(),
etc.
  3. C modules written to support "nonblock" continue to work when
compiled with "plain" Lua, but now can also context switch if compiled
on "nonblock" versions of Lua
  4. coroutine.create() gets a new optional parameter, where true
means "use a separate C stack (if supported)"

To me this would be a major improvement in the status quo and improve
the Lua ecosystem dramatically by allowing all of the modules out
there to be composable.

The only cost to module developers would be match & replace read() ->
lua_read(), etc.

Thoughts?

-Archie

--
Archie L. Cobbs

Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Sean Conner
It was thus said that the Great Archie Cobbs once stated:
> Hello,

  Hello.

> I have a couple of initial observations, plus an idea I'd like to
> throw out there... apologies in advance for the long post. Hopefully
> this has a logical flow to it.

  Don't worry---my reply is just as long, if not longer.  And that's with
snipping out parts I'm not directly replying to.

> The most profound restriction is the limitation to a single C call
> stack (you can't assume pthreads), and so Lua must longjmp() every
> time there is a coroutine context switch. In other words, the stack
> frames of any C functions in the call stack of a coroutine have to
> tossed when that coroutine is switched out.

  The Lua call stack and C call stack are two separate things.  The Lua VM
does not use longjmp() to implement coroutines [a][d].

[a] Here I think you are conflating Lua coroutines with C coroutines. C
        doesn't have the concept of a coroutine, so to implement them, one
        must necessarily dive into unportable code.  Yes, there are
        implementations of C coroutines that use setjmp()/longjmp() but I
        think they rely upon undefined behavior.  The semantics of
        setjmp()/longjmp() is to provide a primitive way to dealing with
        exceptions (returning to a higher point in the call stack) than with
        coroutines.  See [b] for details.

        I think the reason people think that Lua coroutines are handled via
        setjmp()/longjmp() is due to an inability to yield a Lua coroutine
        over a C call (Lua->C->Lua->yield).  Lua 5.2 fixed some issues with
        that (in that 5.2+ can handle some cases where 5.1 can't) [c].

[b] https://stackoverflow.com/questions/14685406/practical-usage-of-setjmp-and-longjmp-in-c

[c] I have some code that can yield properly in Lua 5.2 that will cause
        an error in 5.1.

[d] I'm switching up how I do foot notes because this is a long message.

> For "plain" Lua, restriction to ANSI C creates these problems:
>
>     Problem #1. Lua is single-threaded and can't take advantage of
> multi-core machines

  Yes it can.  Not out of the box certainly, but if you define lua_lock()
and lua_unlock() (and recompile Lua) [a] you can then share a Lua state
across operating system threads.  The downside is that Lua goes from being
one of the fastest scripting langauges down to about the execution speed of
Python.

[a] I wonder ... on some platforms, you could probably get away with
        defining lua_lock() and lua_unlock() as actual functions that do
        nothing, then use LD_PRELOAD to load a different implementation of
        lua_lock()/lua_unlock at runtime.  Then again, you would need to
        compile Lua as a shared object and do some other linking magic.
        Just a thought.

>     Problem #2. If Lua invokes any blocking C function, then the
> entire VM blocks (all coroutines). In other words, coroutine blocking
> is serialized.

  [ snip ]

> Problem #1 speaks for itself and is easy to understand. You can
> address this by creating multiple Lua contexts, but then you have to
> handle the resulting isolation: there are no shared globals or
> upvalues, and so all communication between contexts requires
> serialization of some kind.

  Or implement lua_lock()/lua_unlock() as mentioned.  

  Also, the method you describe is a message based, shared nothing approach
and it works well.  Erlang does nothing but message based, shared nothing
and it's used in the telecom industry.  It is easy to reason about and quite
simple to use.  It does *not* have to be slow [a].

[a] Back in the mid-90s, I was friends with the owners of a small
        software company in Ft. Lauderdale who wrote commercial X Window
        servers for a variety of operating systems.  Their *fastest* version
        was for QNX, which was a message based, shared nothing operating
        system.  In fact, the x86 kernel was about 8K (yes, 8,192 bytes) in
        size, with most services traditionally served by a kernel by user
        processes.

> As for Problem #3, it's more of an inconvenience than a major problem.
>
> Let's talk about Problem #2!
>
> In my view, Problem #2 is the most serious problem. It eliminates a
> huge class of applications, namely, any application supporting
> simultaneous blocking operations. For example, a web server!  

  It does not.  At work, I have a program written in Lua that is handling
over 200,000,000 calls per day [a].  I even have a gopher server [b][c]
written in Lua, and with that, I don't see writing a webserver as much of an
issue.  Granted, in the case of the 200,000,000 phone calls, it's not just a
single process on a single box, but multiple (independent) processes on
multiple boxes dealing with the traffic, but it's probably running fewer
instances than you would expect to handle such traffic. [e]

[a] Literally.  As in phone calls.  The product our company makes is
        part of the call path for a major cell phone company and as such,
        there is Lua code executed when a person makes a cell phone call.

[b] https://en.wikipedia.org/wiki/Gopher_%28protocol%29

[c] gopher://gopher.conman.org/ [d]

[d] Source code:  https://github.com/spc476/port70

[e] If I think too much about it, I want to throw up but that's besides
        the issue.

> I'm
> assuming there that a web server that would serialize every blocking
> I/O call - which is all you can do in plain Lua - does not count as a
> real web server :)

  One *could* in theory use plain Lua to serve up a web page, you just write
a simple program in plain Lua that is invoked via inetd [a] but that's
pretty slow.  Besides, plain Lua can't even do networking out of the box.

[a] A program that can execute a program when a request comes in to a
        certain network port.  It was how some services were done in the 80s
        and early 90s.
 
> For many applications (e.g., web servers) this is too restrictive, and
> so people become willing to compromise on Advantage #1 (i.e.,
> sacrifice portability) in order to get a solution to Problem #2.

  From what I understand, Cqueues will run on POSIX (multiple Unix like
operating systems like Linux, Solaris, Mac OS-X, BSDs) and Windows, so from
a modern perspective, it can be "portable".

> I'm still trying to sort through all the ways people have tried to
> address Problem #2.
>
> But let's take the "luv" module (event library based on non-blocking
> file descriptors) as an exmaple of a reasonable and popular solution.
>
> OK so far so good...  BUT - there's still a larger problem with "luv"
> or any other solution to Problem #2 I've run across so far: none of
> them are COMPOSABLE with other, unrelated Lua modules.

  True, but you have that issue in any language.  Story time.

  Back around 2011, there were two projects being worked on that needed to
make DNS queries [a][b].  At the time, both projects where attemping to use
C-Ares library to do the DNS queries.  Problem one:  it didn't support the
DNS record type so code needed to be added to parse those.  Problem two:  it
wanted to handle *all* the networking releated to DNS queries, which was
problematic because both projects were already dealing with the network, so
both were stuck trying to integrate this monsterous DNS networking library
into their existing program [c].

  And here I was, looking into DNS on my own at the time.  I wasn't
interested in the networking side (boring!) but instead on how to parse the
DNS packets, and as a result, wrote code that just encoded and decoded DNS
packets [d].  When the other two teams saw that code, they had it
integreated into their respective projects in a few hours and it's worked
flawlessly since.

  Oh, and both projects were in C++ by the way, not Lua.

  So to sum up, you have the composability issue regardless of langauge.

[a] In order to look up name information for phone numbers.  I'm
        serious, the telecom industry uses DNS to convert a phone number
        like 561-555-1212 into a name like "John Doe" or "ACME Inc.".

[b] I'm liking this new footnote style.  I'm up to what? 20 footnotes so
        far?

[c] In defense of C-Ares, I think the creators of that project thought
        they should deal with all the complexity of dealing with DNS,
        including multiple servers, TTL managmenet and retries to releave
        users of their project all those details.  The problem was---all the
        *other* DNS libraries out there did the same.

[d] https://github.com/spc476/SPCDNS

> Here's what I mean: take "luv" for example. It only works if all file
> descriptors are set to non-blocking mode. Now suppose you pull some
> random mysql module off the shelf that communicates over a network
> socket to your SQL server. And suppose that mysql library uses
> LuaSocket to create a network connection to the database.
>
> Unless that mysql module is a priori designed to work with "luv", as
> soon as you start using it, it will make some blocking call, and your
> entire application will block - oops, here comes Problem #2 again.

  As mentioned above, this is an issue in any language.  If you need a
library to do X, and it also deals with Y, which your program is already
dealing with, there's a problem.  And the solutions are limited:

        1) Find another library do to X.

        2) Write code to do X.

        3) Adopt librray A to work with your Y.

        4) Change your Y to library A's Y.

        5) Go shopping because this stuff is hard.

> Even if you could somehow access the socket that the mysql module uses
> and set it to non-blocking mode, that still wouldn't work because the
> mysql module wouldn't know what to do with the  new EWOULDBLOCK error
> code it would get back.
>
> In other words, there exist "solutions" to Problem #2, but if you use
> them, then EVERY other module you use that makes blocking calls needs
> to also be designed to work with that solution, or else you haven't
> really solved the problem.
>
> Another example: suppose you want your application to fetch an HTTPS
> URL like "https://foobar.com/" without blocking the entire VM, using
> some simple "request" object API. Then you have to find a module that
> contains ALL THREE of (a) non-blocking I/O, (b) HTTP client support,
> and (c) SSL support... or else you have to find three separate modules
> for (a), (b), and (c) that were all written to work together
> (unlikely).

  Well, while I don't have a module for (b), I do have separate modules for
[a] and [c].  You can use the TLS module without the network module [d] but
you can use the network module if you want to.  In fact, I think you can use
my TLS library with LuaSocket if you really tried to [e].  It was a design
goal of mine to make multiple small libraries that work well together [f]
instead of large all-encompassing libraries.  I *like* composability.

[a] https://github.com/spc476/lua-conmanorg/blob/master/src/net.c

[b] There is no b, only Zuul.

[c] https://github.com/spc476/lua-conmanorg/blob/master/src/tls.c

[d] Because the underlying library I wrapped also does networking, sigh.

[e] I haven't yet.

[f] https://github.com/spc476/LPeg-Parsers
        https://github.com/spc476/lua-conmanorg
        https://github.com/spc476/CBOR
        https://github.com/spc476/SPCDNS

> In my opinion this problem is actually the biggest problem with the
> Lua "ecosystem". As evidence look at the discussion going on in the
> "batteries" thread right now.

  That's a good summary I think.

> PROPOSAL
>
> I tried to think about whether there is a way to address Problem #2
> directly and transparently, i.e., in a way that doesn't require other
> modules to know anything about it. So that, using the above example,
> if you started using some random mysql module that makes blocking
> network calls, it wouldn't lock up the entire VM while it waits.
>
> How could you do this? Here was my first idea:
>
>     1. Give each coroutine it's own pthread and C stack.
>     2. Disable preemption (i.e., context switch only on blocking call
> or yield())
>     3. Disable parallel execution to preserve Advantage #2 (e.g., lock
> all pthreads to one CPU)
>
> The key here is that the combination of #2 and #3 means only one
> coroutine runs at a time, and it runs without being preempted until it
> either (a) yields, or (b) blocks. In effect, a blocking call is
> treated like a yield(), with the corresponding resume() occurring when
> the file descriptor becomes readable (or whatever).

  I think that's how most network event drivers for Lua currently work, only
yield on blocking calls and otherwise let the currently running coroutine do
it's work.  It's how my own network driver [a] works.

[a] https://github.com/spc476/lua-conmanorg/blob/master/lua/nfl.lua


> This type of solution preserves Advantage #2 (and provides a simple
> solution to Problem #3) with no additional locking. There is only one
> small downside compared to "plain" Lua: blocking calls now behave as
> if they could yield() internally. For example, this code could behave
> differently:
>
>     counter = 1;
>     function mycoroutine(client)
>         print(counter)
>         print(counter)     // counter must be the same here
>         client:receive()   // this is a blocking call
>         print(counter)     // counter could have changed here!
>     end
>
> In plain Lua the counter could not have changed in the last print() statement.

  This issue exists in other langauges as well.  This isn't just a Lua
problem.

> Since we can't do it with pthreads, what about GNU pth? It is
> specifically designed for non-preemptive multitasking and gives you
> 3(a), 3(b), and 3(c) by design. Moreover, pth is very portable and
> stable (I used it back in the 90's).

  At least it's LGPL, but how well does it work with pthreads?  Because I
have this module over here that uses pthreads ...

> (If people barf on the GPL, it would be easy to write the same thing
> from scratch under a looser license.)
>
> What would be required of existing C modules to support this? => Any
> blocking calls would have to be changed so they invoke pth's version.
> For example, read becomes pth_read(), sleep() becomes pth_sleep(),
> etc.

  On some systems, one can use LD_PRELOAD tricks to swap out blocking
functions with ones that can yield.  

> Does this mean C modules would end up being littered with #ifdef's and
> incompatible with "plain" Lua?
>
> No. Instead, this addition to the Lua API would define a Lua wrapper
> funtion for each blocking function.

  What functions would those be?  Because there's this other thread trying
to define a minimal set of "batteries" for Lua ...

> So the C module would change read() into lua_read(), sleep() into
> lua_sleep(), etc., after including some appropriate Lua header.
>
> But these wrapper functions could work either way: the same C module
> would compile and work on either "plain" Lua or on this new, optional
> "nonblock" version of Lua: on platforms supporting "nonblock" the
> wrappers would redirect to non-blocking versions; on "plain" Lua
> platforms, the wrappers would just fall back to the original versions
> - so you get exactly what you did before.

  Ah, much like how lua_lock() and lua_unlock() work today.

> To summarize this idea:
>
>   1. Add new, optional "nonblock" support to Lua on platforms that can
> handle it, based on cooperative threading (GNU pth or equivalent)

  Yes, there's another thread about that ...

>   2. There is a new header file #include "lua_nonblock.h" defining
> wrappers for blocking C calls: lua_read(), lua_write(), lua_sleep(),
> etc.
>   3. C modules written to support "nonblock" continue to work when
> compiled with "plain" Lua, but now can also context switch if compiled
> on "nonblock" versions of Lua
>   4. coroutine.create() gets a new optional parameter, where true
> means "use a separate C stack (if supported)"
>
> To me this would be a major improvement in the status quo and improve
> the Lua ecosystem dramatically by allowing all of the modules out
> there to be composable.

  Again, the non-composable problem isn't limited to just Lua.

> The only cost to module developers would be match & replace read() ->
> lua_read(), etc.

  None of my modules call read(), not even my network and file system
modules.  There are some that call fread(), but that's a C call, and on most
POSIX systems today, you can't use a file with select() [a] (any file
descriptions pointing to real files on the disk will always return ready for
both reading and writing).  

[a] or poll(), epoll(), kqueue(), etc.

> Thoughts?

  Lots, but this is long enough as it ...

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Andrew Gierth
>>>>> "Sean" == Sean Conner <[hidden email]> writes:

 Sean>   The Lua call stack and C call stack are two separate things.
 Sean> The Lua VM does not use longjmp() to implement coroutines [a][d].

In 5.4, the VM recurses on the C stack, so it does indeed use longjmp as
part of yield().

https://github.com/lua/lua/blob/1fb4d539254b67e7e35ed698250c66d1edff0e08/ldo.c#L729

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Archie Cobbs
In reply to this post by Sean Conner
Hi Sean, thanks for your reply.

On Sat, Jan 25, 2020 at 6:15 PM Sean Conner <[hidden email]> wrote:
> > The most profound restriction is the limitation to a single C call
> > stack (you can't assume pthreads), and so Lua must longjmp() every
> > time there is a coroutine context switch.
>
>   The Lua call stack and C call stack are two separate things.

Sorry if I wasn't being clear. Still learning all this stuff.

I was referring to this section 29.2 of "Programming in Lua" (4th ed):

    It is easy for the interpreter to have multiple soft stacks,
    but the runtime of ISO C has only one internal stack.
    Therefore, coroutines in Lua cannot suspend the execution
    of a C function: if there is a C function in the call path from
    a resume to its respective yield, Lua cannot save the state
    of that C function to restore it in the next resume....

    Lua 5.2 and later versions ameliorated that difficulty with
    continuations. Lua 5.2 implements yields using long jumps,
    in the same way that it implements errors....

Put another way, the problem I'm getting at boils down to this: a C
Lua function A can invoke a normal Lua function B via lua_pcall(). But
lua_pcall() is a synchronous call, and therefore the C stack frame for
A cannot be discarded until B completes or yields.

Now suppose B invokes a blocking system call. Then the entire VM is
frozen. And even if it wasn't, we couldn't unwind the current C stack
(because A's stack frame is still needed), and there is no other C
stack we can use do meaningful work.

Again, I'm talking about "plain" Lua here.

> > For "plain" Lua, restriction to ANSI C creates these problems:
> >
> >     Problem #1. Lua is single-threaded and can't take advantage of
> > multi-core machines
>
>   Yes it can.  Not out of the box certainly...

Yes, but "out of the box" is what I'm talking here...

Let's take a step back.

The Lua developers have been very careful not to bloat "plain" Lua
with extra dependencies and I totally respect that.

However, a large chunk of the Lua user base is using Lua on more
capable (e.g., POSIX) systems, so it makes sense to try to define a
coherent, standard way to take advantage of those capabilities.

But that's a vague and open-ended statement. So I'm trying to take a
conservative, incremental approach to the idea.

So I'd first ask: What is the single most important problem that is
inherent and inescapable with "plain" Lua that we'd like to solve with
additional capabilities? And what is the minimum additional capability
that could be required in order to solve it?

In other words, what's the lowest hanging fruit here?

To me the most important such problem is the non-composability of modules.

Moreover, there is a solution to this problem which relies on a very
minimal set of "extra capabilities". Ref:
https://www.gnu.org/software/pth/pth-manual.html#implementation_notes

> > Problem #1 speaks for itself and is easy to understand. You can
> > address this by creating multiple Lua contexts, but then you have to
> > handle the resulting isolation: there are no shared globals or
> > upvalues, and so all communication between contexts requires
> > serialization of some kind.
>
>   Or implement lua_lock()/lua_unlock() as mentioned.

Yes, that's a neat idea, but doesn't it represent a much more
aggressive change to the semantics of the language, because
invalidates Advantage #2 (atomic execution between yield()s)?

> > As for Problem #3, it's more of an inconvenience than a major problem.
> >
> > Let's talk about Problem #2!
> >
> > In my view, Problem #2 is the most serious problem. It eliminates a
> > huge class of applications, namely, any application supporting
> > simultaneous blocking operations. For example, a web server!
>
>   It does not.  At work, I have a program written in Lua that is handling
> over 200,000,000 calls per day [a].

But surely you're using luv or cqueues or some equivalent, right?

Or do your Lua VM's still block completely when you read from the network?

> > OK so far so good...  BUT - there's still a larger problem with "luv"
> > or any other solution to Problem #2 I've run across so far: none of
> > them are COMPOSABLE with other, unrelated Lua modules.
>
>   True, but you have that issue in any language.

Not really, at least as it relates to Problem #2.

Take POSIX C for example. I can write a C shared library with a
function F() that invokes a blocking system call, and my library can
be linked into some random program, and F() can be invoked without
freezing the entire program. POSIX C provides a standard way to handle
this (pthreads).

An even more interesting example in this debate is Node.js. They solve
this problem simply by making it impossible to invoke a blocking
system call directly!

But look how composable Node.js is, and how its adoption exploded. I'd
argue composability was an essential enabler of that growth.

> > Since we can't do it with pthreads, what about GNU pth? It is
> > specifically designed for non-preemptive multitasking and gives you
> > 3(a), 3(b), and 3(c) by design. Moreover, pth is very portable and
> > stable (I used it back in the 90's).
>
>   At least it's LGPL, but how well does it work with pthreads?  Because I
> have this module over here that uses pthreads ...

You can safely link to the pth and pthreads libraries at the same time.

> > Instead, this addition to the Lua API would define a Lua wrapper
> > funtion for each blocking function.
>
>   What functions would those be?  Because there's this other thread trying
> to define a minimal set of "batteries" for Lua ...

Here are the POSIX functions that pth provides wrappers for:

pth_nanosleep, pth_usleep, pth_sleep, pth_waitpid, pth_system, pth_sigmask,
pth_sigwait, pth_accept, pth_connect, pth_select, pth_pselect,
pth_poll, pth_read,
pth_readv, pth_write, pth_writev, pth_pread, pth_pwrite, pth_recv, pth_recvfrom,
pth_send, pth_sendto.

Ref: https://www.gnu.org/software/pth/pth-manual.html#item_standard_posix_replacement_api

-Archie

--
Archie L. Cobbs

Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Viacheslav Usov
On Sun, Jan 26, 2020 at 7:05 PM Archie Cobbs <[hidden email]> wrote:
> Moreover, there is a solution to this problem which relies on a very minimal set of "extra capabilities".

Quoting from there: 

On the other hand, it cannot benefit from the existence of multiprocessors, because for this, kernel support would be needed. In practice, this is no problem, because multiprocessor systems are rare, and portability is almost more important than highest concurrency.

(end quote)

That is seriously crazy, in 2020. Is this just the docs that are not up-to-date, or is this really something that still works that way?

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Archie Cobbs
On Sun, Jan 26, 2020 at 12:24 PM Viacheslav Usov <[hidden email]> wrote:

> On Sun, Jan 26, 2020 at 7:05 PM Archie Cobbs <[hidden email]> wrote:
> > Moreover, there is a solution to this problem which relies on a very minimal set of "extra capabilities".
> > Ref: https://www.gnu.org/software/pth/pth-manual.html#implementation_notes
>
> Quoting from there:
>
> On the other hand, it cannot benefit from the existence of multiprocessors, because for this, kernel support would be needed. In practice, this is no problem, because multiprocessor systems are rare, and portability is almost more important than highest concurrency.
>
> (end quote)
>
> That is seriously crazy, in 2020. Is this just the docs that are not up-to-date, or is this really something that still works that way?

Yes that text is seriously out of date :) And yes that's indeed how
pth works... it intentionally does not support or provide for parallel
execution.

However, parallel execution is not relevant to what I'm proposing,
which doesn't intend to change Lua's "single threaded" nature.

-Archie

--
Archie L. Cobbs

Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Sean Conner
In reply to this post by Archie Cobbs
It was thus said that the Great Archie Cobbs once stated:

> Hi Sean, thanks for your reply.
>
> On Sat, Jan 25, 2020 at 6:15 PM Sean Conner <[hidden email]> wrote:
> > > The most profound restriction is the limitation to a single C call
> > > stack (you can't assume pthreads), and so Lua must longjmp() every
> > > time there is a coroutine context switch.
> >
> >   The Lua call stack and C call stack are two separate things.
>
> Sorry if I wasn't being clear. Still learning all this stuff.
>
> I was referring to this section 29.2 of "Programming in Lua" (4th ed):
>
>     It is easy for the interpreter to have multiple soft stacks,
>     but the runtime of ISO C has only one internal stack.
>     Therefore, coroutines in Lua cannot suspend the execution
>     of a C function: if there is a C function in the call path from
>     a resume to its respective yield, Lua cannot save the state
>     of that C function to restore it in the next resume....
>
>     Lua 5.2 and later versions ameliorated that difficulty with
>     continuations. Lua 5.2 implements yields using long jumps,
>     in the same way that it implements errors....

  Good lord!  It even mentions it in the Lua 5.2 manual ... how did I miss
that?  Sigh.

> So I'd first ask: What is the single most important problem that is
> inherent and inescapable with "plain" Lua that we'd like to solve with
> additional capabilities? And what is the minimum additional capability
> that could be required in order to solve it?
>
> In other words, what's the lowest hanging fruit here?
>
> To me the most important such problem is the non-composability of modules.

  I would agree with that.  I know that when I wrote my own modules [1], I
try to focus them to a single concept instead of writing a large
encompassing module.  I've also made sure they work well together.

> > > In my view, Problem #2 is the most serious problem. It eliminates a
> > > huge class of applications, namely, any application supporting
> > > simultaneous blocking operations. For example, a web server!
> >
> >   It does not.  At work, I have a program written in Lua that is handling
> > over 200,000,000 calls per day [a].
>
> But surely you're using luv or cqueues or some equivalent, right?
>
> Or do your Lua VM's still block completely when you read from the network?

  Something equivalent to luv or cqueues (I wrote my own).  And no, the Lua
VMs I run don't block when I read from the network.  I took what you said as
"you can't write a network server in Lua."  

> > > OK so far so good...  BUT - there's still a larger problem with "luv"
> > > or any other solution to Problem #2 I've run across so far: none of
> > > them are COMPOSABLE with other, unrelated Lua modules.
> >
> >   True, but you have that issue in any language.
>
> Not really, at least as it relates to Problem #2.
>
> Take POSIX C for example. I can write a C shared library with a
> function F() that invokes a blocking system call, and my library can
> be linked into some random program, and F() can be invoked without
> freezing the entire program. POSIX C provides a standard way to handle
> this (pthreads).

  Yes, and my colleagues at work had a hell of a time integreating a DNS
library into their projects, even *in* the presense of pthreads, because the
library they selected wanted to do DNS queries its way, existing program be
damned!

> An even more interesting example in this debate is Node.js. They solve
> this problem simply by making it impossible to invoke a blocking
> system call directly!
>
> But look how composable Node.js is, and how its adoption exploded. I'd
> argue composability was an essential enabler of that growth.

  But with callback hell and promises (a form of callback hell).  I'd call
Javascript popular because it comes with every browswer out there, not
because it's any good.

  And Node.js has other issues (leftpad anyone?).

> >   What functions would those be?  Because there's this other thread trying
> > to define a minimal set of "batteries" for Lua ...
>
> Here are the POSIX functions that pth provides wrappers for:
>
> pth_nanosleep, pth_usleep, pth_sleep, pth_waitpid, pth_system, pth_sigmask,
> pth_sigwait, pth_accept, pth_connect, pth_select, pth_pselect,
> pth_poll, pth_read,
> pth_readv, pth_write, pth_writev, pth_pread, pth_pwrite, pth_recv, pth_recvfrom,
> pth_send, pth_sendto.

  Oh, so I have to rewrite code to use the new names (or a preprocessor
hack).

  -spc

[1] https://github.com/spc476/lua-conmanorg

Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Viacheslav Usov
In reply to this post by Archie Cobbs
On Sun, Jan 26, 2020 at 7:55 PM Archie Cobbs <[hidden email]> wrote:

> However, parallel execution is not relevant to what I'm proposing, which doesn't intend to change Lua's "single threaded" nature.

Lua's nature is single-threaded only when compiled with default defines. It can easily be compiled in a way that makes it thread-safe, and then multiple threads can execute in a single Lua VM concurrently.

Secondly, one important use of Lua is an multi-threaded process hosting multiple Lua VMs, where each is "single-threaded". In such a setup, I'd be very cautious about using a technique that assumes that multiple CPUs are rare and irrelevant.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Archie Cobbs
In reply to this post by Sean Conner
On Sun, Jan 26, 2020 at 8:42 PM Sean Conner <[hidden email]> wrote:
> >   What functions would those be?  Because there's this other thread trying
> > to define a minimal set of "batteries" for Lua ...
>
> Here are the POSIX functions that pth provides wrappers for:
>
> pth_nanosleep, pth_usleep, pth_sleep, pth_waitpid, pth_system, pth_sigmask,
> pth_sigwait, pth_accept, pth_connect, pth_select, pth_pselect,
> pth_poll, pth_read,
> pth_readv, pth_write, pth_writev, pth_pread, pth_pwrite, pth_recv, pth_recvfrom,
> pth_send, pth_sendto.

  Oh, so I have to rewrite code to use the new names (or a preprocessor
hack).

Sort-of. Here's how I would propose we approach this.

STEP 1: Add a new header "lnbposix.h" to Lua.

The name "lnbposix" stands for "Lua non-blocking versions of POSIX functions".

The meaning of this file is basically this:

    If you are writing a C module and are using any blocking POSIX
    system calls, and you want your code to be automatically compatible
    with a possible future version or variant of the Lua runtime that
    supports yielding coroutines when they make blocking calls, then
    you can use these functions in place of the normal POSIX functions
    and everything will work out great either way.

There could eventually be other header files for non-POSIX blocking functions as needed.

For now, "lnbposix.h" will simply contain the following definitions:

    #define luaNB_read read
    #define luaNB_write write
    #define luaNB_nanosleep nanosleep
    ...etc...

So far, nothing functions differently, but your module is now "non-blocking ready".

Because this file has no effect, it can be included in even "plain" Lua distributions.

The semantics of function luaNB_foo() would be defined to be the same as foo(), with the exception that luaNB_foo() may cause the coroutine to yield IF that is supported on the current platform. If not, it behaves exactly like foo().

STEP 2: Argue about where and how we should implement real, non-blocking versions of luaNB_read(), luaNB_write(), etc. :)

Is this something we could get added to version 5.4??

-Archie

--
Archie L. Cobbs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Archie Cobbs
On Mon, Jan 27, 2020 at 2:23 PM Archie Cobbs <[hidden email]> wrote:
STEP 1: Add a new header "lnbposix.h" to Lua.
...
Is this something we could get added to version 5.4??

Here's the kind of thing I'm talking about for 5.4:


-Archie

--
Archie L. Cobbs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Sean Conner
In reply to this post by Archie Cobbs
It was thus said that the Great Archie Cobbs once stated:
> On Sun, Jan 26, 2020 at 8:42 PM Sean Conner <[hidden email]> wrote:
>
> >   Oh, so I have to rewrite code to use the new names (or a preprocessor
> > hack).
>
> Sort-of. Here's how I would propose we approach this.
>
> STEP 1: Add a new header "lnbposix.h" to Lua.

  [ snip ]

> For now, "lnbposix.h" will simply contain the following definitions:
>
>     #define luaNB_read read
>     #define luaNB_write write
>     #define luaNB_nanosleep nanosleep
>     ...etc...

  Yes, the preprocessor hack I mentioned.

  There are problems with this (in general, not with the preprocessor hack):

1. It adds a dependency to Lua, namely GNU pth.  Right now, Lua only has one
dependency, and that's the standard C library, which if you are compiling
Lua, you already have.

2. GNU pth is LGPL.  There is a sizable segment of the industry that avoids
anything to do with the GNU.  While there are a number of programmers who
dislike that license, there are some very large companies that actively
avoid any license with the letters G, P, and L in them.  Two in particular
are Google and Apple, which now have a lock on the future of computing
(smart phones and touch pads----seriously, that's the way the industry is
headed for home computing, sad as that is).  If you want Lua to gain
traction, this is NOT the way to go (and I'm saying this as someone who only
uses the GNU licenses).

3. The elephant in the room---Windows.  It *has* been tested with Windows
using Cygwin, which now adds an additional dependency to that platform.
Also, it should be noted that getting C-based modules compiled and working
for Windows is problematic and there have been multiple attempts over the
years to get a working distribution for Lua for Windows (although I may be
overstating the problem here---I don't know as I tend to ignore Windows;
others can probably fill you in on the details).

4. The last release was in June of 2006.  That in and of itself doesn't mean
it doesn't work, but again, there is a segment of the industry that will
conclude that the project is dead and should be avoided.  Also, POSIX has
been updated since 2006, so at the very least, it might be outdated.  Also,
the mailing list last saw activity in June of 2012.  Again, not a good sign.

5. You'll need to patch os.system() and most of io in stock Lua to use this
as well, which is a tall order.  While select() et al. are used on file
descriptors, not every file descriptor can be meaningfully used with
it---notably real files will always return a ready status, even if the files
are on a exceedingly slow floppy drive!  But there are some devices (like
serial ports or the current TTY) that can, and they can be opened via
io.open().  [1]

> STEP 2: Argue about where and how we should implement real, non-blocking
> versions of luaNB_read(), luaNB_write(), etc. :)

  It's hard enough getting "batteries" defined for Lua.

> Is this something we could get added to version 5.4??

  I would be surprised if this came out for Lua 5.4.

  -spc

[1] My own select() et al. wrapper can handle files opened by io.open(),
        but I do have to monkey patch the file object metatable to add a
        function to get the underlying file descriptor.  I have a similar
        function for my socket library, so the select() driver doesn't have
        to know the details of the userdata it receives---it just knows to
        call a special function to get what it needs for the select() et al.
        call.


Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Archie Cobbs
On Mon, Jan 27, 2020 at 4:10 PM Sean Conner <[hidden email]> wrote:
It was thus said that the Great Archie Cobbs once stated:
> On Sun, Jan 26, 2020 at 8:42 PM Sean Conner <[hidden email]> wrote:
>
> >   Oh, so I have to rewrite code to use the new names (or a preprocessor
> > hack).
>
> Sort-of. Here's how I would propose we approach this.
>
> STEP 1: Add a new header "lnbposix.h" to Lua.

  [ snip ]

> For now, "lnbposix.h" will simply contain the following definitions:
>
>     #define luaNB_read read
>     #define luaNB_write write
>     #define luaNB_nanosleep nanosleep
>     ...etc...

  Yes, the preprocessor hack I mentioned.

  There are problems with this (in general, not with the preprocessor hack):

1. It adds a dependency to Lua, namely GNU pth.

I'm not proposing that for Lua itself right now. I'm simply proposing we add the header with the #defines:


Absolutely nothing will change in Lua itself.

The idea is that this header file allows Lua hackers to create future variants of Lua that support non-blocking, and at the same time allows C module hackers to create Lua modules that work properly with those variants (when they arrive) and yet also remain backward compatible with "plain" Lua.

2. GNU pth is LGPL. 

This is an orthogonal issue and can be sorted out later, at worst, by writing a new implementation of the same thing from scratch.

But in any case that's all in the future, pending further discussion. It doesn't stop going ahead and laying the groundwork now with a simple header file.

For now all I want is to start letting C module writers convert their code from read() to luaNP_read() today so they are ready for non-blocking support, whatever form it may take, if/when it occurs sometime in the future.
 
3. The elephant in the room---Windows. 

For any particular feature, either the platform supports it or the platform does not. That's true of any feature.

If Windows supports non-blocking versions of read(), write(), etc. somehow, then it can be done. If not it can't and so you'd necessarily fall back to "blocking" Lua.
 
5. You'll need to patch os.system() and most of io in stock Lua to use this
as well, which is a tall order.  While select() et al. are used on file
descriptors, not every file descriptor can be meaningfully used with
it---notably real files will always return a ready status, even if the files
are on a exceedingly slow floppy drive!  But there are some devices (like
serial ports or the current TTY) that can, and they can be opened via
io.open().  [1]

Yes, to the extent Lua libraries use any blocking POSIX calls, they would also need to be updated like any other C code. It's a simple find & replace.

This doesn't seem very complicated or controversial to me. It's adding a lightweight wrapper layer to allow for very desirable future functionality (namely, composable modules that also support non-blocking coroutines), and has zero runtime cost for current ("plain" Lua) implementations.

Is there some Lua authority out there who can provide feedback on this idea? How are Lua patches supposed to be reviewed around here?

-Archie

--
Archie L. Cobbs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Jonathan Goble
On Mon, Jan 27, 2020, 5:45 PM Archie Cobbs <[hidden email]> wrote:
Is there some Lua authority out there who can provide feedback on this idea? How are Lua patches supposed to be reviewed around here?

-Archie

In general, they aren't, at least not directly. Lua is open source but not open developed. Proposals are normally discussed in terms of ideas and concepts rather than as concrete code and patches (though example code may help facilitate the discussion).

From there, the Lua team (Roberto, Luiz, and one other person who I always forget because he never posts to the list) makes the decision on their own, often without announcement. If they choose to adopt a community-proposed idea, they will write their own implementation rather than use code from the outside. 
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Tim Hill
In reply to this post by Sean Conner


On Jan 25, 2020, at 4:15 PM, Sean Conner <[hidden email]> wrote:

The Lua call stack and C call stack are two separate things.  The Lua VM
does not use longjmp() to implement coroutines [a][d].

Really? I can’t look at the code right now, but I’d always assumed that Lua always used longjmp (as per the OPs point). How can it do otherwise? At any time a Lua state can have any number of intermixed C and Lua frames, and while you are correct in that they dont share the same stack, Lua needs to resume by longjmp to make sure the C stack corresponds to the Lua coroutine state. While I can see that Lua *might* optimize out this condition if it knows there are NO C calls on either coroutine stack, I’m not sure that it does that?

—Tim

Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Sean Conner
It was thus said that the Great Tim Hill once stated:

>
>
> > On Jan 25, 2020, at 4:15 PM, Sean Conner <[hidden email]> wrote:
> >
> > The Lua call stack and C call stack are two separate things.  The Lua VM
> > does not use longjmp() to implement coroutines [a][d].
>
> Really? I can’t look at the code right now, but I’d always assumed that
> Lua always used longjmp (as per the OPs point). How can it do otherwise?
> At any time a Lua state can have any number of intermixed C and Lua
> frames, and while you are correct in that they dont share the same stack,
> Lua needs to resume by longjmp to make sure the C stack corresponds to the
> Lua coroutine state. While I can see that Lua *might* optimize out this
> condition if it knows there are NO C calls on either coroutine stack, I’m
> not sure that it does that?

  I was mistaken.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Archie Cobbs
In reply to this post by Jonathan Goble
On Mon, Jan 27, 2020 at 6:01 PM Jonathan Goble <[hidden email]> wrote:
On Mon, Jan 27, 2020, 5:45 PM Archie Cobbs <[hidden email]> wrote:
Is there some Lua authority out there who can provide feedback on this idea? How are Lua patches supposed to be reviewed around here?

In general, they aren't, at least not directly. Lua is open source but not open developed. Proposals are normally discussed in terms of ideas and concepts rather than as concrete code and patches (though example code may help facilitate the discussion).

From there, the Lua team (Roberto, Luiz, and one other person who I always forget because he never posts to the list) makes the decision on their own, often without announcement. If they choose to adopt a community-proposed idea, they will write their own implementation rather than use code from the outside. 

That sounds pretty lame, to be honest. That kind of model works directly against building a coherent open source community.

Hmm.

-Archie

--
Archie L. Cobbs
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Coda Highland
On Mon, Jan 27, 2020 at 8:43 PM Archie Cobbs <[hidden email]> wrote:
On Mon, Jan 27, 2020 at 6:01 PM Jonathan Goble <[hidden email]> wrote:
On Mon, Jan 27, 2020, 5:45 PM Archie Cobbs <[hidden email]> wrote:
Is there some Lua authority out there who can provide feedback on this idea? How are Lua patches supposed to be reviewed around here?

In general, they aren't, at least not directly. Lua is open source but not open developed. Proposals are normally discussed in terms of ideas and concepts rather than as concrete code and patches (though example code may help facilitate the discussion).

From there, the Lua team (Roberto, Luiz, and one other person who I always forget because he never posts to the list) makes the decision on their own, often without announcement. If they choose to adopt a community-proposed idea, they will write their own implementation rather than use code from the outside. 

That sounds pretty lame, to be honest. That kind of model works directly against building a coherent open source community.

Hmm.

-Archie

That's because PUC-Rio Lua is not a community project. It's maintained by a dedicated group attached to a university (PUC Rio) and the project is designed to satisfy their needs in particular. It's open-source, in that the code is available under a free software license, but the PUC-Rio distribution of Lua is a closed development model. Not accepting third-party contributions makes a number of things more straightforward and less legally risky, and it gives them the flexibility to make decisions based on what's good for the project instead of what's popular with the community.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Viacheslav Usov
In reply to this post by Archie Cobbs
On Mon, Jan 27, 2020 at 9:24 PM Archie Cobbs <[hidden email]> wrote:

> STEP 1: Add a new header "lnbposix.h" to Lua.

This won't work for an important class of libraries known as "wrappers" or "bindings", i.e., libraries that expose (to Lua) the functionality of another library written in some other language. Even if the latter language is C or C++. With shared libraries this is just not possible, full stop.

With static libraries, you would have to modify a library that has nothing to do with Lua with something Lua-specific. First of all, this just does not look right conceptually. Second, the task might just be too difficult. Especially considering that those libraries might depend on more libraries, as is frequently the case.

So this approach is not "composable".

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Blocking calls and module composability

Archie Cobbs
On Tue, Jan 28, 2020 at 11:05 AM Viacheslav Usov <[hidden email]> wrote:
On Mon, Jan 27, 2020 at 9:24 PM Archie Cobbs <[hidden email]> wrote:

> STEP 1: Add a new header "lnbposix.h" to Lua.

This won't work for an important class of libraries known as "wrappers" or "bindings", i.e., libraries that expose (to Lua) the functionality of another library written in some other language. Even if the latter language is C or C++. With shared libraries this is just not possible, full stop.

With static libraries, you would have to modify a library that has nothing to do with Lua with something Lua-specific. First of all, this just does not look right conceptually. Second, the task might just be too difficult. Especially considering that those libraries might depend on more libraries, as is frequently the case.

Correct...

To restate what you're saying: this technique only works if the blocking call is part of the Lua C module itself, where it can be wrapped, but not if the blocking call is "indirect", i.e., invoked by some other unrelated library on which the C module depends and therefore inaccessible to "wrapping".
 
So this approach is not "composable".

More accurate would be to say that it's composable, but only one level deep.

So for example it works fine for a C module that provides a socket library. But if a C module uses libcurl to do network I/O or something then it doesn't help.

This is part of the trade-off with this idea, which is just trying to take an incremental step.

The only way I can see to jump directly to complete composability for non-blocking would be using pthreads (or equivalent), while somehow forcing it into non-preemptive mode. Unfortunately, that doesn't seem possible (at least, not portably).

-Archie

--
Archie L. Cobbs