Embedded lua and yielding to the application run loop (macOS)

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

Embedded lua and yielding to the application run loop (macOS)

Aaron Magill
I am a contributor to Hammerspoon, which uses an embeded lua 5.3 interpreter to allow users to programatically access a fair amount of the macOS API for things like window management, hoteys, etc. and I've been looking at ways to make it seem more responsive when the lua interpreter is actually doing something slow...

A little background before I get to my example and questions (I hope this makes sense):

As I understand it, when things like a timer fires and we want to invoke a lua callback, what happens is an event is queued for the macOS runloop and when the runloop isn't doing anything else, it checks its queue and pops an event off and handles it. In the case of one of our timers, the block attached to the queued event uses `lua_pcall` to call a stored function and the lua interpreter executes the lua function.

There is also a conosle where we can type things in and have them executed by the lua interpreter (much like invoking lua from the command line interactively).

So... I hit upon this hairbrained idea to "mimic" the macOS application runloop for a short period in a C function added to a lua module, defined like this:

          static int hs_yield(lua_State *L) {
              NSTimeInterval interval = (lua_type(L, 1) == LUA_TNUMBER) ? 0.001 : lua_tonumber(L, 1) ;
              NSDate         *date    = [[NSDate date] dateByAddingTimeInterval:interval] ;

              // a melding of code from gnustep's implementation of NSApplication's run and runUntilDate: methods
              // this allows acting on events (hs.eventtap) and keys (hs.hotkey) as well as timers, etc.
              BOOL   mayDoMore = YES ;
              while (mayDoMore) {
                  NSEvent *e = [NSApp nextEventMatchingMask:NSAnyEventMask
                                                  untilDate:date
                                                     inMode:NSDefaultRunLoopMode
                                                    dequeue:YES] ;
                  if (e) [NSApp sendEvent:e] ;

                  mayDoMore = !([date timeIntervalSinceNow] <= 0.0) ;
              }

              return 0 ;
          }

So if I do the following in the Hammerspoon console (pasted in as one large chunk, which means its treated as a single string and also executed within a lua_pcall:

          -- creates a grey rectangular box on the screen with the tet "**" in it
          cv = hs.canvas.new{ x = 100, y = 100, h = 100, w = 100 }:show()
          cv[#cv + 1] = { type = "rectangle", fillColor = { white = .1 } }
          cv[#cv + 1] = { type = "text", text = "**", textSize = 75 }

          st = false
          bl = 0
          dd = 0

          -- every second, update the number displayed in the grey box
          xx = hs.timer.doEvery(1, function()
              dd = dd + 1
              cv[2].text = tostring(dd)
          end)

          -- after 30 seconds, stop while loop below and clean up
          yy = hs.timer.doAfter(30, function()
              st = true   -- stop runaway while
              xx:stop()   -- stop timer updating canvas
              cv:delete() -- delete canvas
          end)

          zz = os.time()
          while (not st) do
              bl = bl + 1
              hs_yield(n) -- defaults to 0.001 if n is nil
          end
          print(bl, dd, os.time() - zz)

And this works as I had hoped -- the canvas is updated each second for 30 seconds and bl ends up being somewhere around 1478052.

My question to the list: is this safe, or am I setting things up for a subtle catastrophe at some point?

As I understand it, what's happening is that the lua interpreter is executing the while loop, and from inside the while loop, pauses for 0.001 seconds to see if an event is queued for the macOS application run loop; when there is, if the event has a lua function attached, it is invoked (from one of the timers) within a new lua_pcall, but before the original lua_pcall, which is executing the while loop, has finished.

Is this safe, since each chunk of lua code is within its own lua_pcall? Are there lua commands I should make sure that hs_yield is never invoked within (haven't tried a for command iterator function yet, for example)? Are there additional sanity checks I should make to ensure the integrity of the lua state and stack?

Figured I should ask here before releasing this into the core Hammerspoon code and let our users start banging on it.

Thanks in advance for any insight or suggestions!

--
Aaron


Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Jerome Vuarand
On Mon, 17 Feb 2020 at 11:34, Aaron Magill <[hidden email]> wrote:
> So... I hit upon this hairbrained idea to "mimic" the macOS application runloop for a short period in a C function added to a lua module, defined like this:
>
>           static int hs_yield(lua_State *L) {
>               NSTimeInterval interval = (lua_type(L, 1) == LUA_TNUMBER) ? 0.001 : lua_tonumber(L, 1) ;

Isn't there a bug in the line above? Or is the ternary operator in
this language different than in C?

>               NSDate         *date    = [[NSDate date] dateByAddingTimeInterval:interval] ;
>
>               // a melding of code from gnustep's implementation of NSApplication's run and runUntilDate: methods
>               // this allows acting on events (hs.eventtap) and keys (hs.hotkey) as well as timers, etc.
>               BOOL   mayDoMore = YES ;
>               while (mayDoMore) {
>                   NSEvent *e = [NSApp nextEventMatchingMask:NSAnyEventMask
>                                                   untilDate:date
>                                                      inMode:NSDefaultRunLoopMode
>                                                     dequeue:YES] ;
>                   if (e) [NSApp sendEvent:e] ;
>
>                   mayDoMore = !([date timeIntervalSinceNow] <= 0.0) ;
>               }
>
>               return 0 ;
>           }
>
> So if I do the following in the Hammerspoon console (pasted in as one large chunk, which means its treated as a single string and also executed within a lua_pcall:
>
>           -- creates a grey rectangular box on the screen with the tet "**" in it
>           cv = hs.canvas.new{ x = 100, y = 100, h = 100, w = 100 }:show()
>           cv[#cv + 1] = { type = "rectangle", fillColor = { white = .1 } }
>           cv[#cv + 1] = { type = "text", text = "**", textSize = 75 }
>
>           st = false
>           bl = 0
>           dd = 0
>
>           -- every second, update the number displayed in the grey box
>           xx = hs.timer.doEvery(1, function()
>               dd = dd + 1
>               cv[2].text = tostring(dd)
>           end)
>
>           -- after 30 seconds, stop while loop below and clean up
>           yy = hs.timer.doAfter(30, function()
>               st = true   -- stop runaway while
>               xx:stop()   -- stop timer updating canvas
>               cv:delete() -- delete canvas
>           end)
>
>           zz = os.time()
>           while (not st) do
>               bl = bl + 1
>               hs_yield(n) -- defaults to 0.001 if n is nil
>           end
>           print(bl, dd, os.time() - zz)
>
> And this works as I had hoped -- the canvas is updated each second for 30 seconds and bl ends up being somewhere around 1478052.
>
> My question to the list: is this safe, or am I setting things up for a subtle catastrophe at some point?

This isn't safe on the Lua side of things.

> As I understand it, what's happening is that the lua interpreter is executing the while loop, and from inside the while loop, pauses for 0.001 seconds to see if an event is queued for the macOS application run loop; when there is, if the event has a lua function attached, it is invoked (from one of the timers) within a new lua_pcall, but before the original lua_pcall, which is executing the while loop, has finished.

This is not very clear to me (is it to you?). The key question is
whether the timer callbacks can only be called from
nextEventMatchingMask, on that same thread, or if the timer callbacks
are asynchronous. It is fine to have "recursive" lua_pcall-s, ie.
pcall function x, and have x pcall function y. But you cannot pcall
function x from thread 1, and pcall function y from thread 2 at the
same time.

> Is this safe, since each chunk of lua code is within its own lua_pcall? Are there lua commands I should make sure that hs_yield is never invoked within (haven't tried a for command iterator function yet, for example)?

lua_pcall doesn't protect against that situation. With a normal build
of Lua (see below for more) there are no lua commands you can use to
protect the state. You must use an external mutex.

So whether it is safe at all depends on the semantics of your timers
and your main loop event pump. Whether this can happen is a question
for experts of your OS APIs.

> Are there additional sanity checks I should make to ensure the integrity of the lua state and stack?

You can make a special build of Lua where you redefine the macros
lua_lock and lua_unlock. In these you can add code that will check a
flag is not set and set it (in lock) and clear it (in unlock). So if
two threads try to lock at the same time, you'll see the flag is
already set and it proves your code is unsafe.

> Figured I should ask here before releasing this into the core Hammerspoon code and let our users start banging on it.

To increase the chance of triggering the bug, you should decrease the
timeout in the main loop (0.001 seconds is a long time, try the
minimum possible value, ideally zero), and reduce the delay in your
timers so they trigger much more frequently. But to be entirely sure
you need to check the documentation for your timer API and your main
loop event polling API.

Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Oliver Schmidt
In reply to this post by Aaron Magill
Hi.

On 17.02.20 12:34, Aaron Magill wrote:
> I am a contributor to Hammerspoon, which uses an embeded lua 5.3 interpreter to allow users to programatically access a fair amount of the macOS API for things like window management, hoteys, etc. and I've been looking at ways to make it seem more responsive when the lua interpreter is actually doing something slow...

I would execute time consuming tasks on separate system threads to keep the GUI
thread responsive. This is also possible with Lua. Is this an option for you?

Another option could be to have the time comsuming task within a coroutine that
yields periodically. The coroutine would be called and continued in an event
queue timer. With this you don't have to handle a local event queue loop.

> As I understand it, when things like a timer fires and we want to invoke a lua callback, what happens is an event is queued for the macOS runloop and when the runloop isn't doing anything else, it checks its queue and pops an event off and handles it. In the case of one of our timers, the block attached to the queued event uses `lua_pcall` to call a stored function and the lua interpreter executes the lua function.

> So... I hit upon this hairbrained idea to "mimic" the macOS application runloop for a short period in a C function added to a lua module, defined like this:
>           zz = os.time()
>           while (not st) do
>               bl = bl + 1
>               hs_yield(n) -- defaults to 0.001 if n is nil
>           end
>           print(bl, dd, os.time() - zz)
>
> My question to the list: is this safe, or am I setting things up for a subtle catastrophe at some point?

In principle this should work, since everything is happening on the same thread.

Special care has to be taken in the implementation of the the callbacks in the
event run loop. On which lua_State do they operate? It could be the same
lua_State than the lua_State that calls hs_yield, or it could also be a separate
lua_State created be lua_newthread (i.e. coroutine). The event queue callback
handling should keep the lua stack balanced and also ensure that there is enough
space on the lua stack. Anyway all this should be happening in the Hammerspoon
codebase, but perhaps you should add an extra call to lua_checkstack in your c
function hs_yield since the Hammespoon probably doen't expect that there is
another function running on the lua_State where the event queue callbacks are
called. This would not be necessary if Hammerspoon uses a separate lua_State for
the callbacks. There could be other pitfalls in the Hammerspoon codebase. You
should ask the Hammerspoon developers or have a closer look on the Hammerspoon
implementation.

You should also consider that this solution doesn't nest very well: if there is
at some point the "local event queue loop" that dispatches events and tries to
handle some background operation and the dispatched events also go into such
kind of "local event queue loop" (could also be the same) the first one has to
wait until the second one finshes. But perhaps this is not the case in your use
case.

Best regards,
Oliver


Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Aaron Magill
In reply to this post by Aaron Magill

On Mon Feb 17 12:39:17 GMT 2020, Jerome Vuarand <jerome.vuarand at gmail.com> wrote:
> >               NSTimeInterval interval = (lua_type(L, 1) == LUA_TNUMBER) ? 0.001 : lua_tonumber(L, 1) ;
>
> Isn't there a bug in the line above? Or is the ternary operator in
> this language different than in C?

Good catch, no wonder I was seeing so little variance in my tests with different delays. Also changing the default to 10 ^ -6.

> This is not very clear to me (is it to you?). The key question is
> whether the timer callbacks can only be called from
> nextEventMatchingMask, on that same thread, or if the timer callbacks
> are asynchronous. It is fine to have "recursive" lua_pcall-s, ie.
> pcall function x, and have x pcall function y. But you cannot pcall
> function x from thread 1, and pcall function y from thread 2 at the
> same time.

I guess I didn't explain things very well...

First, to be clear, Hammerspoon is runnng lua in a single thread -- the main thread of the application.
(A few long running things that are hadnled solely within Objective-C may run in other threads, but
upon completion, they send a notification to the event queue of the main thread which is picked up by
`nextEventMatchingMask:NSAnyEventMask` and thus triggers a lua_pcall to a callback function we've
defined earlier.)

So, (ignoring my new function for the moment) what happens on the main application thread is that either
the main thread is sitting idle waiting for events from the application gui or OS in
`nextEventMatchingMask:NSAnyEventMask`, or it's executing lua code by pushing a function reference onto
the stack and then invoking the function with `lua_pcall`. While lua code is being interpreted, events
to the main thread just queue, they can't be acted upon.



So maybe a better visual for what I think this function adds:

wait for new event from `nextEventMatchingMask:NSAnyEventMask` {
    lua_pcall to user function for event type pulled from queue
        ...lua code interpretted here...
        hs_yield() {
            check `nextEventMatchingMask:NSAnyEventMask` for new event and if found {
                lua_pcall to user function for new event type pulled from queue
                    ...lua code interpretted here...
                        (possibly additional yield/pcall combos, but allways in a stack formation)
                    ...lua code interpretted here...
                end
            }
        }
        ...lua code interpretted here...
    end
} (application thread back to idle, go back to waiting for new event)


And described this way, I think I do see that we're essentially just calling a function from
within a function, in C, rather than lua.

--
Aaron


Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Aaron Magill
In reply to this post by Aaron Magill
On Mon Feb 17 19:09:20 GMT 2020, Oliver <oschmidt-mailinglists at gmx.de> wrote:


> I would execute time consuming tasks on separate system threads to keep the GUI
> thread responsive. This is also possible with Lua. Is this an option for you?

Desgin decisions that predate my joining the project put the lua interpreter on the
same thread as the user interface and application run loop. Can't do much about
it now with a fundamental re-write.

> Another option could be to have the time comsuming task within a coroutine that
> yields periodically. The coroutine would be called and continued in an event
> queue timer. With this you don't have to handle a local event queue loop.

First let me say that I don't have much experience with co-routines... it's a lack
I've been meaning to address, but so far haven't had a need that couldn't be
addressed another way or the time to do it on my own.

But if I'm understanding co-routines correctly, they still require some lua code (or
C code using the lua API) to actually invoke `coroutine.resume` on them for processing
to continue. This won't work for us (at least in the general case) because
the user interface, event queue, and lua interpreter share the main thread --
in Hammerspoon, as long as the lua interpeter is doing something, nothing else on
the main thread can.

> Special care has to be taken in the implementation of the the callbacks in the
> event run loop. On which lua_State do they operate? It could be the same
> lua_State than the lua_State that calls hs_yield, or it could also be a separate
> lua_State created be lua_newthread (i.e. coroutine).

Everything in Hamemrspoon uses the same lua_State -- the one created when Hammerspoon
first started (or a reload occurs, which destroys the state and creates a new one).

That's going to be a problem, isn't it, if someone does try to invoke
hs_yield from within a coroutine... is checking lua_isyieldable == 0 sufficient
to make sure we're not in a coroutine, or is there a better test?

> ... The event queue callback
> handling should keep the lua stack balanced and also ensure that there is enough
> space on the lua stack. Anyway all this should be happening in the Hammerspoon
> codebase, but perhaps you should add an extra call to lua_checkstack in your c
> function hs_yield since the Hammespoon probably doen't expect that there is
> another function running on the lua_State where the event queue callbacks are
> called. This would not be necessary if Hammerspoon uses a separate lua_State for
> the callbacks. There could be other pitfalls in the Hammerspoon codebase. You
> should ask the Hammerspoon developers or have a closer look on the Hammerspoon
> implementation.

Stack guards and tests for this were recently added but I agree additional checks
couldn't hurt, and this may inadvertantly cause the stack to grow, temporarily,
in ways it hasn't before, so thanks for the reminder!

> You should also consider that this solution doesn't nest very well: if there is
> at some point the "local event queue loop" that dispatches events and tries to
> handle some background operation and the dispatched events also go into such
> kind of "local event queue loop" (could also be the same) the first one has to
> wait until the second one finshes. But perhaps this is not the case in your use
> case.

I know, but without a fundamental rewrite, this isn't going to change...

--
Aaron


Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Ką Mykolas
On Tue, Feb 18, 2020 at 2:21 PM Aaron Magill <[hidden email]> wrote:
>
> I know, but without a fundamental rewrite, this isn't going to change...

Well, just jumping in without deeper understanding of the context,
would it be an option to spawn another Lua VM
with it's own state and additional communications with the main one or
just the copies of some more or less
static data?

Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Coda Highland
In reply to this post by Aaron Magill
On Tue, Feb 18, 2020 at 6:21 AM Aaron Magill <[hidden email]> wrote:
> Another option could be to have the time comsuming task within a coroutine that
> yields periodically. The coroutine would be called and continued in an event
> queue timer. With this you don't have to handle a local event queue loop.

First let me say that I don't have much experience with co-routines... it's a lack
I've been meaning to address, but so far haven't had a need that couldn't be
addressed another way or the time to do it on my own.

But if I'm understanding co-routines correctly, they still require some lua code (or
C code using the lua API) to actually invoke `coroutine.resume` on them for processing
to continue. This won't work for us (at least in the general case) because
the user interface, event queue, and lua interpreter share the main thread --
in Hammerspoon, as long as the lua interpeter is doing something, nothing else on
the main thread can.

I have to agree that using yield / resume is probably the best plan for you. There are a couple of points to consider that may help:

(1) You can use a Lua hook to cause a bit of Lua code to run whenever there's a function call or resume, or after each line of code, or after every N bytecode instructions. Inside this hook, you can call yield. This way you can preempt running Lua code without requiring the Lua code to explicitly yield if you don't want to. (You might need to come up with some sort of serialization primitive to make sure it doesn't leave you in an inconsistent state, but that also might not be a problem depending on how the rest of the system is architected.)


(2) Every event loop system I know of has a way to create synthetic events. If you just want to yield to the event loop and get called right back after the pending events have been processed, just create an event that calls coroutine.resume whenever the Lua code yields.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Aaron Magill


On Feb 18, 2020, at 2:39 PM, Coda Highland <[hidden email]> wrote:

(1) You can use a Lua hook to cause a bit of Lua code to run whenever there's a function call or resume, or after each line of code, or after every N bytecode instructions. Inside this hook, you can call yield. This way you can preempt running Lua code without requiring the Lua code to explicitly yield if you don't want to. (You might need to come up with some sort of serialization primitive to make sure it doesn't leave you in an inconsistent state, but that also might not be a problem depending on how the rest of the system is architected.)



Been considering that, if I can prove to myself that this `hs_yield` function isn't introducing other issues, and it has some appeal because a lot of our users aren't programmers (or weren't to begin with) beyond simple scripting, so keeping implementation details simple and hidden can be a plus.

(2) Every event loop system I know of has a way to create synthetic events. If you just want to yield to the event loop and get called right back after the pending events have been processed, just create an event that calls coroutine.resume whenever the Lua code yields.


That might be a reasonable approach if I hadn't just discovered that we have a fundamental issue with coroutines to begin with in Hammerspoon: see https://github.com/Hammerspoon/hammerspoon/issues/2306, if you want the nitty-gritty, but it boils down to any approach with leverages coroutines will have to wait until that's fixed first...

/s/ Adam

--
Aaron

Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Coda Highland
On Tue, Feb 18, 2020 at 8:53 AM Aaron Magill <[hidden email]> wrote:
(2) Every event loop system I know of has a way to create synthetic events. If you just want to yield to the event loop and get called right back after the pending events have been processed, just create an event that calls coroutine.resume whenever the Lua code yields.

That might be a reasonable approach if I hadn't just discovered that we have a fundamental issue with coroutines to begin with in Hammerspoon: see https://github.com/Hammerspoon/hammerspoon/issues/2306, if you want the nitty-gritty, but it boils down to any approach with leverages coroutines will have to wait until that's fixed first...

Ah, yeah, that does look like an issue, but I think you might have an easier solution available to you than the ones you're considering now: replace coroutine.create(). If the problem is that the new state isn't set up in LuaSkin, then just make a function that calls lua_newthread(), sets it up appropriately, and returns it to the caller.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Aaron Magill


On Feb 18, 2020, at 3:20 PM, Coda Highland <[hidden email]> wrote:

On Tue, Feb 18, 2020 at 8:53 AM Aaron Magill <[hidden email]> wrote:
(2) Every event loop system I know of has a way to create synthetic events. If you just want to yield to the event loop and get called right back after the pending events have been processed, just create an event that calls coroutine.resume whenever the Lua code yields.

That might be a reasonable approach if I hadn't just discovered that we have a fundamental issue with coroutines to begin with in Hammerspoon: see https://github.com/Hammerspoon/hammerspoon/issues/2306, if you want the nitty-gritty, but it boils down to any approach with leverages coroutines will have to wait until that's fixed first...

Ah, yeah, that does look like an issue, but I think you might have an easier solution available to you than the ones you're considering now: replace coroutine.create(). If the problem is that the new state isn't set up in LuaSkin, then just make a function that calls lua_newthread(), sets it up appropriately, and returns it to the caller.


Unfortunately, LuaSkin uses a shared singleton instance which contains a number of references to helper functions (for moving Objective-C data types into and out of lua as userdata) that get added at run time as lua modules are loaded -- in its current incarnation, it's not possible to create a new instance with just the stored reference to the lua_State being different.

I think we've been lucky so far that none of our users have seriously tried to use coroutines; but going forward, I think we need to, so... it's time for some thinking and probably some restructuring and recoding :-)

--
Aaron





Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Coda Highland


On Tue, Feb 18, 2020 at 12:05 PM Aaron Magill <[hidden email]> wrote:


On Feb 18, 2020, at 3:20 PM, Coda Highland <[hidden email]> wrote:

On Tue, Feb 18, 2020 at 8:53 AM Aaron Magill <[hidden email]> wrote:
(2) Every event loop system I know of has a way to create synthetic events. If you just want to yield to the event loop and get called right back after the pending events have been processed, just create an event that calls coroutine.resume whenever the Lua code yields.

That might be a reasonable approach if I hadn't just discovered that we have a fundamental issue with coroutines to begin with in Hammerspoon: see https://github.com/Hammerspoon/hammerspoon/issues/2306, if you want the nitty-gritty, but it boils down to any approach with leverages coroutines will have to wait until that's fixed first...

Ah, yeah, that does look like an issue, but I think you might have an easier solution available to you than the ones you're considering now: replace coroutine.create(). If the problem is that the new state isn't set up in LuaSkin, then just make a function that calls lua_newthread(), sets it up appropriately, and returns it to the caller.


Unfortunately, LuaSkin uses a shared singleton instance which contains a number of references to helper functions (for moving Objective-C data types into and out of lua as userdata) that get added at run time as lua modules are loaded -- in its current incarnation, it's not possible to create a new instance with just the stored reference to the lua_State being different.

I think we've been lucky so far that none of our users have seriously tried to use coroutines; but going forward, I think we need to, so... it's time for some thinking and probably some restructuring and recoding :-)

--
Aaron 

Couldn't you just keep track of the running thread so you know which stack you need to address?

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Eric Wing
In reply to this post by Aaron Magill
Manually pumping the event loop on macOS can work in limited cases,
but can/will break down if your application uses/interacts other
system frameworks that depend on the native event loop. Apple has
private implementation details in their event loop pump which can't be
recreated through public APIs and some of their own implementations
seem to depend on these things (or have some kind of implicit
assumptions). For example, things with modal windows, file dialogs,
and menu bar interaction may no longer work correctly. I remember a
very old bug where Game Center just wouldn't work.

So generally speaking, you should not manually pump the event loop on
Apple. If you do, it should be for a very limited, and very specific
time, and your app should be avoiding a lot of native AppKit/system
related features, and your application should resume using the normal
Apple runloop.

Also remember that, when you manually pump the event loop, other event
responder code that may have been written elsewhere may completely
miss seeing the events if they occurred while you were manually
pumping the event loop.


Also, a note about the main thread...this is probably the correct
thing in general. If you are calling any AppKit (GUI) APIs, they must
be on the main thread. My impression of Hammerspoon was that this is
what it is doing, so the design of it being on the main thread is the
correct one. If you want to use Lua on a background thread, make sure
that that thread never calls AppKit APIs or anything else that is not
background-thread-safe on Apple.


Personally, I've seen too many projects get into trouble manually
pumping the event loop and using background threads incorrectly on
Apple. (Coincidentally, I recently got contracted to help fix these
class of problems in the Mac port of a very popular game.)

Since coroutines have been mentioned here, they sound like a much
better path to go down to me.

-Eric

Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Coda Highland


On Wed, Feb 19, 2020 at 1:16 PM Eric Wing <[hidden email]> wrote:
Manually pumping the event loop on macOS can work in limited cases,
but can/will break down if your application uses/interacts other
system frameworks that depend on the native event loop. Apple has
private implementation details in their event loop pump which can't be
recreated through public APIs and some of their own implementations
seem to depend on these things (or have some kind of implicit
assumptions). For example, things with modal windows, file dialogs,
and menu bar interaction may no longer work correctly. I remember a
very old bug where Game Center just wouldn't work.

I'm well aware of that fact, but... who's suggesting doing that? Queueing up continuations on the event loop is a far better way of going about it.

/s/ Adam 
Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Eric Wing
On 2/19/20, Coda Highland <[hidden email]> wrote:

> On Wed, Feb 19, 2020 at 1:16 PM Eric Wing <[hidden email]> wrote:
>
>> Manually pumping the event loop on macOS can work in limited cases,
>> but can/will break down if your application uses/interacts other
>> system frameworks that depend on the native event loop. Apple has
>> private implementation details in their event loop pump which can't be
>> recreated through public APIs and some of their own implementations
>> seem to depend on these things (or have some kind of implicit
>> assumptions). For example, things with modal windows, file dialogs,
>> and menu bar interaction may no longer work correctly. I remember a
>> very old bug where Game Center just wouldn't work.
>>
>
> I'm well aware of that fact, but... who's suggesting doing that? Queueing
> up continuations on the event loop is a far better way of going about it.
>

The op's proposed code using NSApp nextEventMatchingMask is manually
pumping the event loop.

Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Aaron Magill


On Feb 19, 2020, at 9:25 PM, Eric Wing <[hidden email]> wrote:

On 2/19/20, Coda Highland <[hidden email]> wrote:
On Wed, Feb 19, 2020 at 1:16 PM Eric Wing <[hidden email]> wrote:

Manually pumping the event loop on macOS can work in limited cases,
but can/will break down if your application uses/interacts other
system frameworks that depend on the native event loop. Apple has
private implementation details in their event loop pump which can't be
recreated through public APIs and some of their own implementations
seem to depend on these things (or have some kind of implicit
assumptions). For example, things with modal windows, file dialogs,
and menu bar interaction may no longer work correctly. I remember a
very old bug where Game Center just wouldn't work.


I'm well aware of that fact, but... who's suggesting doing that? Queueing
up continuations on the event loop is a far better way of going about it.


The op's proposed code using NSApp nextEventMatchingMask is manually
pumping the event loop.

Looking deeper coroutines as a possible alternative, I discovered that they are currently fundamentally broken in Hammerspoon, so working on fixing that little snafu atm... one crisis at a time, please :-)

--
Aaron
Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Robert Burke
On Thu, Feb 20, 2020 at 6:53 AM Aaron Magill <[hidden email]> wrote:
>
> Looking deeper coroutines as a possible alternative, I discovered that they are currently fundamentally broken in Hammerspoon, so working on fixing that little snafu atm... one crisis at a time, please :-)
>
> --
> Aaron

If you would like an additional crisis, and the original question is
about Lua code that ends up running slowly on its own (not because it
is waiting for a lot of I/O or remote operations that are inherently
slow), another option is to try using luajit. I understand this is
also very hard in Hammerspoon's current setup.

Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Coda Highland


On Wed, Feb 19, 2020 at 8:12 PM Robert Burke <[hidden email]> wrote:
On Thu, Feb 20, 2020 at 6:53 AM Aaron Magill <[hidden email]> wrote:
>
> Looking deeper coroutines as a possible alternative, I discovered that they are currently fundamentally broken in Hammerspoon, so working on fixing that little snafu atm... one crisis at a time, please :-)
>
> --
> Aaron

If you would like an additional crisis, and the original question is
about Lua code that ends up running slowly on its own (not because it
is waiting for a lot of I/O or remote operations that are inherently
slow), another option is to try using luajit. I understand this is
also very hard in Hammerspoon's current setup.

It's basically out of the question; Hammerspoon is uses 5.3, so LuaJIT won't necessarily be compatible with the scripts.

Also embedding LuaJIT on macOS is a headache.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Chris Smith

> On 20 Feb 2020, at 03:05, Coda Highland <[hidden email]> wrote:
>
> Also embedding LuaJIT on macOS is a headache.


Really?  I tend not to use LuaJIT, but I’ve done this before and I seem to recall it just needed pagezero_size and image_base linker flags to work.  What else don’t I know that might bite me?

Regards,
Chris

Chris Smith <[hidden email]>

Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Coda Highland


On Thu, Feb 20, 2020 at 2:27 AM Chris Smith <[hidden email]> wrote:

> On 20 Feb 2020, at 03:05, Coda Highland <[hidden email]> wrote:
>
> Also embedding LuaJIT on macOS is a headache.


Really?  I tend not to use LuaJIT, but I’ve done this before and I seem to recall it just needed pagezero_size and image_base linker flags to work.  What else don’t I know that might bite me?

Regards,
Chris

Those linker flags only work for applications. If you're linking LuaJIT into a library/plugin and you don't control the application loading it, you're out of luck. That's why I had to abandon it.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: Embedded lua and yielding to the application run loop (macOS)

Sam Putman


On Thu, Feb 20, 2020 at 3:09 AM Coda Highland <[hidden email]> wrote:


On Thu, Feb 20, 2020 at 2:27 AM Chris Smith <[hidden email]> wrote:

> On 20 Feb 2020, at 03:05, Coda Highland <[hidden email]> wrote:
>
> Also embedding LuaJIT on macOS is a headache.


Really?  I tend not to use LuaJIT, but I’ve done this before and I seem to recall it just needed pagezero_size and image_base linker flags to work.  What else don’t I know that might bite me?

Regards,
Chris

Those linker flags only work for applications. If you're linking LuaJIT into a library/plugin and you don't control the application loading it, you're out of luck. That's why I had to abandon it.

/s/ Adam

Mike Pall said on the LuaJIT mailing list, last week, that this is obsolete as of the current v2.1 branch, which uses GC64 mode by default.

I admittedly haven't been able to get this to work in my own application, but I'm confident that's user error...

cheers,
-Sam.