FFI and Objective C

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

FFI and Objective C

Chris Jones
Hey folks

Has anyone out there successfully managed to get some kind of FFI module talking to Objective C?

I had a quick go with Facebook's fork of luaffi, and luapower's objc.lua, but all I got was segfaults when introspecting ObjC object properties .

--
Cheers,

Chris
Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Simon Cozens
On 18/11/2016 21:19, Chris Jones wrote:
> Has anyone out there successfully managed to get some kind of FFI
> module talking to Objective C?

Yep, my NSLua (https://github.com/simoncozens/NSLua - heavily based on
EasyLua) gives you a bidirectional bridge between Lua and ObjC.

What exactly are you looking for? Are you embedding Lua in an ObjC app,
or something else?

S

Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Chris Jones
Hey

> On 18 Nov 2016, at 10:24, Simon Cozens <[hidden email]> wrote:
> Yep, my NSLua (https://github.com/simoncozens/NSLua - heavily based on
> EasyLua) gives you a bidirectional bridge between Lua and ObjC.
>
> What exactly are you looking for? Are you embedding Lua in an ObjC app,
> or something else?

Thanks for the swift reply!

I currently maintain Hammerspoon (http://www.hammerspoon.org/) which uses Lua to expose a whole raft of OS X APIs to users (window management being probably the most used part).

Right now I'm investigating how reasonable it would be to allow users to interact directly with ObjC objects from Lua (effectively allowing new OS API plugins to be written in pure Lua). What potentially makes this harder is that we already have a homegrown NSObject wrapper around Lua (https://github.com/Hammerspoon/hammerspoon/tree/master/LuaSkin), which provides some interfacing between the two, but not anything like FFI (mainly around converting data back and forth). I had looked at some of the Lua<->Cocoa bridges before, but since they generally also want to manage the lua_State object, it's not a simple drop-in thing :)

Cheers,
Chris
Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Eric Wing
On 11/18/16, Chris Jones <[hidden email]> wrote:

> Hey
>
>> On 18 Nov 2016, at 10:24, Simon Cozens <[hidden email]> wrote:
>> Yep, my NSLua (https://github.com/simoncozens/NSLua - heavily based on
>> EasyLua) gives you a bidirectional bridge between Lua and ObjC.
>>
>> What exactly are you looking for? Are you embedding Lua in an ObjC app,
>> or something else?
>
> Thanks for the swift reply!
>
> I currently maintain Hammerspoon (http://www.hammerspoon.org/) which uses
> Lua to expose a whole raft of OS X APIs to users (window management being
> probably the most used part).
>
> Right now I'm investigating how reasonable it would be to allow users to
> interact directly with ObjC objects from Lua (effectively allowing new OS
> API plugins to be written in pure Lua). What potentially makes this harder
> is that we already have a homegrown NSObject wrapper around Lua
> (https://github.com/Hammerspoon/hammerspoon/tree/master/LuaSkin), which
> provides some interfacing between the two, but not anything like FFI (mainly
> around converting data back and forth). I had looked at some of the
> Lua<->Cocoa bridges before, but since they generally also want to manage the
> lua_State object, it's not a simple drop-in thing :)
>
> Cheers,
> Chris
>

You might also check out LuaCocoa. It uses libffi under the hood for
some things. Maybe it could give you some ideas.

Or maybe you might be able to use it. While not perfect (some things I
would like to clean up), LuaCocoa does very hard to adhere to the Lua
philosophy of "mechanisms, not policy" and not impose on you to
conform to it (be a library and not a framework). LuaCocoa does allow
you to use a lua_State that you created yourself.

LuaCocoa uses BridgeSupport (which is part of Mac OS X) to fully and
automatically and correctly bridge all APIs on the system.

http://playcontrol.net/opensource/LuaCocoa/

I gave a talk on its design last month at the Lua Workshop. The videos
are supposed to go up soon.

-Eric

Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Chris Jones
Hey Eric

Thanks for the reply. I hadn't spotted that LuaCocoa could use a pre-existing state. I'm having a quick go at integrating it into Hammerspoon at the moment, although there are some compatibility issues to work through since we're using Lua 5.3 :)

Many thanks,
Chris

On 18 November 2016 at 13:26, Eric Wing <[hidden email]> wrote:
On 11/18/16, Chris Jones <[hidden email]> wrote:
> Hey
>
>> On 18 Nov 2016, at 10:24, Simon Cozens <[hidden email]> wrote:
>> Yep, my NSLua (https://github.com/simoncozens/NSLua - heavily based on
>> EasyLua) gives you a bidirectional bridge between Lua and ObjC.
>>
>> What exactly are you looking for? Are you embedding Lua in an ObjC app,
>> or something else?
>
> Thanks for the swift reply!
>
> I currently maintain Hammerspoon (http://www.hammerspoon.org/) which uses
> Lua to expose a whole raft of OS X APIs to users (window management being
> probably the most used part).
>
> Right now I'm investigating how reasonable it would be to allow users to
> interact directly with ObjC objects from Lua (effectively allowing new OS
> API plugins to be written in pure Lua). What potentially makes this harder
> is that we already have a homegrown NSObject wrapper around Lua
> (https://github.com/Hammerspoon/hammerspoon/tree/master/LuaSkin), which
> provides some interfacing between the two, but not anything like FFI (mainly
> around converting data back and forth). I had looked at some of the
> Lua<->Cocoa bridges before, but since they generally also want to manage the
> lua_State object, it's not a simple drop-in thing :)
>
> Cheers,
> Chris
>

You might also check out LuaCocoa. It uses libffi under the hood for
some things. Maybe it could give you some ideas.

Or maybe you might be able to use it. While not perfect (some things I
would like to clean up), LuaCocoa does very hard to adhere to the Lua
philosophy of "mechanisms, not policy" and not impose on you to
conform to it (be a library and not a framework). LuaCocoa does allow
you to use a lua_State that you created yourself.

LuaCocoa uses BridgeSupport (which is part of Mac OS X) to fully and
automatically and correctly bridge all APIs on the system.

http://playcontrol.net/opensource/LuaCocoa/

I gave a talk on its design last month at the Lua Workshop. The videos
are supposed to go up soon.

-Eric




--
Cheers,

Chris
Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Eric Wing
On 11/18/16, Chris Jones <[hidden email]> wrote:

> Hey Eric
>
> Thanks for the reply. I hadn't spotted that LuaCocoa could use a
> pre-existing state. I'm having a quick go at integrating it into
> Hammerspoon at the moment, although there are some compatibility issues to
> work through since we're using Lua 5.3 :)
>
> Many thanks,
> Chris
>
> On 18 November 2016 at 13:26, Eric Wing <[hidden email]> wrote:
>
>> On 11/18/16, Chris Jones <[hidden email]> wrote:
>> > Hey
>> >
>> >> On 18 Nov 2016, at 10:24, Simon Cozens <[hidden email]> wrote:
>> >> Yep, my NSLua (https://github.com/simoncozens/NSLua - heavily based on
>> >> EasyLua) gives you a bidirectional bridge between Lua and ObjC.
>> >>
>> >> What exactly are you looking for? Are you embedding Lua in an ObjC
>> >> app,
>> >> or something else?
>> >
>> > Thanks for the swift reply!
>> >
>> > I currently maintain Hammerspoon (http://www.hammerspoon.org/) which
>> uses
>> > Lua to expose a whole raft of OS X APIs to users (window management
>> > being
>> > probably the most used part).
>> >
>> > Right now I'm investigating how reasonable it would be to allow users
>> > to
>> > interact directly with ObjC objects from Lua (effectively allowing new
>> > OS
>> > API plugins to be written in pure Lua). What potentially makes this
>> harder
>> > is that we already have a homegrown NSObject wrapper around Lua
>> > (https://github.com/Hammerspoon/hammerspoon/tree/master/LuaSkin), which
>> > provides some interfacing between the two, but not anything like FFI
>> (mainly
>> > around converting data back and forth). I had looked at some of the
>> > Lua<->Cocoa bridges before, but since they generally also want to
>> > manage
>> the
>> > lua_State object, it's not a simple drop-in thing :)
>> >
>> > Cheers,
>> > Chris
>> >
>>
>> You might also check out LuaCocoa. It uses libffi under the hood for
>> some things. Maybe it could give you some ideas.
>>
>> Or maybe you might be able to use it. While not perfect (some things I
>> would like to clean up), LuaCocoa does very hard to adhere to the Lua
>> philosophy of "mechanisms, not policy" and not impose on you to
>> conform to it (be a library and not a framework). LuaCocoa does allow
>> you to use a lua_State that you created yourself.
>>
>> LuaCocoa uses BridgeSupport (which is part of Mac OS X) to fully and
>> automatically and correctly bridge all APIs on the system.
>>
>> http://playcontrol.net/opensource/LuaCocoa/
>>
>> I gave a talk on its design last month at the Lua Workshop. The videos
>> are supposed to go up soon.
>>
>> -Eric
>>
>>
>
>
> --
> Cheers,
>
> Chris
>


I've been interested in moving to Lua 5.3. So I'd love to incorporate
your changes. Lua 5.3 integers removes the need for the LNUM patch I
currently use.

I have an updated version of my Lua patch (now for 5.3) to make it use
Objective-C exception handling for pcall. Just to get things basically
working, you won't need it immediately, but you'll want it for any
real shipping product on any Apple platforms. My new one still isn't
heavily tested so I haven't posted it yet. But when you get to that
point, let me know.

FYI, until the video gets posted, these are the slides from my
LuaCocoa talk at the Lua Workshop 2016.
http://playcontrol.net/tempdownload/video/LuaCocoa.pdf

-Eric

Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Chris Jones
Hey

On 19 November 2016 at 05:22, Eric Wing <[hidden email]> wrote:
I've been interested in moving to Lua 5.3. So I'd love to incorporate
your changes. Lua 5.3 integers removes the need for the LNUM patch I
currently use.

I now have a fairly hacky conversion that appears to pass the TestApp, and AFAICS, runs all of the examples correctly, with the possible exception of the Action 4 button in BlocksExample (which just seems to hang after quite a few block invocations). I've not had a chance to dig into that yet.

I've tried to avoid keeping 5.1 API calls around via #defines where I could, but e.g. {get,set}fenv() are currently being handled with a mixture of #define and some subsequent boilerplate in the getfenv() case. This would be pretty easy to change

There's also something weird going on with my whitespace, annoyingly!

Excluding the diff from replacing Lua 5.1 src with 5.3, the diffstat is pretty reasonable:  15 files changed, 218 insertions(+), 128 deletions(-)

I've pushed up my changes to: https://bitbucket.org/cmsj/luacocoa - happy to get whatever feedback you have :)
 
I have an updated version of my Lua patch (now for 5.3) to make it use
Objective-C exception handling for pcall. Just to get things basically

I'm curious about this part. I didn't pay any attention to it for this first stage. What does it do?

--
Cheers,

Chris
Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Eric Wing
On 11/22/16, Chris Jones <[hidden email]> wrote:

> Hey
>
> On 19 November 2016 at 05:22, Eric Wing <[hidden email]> wrote:
>>
>> I've been interested in moving to Lua 5.3. So I'd love to incorporate
>> your changes. Lua 5.3 integers removes the need for the LNUM patch I
>> currently use.
>>
>
> I now have a fairly hacky conversion that appears to pass the TestApp, and
> AFAICS, runs all of the examples correctly, with the possible exception of
> the Action 4 button in BlocksExample (which just seems to hang after quite
> a few block invocations). I've not had a chance to dig into that yet.
>

Fantastic!

Action 4 is known to deadlock so don't worry about it. (There is a
comment in the Lua code about it.) Basically this is the Cocoa API
trying to concurrently operate on each element in the array. I tried
some tricks to try to deal with this, but they were unsuccessful. I'm
thinking this may only be solvable with a global interpretor lock, but
I really don't want to do that. I was holding out hope there may be
some other libdispatch trick. For now, these Cocoa APIs have a
parameter to say to not do things concurrently which avoids this
entire problem.


> I've tried to avoid keeping 5.1 API calls around via #defines where I
> could, but e.g. {get,set}fenv() are currently being handled with a mixture
> of #define and some subsequent boilerplate in the getfenv() case. This
> would be pretty easy to change

My only thinking here is it would be nice if we could make it easy for
somebody to switch back to 5.1. Though honestly, at this point in
time, I don't expect there to be many changes in LuaCocoa and I don't
really expect anybody new to ask for 5.1 at this point.


> There's also something weird going on with my whitespace, annoyingly!
>

I probably should just use clang format.

> Excluding the diff from replacing Lua 5.1 src with 5.3, the diffstat is
> pretty reasonable:  15 files changed, 218 insertions(+), 128 deletions(-)
>
> I've pushed up my changes to: https://bitbucket.org/cmsj/luacocoa - happy
> to get whatever feedback you have :)
>
>> I have an updated version of my Lua patch (now for 5.3) to make it use
>> Objective-C exception handling for pcall. Just to get things basically
>>
>
> I'm curious about this part. I didn't pay any attention to it for this
> first stage. What does it do?
>


My patch modifies Lua so it will use Objective-C's exception handling
mechanism instead of _setjmp/_longjmp for pcall.

This protects you in case something throws an exception on the system
when you are in a pcall. This could happen if either a user or system
library throws an exception. I’ve also seen it happen in cases where
there was an internal Apple bug in one of the system frameworks and it
threw an exception.

With setjmp/longjmp, Lua has no idea that an exception was thrown and
the execution jumped out of the Lua VM. This could lead to unexpected
behavior, memory leaks, and even crashes.

Switching to Objective-C's exception handling mechansism allows Lua to catch
all exceptions during a lua_pcall also gives the ability for Lua script writers
to trap/handle Obj-C exceptions from their own pcall's.


There are also two bonuses:

- On the “modern” Objective-C runtime, Obj-C and C++ exceptions are
unified, so this can be used for C++ exceptions too.

- The “modern” Obj-C runtime uses “zero-cost exceptions”, which means
the @try setup is essentially free (performance benefit for pcall)


So generally everybody should be using this pcall/Obj-C exception
patch on all Apple platforms (macOS, iOS, tvOS, watchOS). It is the
most correct code for Apple platforms, it has a bunch of benefits, and
no drawbacks (except that it cannot be used for Mac OS X 10.4 or
below, though it is now extremely hard to even find Macs that can run
that low since new Macs cannot run anything lower than they ship
with.).



I've pushed up a new working repo with your changes and my pcall/Obj-C
patch applied here:
https://bitbucket.org/ewing/luacocoa53

It's not really tested...probably should add some tests to LuaCocoa for this.


I also made a few fixes to the Xcode project and moved the defines to
the preprocessor section.

One additional thought, I would like to try to stay as close to the
Lua 5.3 default definitions as possible (just to simplify people
having to remember what definitions were used). I don't think
LUA_COMPAT_MODULE is a default, so it might be nice to work towards
removing this.

Thanks,
Eric

Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Daurnimator
On 24 November 2016 at 01:08, Eric Wing <[hidden email]> wrote:
> Action 4 is known to deadlock so don't worry about it. (There is a
> comment in the Lua code about it.) Basically this is the Cocoa API
> trying to concurrently operate on each element in the array. I tried
> some tricks to try to deal with this, but they were unsuccessful. I'm
> thinking this may only be solvable with a global interpretor lock, but
> I really don't want to do that. I was holding out hope there may be
> some other libdispatch trick. For now, these Cocoa APIs have a
> parameter to say to not do things concurrently which avoids this
> entire problem.

I mentioned it to you at the workshop, did you get to look at
http://daurnimator.com/post/147024385399/using-your-own-main-loop-on-osx

Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Chris Jones
In reply to this post by Eric Wing
Hey Eric

On 23 November 2016 at 14:08, Eric Wing <[hidden email]> wrote:
My only thinking here is it would be nice if we could make it easy for
somebody to switch back to 5.1. Though honestly, at this point in
time, I don't expect there to be many changes in LuaCocoa and I don't
really expect anybody new to ask for 5.1 at this point.

I think it could mostly all be handled with #ifdefs. Your call - I don't ever plan on going back to 5.1, but I know a lot of people do still use it.
 
This protects you in case something throws an exception on the system
when you are in a pcall. This could happen if either a user or system

Oh that's very cool! Thanks for the explanation :)
 
One additional thought, I would like to try to stay as close to the
Lua 5.3 default definitions as possible (just to simplify people
having to remember what definitions were used). I don't think
LUA_COMPAT_MODULE is a default, so it might be nice to work towards
removing this.

So I set that mainly so it would make the changes to luaL_register() call sites as small as possible. If I understand LuaCocoa correctly, the various Function/Selector/Struct/Subclass parts of the code, use luaL_register() (and now luaL_openlib()) to layer up a single "LuaCocoa" table with all of their functions.

This could certainly be done in other ways - possibly the simplest being to pull over the functions that LUA_COMPAT_MODULE includes, to LuaCocoa's source.

I'm not sure I grok the complete mechanisms going on there, to rewrite the Function/Selector/Struct/Subclass code to do this in a different way, but as with the 5.1 compatibility, it's your project, so I defer to your judgement :)

Thanks very much for your feedback!

--
Cheers,

Chris
Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Eric Wing
In reply to this post by Daurnimator
On 11/23/16, Daurnimator <[hidden email]> wrote:

> On 24 November 2016 at 01:08, Eric Wing <[hidden email]> wrote:
>> Action 4 is known to deadlock so don't worry about it. (There is a
>> comment in the Lua code about it.) Basically this is the Cocoa API
>> trying to concurrently operate on each element in the array. I tried
>> some tricks to try to deal with this, but they were unsuccessful. I'm
>> thinking this may only be solvable with a global interpretor lock, but
>> I really don't want to do that. I was holding out hope there may be
>> some other libdispatch trick. For now, these Cocoa APIs have a
>> parameter to say to not do things concurrently which avoids this
>> entire problem.
>
> I mentioned it to you at the workshop, did you get to look at
> http://daurnimator.com/post/147024385399/using-your-own-main-loop-on-osx
>
>

Thank you for the link. As somebody who has had to fight frameworks
taking over event loops, that is interesting information. (Though
unfortunately, I think the Mac App Store rejects on private API use.)


But for LuaCocoa, there is no event loop. LuaCocoa tries very hard to
not force decisions on how you write your app, allowing it to be
integrated in all sort of environments. Ultimately, it is your app
that creates the run-loop.


The way blocks work with LuaCocoa is that when you create a Lua
function to be used as a block, LuaCocoa dynamically creates a generic
wrapper/proxy block (ffi closure). In this wrapper block, the Lua
function is stored. This wrapper block is what gets passed to the
native API accepting the block parameter.

So when the system calls-back this block, it is indeed this wrapper
block. My wrapper block has code which in turn calls the stored Lua
function.

One extra thing my code tries to do before it invokes the Lua
function, is track which thread we’re being invoked on on and compare
to the originating thread. If they are different, there is some awful
things I do to try to perform the call on the originating thread. For
typical cases, this seems to work, but for the concurrent stuff, I can
seem to run into deadlocks.


I infer that you prompting me about controlling the event loop may
allow me to pre-sort/schedule how to deal with blocks before the
callback is invoked which maybe could avoid the need to deal with the
problem after the callback has been invoked. I'm not particularly keen
on LuaCocoa taking control of the event-loop, but it is an interesting
idea in general.


Thanks,
Eric

Reply | Threaded
Open this post in threaded view
|

Re: FFI and Objective C

Eric Wing
In reply to this post by Chris Jones
On 11/23/16, Chris Jones <[hidden email]> wrote:

> Hey Eric
>
> On 23 November 2016 at 14:08, Eric Wing <[hidden email]> wrote:
>
>> My only thinking here is it would be nice if we could make it easy for
>> somebody to switch back to 5.1. Though honestly, at this point in
>> time, I don't expect there to be many changes in LuaCocoa and I don't
>> really expect anybody new to ask for 5.1 at this point.
>>
>
> I think it could mostly all be handled with #ifdefs. Your call - I don't
> ever plan on going back to 5.1, but I know a lot of people do still use it.
>


I’m fine with #ifdefs for this. The default path can be Lua 5.3.



>> This protects you in case something throws an exception on the system
>> when you are in a pcall. This could happen if either a user or system
>>
>
> Oh that's very cool! Thanks for the explanation :)
>
>
>> One additional thought, I would like to try to stay as close to the
>> Lua 5.3 default definitions as possible (just to simplify people
>> having to remember what definitions were used). I don't think
>> LUA_COMPAT_MODULE is a default, so it might be nice to work towards
>> removing this.
>>
>
> So I set that mainly so it would make the changes to luaL_register() call
> sites as small as possible. If I understand LuaCocoa correctly, the various
> Function/Selector/Struct/Subclass parts of the code, use luaL_register()
> (and now luaL_openlib()) to layer up a single "LuaCocoa" table with all of
> their functions.
>
> This could certainly be done in other ways - possibly the simplest being to
> pull over the functions that LUA_COMPAT_MODULE includes, to LuaCocoa's
> source.
>
> I'm not sure I grok the complete mechanisms going on there, to rewrite the
> Function/Selector/Struct/Subclass code to do this in a different way, but
> as with the 5.1 compatibility, it's your project, so I defer to your
> judgement :)
>
> Thanks very much for your feedback!

Removing the need for LUA_COMPAT_MODULE is a ‘nice to have’. A pipe
dream of mine is that LuaCocoa could eventually just use the stock Lua
distribution and do something like require(‘LuaCocoa’). The
LUA_COMPAT_MODULE is one more hurdle. So pulling over the functions
into LuaCocoa (and renaming them slightly to avoid duplicate symbols)
certainly sounds like the easiest thing to do. However, this isn’t a
requirement for me and is just another ‘nice to have’. Realistically,
LUA_COMPAT_MODULE won’t likely be a real issue for a long time (Lua
5.4?). But if you feel inclined to implement this, don’t let me stop
you :P

Thanks,
Eric