Lua distros again

classic Classic list List threaded Threaded
56 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Daurnimator
On 30 January 2018 at 08:14, Dibyendu Majumdar <[hidden email]> wrote:
> I am probably missing something in the openssl space.

luaossl? https://github.com/wahern/luaossl/

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
On 29 January 2018 at 21:24, Daurnimator <[hidden email]> wrote:
> On 30 January 2018 at 08:14, Dibyendu Majumdar <[hidden email]> wrote:
>> I am probably missing something in the openssl space.
>
> luaossl? https://github.com/wahern/luaossl/
>

Thank you - I will have a look at above.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
> I have always wanted to (but haven't managed to yet) bundle some high
> quality libraries with Ravi in a well tested combination with support
> for Windows, Linux and Mac OSX. Is there a list of the best essential
> libraries for Lua? I want to bundle a small set of high quality
> libraries that I will test with Ravi, rather than a huge set of
> untested libraries of varying quality.
>

Thank you all for the feedback. My shortlist now consists of:

- lpeg
- luafilesystem
- luasocket
- libuv (Luvit)
- libcurl (wrapper tbc)
- lua-cjson
- torch7
- luaossl
- cephes (wrapper tbc)
- luaffifb (port of LuaJIT FFI interface)

Well this list although short is probably going to keep me busy for
several months.

One of the realizations I had this year (thanks to some posts here) is
that we need to stop tweaking Lua (let Lua 5.4 be delayed by 10
years!) and instead focus on building an eco-system and community and
tooling. So that's going to be my focus for Ravi anyway - I have
merged the generational GC from Lua 5.4 but am pretty sure I will stop
there.

Thanks and Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dirk Laurie-2
2018-01-30 0:55 GMT+02:00 Dibyendu Majumdar <[hidden email]>:

> On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
>> I have always wanted to (but haven't managed to yet) bundle some high
>> quality libraries with Ravi in a well tested combination with support
>> for Windows, Linux and Mac OSX. Is there a list of the best essential
>> libraries for Lua? I want to bundle a small set of high quality
>> libraries that I will test with Ravi, rather than a huge set of
>> untested libraries of varying quality.
>>
>
> Thank you all for the feedback. My shortlist now consists of:
>
> - lpeg
> - luafilesystem
> - luasocket
> - libuv (Luvit)
> - libcurl (wrapper tbc)
> - lua-cjson
> - torch7
> - luaossl
> - cephes (wrapper tbc)
> - luaffifb (port of LuaJIT FFI interface)
>
> Well this list although short is probably going to keep me busy for
> several months.

That list demonstrates perfectly the problem I have with most
lua modules that people publish. With the exception of luafilesystem
and luasocket, the non-expert cannot tell what the package does
unless you already know the underlying software in another incarnation.
The name is either an acronym or a cute codename. We get coy
announcements on the list like "Version 0.7.0 of lua-bllsht, a set of
bindings for libbllsht,  has been released. Enjoy!"

So whatever else you do, Dibyendu, please add a short descriptive
phrase or sentence to each, e.g. "Call precompiled C libraries directly,
bypassing the Lua API".

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 29 January 2018 at 22:55, Dibyendu Majumdar <[hidden email]> wrote:

> Thank you all for the feedback. My shortlist now consists of:
>
> - lpeg
> - luafilesystem
> - luasocket
> - libuv (Luvit)
> - libcurl (wrapper tbc)
> - lua-cjson
> - torch7
> - luaossl
> - cephes (wrapper tbc)
> - luaffifb (port of LuaJIT FFI interface)
>

I have been thinking about how to go about with this. I want to create
a harmonious whole on the one hand, on the other, want the ability to
easily pull changes from the upstream projects.

Seems like the Luadist model is a good fit:

a) Fork an upstream project (possibly create a branch called 'Ravi' as
I expect to have to tweak some aspects of the project).
b) Create a submodule in the distro project for each upstream project.
c) Hook it all up in a CMake based build system.

My goal is not to create a generic package manager for Lua. Nor to
make it easy for users to pull in additional modules easily. Instead
the goals are:

1. Well tested and supported product on Windows, Linux and Mac OS -
all supported modules installed up-front.
2. Binary distros for each supported platform (native installers would
be nice but I don't know how to do that so initially it may be in the
form of a self contained compressed archive).
3. The installation should not require any system privileges, and
should just work.
4. Long term support and commitment for the distro (i.e. best effort
to fix bugs and incorporate additional high quality libs) - however
this does not imply Ravi will keep up with Lua changes, because to me
100% backwards compatibility is of paramount importance. I may
selectively apply changes that are non disruptive.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Russell Haley
On Thu, Feb 1, 2018 at 1:23 PM, Dibyendu Majumdar
<[hidden email]> wrote:

> On 29 January 2018 at 22:55, Dibyendu Majumdar <[hidden email]> wrote:
>> Thank you all for the feedback. My shortlist now consists of:
>>
>> - lpeg
>> - luafilesystem
>> - luasocket
>> - libuv (Luvit)
>> - libcurl (wrapper tbc)
>> - lua-cjson
>> - torch7
>> - luaossl
>> - cephes (wrapper tbc)
>> - luaffifb (port of LuaJIT FFI interface)
>>
>
> I have been thinking about how to go about with this. I want to create
> a harmonious whole on the one hand, on the other, want the ability to
> easily pull changes from the upstream projects.
>
> Seems like the Luadist model is a good fit:
>
> a) Fork an upstream project (possibly create a branch called 'Ravi' as
> I expect to have to tweak some aspects of the project).
> b) Create a submodule in the distro project for each upstream project.
> c) Hook it all up in a CMake based build system.
>
> My goal is not to create a generic package manager for Lua. Nor to
> make it easy for users to pull in additional modules easily. Instead
> the goals are:
>
> 1. Well tested and supported product on Windows, Linux and Mac OS -
> all supported modules installed up-front.
> 2. Binary distros for each supported platform (native installers would
> be nice but I don't know how to do that so initially it may be in the
> form of a self contained compressed archive).
> 3. The installation should not require any system privileges, and
> should just work.
> 4. Long term support and commitment for the distro (i.e. best effort
> to fix bugs and incorporate additional high quality libs) - however
> this does not imply Ravi will keep up with Lua changes, because to me
> 100% backwards compatibility is of paramount importance. I may
> selectively apply changes that are non disruptive.
>
> Regards
>

Josh Jensen has done much of this with LuaPlus too:

https://github.com/jjensen/luaplus51-all

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Joshua Jensen-2
Russell Haley wrote on 2/1/2018 2:40 PM:

> On Thu, Feb 1, 2018 at 1:23 PM, Dibyendu Majumdar
> <[hidden email]> wrote:
>> I have been thinking about how to go about with this. I want to create
>> a harmonious whole on the one hand, on the other, want the ability to
>> easily pull changes from the upstream projects.
>>
>> Seems like the Luadist model is a good fit:
>>
>> a) Fork an upstream project (possibly create a branch called 'Ravi' as
>> I expect to have to tweak some aspects of the project).
>> b) Create a submodule in the distro project for each upstream project.
>> c) Hook it all up in a CMake based build system.
>>
>> My goal is not to create a generic package manager for Lua. Nor to
>> make it easy for users to pull in additional modules easily. Instead
>> the goals are:
>>
>> 1. Well tested and supported product on Windows, Linux and Mac OS -
>> all supported modules installed up-front.
>> 2. Binary distros for each supported platform (native installers would
>> be nice but I don't know how to do that so initially it may be in the
>> form of a self contained compressed archive).
>> 3. The installation should not require any system privileges, and
>> should just work.
>> 4. Long term support and commitment for the distro (i.e. best effort
>> to fix bugs and incorporate additional high quality libs) - however
>> this does not imply Ravi will keep up with Lua changes, because to me
>> 100% backwards compatibility is of paramount importance. I may
>> selectively apply changes that are non disruptive.
> Josh Jensen has done much of this with LuaPlus too:
>
> https://github.com/jjensen/luaplus51-all
What LuaPlus is to me and what it does:

1) It's a tool set that has been around for a long time. It has been
used in some very large applications and games over the years. It is
also used as a tight Lua distribution for tools and scripting within
those places. It's mature, it just works, and that makes me happy.
2) It bootstraps across multiple OSes out the gate... the build system,
the Lua version, the Lua modules. I'm all kinds of giddy happy about that.
3) It supports multiple Lua distributions. Lua 5.1 is never going to go
away, but people use Lua 5.2 and 5.3, too. It gets version updates when
I have time or when they're contributed. I think I'm behind on Lua 5.3.x
a bit right now. I'm not happy about that, but having it all in one
place is a happy moment.
4) It has enhancements to Lua; that's why it is called LuaPlus. They're
nicely stored in different directories than the actual Lua versions,
also supported. The LuaPlus enhancements make me happy.
5) It has a bunch of Lua modules that I (and others) regularly use, and
the versions of those modules embedded in the Git repository work in
concert with one another. lrexlib is not at latest, for instance,
because there were some major API changes once, and they broke a ton of
production scripts at an organization. There also might be a submodule
or two, but using them is not a priority. When a Lua module needs a
tweak for better behavior (compilation issue/bugfix/Lua 5.3
support/whatever), I do it right there. I manually upgrade Lua modules
with WinMerge when needed. All of this is happiness-inducing.
6) I do not use nor care to use CMake nor are my personal opinions of
many years of CMake usage valid here. I use an enhanced version of
Perforce's Jam build system to build up the LuaPlus distribution. It's
called JamPlus. It does all kinds of cool stuff. I like it, I use it,
and I wish the world would use it. In any case, JamPlus as LuaPlus'
build system also makes me happy.
7) Binaries can be easily built against Debug or Release versions for
separate compilers on your own system where it is easy to debug. The Lua
executable and the Lua modules are all built against the same compiler,
so it is spiffy keen and easy to debug them. Oh, it also does the whole
rpath thing on a Unix-y file system. << Happy about all of that.
8) It is stable. Stable = Good. Good = less stress = more happiness.

If any of these serve as
strategies/policies/workflows/arrangements/whatever for creating your
Ravi distribution, then awesome. Steal as you see fit.

If it seems like a bunch of mumbo jumbo and brouhaha, then you're
probably right. But hey, it's my mumbo jumbo. I like it, some others
like it, and I and others get to help evolve it into forms that fit our
needs, and that's really neat, especially to have it last as long as it has.

Anyway, off my soapbox for now.

-Josh

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
On 1 February 2018 at 23:11, Joshua Jensen <[hidden email]> wrote:

>> https://github.com/jjensen/luaplus51-all
>
> What LuaPlus is to me and what it does:
>
> 5) It has a bunch of Lua modules that I (and others) regularly use, and the
> versions of those modules embedded in the Git repository work in concert
> with one another. lrexlib is not at latest, for instance, because there were
> some major API changes once, and they broke a ton of production scripts at
> an organization. There also might be a submodule or two, but using them is
> not a priority. When a Lua module needs a tweak for better behavior
> (compilation issue/bugfix/Lua 5.3 support/whatever), I do it right there. I
> manually upgrade Lua modules with WinMerge when needed. All of this is
> happiness-inducing.

I thought about copying the projects into my own repository like you
have done, but it seems that one can achieve the same thing by forking
a project and referencing it. The main advantage is that all the
original history is retained.

> 6) I do not use nor care to use CMake nor are my personal opinions of many
> years of CMake usage valid here. I use an enhanced version of Perforce's Jam
> build system to build up the LuaPlus distribution. It's called JamPlus. It
> does all kinds of cool stuff. I like it, I use it, and I wish the world
> would use it. In any case, JamPlus as LuaPlus' build system also makes me
> happy.

Looks like you have put in a lot of effort into this ... despite
CMake's issues it seems the right tool to me at this time.

> If any of these serve as strategies/policies/workflows/arrangements/whatever
> for creating your Ravi distribution, then awesome. Steal as you see fit.
>

Thank you - I will learn from your experiences. If you could point out
what the pain points were that would be very helpful.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Paige DePol
Dibyendu Majumdar <[hidden email]> wrote:

> On 1 February 2018 at 23:11, Joshua Jensen <[hidden email]> wrote:
>>
>> 5) It has a bunch of Lua modules that I (and others) regularly use, and the
>> versions of those modules embedded in the Git repository work in concert
>> with one another. lrexlib is not at latest, for instance, because there were
>> some major API changes once, and they broke a ton of production scripts at
>> an organization. There also might be a submodule or two, but using them is
>> not a priority. When a Lua module needs a tweak for better behavior
>> (compilation issue/bugfix/Lua 5.3 support/whatever), I do it right there. I
>> manually upgrade Lua modules with WinMerge when needed. All of this is
>> happiness-inducing.
>
> I thought about copying the projects into my own repository like you
> have done, but it seems that one can achieve the same thing by forking
> a project and referencing it. The main advantage is that all the
> original history is retained.

Yes, preserving history is a great reason to fork repositories under GitHub.
According to some recent research[1] about 70% of all code on their platform
is duplicated via the fork mechanism. In reality even more code is probably
duplicated as they can't easily count code that is manually copy/pasted.

I have also forked LuaPlus for use in my own project as it does have some
functionality I will need at a future point as well. Thanks for sharing your
source Joshua, it is very appreciated!

One thing I did wonder about, on GitHub you have the URL http://luaplus.org/
listed, however, that URL just seems to redirect back to GitHub. Did you
have anything on that site in the past, or has it always just been a redir?


>> If any of these serve as strategies/policies/workflows/arrangements/whatever
>> for creating your Ravi distribution, then awesome. Steal as you see fit.
>
> Thank you - I will learn from your experiences. If you could point out
> what the pain points were that would be very helpful.

I am also interested in any extra information you may care to share!

~Paige

[1] https://www.theregister.co.uk/2017/11/21/github_duplicate_code/



Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Russell Haley
In reply to this post by Dibyendu Majumdar
‎Hi ‎Dibyendu,

You *really* need to look at JamPlus. Unless, of course, you don't want to script your builds with Lua...

Jamplus.org

Russ

Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
  Original Message  
From: Dibyendu Majumdar
Sent: Monday, January 29, 2018 2:56 PM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Re: Lua distros again

On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
> I have always wanted to (but haven't managed to yet) bundle some high
> quality libraries with Ravi in a well tested combination with support
> for Windows, Linux and Mac OSX. Is there a list of the best essential
> libraries for Lua? I want to bundle a small set of high quality
> libraries that I will test with Ravi, rather than a huge set of
> untested libraries of varying quality.
>

Thank you all for the feedback. My shortlist now consists of:

- lpeg
- luafilesystem
- luasocket
- libuv (Luvit)
- libcurl (wrapper tbc)
- lua-cjson
- torch7
- luaossl
- cephes (wrapper tbc)
- luaffifb (port of LuaJIT FFI interface)

Well this list although short is probably going to keep me busy for
several months.

One of the realizations I had this year (thanks to some posts here) is
that we need to stop tweaking Lua (let Lua 5.4 be delayed by 10
years!) and instead focus on building an eco-system and community and
tooling. So that's going to be my focus for Ravi anyway - I have
merged the generational GC from Lua 5.4 but am pretty sure I will stop
there.

Thanks and Regards
Dibyendu


Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
Hi Russell,

On 6 February 2018 at 16:42, Russell Haley <[hidden email]> wrote:
> You *really* need to look at JamPlus. Unless, of course, you don't want to script your builds with Lua...
>

No doubt JamPlus is a great product but life is too short for spending
time on build tools - I spend as little time as possible, and cannot
really look at new build tools as CMake works fine for me. Hope you
can find good use of JamPlus in your projects.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:

> I have always wanted to (but haven't managed to yet) bundle some high
> quality libraries with Ravi in a well tested combination with support
> for Windows, Linux and Mac OSX. Is there a list of the best essential
> libraries for Lua? I want to bundle a small set of high quality
> libraries that I will test with Ravi, rather than a huge set of
> untested libraries of varying quality.
>

I wanted to ask your views on couple of points:

1. Should I force all libraries to have a require string that is of
the format 'supplier.module' ? The problem in enforcing this is that
it makes the distro incompatible with existing code. A better approach
may be to use the require string that the module provides but prohibit
any other use of the same module path. This is easy as I have control
over which libraries go in.

2. Folks that have experience with modules implemented as shared
libraries - what are the gotchas? I found two potential issues
already:

a) in some cases the shared library needs to be on OS specific
environment PATH (or library PATH), and
b) the path resolution doesn't seem to like it if the build adds a
'lib' prefix to the library (as it does on Mac OSX).

Maybe these aren't issues and just my ignorance on how this is meant
to work - happy to be corrected.

Thanks and Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:

> On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
>
> > I have always wanted to (but haven't managed to yet) bundle some high
> > quality libraries with Ravi in a well tested combination with support
> > for Windows, Linux and Mac OSX. Is there a list of the best essential
> > libraries for Lua? I want to bundle a small set of high quality
> > libraries that I will test with Ravi, rather than a huge set of
> > untested libraries of varying quality.
> >
>
> I wanted to ask your views on couple of points:
>
> 1. Should I force all libraries to have a require string that is of
> the format 'supplier.module' ? The problem in enforcing this is that
> it makes the distro incompatible with existing code. A better approach
> may be to use the require string that the module provides but prohibit
> any other use of the same module path. This is easy as I have control
> over which libraries go in.

  What is the concern here?  So far, Lua has managed without a conherent
namespace for modules.  If Ravi is meant to run existing Lua code then you
pretty much *can't* enforce a new requirement.

  The stock Lua interpreter provides a mechanism but no real policy.  That
allows experimentation, but it also allows conflicting modules to exist.
Lua also doesn't care about versions (or version numbers) of modules.  The
ULua distribution [1] does enforce semantic versioning for any modules it
supports, so there are Lua distributions that are opinionated.

> 2. Folks that have experience with modules implemented as shared
> libraries - what are the gotchas? I found two potential issues
> already:
>
> a) in some cases the shared library needs to be on OS specific
> environment PATH (or library PATH), and

  Have you actually encountered this problem?  And if so, which Lua module
was it?

> b) the path resolution doesn't seem to like it if the build adds a
> 'lib' prefix to the library (as it does on Mac OSX).

  It sounds like you are still unclear on how dynamic linking works on Unix.

  Lua uses dlopen() (a POSIX function) to load Lua modules written in C.
Only in the case when the filename of a shared object does *NOT* contain a
'/' does dlopen() check the environment variable LD_LIBRARY_PATH (and only
if the program is not setuid or setgid).  If not found (or LD_LIBRARY_PATH
isn't set) does dlopen() then check system wide locations (under Linux, the
paths in /etc/ld.so.conf).

  The 'lib' prefix is a result of the C toolchain.  When you invoke the C
compiler like:

        cc -o myprog *.o -lfoo

it's the toolchain that adds the 'lib' to 'foo' and looks for 'libfoo.so'
(and if not found, 'libfoo.a').  If you don't specify the '-L' option, then
the system defaults are searched for 'libfoo.*'.  If a '-L' option *is*
given, said directories are searched first.

  Again, the whole 'lib' prefix is added by the C compiler toolchain.

  Also, shared objects can reference other shared objects.  

  What problems (errors, etc.) have you actually encounted with this issue?

  Have you tried writing a Lua module in C?  Even a simple one?  That might
answer most of your questions.

  -spc

[1] http://ulua.io/index.html

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
On 7 February 2018 at 23:30, Sean Conner <[hidden email]> wrote:
>> a) in some cases the shared library needs to be on OS specific
>> environment PATH (or library PATH), and
>
>   Have you actually encountered this problem?  And if so, which Lua module
> was it?

Yes of course as soon as I started applying this to the first
component in the proposed distro - luaffifb. I modified the library so
that it would support:

require 'ravi.ffi'

This library comes with another shared library used for testing - this
is referenced from test.lua as:

dlls.__cdecl = ffi.load(package.searchpath('ravi.ffi.libtest', package.cpath))

This immediately fails as 'libtest' has a dependency on 'ffi' shared
library but cannot find it. So one has to set following in the
environment:

export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:$HOME/lib/ravi

While this is maybe not a good example as 'libtest' is not a Lua
module per se - it immediately confirmed to me the theoretical
analysis that there is a potential problem.

>
>> b) the path resolution doesn't seem to like it if the build adds a
>> 'lib' prefix to the library (as it does on Mac OSX).
>
>   It sounds like you are still unclear on how dynamic linking works on Unix.

My knowledge is not perfect of course but I also did not explain the
problem fully.

In order for Ravi to find the libraries on Mac OSX I need to set something like:

export LUA_CPATH="$RAVI_HOME/lib/?.dylib;$RAVI_HOME/lib/lib?.dylib"

Now in my CMake build by default the library gets created as:

$HOME/ravi/lib/ravi/libffi.dylib

It appears that Lua searches for:

$HOME/ravi/lib/ravi/ffi.dylib as it substitutes 'ravi/ffi' in place of
'?'. So it cannot handle the 'lib' prefix.

I have amended the CMake script to omit the 'lib' prefix and now it
works fine, but this is not standard naming on Mac OSX I think.

>   Have you tried writing a Lua module in C?  Even a simple one?  That might
> answer most of your questions.
>

Of course - since my first interest in Lua. But I never used the
dotted require statement to access shared libraries before as I was
unaware of this - all my attempts were based on shared libraries being
'top-level' which worked fine.


Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:

>
> My knowledge is not perfect of course but I also did not explain the
> problem fully.
>
> In order for Ravi to find the libraries on Mac OSX I need to set something like:
>
> export LUA_CPATH="$RAVI_HOME/lib/?.dylib;$RAVI_HOME/lib/lib?.dylib"
>
> Now in my CMake build by default the library gets created as:
>
> $HOME/ravi/lib/ravi/libffi.dylib
>
> It appears that Lua searches for:
>
> $HOME/ravi/lib/ravi/ffi.dylib as it substitutes 'ravi/ffi' in place of
> '?'. So it cannot handle the 'lib' prefix.
>
> I have amended the CMake script to omit the 'lib' prefix and now it
> works fine, but this is not standard naming on Mac OSX I think.

  At work we develop on Mac OS-X, and while normally I package everything
into a single executable, I do have the ability to install individual
modules [1] and they all have a .so extension (just like they do on Linux
and Solaris) and Mac OS-X can load them (the Lua modules written in C) just
fine.  I did check other, standard shared libraries on the Mac, and yes,
they have the .dylib extension [2].  So for Lua modules I don't think it
matters all that much what the extension is.  

  -spc

[1] Just because.

[2] Learn something new every day.  Our build products are for the most
        part complete and any dynamic linking is against standard libraries
        that come with the OS like the C standard library, etc.  Since our
        production systems are Linux or Solaris, I never really bothered to
        look that deep into Mac development.

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
On 9 February 2018 at 08:08, Sean Conner <[hidden email]> wrote:

> It was thus said that the Great Dibyendu Majumdar once stated:
>>
>> My knowledge is not perfect of course but I also did not explain the
>> problem fully.
>>
>> In order for Ravi to find the libraries on Mac OSX I need to set something like:
>>
>> export LUA_CPATH="$RAVI_HOME/lib/?.dylib;$RAVI_HOME/lib/lib?.dylib"
>>
>> Now in my CMake build by default the library gets created as:
>>
>> $HOME/ravi/lib/ravi/libffi.dylib
>>
>> It appears that Lua searches for:
>>
>> $HOME/ravi/lib/ravi/ffi.dylib as it substitutes 'ravi/ffi' in place of
>> '?'. So it cannot handle the 'lib' prefix.
>>
>> I have amended the CMake script to omit the 'lib' prefix and now it
>> works fine, but this is not standard naming on Mac OSX I think.
>
>   At work we develop on Mac OS-X, and while normally I package everything
> into a single executable, I do have the ability to install individual
> modules [1] and they all have a .so extension (just like they do on Linux
> and Solaris) and Mac OS-X can load them (the Lua modules written in C) just
> fine.  I did check other, standard shared libraries on the Mac, and yes,
> they have the .dylib extension [2].  So for Lua modules I don't think it
> matters all that much what the extension is.
>

Sure but that was not the issue I was high lighting. Do you have a
solution for the problem I described above?

Thanks and Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:

> On 9 February 2018 at 08:08, Sean Conner <[hidden email]> wrote:
> > It was thus said that the Great Dibyendu Majumdar once stated:
> >>
> >> My knowledge is not perfect of course but I also did not explain the
> >> problem fully.
> >>
> >> In order for Ravi to find the libraries on Mac OSX I need to set something like:
> >>
> >> export LUA_CPATH="$RAVI_HOME/lib/?.dylib;$RAVI_HOME/lib/lib?.dylib"
> >>
> >> Now in my CMake build by default the library gets created as:
> >>
> >> $HOME/ravi/lib/ravi/libffi.dylib
> >>
> >> It appears that Lua searches for:
> >>
> >> $HOME/ravi/lib/ravi/ffi.dylib as it substitutes 'ravi/ffi' in place of
> >> '?'. So it cannot handle the 'lib' prefix.
> >>
> >> I have amended the CMake script to omit the 'lib' prefix and now it
> >> works fine, but this is not standard naming on Mac OSX I think.
> >
> >   At work we develop on Mac OS-X, and while normally I package everything
> > into a single executable, I do have the ability to install individual
> > modules [1] and they all have a .so extension (just like they do on Linux
> > and Solaris) and Mac OS-X can load them (the Lua modules written in C) just
> > fine.  I did check other, standard shared libraries on the Mac, and yes,
> > they have the .dylib extension [2].  So for Lua modules I don't think it
> > matters all that much what the extension is.
> >
>
> Sure but that was not the issue I was high lighting. Do you have a
> solution for the problem I described above?

  It could be that I don't have a full grasp of your problem.  I've read
over your message, and as best as I could tell:

        1. You have a Lua module written in C called ravi/ffi, with a
           filename of "ravi/ffi.dylib".

        2. You have another shared object, NOT a Lua module, called libtest
            (filename: "ravi/ffi/libtest.dylib") that references functions
            defined in ravi/ffi

        3. You are having issues loading libtest beacuse of naming issues.
           Specifically, libtest has a depenency on ffi
           (ravi/ffi/libtest.dylib has a reference to "ffi.dylib") without
           any path information so it needs to be searched for.

  Is that the core issue?  And a secondary issue you have is the feeling
that shared libraries need to be prefixed with 'lib'?

  I'm not as familar with Mac OS-X as I am with Linux.  With that, I do load
C-based Lua modules with an extension of ".so" so the name isn't *that* much
of an issue:

[spc]saltmine-2:~/bin>luawhich org.conman.fsys
/usr/local/lib/lua/5.1/org/conman/fsys.so
[spc]saltmine-2:~/bin>file `luawhich org.conman.fsys`
/usr/local/lib/lua/5.1/org/conman/fsys.so: Mach-O 64-bit executable x86_64
[spc]saltmine-2:~/bin>

(saltmine-2 is my work Mac laptop; "luawhich" is a Lua script that displays
the location of Lua modules---code available upon request)

  Now, I'm more used to Linux than Mac OS-X, so to find out what a shared
object's depencies are, I'm used to using ldd:

[spc]lucy:~/bin>ldd `luawhich org.conman.hash`
        libcrypto.so.4 => /lib/libcrypto.so.4 (0x00bfb000)
        libc.so.6 => /lib/tls/libc.so.6 (0x00a35000)
        libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x0050b000)
        libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x00259000)
        libcom_err.so.2 => /lib/libcom_err.so.2 (0x004e4000)
        libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x0081a000)
        libresolv.so.2 => /lib/libresolv.so.2 (0x0052d000)
        libdl.so.2 => /lib/libdl.so.2 (0x00111000)
        libz.so.1 => /usr/lib/libz.so.1 (0x00d30000)
        /lib/ld-linux.so.2 (0x00b76000)

And here, we see that org.conman.hash relies upon libcrypto.so.4 but because
it lacks any path information (and ldd does the search and reports back the
final location), a search is required.  On Mac OS-X ldd doesn't exist, but I
did find that otool does a similar thing:

[spc]saltmine-2:~/bin>otool -L `luawhich org.conman.hash`
/usr/local/lib/lua/5.1/org/conman/hash.so:
        /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

Poking around various programs and shared objects, I found that the full
paths to dependencies are always present, so I'm not sure of the rules for
$DYLD_LIBRARY_PATH are.  You might have some more digging to go through but
if what you have works, I say go for it for now and somebody with more
knowledge might be able to help.

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Jay Carlson
In reply to this post by Sean Conner


On Feb 7, 2018 18:30, "Sean Conner" <[hidden email]> wrote:
It was thus said that the Great Dibyendu Majumdar once stated:
> On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:

> 1. Should I force all libraries to have a require string that is of
> the format 'supplier.module' ? The problem in enforcing this is that
> it makes the distro incompatible with existing code. A better approach
> may be to use the require string that the module provides but prohibit
> any other use of the same module path. This is easy as I have control
> over which libraries go in.

  What is the concern here?  So far, Lua has managed without a conherent
namespace for modules.  If Ravi is meant to run existing Lua code then you
pretty much *can't* enforce a new requirement.

You can have multiple namespaces covering the same objects in Lua. So along with

  pl = require "pl"

I could have

  pl = nop_require "general/penlight"

which return the same object. Presumably you bootstrap the second with 

  nop_require = require "nop_require"

I think the painful issue then is fighting over how the underlying loader handles "real" module names in the "loaded modules" table. There could be other landmines too, though.

-- 
Jay
Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Sean Conner
On 10 February 2018 at 00:34, Sean Conner <[hidden email]> wrote:

> It was thus said that the Great Dibyendu Majumdar once stated:
>> On 9 February 2018 at 08:08, Sean Conner <[hidden email]> wrote:
>> > It was thus said that the Great Dibyendu Majumdar once stated:
>> >>
>> >> My knowledge is not perfect of course but I also did not explain the
>> >> problem fully.
>> >>
>> >> In order for Ravi to find the libraries on Mac OSX I need to set something like:
>> >>
>> >> export LUA_CPATH="$RAVI_HOME/lib/?.dylib;$RAVI_HOME/lib/lib?.dylib"
>> >>
>> >> Now in my CMake build by default the library gets created as:
>> >>
>> >> $HOME/ravi/lib/ravi/libffi.dylib
>> >>
>> >> It appears that Lua searches for:
>> >>
>> >> $HOME/ravi/lib/ravi/ffi.dylib as it substitutes 'ravi/ffi' in place of
>> >> '?'. So it cannot handle the 'lib' prefix.
>> >>
>> >> I have amended the CMake script to omit the 'lib' prefix and now it
>> >> works fine, but this is not standard naming on Mac OSX I think.
>> >
>> >   At work we develop on Mac OS-X, and while normally I package everything
>> > into a single executable, I do have the ability to install individual
>> > modules [1] and they all have a .so extension (just like they do on Linux
>> > and Solaris) and Mac OS-X can load them (the Lua modules written in C) just
>> > fine.  I did check other, standard shared libraries on the Mac, and yes,
>> > they have the .dylib extension [2].  So for Lua modules I don't think it
>> > matters all that much what the extension is.
>> >
>>
>> Sure but that was not the issue I was high lighting. Do you have a
>> solution for the problem I described above?
>
>   It could be that I don't have a full grasp of your problem.  I've read
> over your message, and as best as I could tell:
>

Above the problem I was describing is that Lua's shared library
resolution does not work if the library has a 'lib' prefix, and the
resolution has a path component in it. That is:

Lua looks for '$HOME/ravi/lib/ravi/ffi.dylib' or
'$HOME/ravi/lib/ravi/ffi.so' - depending on CPATH setting - and this
doesn't support the 'lib' prefix. Is this any clearer?

Anyway I thought there was a view that there isn't a problem - so I
thought I was missing something, and I was looking to confirm whether
my finding is correct or not.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:

> On 10 February 2018 at 00:34, Sean Conner <[hidden email]> wrote:
> >
> >   It could be that I don't have a full grasp of your problem.  I've read
> > over your message, and as best as I could tell:
>
> Above the problem I was describing is that Lua's shared library
> resolution does not work if the library has a 'lib' prefix, and the
> resolution has a path component in it. That is:
>
> Lua looks for '$HOME/ravi/lib/ravi/ffi.dylib' or
> '$HOME/ravi/lib/ravi/ffi.so' - depending on CPATH setting - and this
> doesn't support the 'lib' prefix. Is this any clearer?

  Yes.  But why are you building Lua modules prefixed with 'lib'?  You would
have that issue on other platforms as well.  That's what makes this
perplexing (at least to me).

  As far as I know, there are *no* Lua modules in C that are prefixed with
'lib'.

  -spc

123