Lua package manager(s)

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua package manager(s)

Steven Degutis
I recently wrote a Mac OS X app[1] that just acts as a Lua host, and I
wrote some packages for it that let you do things like inspect and
move windows, bind global hotkeys to arbitrary Lua functions, etc...
It was called Hydra but the name had to be changed to Mjolnir for
trademark reasons.

Anyway, there was a lot of disagreement among my app's users for how
to install packages. So I took out the internal package management
code I wrote, moved all my modules to LuaRocks, and told all my app's
users to just use that for their packages.

But there's been a lot of discussion on my app's mailing list about
wanting a different package manager. Partly because LuaRocks (1) is
missing some features like being able to know what packages can be
upgraded, and being able to upgrade them, and (2) is designed for the
command-line primarily, and we're not sure it's reasonable to embed it
into my application and use it in a Lua-prompt (i.e. REPL).

As for #1, there has already been an open issue about this for
years[2] which has not seen any activity, and I've posted a new
issue[3] requesting this feature, which also has not seen any
response. So this gives me no hope that this feature will be added any
time soon, or that a pull-request including it is welcome.

So we've been looking into all our available options. Some of the
options that users are suggesting and brainstorming involve writing a
brand-new Lua package manager. Some involve forking LuaRocks itself.
Some involve writing new tools on top of LuaRocks.

Because of the scope the discussion in my mailing list[4][5][6] is
taking on, it seemed fitting to me to write this email to lua-l to
bring up the issue to the wider Lua community.

[1] http://mjolnir.io/
[2] https://github.com/keplerproject/luarocks/issues/22
[3] https://github.com/keplerproject/luarocks/issues/282
[4] https://groups.google.com/forum/#!topic/mjolnir-io/9Bg5JNa_MiU
[5] https://groups.google.com/forum/#!topic/mjolnir-io/Q200sDBapxE
[6] https://groups.google.com/forum/#!topic/mjolnir-io/TyKRqMpgO5U

-Steven

Reply | Threaded
Open this post in threaded view
|

Re: Lua package manager(s)

Hisham
On 8 September 2014 15:14, Steven Degutis <[hidden email]> wrote:

> I recently wrote a Mac OS X app[1] that just acts as a Lua host, and I
> wrote some packages for it that let you do things like inspect and
> move windows, bind global hotkeys to arbitrary Lua functions, etc...
> It was called Hydra but the name had to be changed to Mjolnir for
> trademark reasons.
>
> Anyway, there was a lot of disagreement among my app's users for how
> to install packages. So I took out the internal package management
> code I wrote, moved all my modules to LuaRocks, and told all my app's
> users to just use that for their packages.
>
> But there's been a lot of discussion on my app's mailing list about
> wanting a different package manager. Partly because LuaRocks (1) is
> missing some features like being able to know what packages can be
> upgraded, and being able to upgrade them, and (2) is designed for the
> command-line primarily, and we're not sure it's reasonable to embed it
> into my application and use it in a Lua-prompt (i.e. REPL).
>
> As for #1, there has already been an open issue about this for
> years[2] which has not seen any activity, and I've posted a new
> issue[3] requesting this feature, which also has not seen any
> response. So this gives me no hope that this feature will be added any
> time soon, or that a pull-request including it is welcome.

A pull request is welcome. I have mixed feelings about feature
requests in the bugtracker (and the fact that Github calls it all
"issues"), because these are not really bugs or issues, they're
wishlist items.

People have at times requested different things: a command to list
what packages have more recent versions, and a command that
auto-upgrades everything. The latter I think is very questionable. The
former is less dangerous, but who says that a newer version of a
package is compatible with the previous ones?

Scanning the repository and telling you what rocks have a version with
the same name and a different number is easy. Cross-referencing all
your existing installed rocks with those versions and figuring out if
any dependency chains break if the older rock is replaced by the new
one is much harder.

Add to that the possibility that two available upgrades may be
mutually incompatible: say you have A 1.0, B 1.0 and X 1.0; A 2.0 is
available, but it requires X <= 1.0; B 2.0 is available but it
requires X >= 2.0. What to do? In LuaRocks we can keep them all
installed at the same time if you use luarocks.loader, but users have
complained about our feature of keeping versions simultaneously, so
that's not the default anymore. Users who ask for auto-upgrade want
the old versions to be removed.

(And no, "enforce semver to all rock authors!" is not possible.)

So, (1) it's not trivial to get the upgrade feature 100% right, (2)
whet it is not 100% it has great systeam-breaking potential, (3) it
may be impossible to make everyone happy since people want different
things.

Sorry about the pessimistic description, but that's one of those
"obvious features" that everyone wants but that's much harder to do
than it seems. Having said that, a simple list-available-upgrades
command wouldn't be that hard. Obviously, once that's in, people will
instantly ask "why can't I give it a command to install all these at
once?"

So that's why _I_ don't have plans to implement "luarocks upgrade" in
sight. If you'd like to implement it (or at least "list-upgrades") and
send a pull request our way, please document all design decisions so
we can point users in the direction of the documentation when they ask
for auto-upgrade (or ask why upgrade doesn't work the way they want)
:)

(Looks like it's unavoidable that long-time maintainers start sounding
grumpy... I hate to see it happening to myself!)

What I want you to take from this is: it's not due to simple lack of
goodwill that we lack an "upgrade" command. But we want to make
LuaRocks useful for many use cases including yours, and I hope we can
collaborate towards that. The goal for the next releases is to make it
more extensible and more embeddable. It would be awesome if we could
have your help to make that happen, and making sure LuaRocks serves
your needs; it would certainly be less work for everyone involved.

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Lua package manager(s)

Steven Degutis
> So, (1) it's not trivial to get the upgrade feature 100% right, (2)
> whet it is not 100% it has great systeam-breaking potential, (3) it
> may be impossible to make everyone happy since people want different
> things.

Understood. Thanks for the clarification.

> Sorry about the pessimistic description, but that's one of those
> "obvious features" that everyone wants but that's much harder to do
> than it seems.

As a maintainer myself, I think I fully understand.

> So that's why _I_ don't have plans to implement "luarocks upgrade" in
> sight. If you'd like to implement it (or at least "list-upgrades") and
> send a pull request our way, please document all design decisions so
> we can point users in the direction of the documentation when they ask
> for auto-upgrade (or ask why upgrade doesn't work the way they want)
> :)

No, you're right, it doesn't fully make sense in this context.

> (Looks like it's unavoidable that long-time maintainers start sounding
> grumpy... I hate to see it happening to myself!)

Heh, it's started happening to me too this month :)

> What I want you to take from this is: it's not due to simple lack of
> goodwill that we lack an "upgrade" command. But we want to make
> LuaRocks useful for many use cases including yours, and I hope we can
> collaborate towards that.

Fully understood. This is what I've been saying on my mailing list,
that LuaRocks covers a whole lot more use-cases than the simple ones
that my app's user have. That's actually why there's some debate about
whether using it as a starting-point makes sense for our use-case.
Some people suggest that we should start fresh, given we have
different limitations.

For example, as the author of the app which people are writing modules
for, I'm completely free to say things like "you must use semantic
versioning" and "you must not break backwards compatibility ever" and
stuff like that, with some reasonable expectation that module authors
will follow these guidelines.

That frees up a lot of options that otherwise LuaRocks doesn't have,
at least when installing modules for my app. So I can't help but
wonder if it's still possible to go down that route and take advantage
of those possibilities, while still using LuaRocks.

> The goal for the next releases is to make it
> more extensible and more embeddable. It would be awesome if we could
> have your help to make that happen, and making sure LuaRocks serves
> your needs; it would certainly be less work for everyone involved.

I'm definitely open to some kind of collaboration.

-Steven

Reply | Threaded
Open this post in threaded view
|

Re: Lua package manager(s)

Paul K-2
In reply to this post by Steven Degutis
Hi Steven,

> So we've been looking into all our available options. Some of the
> options that users are suggesting and brainstorming involve writing a
> brand-new Lua package manager. Some involve forking LuaRocks itself.
> Some involve writing new tools on top of LuaRocks.

I went through a similar process with ZeroBrane Studio IDE. Some of my
users wanted to be able to install new modules to make them available
from the IDE (especially on Windows and OSX). I looked at the existing
package managers [1] and decided to integrate with luadist; one of the
important points for me was the small number of dependencies (<200k of
lua and <200k of binary modules), which allowed me to include them
with the IDE itself.

> being able to know what packages can be upgraded

I don't think LuaDist provides this "out of the box" either, but the
information about installed and available packages is there, so it
should be possible to write a command that takes care of installing
updated versions.

> it's reasonable to embed it into my application and use it in a Lua-prompt (i.e. REPL).

This is exactly what I was looking for and implemented for my project.
ZBS provides a Lua console and with the plugin I wrote users can type
"luadist.install 'penlight'" and get the required modules and their
dependencies installed. The plugin is a wrapper around commands that
luadist provides; something similar can probably be done for LuaRocks.
By default Windows and OSX users will get binary versions, so no other
dependencies are needed.

This works quite well (both Lua 5.1 and Lua 5.2 modules are supported)
and the biggest issue for me is with binary modules being outdated in
the repository (for example, lpeg is at version 0.10), although you
can always compile from source similar to LuaRocks if you have dev
tools available. Another issue that I've seen is that requiring binary
modules for Lua 5.2 returns Lua 5.1 modules, which obviously doesn't
work.

I'd say LuaRocks is under more active development than LuaDist; even
if you don't want to modify the core modules directly, consider adding
a wrapper that implements the required functionality just for your
product and still reuses the rest of the infrastructure.

There was a long discussion on this very subject about a year ago that
may be of interest [2].

Paul.

[1] http://notebook.kulchenko.com/zerobrane/lua-package-managers-luadist-luarocks-and-integration-with-zerobrane-studio
[2] http://lua-users.org/lists/lua-l/2013-09/msg00740.html

On Mon, Sep 8, 2014 at 11:14 AM, Steven Degutis <[hidden email]> wrote:

> I recently wrote a Mac OS X app[1] that just acts as a Lua host, and I
> wrote some packages for it that let you do things like inspect and
> move windows, bind global hotkeys to arbitrary Lua functions, etc...
> It was called Hydra but the name had to be changed to Mjolnir for
> trademark reasons.
>
> Anyway, there was a lot of disagreement among my app's users for how
> to install packages. So I took out the internal package management
> code I wrote, moved all my modules to LuaRocks, and told all my app's
> users to just use that for their packages.

Reply | Threaded
Open this post in threaded view
|

Re: Lua package manager(s)

Steven Degutis
Thanks Paul. I'll look into this.

On Mon, Sep 8, 2014 at 2:10 PM, Paul K <[hidden email]> wrote:

> Hi Steven,
>
>> So we've been looking into all our available options. Some of the
>> options that users are suggesting and brainstorming involve writing a
>> brand-new Lua package manager. Some involve forking LuaRocks itself.
>> Some involve writing new tools on top of LuaRocks.
>
> I went through a similar process with ZeroBrane Studio IDE. Some of my
> users wanted to be able to install new modules to make them available
> from the IDE (especially on Windows and OSX). I looked at the existing
> package managers [1] and decided to integrate with luadist; one of the
> important points for me was the small number of dependencies (<200k of
> lua and <200k of binary modules), which allowed me to include them
> with the IDE itself.
>
>> being able to know what packages can be upgraded
>
> I don't think LuaDist provides this "out of the box" either, but the
> information about installed and available packages is there, so it
> should be possible to write a command that takes care of installing
> updated versions.
>
>> it's reasonable to embed it into my application and use it in a Lua-prompt (i.e. REPL).
>
> This is exactly what I was looking for and implemented for my project.
> ZBS provides a Lua console and with the plugin I wrote users can type
> "luadist.install 'penlight'" and get the required modules and their
> dependencies installed. The plugin is a wrapper around commands that
> luadist provides; something similar can probably be done for LuaRocks.
> By default Windows and OSX users will get binary versions, so no other
> dependencies are needed.
>
> This works quite well (both Lua 5.1 and Lua 5.2 modules are supported)
> and the biggest issue for me is with binary modules being outdated in
> the repository (for example, lpeg is at version 0.10), although you
> can always compile from source similar to LuaRocks if you have dev
> tools available. Another issue that I've seen is that requiring binary
> modules for Lua 5.2 returns Lua 5.1 modules, which obviously doesn't
> work.
>
> I'd say LuaRocks is under more active development than LuaDist; even
> if you don't want to modify the core modules directly, consider adding
> a wrapper that implements the required functionality just for your
> product and still reuses the rest of the infrastructure.
>
> There was a long discussion on this very subject about a year ago that
> may be of interest [2].
>
> Paul.
>
> [1] http://notebook.kulchenko.com/zerobrane/lua-package-managers-luadist-luarocks-and-integration-with-zerobrane-studio
> [2] http://lua-users.org/lists/lua-l/2013-09/msg00740.html
>
> On Mon, Sep 8, 2014 at 11:14 AM, Steven Degutis <[hidden email]> wrote:
>> I recently wrote a Mac OS X app[1] that just acts as a Lua host, and I
>> wrote some packages for it that let you do things like inspect and
>> move windows, bind global hotkeys to arbitrary Lua functions, etc...
>> It was called Hydra but the name had to be changed to Mjolnir for
>> trademark reasons.
>>
>> Anyway, there was a lot of disagreement among my app's users for how
>> to install packages. So I took out the internal package management
>> code I wrote, moved all my modules to LuaRocks, and told all my app's
>> users to just use that for their packages.
>