Lua distros again

classic Classic list List threaded Threaded
56 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Lorenzo Donati-3
On 29/01/2018 10:27, Pierre Chapuis wrote:

>> Is there a list of the best essential libraries for Lua?
>> I want to bundle a small set of high quality
>> libraries that I will test with Ravi, rather than a huge set of
>> untested libraries of varying quality.
>
>>From the end of 2013 to the end of 2016 I was running a website
> called Lua Toolbox where community members could endorse
> the modules they used. The website has been merged into
> the main LuaRocks website and sadly this feature has been lost
> in the process, but the top 5 modules at the time were:
>
> - luafilesystem
> - luasocket
> - lpeg
> - luarocks
> - luaposix
>
> I think they represent what I would expect in such a distribution
> pretty well. Maybe not LuaRocks given that you want a different
> build system, and maybe winapi as the pendent of luaposix for
> Windows users... Some OpenSSL binding is also a must have.
>

[warning: long post ahead - with some ranting, too]


I think that one of the problem with actual library usage is that the
average Lua (the language) power-user, if I may call him that way, is
that he has been taught to "roll your own", so I suspect he is using
lots of home-brewed code that has lots of commonality with "self-rolled"
code used by other users.

Therefore he really downloads libraries that he cannot (by lack of
expertise or time) roll by himself. LFS is a clear example. It is
symptomatic that no string or table library comes first, although
probably each one of us has some kind of custom-made self-rolled
"stringx" or "tablex" library, because I guess most programming task
involving Lua may well imply non-trivial string or table manipulation.
Hence LFS /appears/ to be the most used library, although you can go a
long way without its functionalities.

This, IMHO, is /the/ big problem with Lua ecosystem. The most common
libraries are actually not published because they are often half-baked
solutions that are too author-specific and not worth maintaining in a
public way (who wants to support his own string library if it hasn't
enough traction from the community?)

It should seem strange that more general-purpose libraries are /not/ the
most downloaded ("penlight", "stdlib"), as if people didn't usually use
those functionalities.

After years of following Lua-l I've seen this discussion coming up
several times. And every time died off or gave rise to initiatives that
lasted just for a short while or didn't gain enough traction to become
de-facto standards. Every time.

Sadly I think this is a direct consequence of Lua team not wanting to
endorse any specific initiative.

I understand their decision of not wanting to create and maintain a "PUC
standard" library collection, and I also understand their attitude
toward backwards compatibility. That's ok. It's Lua team's way.

But there is a big "but": not endorsing, say, a charter of committed
people that took responsibility for creating a basic set of standard
library was and is a big mistake for the future of Lua, IMO.

I still remember some years ago when Lua was considered (I can't recall
exactly the source) one of the three major scripting language, together
with Python and Javascript.

I fear we are steadily losing terrain. I haven't hard data to back up
this feeling, so I may well stand to be corrected (and I'd like to be,
really), but I see lots of news about Python gaining ever more users
among scientists and non-programmers, whereas nothing about Lua outside
very specific "circles".

Mind, I don't think Lua user base is shrinking, on the contrary, I think
it is increasing, but much, much more slowly than other languages.
This means that a widespread diffusion of Lua (as was the intention of
their authors, at least years ago) is not going to happen, IMO.

Even when thinking about embedding a language in an application, where
Lua should be first choice, I hear people talking about Python and using
Python (urgh!)! Sometimes because they don't even know Lua, and
sometimes because they need better supported libraries, so they rule-out
Lua!

I didn't even understand the rationale behind the latest changes in the
language. I did appreciate the addition of "goto" in 5.2 and bitwise
operators in 5.3, but, really, what's the use of the utf8 library in a
language that doesn't support unicode natively (relying on underlying C
reading files in text mode - I know that in Linux that doesn't matter-
but Windows is not a marginal system)? The minimalistic mantra of
keeping unneeded stuff out of the language wasn't applied here. (why?)
Utf8 was an ideal candidate for an external library.

Lua team stripped the math library of world-standard hyperbolic
functions because they were little used (?!?) and they were a simple
forwarding to C libs. "OK, if you are a C programmer you can re-add
them, or you can implement them using exp". Sadly who usually needs the
optimized C versions that map to math processor instructions
(scientists) is typically /not/ a C programmer, whereas a C programmer
rarely uses hyperbolic functions!

Then table.move (good addition but why the complicated usage pattern,
resembling the C memmove, and why "move", when it copies things
around?!? Another wink to C programmers?).

Then also the string.pack/unpack/packsize, things. Why? That may make
sense to some, but is it really a "general mechanism"? For what? The
manual not even explains what is going on:

"The first argument to string.pack, string.packsize, and string.unpack
is a format string, which describes the layout of the structure being
created or read."

What structure? Not even an example is given! And Lua hasn't got any
"structure" datatype. Yes, I know that is some kind of inclusion of
Roberto's "struct" library. Of course a C programmer or someone using
protocols could discover this, but another wink to C programmers and
more unfriendliness toward a novice Lua programmer that is not already a
C programmer. Another very specialized thing (not really a "general
mechanism") that seems well fit for an external library that gets in the
language, instead. Moreover it is a fairly low-level mechanism. Another
thing that doesn't appeal to "higher level" programmers. Why?

Really, I can't understand where Lua is heading in the intention of
their creators. What is the target user base of Lua?

These eternal discussions about a "standard set of library", recurring
for decades and always leading to nothing lasting more than a couple of
years should be a warning for Lua team.

Judging by lua-l alone there are more attempts at creating
customized/patched/jitted Lua variants than to create a set of
standardized libraries having the consensus of the community!

What if Donald Knuth kept TeX for himself and didn't endorse the efforts
of the community to build that enormous, reliable and widespread code-base?


Cheers!

-- Lorenzo
(who still loves Lua a lot, but is ever more puzzled)














Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Pierre Chapuis
On Tue, Feb 13, 2018, at 11:46, Lorenzo Donati wrote:

> Therefore he really downloads libraries that he cannot (by lack of
> expertise or time) roll by himself. LFS is a clear example. It is
> symptomatic that no string or table library comes first.

In other words, users download libraries written in C that deal with
platform-specific stuff, not small Lua snippets like in the Node.JS
ecosystem.

> This, IMHO, is /the/ big problem with Lua ecosystem. The most common
> libraries are actually not published because they are often half-baked
> solutions that are too author-specific and not worth maintaining in a
> public way (who wants to support his own string library if it hasn't
> enough traction from the community?)

Well, what do you expect from a "string library"? Because there is
a standard string library that ships with the language...

> It should seem strange that more general-purpose libraries are /not/ the
> most downloaded ("penlight", "stdlib"), as if people didn't usually use
> those functionalities.

I use Penlight, but mostly as a debugging tool (especially `pretty`), not
something I ship with my projects.

> After years of following Lua-l I've seen this discussion coming up
> several times. And every time died off or gave rise to initiatives that
> lasted just for a short while or didn't gain enough traction to become
> de-facto standards. Every time.
>
> Sadly I think this is a direct consequence of Lua team not wanting to
> endorse any specific initiative.
>
> I understand their decision of not wanting to create and maintain a "PUC
> standard" library collection, and I also understand their attitude
> toward backwards compatibility. That's ok. It's Lua team's way.
>
> But there is a big "but": not endorsing, say, a charter of committed
> people that took responsibility for creating a basic set of standard
> library was and is a big mistake for the future of Lua, IMO.

I understand the sentiment, I thought the same 5-10 years ago,
I don't anymore. Maybe it's a chichen-and-egg project but I don't
think it is easy to "bless" a project that hasn't emerged as a clear
winner from the community itself. Moreover, even if the Lua team
(basically Roberto in this case I guess) did that, what effect would
it have? I mean, LuaJIT is still not implementing Lua 5.3, is it?

It makes sense that the community

> I still remember some years ago when Lua was considered (I can't recall
> exactly the source) one of the three major scripting language, together
> with Python and Javascript.

Well, I'd be happy to know the source for that and how many years
"some" is, because I don't think that was ever the case for most
people's definition of "scripting". Lua has always had very strong
niches (games, embedded...), more and more recently (with e.g.
Software Defined Networking and Machine Learning) but I can't
think of a time where it was much higher in mindshare than now.

> I fear we are steadily losing terrain. I haven't hard data to back up
> this feeling, so I may well stand to be corrected (and I'd like to be,
> really), but I see lots of news about Python gaining ever more users
> among scientists and non-programmers, whereas nothing about
> Lua outside  very specific "circles".
>
> Mind, I don't think Lua user base is shrinking, on the contrary, I think
> it is increasing, but much, much more slowly than other languages.
> This means that a widespread diffusion of Lua (as was the intention of
> their authors, at least years ago) is not going to happen, IMO.

Scientific people is a specific case IMO, related to ML frameworks.
I have been pretty much involved in this so how is how I see things:

- Once upon a time people were using various things including
  Matlab, R, Python (Scipy / Numpy), C++, etc.
- Deep Learning happened, and the market split between three
  major frameworks: Caffe (Java), Theano (Python) and Torch7 (LuaJIT).
- Important companies like Facebook and Twitter chose Torch7 so Lua
  grew in popularity.
- Some people at Google were not happy because Lua is not one of
  their blessed languages, but even their own team was using it
  nevertheless (AlphaGo).
- They pushed for new C++ / Python frameworks with TensorFlow
  and Keras.
- Meanwhile some people who preferred Python to Lua made
  PyTorch which got a lot of adoption too.

So tl;dr Lua had a short-lived bubble in that space because of one
framework, but now it's not the only one on the stage anymore so
Lua usage is going back down.

Add to that the (again) chicken-and-egg problem that few people
use Lua hence few people *teach* Lua at universities (as opposed
to Python which is on basically every curriculum now)...

> Even when thinking about embedding a language in an application, where
> Lua should be first choice, I hear people talking about Python and using
> Python (urgh!)! Sometimes because they don't even know Lua, and
> sometimes because they need better supported libraries, so they rule-out
> Lua!

Have you already embedded Python, packaged it and shipped it to people?
Because I have done it and I can tell you using most of the C-based modules
on the package index is much, much harder than in Lua.

> I didn't even understand the rationale behind the latest changes in the
> language. I did appreciate the addition of "goto" in 5.2 and bitwise
> operators in 5.3, but, really, what's the use of the utf8 library in a
> language that doesn't support unicode natively (relying on underlying C
> reading files in text mode - I know that in Linux that doesn't matter-
> but Windows is not a marginal system)? The minimalistic mantra of
> keeping unneeded stuff out of the language wasn't applied here. (why?)
> Utf8 was an ideal candidate for an external library.

I think UTF-8 is a very good candidate for an internal library, but I agree
that it didn't go far enough in supporting Windows (but that has already
been discussed in other threads).

> Lua team stripped the math library of world-standard hyperbolic
> functions because they were little used (?!?) and they were a simple
> forwarding to C libs. "OK, if you are a C programmer you can re-add
> them, or you can implement them using exp". Sadly who usually needs the
> optimized C versions that map to math processor instructions
> (scientists) is typically /not/ a C programmer, whereas a C programmer
> rarely uses hyperbolic functions!

Well, scientific stuff is a very good candidate for an external library...
as it is the case in Python by the way. We don't have SciPy / NumPy
in Lua but is that the core team's fault?

> Then also the string.pack/unpack/packsize, things. Why? That may make
> sense to some, but is it really a "general mechanism"? For what? The
> manual not even explains what is going on:
>
> "The first argument to string.pack, string.packsize, and string.unpack
> is a format string, which describes the layout of the structure being
> created or read."
>
> What structure? Not even an example is given! And Lua hasn't got any
> "structure" datatype. Yes, I know that is some kind of inclusion of
> Roberto's "struct" library. Of course a C programmer or someone using
> protocols could discover this, but another wink to C programmers and
> more unfriendliness toward a novice Lua programmer that is not already a
> C programmer. Another very specialized thing (not really a "general
> mechanism") that seems well fit for an external library that gets in the
> language, instead. Moreover it is a fairly low-level mechanism. Another
> thing that doesn't appeal to "higher level" programmers. Why?

Well, to parse or generate any kind of real world data? Network
protocols, file formats... Again, look at Python: the exact same library
is called "struct" and is included in the standard library. Moreover
both "struct" and "pack" were some of the most used libraries
in former Lua versions.

> Judging by lua-l alone there are more attempts at creating
> customized/patched/jitted Lua variants than to create a set of
> standardized libraries having the consensus of the community!

There *is* a standard library. You just don't like what's in it.

Again, I am not saying you are wrong. I used to think like you,
and now I don't anymore. I am extremely happy with the additions
in 5.3 (64 bit integers, binary operators and string.{un,}pack)
which cater to the "networks and systems" niche. The only
thing I still miss is a much better defined standard way to deal
with sequences (i.e. "I have gotten a table from somewhere,
is it a list or a map?"), but otherwise I think the community can
do the job, and is doing it judging from the increase in number of
modules on LuaRocks.

That being said, on the community end, we are still bad at
making the core libraries and tools we all need. We have several
package managers, network libraries, crypto libraries... and most
of them do not cover all platforms and use cases. And that's the
curse of Lua not being a generic programming language: we don't
have people paid to (as part of their job) maintain libraries which
do one of those things well *in all cases*. Writing a (C-based) library
that works well on, say, only Windows is easy and requires few
resources (a Windows machine). But when you want to support
Linux, MacOS, iOS, Android etc as well, you incur costs in hardware
and time that can't be justified in many cases.

I think that's the problem we have to solve, and I don't know how
we can do it. But I don't mind that everyone has their 150 lines
table / string library anymore.

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dirk Laurie-2
In reply to this post by Lorenzo Donati-3
2018-02-13 12:46 GMT+02:00 Lorenzo Donati <[hidden email]>:

> It should seem strange that more general-purpose libraries are /not/ the
> most downloaded ("penlight", "stdlib"), as if people didn't usually use
> those functionalities.

They are very idiosyncratic, in the sense of being tailored to the
predilections of their respective authors. I steal lots of code from
them, but I don't require the libraries themselves.

> Sadly I think this is a direct consequence of Lua team not wanting to
> endorse any specific initiative.
...
> But there is a big "but": not endorsing, say, a charter of committed people
> that took responsibility for creating a basic set of standard library was
> and is a big mistake for the future of Lua, IMO.

Only for Lua as a possible replacement for Python. The future of Lua
lies where the past of Lua also lies, as a scripting language embedded
in an application, not as the backbone of standalone applications.

> Even when thinking about embedding a language in an application, where Lua
> should be first choice, I hear people talking about Python and using Python
> (urgh!)! Sometimes because they don't even know Lua, and sometimes because
> they need better supported libraries, so they rule-out Lua!

I haven't seen anybody suggesting yet that a PyTeX effort like LuaTeX
should be made.

> Utf8 was an ideal candidate for an external library.

It still is. Nothing in the world stops you from loading one. On the other
hand, if all you need is to count characters for the sake of you own-rolled
linewrap utility, the Lua 5.3 libray is adequate.

> Lua team stripped the math library of world-standard hyperbolic functions
> because they were little used (?!?) and they were a simple forwarding to C
> libs. "OK, if you are a C programmer you can re-add them, or you can
> implement them using exp". Sadly who usually needs the optimized C versions
> that map to math processor instructions (scientists) is typically /not/ a C
> programmer, whereas a C programmer rarely uses hyperbolic functions!

Although I agree with you, it is nevertheless also true that a member of the
Lua team maintains a math library that has not only those, but all of what C99
has to offer.

> Then table.move (good addition but why the complicated usage pattern,
> resembling the C memmove, and why "move", when it copies things around?!?
> Another wink to C programmers?).

I like it, having written something similar (except that mine can reverse,
i.e. table.move(x,i,j,x,j,i). Note that the Lua table.move returns the
destination
table, encouraging you to chain self-calls.

> What if Donald Knuth kept TeX for himself and didn't endorse the efforts of
> the community to build that enormous, reliable and widespread code-base?

He did indeed keep TeX for himself. He has been a mere participant in
that codebase. He has not endorsed anything build on top of TeX, not
even LaTeX, and seems to write a new style file, even new Metafonts,
for every project. That's very Lua-ish :-)

The difference is that TeX converges to TeX version math.pi, so that
eternal backwards compatibility is guaranteed.

<tldr> I haven't yet read Pierre's response which I now see peeping out
as the screen scrolls up, so please forgive my if I have duplicated
some of his points. </tldr>

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Lorenzo Donati-3
On 13 February 2018 at 10:46, Lorenzo Donati <[hidden email]> wrote:

> Sadly I think this is a direct consequence of Lua team not wanting to
> endorse any specific initiative.
>
> I understand their decision of not wanting to create and maintain a "PUC
> standard" library collection, and I also understand their attitude toward
> backwards compatibility. That's ok. It's Lua team's way.
>
> But there is a big "but": not endorsing, say, a charter of committed people
> that took responsibility for creating a basic set of standard library was
> and is a big mistake for the future of Lua, IMO.
>

Hi, I think this depends on what you see as the future of Lua. I
cannot speak for the Lua team, but my impression is that they are not
looking to win the popularity game. Their intention appears to be to
create a minimal language / library that is still a powerful language.
The niche of Lua is that it comes with no baggage, and is therefore
eminently suitable for embedding as a scripting language in a larger
C/C++ application (I chose those languages as host languages as
embedding Lua in say Java or C# appears pointless).

I think Lua is still the strongest language in this niche.

> Even when thinking about embedding a language in an application, where Lua
> should be first choice, I hear people talking about Python and using Python
> (urgh!)! Sometimes because they don't even know Lua, and sometimes because
> they need better supported libraries, so they rule-out Lua!
>

In some domains where Python is already strong, there is an obvious
logic to using Python. For instance in the financial sector Python is
now becoming popular; I chose Lua as the scripting language in this
domain, but the response from users is 'why?'. My convenience (as
developer) in using Lua is immaterial, as you have to think of what
your users want.

Availability of an extended standard library would make Lua more
friendly to larger range of users. But keeping with Lua's spirit it
needs to be done outside of Lua as an add-on - and this is where the
problem lies, as without a clear leading solution, there are many
competing solutions with different objectives (I am adding my own to
the mix too!).

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

aryajur
In reply to this post by Dirk Laurie-2


Only for Lua as a possible replacement for Python. The future of Lua
lies where the past of Lua also lies, as a scripting language embedded
in an application, not as the backbone of standalone applications.

I think this may be a false hope. Even though Lua is extremely good at that I have failed to convince popular (in my domain) open source software to use that as their scripting language when they decided they want to add one.

I am an Electrical Engineer designing Power ICs and 90% of my colleagues  don't need to know any programming in any language. I use Lua as an application language to manage my project, generate corner simulation scripts and data analysis on all my chips. I generate and serve my website using Lua[1]. I even used it with IUP and IM to generate Bingo game cards with cartoons for my Son's Birthday.
    The reason I chose Lua was because it was much more elegant and clean than Python. And slowly I realized that I can do anything in it and I dont need to learn any other language. I can use it in embedded projects to making a full website, to making a full blown GUI application to doing complex math and data analysis. I had come to Lua after riding on Basic/Qbasic/Visual Basic/C/Matlab and very briefly Python. After finding Lua haven't looked at any other language but now sometimes get worried that it may die out. 
     So which open source software? I tried to convince KiCAD[2], QUCS[3], Klayout[4] teams to embed Lua but they rejected my proposal with a majority of their developers favoring Python (Klayout used Ruby). Main reason is since it is more popular and better ecosystem. 
     I tried convincing a Semiconductor IP startup I was advising to base their web platform on Lua but they didnt want to take up a lesser used language and risk not finding developers for it plus I was the only one that seemed to think it was the good idea.

Google just recently announced their TPU chipset [5] which makes Tensorflow and Python more desirable.

I am afraid that just having the best technology is not enough. If nobody uses it then it is simply useless. So marketing the technology and removing any pain points even if they are not at all related to the technology (In Lua's case being it's ecosystem and a standardisation path and guidance) is sometimes critical for the technology to survive in the long term and benefit everyone. 

  


Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Pierre Chapuis
On Tue, Feb 13, 2018, at 17:51, Milind Gupta wrote:


Only for Lua as a possible replacement for Python. The future of Lua
lies where the past of Lua also lies, as a scripting language embedded
in an application, not as the backbone of standalone applications.

I think this may be a false hope. Even though Lua is extremely good at that I have failed to convince popular (in my domain) open source software to use that as their scripting language when they decided they want to add one.

We have gone way off topic here but since we are talking about convincing people to choose Lua instead of Python for an embedded language, here are some arguments that worked for me.

- Security is a good one. Lua is much, much easier to sandbox than Python. Also, should you ever audit the source, it's 16,000 lines of C. Good luck auditing CPython.

- Open Source references. Yes, there are some projects that use Python. But they aren't OpenResty, Redis, HAProxy, nmap, wireshark, VLC, Asterisk, OpenWRT, PowerDNS, RPM, the NetBSD kernel...

Every year we hear of large Open Source projects successfully embedding Lua. For instance a friend of mine added it to the neomutt MUA last year: https://www.neomutt.org/2017/04/29/lua A few days ago at FOSDEM there was this talk about a Lua plugin for Janus, a WebRTC server: https://fosdem.org/2018/schedule/event/janus



Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

aryajur

- Open Source references. Yes, there are some projects that use Python. But they aren't OpenResty, Redis, HAProxy, nmap, wireshark, VLC, Asterisk, OpenWRT, PowerDNS, RPM, the NetBSD kernel...

Yes some of these are what I cite to everyone when describing Lua I just hope the list grows and not shrink. Also the ability to do mobile phone apps for all major platforms also once you know Lua is also very attractive.


Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 29 January 2018 at 22:55, Dibyendu Majumdar <[hidden email]> wrote:

> On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
>> I have always wanted to (but haven't managed to yet) bundle some high
>> quality libraries with Ravi in a well tested combination with support
>> for Windows, Linux and Mac OSX. Is there a list of the best essential
>> libraries for Lua? I want to bundle a small set of high quality
>> libraries that I will test with Ravi, rather than a huge set of
>> untested libraries of varying quality.
>>
>
> Thank you all for the feedback. My shortlist now consists of:
>
> - lpeg
> - luafilesystem
> - luasocket
> - libuv (Luvit)
> - libcurl (wrapper tbc)
> - lua-cjson
> - torch7
> - luaossl
> - cephes (wrapper tbc)
> - luaffifb (port of LuaJIT FFI interface)
>
> Well this list although short is probably going to keep me busy for
> several months.
>

An update on this:

I have been working on this in my spare time. The first module I
decided to tackle is luaffifb / luaffi - which provides an 'ffi'
library similar to that in LuaJIT. This has turned out to be more
involved as I am having to fix some issues with this module (see my
other post for details).

The main thing I want to mention is that I decided to support stock
Lua 5.3 too - that is the distro will have a Ravi and a Lua 5.3
output, although naturally I will first test everything with Ravi.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:

> I have always wanted to (but haven't managed to yet) bundle some high
> quality libraries with Ravi in a well tested combination with support
> for Windows, Linux and Mac OSX. Is there a list of the best essential
> libraries for Lua? I want to bundle a small set of high quality
> libraries that I will test with Ravi, rather than a huge set of
> untested libraries of varying quality.
>
> My preference is to use CMake based build system - such as used by
> VCPKG by Microsoft, or LuaDist.
>

Hi - this is now slowly taking shape here:

https://github.com/dibyendumajumdar/ravi-distro

All feedback welcome.

Thanks and Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
On 15 February 2018 at 18:18, Dibyendu Majumdar <[hidden email]> wrote:

> On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
>> I have always wanted to (but haven't managed to yet) bundle some high
>> quality libraries with Ravi in a well tested combination with support
>> for Windows, Linux and Mac OSX. Is there a list of the best essential
>> libraries for Lua? I want to bundle a small set of high quality
>> libraries that I will test with Ravi, rather than a huge set of
>> untested libraries of varying quality.
>>
>> My preference is to use CMake based build system - such as used by
>> VCPKG by Microsoft, or LuaDist.
>>
>
> Hi - this is now slowly taking shape here:
>
> https://github.com/dibyendumajumdar/ravi-distro
>
> All feedback welcome.

I have been busy trying to get the torch library to work on all
platforms. It appears that the Lua version uses 'long' data type - but
this is not portable as on Windows, MSVC treats longs as 32-bit
values. I found that PyTorch has had fixes updates to resolve this so
I decided to merge those changes. However I am still getting test
failures and am  working on those.

Prior to this I spent some time fixing bugs in the FFI library.

I wanted to say that this highlights what kind of distro I am trying
to create - the aim is not just throw packages together, but to have a
distro that works reliably on all three major platforms. This requires
some amount of effort - and in some cases I have take on the
maintenance of the packages that are no longer being maintained
upstream.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 29 January 2018 at 22:55, Dibyendu Majumdar <[hidden email]> wrote:

> On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
>> I have always wanted to (but haven't managed to yet) bundle some high
>> quality libraries with Ravi in a well tested combination with support
>> for Windows, Linux and Mac OSX. Is there a list of the best essential
>> libraries for Lua? I want to bundle a small set of high quality
>> libraries that I will test with Ravi, rather than a huge set of
>> untested libraries of varying quality.
>>
>
> Thank you all for the feedback. My shortlist now consists of:
>
> - lpeg
> - luafilesystem
> - luasocket
> - libuv (Luvit)
> - libcurl (wrapper tbc)
> - lua-cjson
> - torch7
> - luaossl
> - cephes (wrapper tbc)
> - luaffifb (port of LuaJIT FFI interface)
>

Hi - planning to add Penlight to the distro.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Hisham
In reply to this post by Pierre Chapuis
On 13 February 2018 at 09:41, Pierre Chapuis <[hidden email]> wrote:
> Well, I'd be happy to know the source for that and how many years
> "some" is, because I don't think that was ever the case for most
> people's definition of "scripting". Lua has always had very strong
> niches (games, embedded...), more and more recently (with e.g.
> Software Defined Networking and Machine Learning) but I can't
> think of a time where it was much higher in mindshare than now.

Can we even still put Machine Learning in that list, with the demise of Torch?

>> I fear we are steadily losing terrain. I haven't hard data to back up
>> this feeling, so I may well stand to be corrected (and I'd like to be,
>> really), but I see lots of news about Python gaining ever more users
>> among scientists and non-programmers, whereas nothing about
>> Lua outside  very specific "circles".
>>
>> Mind, I don't think Lua user base is shrinking, on the contrary, I think
>> it is increasing, but much, much more slowly than other languages.
>> This means that a widespread diffusion of Lua (as was the intention of
>> their authors, at least years ago) is not going to happen, IMO.
>
> Scientific people is a specific case IMO, related to ML frameworks.
> I have been pretty much involved in this so how is how I see things:
>
> - Once upon a time people were using various things including
>   Matlab, R, Python (Scipy / Numpy), C++, etc.
> - Deep Learning happened, and the market split between three
>   major frameworks: Caffe (Java), Theano (Python) and Torch7 (LuaJIT).
> - Important companies like Facebook and Twitter chose Torch7 so Lua
>   grew in popularity.
> - Some people at Google were not happy because Lua is not one of
>   their blessed languages, but even their own team was using it
>   nevertheless (AlphaGo).
> - They pushed for new C++ / Python frameworks with TensorFlow
>   and Keras.
> - Meanwhile some people who preferred Python to Lua made
>   PyTorch which got a lot of adoption too.

Small clarification: these "some people" are the same people who made
Torch in the first place. PyTorch is developed by the original Torch
team and the Lua version is no longer being developed (it's in
"maintenance mode").

> So tl;dr Lua had a short-lived bubble in that space because of one
> framework, but now it's not the only one on the stage anymore so
> Lua usage is going back down.

And when I contacted members of the Torch team to learn more about
this, the reasoning I was given for abandoning Lua was twofold:

* most of the high-performance execution in this field happens in the
GPU anyway; using LuaJIT is not such a big win as opposed to Python
* Python's more mature ecosystem with the gazillion libraries for
visualization etc already available

(This is my recollection of the exchange, please don't quote them on this.)

> Add to that the (again) chicken-and-egg problem that few people
> use Lua hence few people *teach* Lua at universities (as opposed
> to Python which is on basically every curriculum now)...

Funnily enough, that includes PUC-Rio, which has Python in its
standard curriculum but not Lua. (Granted, Lua is offered as an
elective course every now and then, but Python is CS101).

>> "The first argument to string.pack, string.packsize, and string.unpack
>> is a format string, which describes the layout of the structure being
>> created or read."
>>
>> What structure? Not even an example is given! And Lua hasn't got any
>> "structure" datatype. Yes, I know that is some kind of inclusion of
>> Roberto's "struct" library. Of course a C programmer or someone using
>> protocols could discover this, but another wink to C programmers and
>> more unfriendliness toward a novice Lua programmer that is not already a
>> C programmer. Another very specialized thing (not really a "general
>> mechanism") that seems well fit for an external library that gets in the
>> language, instead. Moreover it is a fairly low-level mechanism. Another
>> thing that doesn't appeal to "higher level" programmers. Why?
>
> Well, to parse or generate any kind of real world data? Network
> protocols, file formats...

Lorenzo's examples do paint an interesting picture that I haven't
visualized as clearly before: the evolution of the Lua standard
library does seem to be biased towards the needs of embedded systems.
Which is a good bet, market-wise. There will always be a niche for
resource-constrained embedded systems where the computing budget is as
tight as possible: some applications that required a custom IC decades
ago may be doable today with a RPi running a full Linux distro, but
some new applications that were unthinkable back then are now doable
with a penny-sized computer where CPU and memory are still at a
premium. From credit-card sized, to penny-sized, to
grain-of-sand-sized, the point will still stand.

But it's understandable that many of us share this sort of frustration
about the language's popularity, because it is a great design as far
as languages go. It's the same feeling as turning on the radio and
hearing whatever is #1 at the moment instead of our favorite music
genres. Some people go "eww", some go "why world, why??" and some feel
good about themselves because they consider themselves the hipsters
with better taste than the mainstream. :)

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
On 11 March 2018 at 23:34, Dibyendu Majumdar <[hidden email]> wrote:

> On 29 January 2018 at 22:55, Dibyendu Majumdar <[hidden email]> wrote:
>> On 28 January 2018 at 16:40, Dibyendu Majumdar <[hidden email]> wrote:
>>> I have always wanted to (but haven't managed to yet) bundle some high
>>> quality libraries with Ravi in a well tested combination with support
>>> for Windows, Linux and Mac OSX. Is there a list of the best essential
>>> libraries for Lua? I want to bundle a small set of high quality
>>> libraries that I will test with Ravi, rather than a huge set of
>>> untested libraries of varying quality.
>>>
>>
>> Thank you all for the feedback. My shortlist now consists of:
>>
>> - lpeg
>> - luafilesystem
>> - luasocket
>> - libuv (Luvit)
>> - libcurl (wrapper tbc)
>> - lua-cjson
>> - torch7
>> - luaossl
>> - cephes (wrapper tbc)
>> - luaffifb (port of LuaJIT FFI interface)
>>
>
> Hi - planning to add Penlight to the distro.
>

I am pleased to report that Penlight is now part of the distro.
Testing Penlight uncovered a bug in Ravi which I have fixed but sadly
it was not part of the recent 0.23 release of Ravi.

I have a question re any pure Lua libraries that people care about:

a) Busted has already been suggested
(https://github.com/dibyendumajumdar/ravi-distro/issues/1).
b) Any others that I should look at?

Bear in mind that I am keen to keep the distro small and ideally avoid
duplicating functionality.

I was thinking whether to include a scripting language such as
Moonscript. Any views?

Thanks and Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar
In reply to this post by Hisham
On 19 March 2018 at 22:16, Hisham <[hidden email]> wrote:
> Small clarification: these "some people" are the same people who made
> Torch in the first place. PyTorch is developed by the original Torch
> team and the Lua version is no longer being developed (it's in
> "maintenance mode").
>

Hi, I am hoping to maintain the Lua version of Torch in my forked repo
(https://github.com/dibyendumajumdar/ravi-torch7). I fact I merged the
recent updates from PyTorch into my fork. There are additional Torch
libraries that need to be added. It would be shame to let Torch for
Lua die; so if anyone wants to help me maintain the Lua version of
Torch, please do not hesitate to join the project.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Russell Haley


On Mon, Mar 19, 2018 at 3:58 PM, Dibyendu Majumdar <[hidden email]> wrote:
On 19 March 2018 at 22:16, Hisham <[hidden email]> wrote:
> Small clarification: these "some people" are the same people who made
> Torch in the first place. PyTorch is developed by the original Torch
> team and the Lua version is no longer being developed (it's in
> "maintenance mode").
>

Hi, I am hoping to maintain the Lua version of Torch in my forked repo
(https://github.com/dibyendumajumdar/ravi-torch7). I fact I merged the
recent updates from PyTorch into my fork. There are additional Torch
libraries that need to be added. It would be shame to let Torch for
Lua die; so if anyone wants to help me maintain the Lua version of
Torch, please do not hesitate to join the project.

I was under the impression Torch is only supported on LuaJIT. Am I incorrect?

Russ 

Regards
Dibyendu


Reply | Threaded
Open this post in threaded view
|

Re: Lua distros again

Dibyendu Majumdar


On Wednesday, 21 March 2018, Russell Haley <[hidden email]> wrote:

>
>
> On Mon, Mar 19, 2018 at 3:58 PM, Dibyendu Majumdar <[hidden email]> wrote:
>>
>> On 19 March 2018 at 22:16, Hisham <[hidden email]> wrote:
>> > Small clarification: these "some people" are the same people who made
>> > Torch in the first place. PyTorch is developed by the original Torch
>> > team and the Lua version is no longer being developed (it's in
>> > "maintenance mode").
>> >
>>
>> Hi, I am hoping to maintain the Lua version of Torch in my forked repo
>> (https://github.com/dibyendumajumdar/ravi-torch7). I fact I merged the
>> recent updates from PyTorch into my fork. There are additional Torch
>> libraries that need to be added. It would be shame to let Torch for
>> Lua die; so if anyone wants to help me maintain the Lua version of
>> Torch, please do not hesitate to join the project.
>
> I was under the impression Torch is only supported on LuaJIT. Am I incorrect?
>
>

Yes
123