Comparison of numbers of 3rd party libraries

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Comparison of numbers of 3rd party libraries

zejian ju
Several days ago, I incidentally checked the numbers of 3rd party libraries
for node.js, python, ruby and lua. I was sad and puzzling about so small the
number of lua packages.

node.js npm packages:  146310 (https://www.npmjs.com/)
python pypi packages:  59223 (https://pypi.python.org/pypi)
ruby gems packages:    6348 (https://rubygems.org/gems)
lua luarocks packages: 957 (https://luarocks.org/modules)

Regards,
Zenk Ju
Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

Brigham Toskin
On Sat, May 9, 2015 at 2:38 AM, zejian ju <[hidden email]> wrote:
Several days ago, I incidentally checked the numbers of 3rd party libraries
for node.js, python, ruby and lua. I was sad and puzzling about so small the
number of lua packages.

node.js npm packages:  146310 (https://www.npmjs.com/)
python pypi packages:  59223 (https://pypi.python.org/pypi)
ruby gems packages:    6348 (https://rubygems.org/gems)
lua luarocks packages: 957 (https://luarocks.org/modules)

For whatever reasons, I get the impression that rocks is "less official" in the Lua community than many of these other repos/package managers. Even CPAN, which at least started out completely unofficial quickly filled that niche. Maybe my impression is wrong, but I just feel like less people use rocks as their primary source of Lua packages (or conversely, the primary distribution mechanism for their code).

For more concrete context, Search github for "lua": https://github.com/search?utf8=%E2%9C%93&q=lua

> "We've found 11,432 repository results"

Search sourceforge for "lua": http://sourceforge.net/directory/?q=lua

> "Showing page 1 of 68." (x 25 results per page = 1700 results)

Google code projects tagged with "Lua": https://code.google.com/hosting/search?q=label%3aLua

> "Results 1 - 10 of 928"

--
Brigham Toskin
Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

Tim Hill
In reply to this post by zejian ju

On May 9, 2015, at 2:38 AM, zejian ju <[hidden email]> wrote:

Several days ago, I incidentally checked the numbers of 3rd party libraries
for node.js, python, ruby and lua. I was sad and puzzling about so small the
number of lua packages.

node.js npm packages:  146310 (https://www.npmjs.com/)
python pypi packages:  59223 (https://pypi.python.org/pypi)
ruby gems packages:    6348 (https://rubygems.org/gems)
lua luarocks packages: 957 (https://luarocks.org/modules)

Regards,
Zenk Ju

bigger != better

—Tim

Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

zejian ju
In reply to this post by Brigham Toskin

I do not know what others think about luarocks, but I just do not like to use it because of the long command name....

Look at npm, rpm, yum, it is so easy to type. Even apt-get is one character less than luarocks :-)

Cheers,
Zenk Ju
Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

steve donovan
On Mon, May 11, 2015 at 8:48 AM, zejian ju <[hidden email]> wrote:
> Look at npm, rpm, yum, it is so easy to type. Even apt-get is one character
> less than luarocks :-)

$ alias lr=luarocks

two characters ;)

Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

slayer beast
Also lots of Lua code is developed as scripts for applications (like
WoW addons), and those don't interact with luarocks

On Mon, May 11, 2015 at 4:25 AM, steve donovan
<[hidden email]> wrote:
> On Mon, May 11, 2015 at 8:48 AM, zejian ju <[hidden email]> wrote:
>> Look at npm, rpm, yum, it is so easy to type. Even apt-get is one character
>> less than luarocks :-)
>
> $ alias lr=luarocks
>
> two characters ;)
>

Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

zejian ju
Yes, there's no common application platform for lua, as node.js for javascript.

Openresty provides a web server platform for lua based on nginx, but there's no
suitable package manager for it.

Comparing to js, I like lua because of lexical scoping, true coroutine, simple and
consistent design, and without the ugly and strange parts of js. So I hope lua can
be used pervasively.

But js seems catching up, let scoping/module/generator/async function will
appear in the coming ES6.

2015-05-13 6:46 GMT+08:00 slayer beast <[hidden email]>:
Also lots of Lua code is developed as scripts for applications (like
WoW addons), and those don't interact with luarocks


Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

Andrew Starks


On Tuesday, May 12, 2015, zejian ju <[hidden email]> wrote:
Yes, there's no common application platform for lua, as node.js for javascript.

Openresty provides a web server platform for lua based on nginx, but there's no
suitable package manager for it.

Comparing to js, I like lua because of lexical scoping, true coroutine, simple and
consistent design, and without the ugly and strange parts of js. So I hope lua can
be used pervasively.

But js seems catching up, let scoping/module/generator/async function will
appear in the coming ES6.

2015-05-13 6:46 GMT+08:00 slayer beast <<a href="javascript:_e(%7B%7D,&#39;cvml&#39;,&#39;slayerbeast@gmail.com&#39;);" target="_blank">slayerbeast@...>:
Also lots of Lua code is developed as scripts for applications (like
WoW addons), and those don't interact with luarocks



First, LuaRocks is very nice, and has made some excellent improvements. Also, LuaDist and ZeroBrainStudio are extremely useful. The quality of the little eccosystem that Lua does have deserves a lot of credit for the noticeable improvements that people have made. 

I'll point out that, unlike JavaScript, a defining feature of Lua is its C-API, which should be thought of as a part of the language. This means that large eccosystems will always be a side show. Very often the eccosystems belong to the application that Lua is embedded into (Wikimedia, ConTeX, Lightroom plugins). Also, developers are likely to disable many features that module writers use: coroutines, access to other modules, etc. 

The second difference is Lua's focus on embedding, especially where resources are tight. I have not heard much in the way of JS advancing the way that it is embedded into applications, outside of webbrowsers. My understanding is that decisions made early on make JavaScript more difficult to interact with from C, than Lua, but I have not researched that. 

Lua is best compared against language libraries with similar constraints/missions, although your observations do have merit. 

-Andrew

Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

zejian ju


2015-05-13 11:11 GMT+08:00 Andrew Starks <[hidden email]>:


First, LuaRocks is very nice, and has made some excellent improvements. Also, LuaDist and ZeroBrainStudio are extremely useful. The quality of the little eccosystem that Lua does have deserves a lot of credit for the noticeable improvements that people have made. 

I'll point out that, unlike JavaScript, a defining feature of Lua is its C-API, which should be thought of as a part of the language. This means that large eccosystems will always be a side show. Very often the eccosystems belong to the application that Lua is embedded into (Wikimedia, ConTeX, Lightroom plugins). Also, developers are likely to disable many features that module writers use: coroutines, access to other modules, etc. 

The second difference is Lua's focus on embedding, especially where resources are tight. I have not heard much in the way of JS advancing the way that it is embedded into applications, outside of webbrowsers. My understanding is that decisions made early on make JavaScript more difficult to interact with from C, than Lua, but I have not researched that. 

Agree. Lua's fucus on embedding (IMO its C-API is also defined for this goal) makes it always a "side" language to assist the main applications, which are usually written in C/C++. But as the hardware is becoming more powerful, Lua (with so good design and implementation) deserves more general usage and acts as a "main" language for general application.

Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

William Ahern
In reply to this post by zejian ju
On Wed, May 13, 2015 at 09:31:18AM +0800, zejian ju wrote:

> Yes, there's no common application platform for lua, as node.js for
> javascript.
>
> Openresty provides a web server platform for lua based on nginx, but
> there's no
> suitable package manager for it.
>
> Comparing to js, I like lua because of lexical scoping, true coroutine,
> simple and
> consistent design, and without the ugly and strange parts of js. So I hope
> lua can
> be used pervasively.
>
> But js seems catching up, let scoping/module/generator/async function will
> appear in the coming ES6.
>

ES6 generators are a far cry from Lua's coroutines:

1) ES6 generators require a special function definition.

2) Lua's coroutines are symmetric--yield can return values provided by
resume.

3) ES6 yield can only be used in the generator function, whereas you can
yield from any function in the Lua coroutine call stack.

Javascript will never have Lua's coroutines because it would require huge
refactoring of the existing vendor VMs. Instead, Javascript chose
callback-based futures, and then tacked on generators as a band-aid to help
with the mess of callbacks. But generators will never be as elegant or as
useful as coroutines.

Furthermore, Lua's coroutine implementation is unique in real-world
languages in that it's stackless--it doesn't rely on the C stack to
implement coroutines, even when using the C API. (Stackless Python is only
stackless for Python code.) That has useful performance, memory, and
debugging characteristics. On some older platforms (notably the old
LinuxThreads implementation) you can't mix POSIX threads with fibers (the
Windows term for application-created C stacks), and in the future this
limitation may return in the form of security measures on more popular
platforms.


Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

zejian ju


2015-05-14 3:16 GMT+08:00 William Ahern <[hidden email]>:

ES6 generators are a far cry from Lua's coroutines:

1) ES6 generators require a special function definition.
2) Lua's coroutines are symmetric--yield can return values provided by 
resume.

3) ES6 yield can only be used in the generator function, whereas you can
yield from any function in the Lua coroutine call stack.

Yes, you are right. Even the proposed ES7 async/await should add async keyword
to the function definition.

Furthermore, Lua's coroutine implementation is unique in real-world
languages in that it's stackless--it doesn't rely on the C stack to
implement coroutines, even when using the C API. (Stackless Python is only
stackless for Python code.) That has useful performance, memory, and
debugging characteristics. On some older platforms (notably the old
LinuxThreads implementation) you can't mix POSIX threads with fibers (the
Windows term for application-created C stacks), and in the future this
limitation may return in the form of security measures on more popular
platforms.

Lua's coroutine is great for its true stackless. Every normal function can be used
in coroutine thread. But stackless means a full separate Lua stack should be maintained
for each coroutine, which may cause more garbage memory refered from the earlier
frames of these stacks will be kept. Considering the not-so-great GC in Lua, this
may not be a good thing in some conditions.

Regards,
Zenk Ju
Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

Tim Hill

On May 14, 2015, at 8:18 PM, zejian ju <[hidden email]> wrote:

Lua's coroutine is great for its true stackless. Every normal function can be used
in coroutine thread. But stackless means a full separate Lua stack should be maintained
for each coroutine, which may cause more garbage memory refered from the earlier
frames of these stacks will be kept. Considering the not-so-great GC in Lua, this
may not be a good thing in some conditions.

What are your issues with the Lua GC?

—Tim

Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

KHMan
On 5/16/2015 10:32 AM, Tim Hill wrote:

>
>> On May 14, 2015, at 8:18 PM, zejian ju wrote:
>>
>> Lua's coroutine is great for its true stackless. Every normal
>> function can be used
>> in coroutine thread. But stackless means a full separate Lua
>> stack should be maintained
>> for each coroutine, which may cause more garbage memory refered
>> from the earlier
>> frames of these stacks will be kept. Considering the
>> not-so-great GC in Lua, this
>> may not be a good thing in some conditions.
>
> What are your issues with the Lua GC?

He wishes the app was running at a sustained 4
instructions/clock/core... :-P

Or, he hopes someone else would do the hard work of making a
cache-aware GC...

[intended as good-natured ribbing, let us be able to keep things
slightly loose...:-p]

I wonder what is the _actual_ slowdown, wall clock, not profiler
reports. Often a tip in Android development is to avoid object
creation, fix everything when running the main game loop for
instance. If in his Lua app he fixed objects in
performance-critical sections, how much would that change running
time? Canonical Lua is a great multi-platform interpreter, the
question is do users expect it to juggle knives while doing
cartwheels...

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia


Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

zejian ju
In reply to this post by Tim Hill


2015-05-16 10:32 GMT+08:00 Tim Hill <[hidden email]>:

What are your issues with the Lua GC?
1. I hope the Lua GC-related data structure can be more cache-friendly, array/stack-based
    instead of singlle-linklist-based structure could be used for it. But I'm afraid this implementation
    may be OS-related and conflit with the goal of Lua as an embedded language.
2. I hope Lua have a incremental generational compact/move GC. I know there's one experimental
    generational GC in Lua 5.2, but it is not better than the wonderful incremental one. In our
    application the full GC was triggered very often, so we switched back to incremental GC eventually.

Some guys assert that Lua is not suitable for move GC because of the Lua/C API, I'm not very
sure about this. Yes, the return value of lua_tostring may be changed during GC, but we can fix
it in memory, or change some part of the API. Anyway the API is not always stable between Lua vesions.

Lua's current GC implementation is suitable for many situations, but for our server usage, we
hope it can be more efficient. I write one reference counting version of Lua5.1[1], sometimes
it is more efficient than the vanilla Lua.

[1] https://github.com/zenkj/luarc-5.1

Regards,
Zenk Ju
Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

zejian ju
In reply to this post by KHMan


2015-05-16 11:50 GMT+08:00 KHMan <[hidden email]>:

He wishes the app was running at a sustained 4 instructions/clock/core... :-P

Or, he hopes someone else would do the hard work of making a cache-aware GC...

[intended as good-natured ribbing, let us be able to keep things slightly loose...:-p]

Well..., I take back my words about "not-so-great Lua GC". Lua's GC is great. I read all the GC code of Lua5.1/5.2/5.3. Its authors put great effort to improve it and I like the elegant implementation. what I mean is it can be made faster for my server application.
 
Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

KHMan
On 5/16/2015 3:41 PM, zejian ju wrote:

> 2015-05-16 11:50 GMT+08:00 KHMan:
>
>     He wishes the app was running at a sustained 4
>     instructions/clock/core... :-P
>
>     Or, he hopes someone else would do the hard work of making a
>     cache-aware GC...
>
>     [intended as good-natured ribbing, let us be able to keep
>     things slightly loose...:-p]
>
> Well..., I take back my words about "not-so-great Lua GC". Lua's
> GC is great. I read all the GC code of Lua5.1/5.2/5.3. Its authors
> put great effort to improve it and I like the elegant
> implementation. what I mean is it can be made faster for my server
> application.

I'm not totally sold on the idea that an awesome GC is the best
solution. The node.js ecosystem works because of V8's (is it still
V8?) performance.

Here, interpreted Lua is still interpreted, and if your dataset is
big, assuming for example for an online multiuser app backend you
will still touch plenty of data and churn the CPU cache. Now if
you write a test script that has a workload not too different from
your server load, then test GC-running and no-GC-running versions,
then we'd have an idea what kind of difference we can get. I love
data (de best thing to back up any claims), some timing data would
be lovely here. But remember you can't get anywhere close to JIT
performance with an interpreter...

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia


Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

Andrew Starks


On Saturday, May 16, 2015, KHMan <[hidden email]> wrote:
On 5/16/2015 3:41 PM, zejian ju wrote:
2015-05-16 11:50 GMT+08:00 KHMan:

    He wishes the app was running at a sustained 4
    instructions/clock/core... :-P

    Or, he hopes someone else would do the hard work of making a
    cache-aware GC...

    [intended as good-natured ribbing, let us be able to keep
    things slightly loose...:-p]

Well..., I take back my words about "not-so-great Lua GC". Lua's
GC is great. I read all the GC code of Lua5.1/5.2/5.3. Its authors
put great effort to improve it and I like the elegant
implementation. what I mean is it can be made faster for my server
application.

I'm not totally sold on the idea that an awesome GC is the best solution. The node.js ecosystem works because of V8's (is it still V8?) performance.

Here, interpreted Lua is still interpreted, and if your dataset is big, assuming for example for an online multiuser app backend you will still touch plenty of data and churn the CPU cache. Now if you write a test script that has a workload not too different from your server load, then test GC-running and no-GC-running versions, then we'd have an idea what kind of difference we can get. I love data (de best thing to back up any claims), some timing data would be lovely here. But remember you can't get anywhere close to JIT performance with an interpreter...

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia



Is it possible that the issues that people experience with the GC might be more accurately framed as integration issues, rather than more generically? Integration might mean collecting userdata that points to large blocks, dealing with reference counted objects and errors, etc. I'm giving sub-optimal examples.l, but it does seemsl like GC issues are either nebulous, not really there, or are there, but out of Lua's hands. 

 Maybe the GC is fine as is, but could be improved with hinting or some other mechanism?

-Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Comparison of numbers of 3rd party libraries

KHMan
On 5/17/2015 12:10 PM, Andrew Starks wrote:

>
> On Saturday, May 16, 2015, KHMan wrote:
>
>     On 5/16/2015 3:41 PM, zejian ju wrote:
>
>         2015-05-16 11:50 GMT+08:00 KHMan:
>
>              He wishes the app was running at a sustained 4
>              instructions/clock/core... :-P
>
>              Or, he hopes someone else would do the hard work of
>         making a
>              cache-aware GC...
>
>              [intended as good-natured ribbing, let us be able to keep
>              things slightly loose...:-p]
>
>         Well..., I take back my words about "not-so-great Lua GC".
>         Lua's
>         GC is great. I read all the GC code of Lua5.1/5.2/5.3. Its
>         authors
>         put great effort to improve it and I like the elegant
>         implementation. what I mean is it can be made faster for
>         my server
>         application.
>
>     I'm not totally sold on the idea that an awesome GC is the
>     best solution. The node.js ecosystem works because of V8's (is
>     it still V8?) performance.
>
>     Here, interpreted Lua is still interpreted, and if your
>     dataset is big, assuming for example for an online multiuser
>     app backend you will still touch plenty of data and churn the
>     CPU cache. Now if you write a test script that has a workload
>     not too different from your server load, then test GC-running
>     and no-GC-running versions, then we'd have an idea what kind
>     of difference we can get. I love data (de best thing to back
>     up any claims), some timing data would be lovely here. But
>     remember you can't get anywhere close to JIT performance with
>     an interpreter...
>[snip]
>
> Is it possible that the issues that people experience with the GC
> might be more accurately framed as integration issues, rather than
> more generically? Integration might mean collecting userdata that
> points to large blocks, dealing with reference counted objects and
> errors, etc. I'm giving sub-optimal examples.l, but it
> does seemsl like GC issues are either nebulous, not really
> there, or are there, but out of Lua's hands.
>
>   Maybe the GC is fine as is, but could be improved with hinting
> or some other mechanism?

Yeah, I rather like Roberto's reminder about the dearth of GC
benchmarks.

An artificial test script would clear up plenty of ambiguities --
code quality, data structure, object churn, dataset size,
higher-level optimization opportunities, wall clock effect of GC, etc.

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia