Lua has reached a local optimum

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Lua has reached a local optimum

Dirk Laurie-2
Lua 5.4 has NOCVTS2N as default and tiny enhancements to the API,
but the real changes are not visible at the language level. The
substantial enhancements are mainly in its implementation.

With that, we have reached the end of the road for the Lua 5.x series.
Lua 5.3/Lua5.4 as a language has reached a local optimum, in the
mathematical sense that moving a small amount away from where it is
now in any direction makes it worse. We've had a few of that recently.

* The undef idea, already put on the back burner by its inventors.
* The n+1-st discussion, futile before it started, on global-by-default.
* Nitpicking about error messages or the lack of them from library functions.

It may, of course, not be a global optimum. But the next version can't be
another minor release. It will have to be Lua 6.0, as far from the Lua 5.x
series as Lua 5.0 was from Lua 4.0.

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Dibyendu Majumdar
On 4 July 2018 at 21:57, Dirk Laurie <[hidden email]> wrote:
> With that, we have reached the end of the road for the Lua 5.x series.
> Lua 5.3/Lua5.4 as a language has reached a local optimum, in the
> mathematical sense that moving a small amount away from where it is
> now in any direction makes it worse. We've had a few of that recently.
>

http://lua-users.org/lists/lua-l/2018-01/msg00392.html

:-)

I think 5.3 may well be the greatest of Lua 5.x releases as I can't
think of anything in 5.4 that was really necessary. Did people really
need a generational GC? Is the additional complexity worth it? Did Lua
need the performance optimizations - which are not free either?

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

云风 Cloud Wu


Dibyendu Majumdar <[hidden email]>于2018年7月5日周四 上午5:44写道:
I think 5.3 may well be the greatest of Lua 5.x releases as I can't
think of anything in 5.4 that was really necessary. Did people really
need a generational GC? Is the additional complexity worth it? Did Lua


A user of our online game server framework (skynet) report lua 5.4's  generational GC could improve the memory usage obviously [1]. It reduce nearly half physical memory usage, because the allocator (sush as jemalloc) would use much more memory if we haven't collect it as soon as possible.

[1] : https://github.com/cloudwu/skynet/issues/827  (The discussion is in Chinese)




Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Rodrigo Azevedo


2018-07-05 0:55 GMT-03:00 云风 Cloud Wu <[hidden email]>:


Dibyendu Majumdar <[hidden email]>于2018年7月5日周四 上午5:44写道:
I think 5.3 may well be the greatest of Lua 5.x releases as I can't
think of anything in 5.4 that was really necessary. Did people really
need a generational GC? Is the additional complexity worth it? Did Lua


A user of our online game server framework (skynet) report lua 5.4's  generational GC could improve the memory usage obviously [1]. It reduce nearly half physical memory usage, because the allocator (sush as jemalloc) would use much more memory if we haven't collect it as soon as possible.

[1] : https://github.com/cloudwu/skynet/issues/827  (The discussion is in Chinese)

I can confirm that the use of generational GC with tcmalloc.so improves the performance (less CPU time and memory usage) of complex programs (douzens GB of RAM) considerable. Moreover, the new random number generator algorithm and the new opcodes of lua 5.4 are absolutelly essential, sorry Dibyendu.

--
Rodrigo Azevedo Moreira da Silva
Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Jakub Jirutka
> I can confirm that the use of generational GC with tcmalloc.so improves
> the performance (less CPU time and memory usage) of complex programs
> (douzens GB of RAM) considerable.

Who develops programs consuming dozens GiB of RAM in Lua?! O.O
I don't have any objections against generational GC in Lua, but arguing
with dozens GiB of RAM looks more like discussion about Java VM than Lua...

Jakub

On 05/07/18 12:13, Rodrigo Azevedo wrote:

>
>
> 2018-07-05 0:55 GMT-03:00 云风 Cloud Wu <[hidden email]
> <mailto:[hidden email]>>:
>
>
>
>     Dibyendu Majumdar <[hidden email]
>     <mailto:[hidden email]>>于2018年7月5日周四 上午5:44写道:
>
>         I think 5.3 may well be the greatest of Lua 5.x releases as I can't
>         think of anything in 5.4 that was really necessary. Did people
>         really
>         need a generational GC? Is the additional complexity worth it?
>         Did Lua
>
>
>     A user of our online game server framework (skynet) report lua
>     5.4's  generational GC could improve the memory usage obviously [1].
>     It reduce nearly half physical memory usage, because the allocator
>     (sush as jemalloc) would use much more memory if we haven't collect
>     it as soon as possible.
>
>     [1] : https://github.com/cloudwu/skynet/issues/827
>     <https://github.com/cloudwu/skynet/issues/827>  (The discussion is
>     in Chinese)
>
>
> I can confirm that the use of generational GC with tcmalloc.so improves
> the performance (less CPU time and memory usage) of complex programs
> (douzens GB of RAM) considerable. Moreover, the new random number
> generator algorithm and the new opcodes of lua 5.4 are absolutelly
> essential, sorry Dibyendu.
>
> --
> Rodrigo Azevedo Moreira da Silva

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Viacheslav Usov
On Thu, Jul 5, 2018 at 12:20 PM Jakub Jirutka <[hidden email]> wrote:

> Who develops programs consuming dozens GiB of RAM in Lua?!

I do, or, rather, my customers do in a system that embeds Lua.

>  arguing with dozens GiB of RAM looks more like discussion about Java VM than Lua...

False.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Luiz Henrique de Figueiredo
In reply to this post by Jakub Jirutka
> Who develops programs consuming dozens GiB of RAM in Lua?!

I did for this paper (see page 19):

Rigorous bounds for polynomial Julia sets
http://dx.doi.org/10.3934/jcd.2016006
http://lhf.impa.br/ftp/papers/ij.pdf

We built huge graphs. One of the examples required 12Gb for a graph
with 16560993 vertices and 127668413 edges.

I remember performance (time and memory) improved significantly when I
switched from 5.2 to 5.3. Perhaps I should run this again with 5.4.

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

云风 Cloud Wu
In reply to this post by Jakub Jirutka


Jakub Jirutka <[hidden email]>于2018年7月5日周四 下午6:20写道:
> I can confirm that the use of generational GC with tcmalloc.so improves
> the performance (less CPU time and memory usage) of complex programs
> (douzens GB of RAM) considerable.

Who develops programs consuming dozens GiB of RAM in Lua?! O.O
I don't have any objections against generational GC in Lua, but arguing
with dozens GiB of RAM looks more like discussion about Java VM than Lua...


Our game server use more than 100GiB RAM in lua  (But separate into thousands lua vm in one  process)

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Jakub Jirutka
On 05/07/18 12:33, Luiz Henrique de Figueiredo wrote:
> I did for this paper (see page 19):
>
> Rigorous bounds for polynomial Julia sets
> http://dx.doi.org/10.3934/jcd.2016006
> http://lhf.impa.br/ftp/papers/ij.pdf
>
> We built huge graphs. One of the examples required 12Gb for a graph
> with 16560993 vertices and 127668413 edges.

This is really interesting, thanks for sharing it! I understand that it
probably wasn't the goal of the research, but any chance you have some
comparison (time and memory usage) with the same algorithm implemented
in some different language(s)?

On 05/07/18 12:29, Viacheslav Usov wrote:
> I do, or, rather, my customers do in a system that embeds Lua.

On 05/07/18 12:53, 云风 Cloud Wu wrote:
> Our game server use more than 100GiB RAM in lua  (But separate into
> thousands lua vm in one  process)

Okay, my bad, I should be more specific. What matters here is how much
memory is allocated for single Lua heap, not how much memory consumes
some program that embeds Lua.

Jakub

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Viacheslav Usov
On Thu, Jul 5, 2018 at 1:09 PM Jakub Jirutka <[hidden email]> wrote:

> Okay, my bad, I should be more specific. What matters here is how much memory is allocated for single Lua heap, not how much memory consumes some program that embeds Lua.

See https://www.lua.org/bugs.html#5.3.4-2 . Even if that was reported by me with an artificial repro case, it originally hit a customer installation, with a table holding some fat values, all in one Lua state, consuming quite a few gigs before it went down.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

云风 Cloud Wu
In reply to this post by Jakub Jirutka
Jakub Jirutka <[hidden email]>于2018年7月5日周四 下午7:09写道:

On 05/07/18 12:53, 云风 Cloud Wu wrote:
> Our game server use more than 100GiB RAM in lua  (But separate into
> thousands lua vm in one  process)

Okay, my bad, I should be more specific. What matters here is how much
memory is allocated for single Lua heap, not how much memory consumes
some program that embeds Lua.


I know some other game server projects use more than 10GB RAM in one lua vm, and the cost of gc is always a trouble. Someone choose python instead of lua because python can use reference count only (turn off mark-sweep gc) to manage life time of objects. 

My opinion is using multiple lua vm and keeping each lua vm small is a better way. Lua can do it simpler than other dynamic language such as python, javascript, etc. Sometimes, we use a few vm to store large data. These vm consume a lot of memory, but not mutable. We don't need mark these objects constantly.

Even for small vm, lua 5.4's gc is very valuable because the total memory of thousands lua vm can be very large.


 
Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

szbnwer@gmail.com
In reply to this post by Dirk Laurie-2
hi there! :)

2018-07-04 20:57 GMT+00:00 Dirk Laurie <[hidden email]>:
[...]
> It may, of course, not be a global optimum. But the next version can't be
> another minor release. It will have to be Lua 6.0, as far from the Lua 5.x
> series as Lua 5.0 was from Lua 4.0.

[feature request]
i could imagine as 5.x that paths in error messages wouldn't be
trimmed when they are too long, so they can be parsed for whatever
purpose. just like 'str':fun() and {}:fun() would be nice to see. i
can only remember that the reasoning against this was 'cuz the refman
says so' :D (probably that was you, Dirk, you are the local refman
guru (that i like to see! :) ))

otherwise in general i can totally agree, when i came here a bit
before 5.2, lua was already great for whatever a mind can imagine, and
a nice toy to play with :D

bests for all! :)

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Dirk Laurie-2
2018-07-05 14:52 GMT+02:00 [hidden email] <[hidden email]>:

> 'str':fun() and {}:fun() would be nice to see. i
> can only remember that the reasoning against this was 'cuz the refman
> says so' :D (probably that was you, Dirk, you are the local refman
> guru (that i like to see! :) ))

I can't find the thread but the clincher for me was a demonstration
(not by myself) that a syntax ambiguity could arise.

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Roberto Ierusalimschy
> 2018-07-05 14:52 GMT+02:00 [hidden email] <[hidden email]>:
>
> > 'str':fun() and {}:fun() would be nice to see. i
> > can only remember that the reasoning against this was 'cuz the refman
> > says so' :D (probably that was you, Dirk, you are the local refman
> > guru (that i like to see! :) ))
>
> I can't find the thread but the clincher for me was a demonstration
> (not by myself) that a syntax ambiguity could arise.

A syntax ambiguity is already there; these cases would make it worse:

  -- already ambiguous:
  a = b + c
  ('str'):fun()

  -- new ambiguity:
  a = b + c
  'str':fun()

  -- new ambiguity:
  a = b + c
  {}:fun()

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Rena
On Thu, Jul 5, 2018, 12:04 Roberto Ierusalimschy, <[hidden email]> wrote:
> 2018-07-05 14:52 GMT+02:00 [hidden email] <[hidden email]>:
>
> > 'str':fun() and {}:fun() would be nice to see. i
> > can only remember that the reasoning against this was 'cuz the refman
> > says so' :D (probably that was you, Dirk, you are the local refman
> > guru (that i like to see! :) ))
>
> I can't find the thread but the clincher for me was a demonstration
> (not by myself) that a syntax ambiguity could arise.

A syntax ambiguity is already there; these cases would make it worse:

  -- already ambiguous:
  a = b + c
  ('str'):fun()

  -- new ambiguity:
  a = b + c
  'str':fun()

  -- new ambiguity:
  a = b + c
  {}:fun()

-- Roberto

All three of those look like the same case, really.
Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

szbnwer@gmail.com
In reply to this post by Roberto Ierusalimschy
2018-07-05 16:03 GMT+00:00 Roberto Ierusalimschy <[hidden email]>:

>> 2018-07-05 14:52 GMT+02:00 [hidden email] <[hidden email]>:
>>
>> > 'str':fun() and {}:fun() would be nice to see. i
>> > can only remember that the reasoning against this was 'cuz the refman
>> > says so' :D (probably that was you, Dirk, you are the local refman
>> > guru (that i like to see! :) ))
>>
>> I can't find the thread but the clincher for me was a demonstration
>> (not by myself) that a syntax ambiguity could arise.
>
> A syntax ambiguity is already there; these cases would make it worse:
>
>   -- already ambiguous:
>   a = b + c
>   ('str'):fun()
>
>   -- new ambiguity:
>   a = b + c
>   'str':fun()
>
>   -- new ambiguity:
>   a = b + c
>   {}:fun()
>
> -- Roberto

if `c` is a callable, then it can be called w/o parenthesis just as
well the same way. there is already the need for semicolons here and
there in some corner cases, but i'd feel this more consistent around
the "luaverse"

however if any1 can find past talks, or give new pro-contra, then
don't hold back nothing :) i think it was around late 2017, but this
can be misleading if not, and i'd guess it came from Soni, or maybe
someone who is rare around, and i even have an impression that a yahoo
mail was involved, cuz ive got the vision that i've found this in my
spam folder with additional suggestions for some sugar... and fine, i
will do my part of research as well, and throw a pointer if i have it
:)

btw an another stuff that came into my mind is that we need an `if`
block or a `local _=` for an expression like `option and fun()` or
`fun1() and fun2()`... the `local _=` is just too ugly to use, however
we already can call functions without capturing any value that it
returns, and it is not rare at all, and i never had to use a
semicolons cuz of this, however i think its not that rare. so this is
something else that we could think about :D

anyway my use case for the error messages without being trimmed is to
show the place of an error with some extra lines around, however it
can be ambiguous anyhow, like
a/longpath/x.lua --> ...ongpath/x.lua
b/longpath/x.lua --> ...ongpath/x.lua
this way a traceback can be hard to follow in some cornercases...
however even giving fast access can be hard/impossible :/
currently my best guess is to wrap require and co. to keep the paths,
and the worse is when there are more places to visit in some rare
cases, so it's kinda fine to live with, just like a splay-foot :D

if im right, then none of these 3 suggestions would make impossible to
hold compatibility, but there are other points of view for evaluating
these possibilities, but that is the business of the community to
autopsy whatever comes here :D

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Gé Weijers
In reply to this post by Roberto Ierusalimschy


On Thu, Jul 5, 2018 at 09:03 Roberto Ierusalimschy <[hidden email]> wrote:

A syntax ambiguity is already there; these cases would make it worse:

  -- already ambiguous:
  a = b + c
  ('str'):fun()

  -- new ambiguity:
  a = b + c
  'str':fun()

  -- new ambiguity:
  a = b + c
  {}:fun()

-- Roberto


The language BCPL used a fairly simple strategy to detect statement ends:

Let E be the set of symbols that can end a statement, and S be the set of symbols that can start a statement. If the lexical analyzer sees a symbol in E followed by a newline, follow by a symbol in S it inserts a semicolon between them. 

For Lua E would include identifiers, literals and all the closing brackets, and S would include identifiers, literals and opening brackets.

It would be interesting to see whether this could be made to work well for Lua. In terms of LR parsing, the parser would resolve these shift-reduce ambiguities with a shift, except when a newline occurs.

The Go language uses a simplified version of this strategy: it only looks at the symbol before the newline.




--
-- Gė
Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Pierre Chapuis
On Fri, Jul 6, 2018, at 05:18, Gé Weijers wrote:

The language BCPL used a fairly simple strategy to detect statement ends:

Let E be the set of symbols that can end a statement, and S be the set of symbols that can start a statement. If the lexical analyzer sees a symbol in E followed by a newline, follow by a symbol in S it inserts a semicolon between them. 

For Lua E would include identifiers, literals and all the closing brackets, and S would include identifiers, literals and opening brackets.

It would be interesting to see whether this could be made to work well for Lua. In terms of LR parsing, the parser would resolve these shift-reduce ambiguities with a shift, except when a newline occurs.


This would be a pretty big change to the language, since as of today newlines have no special meaning in Lua.

For reference, the manual documents the ambiguity and how it is resolved ( https://www.lua.org/manual/5.3/manual.html#3.3.1 ).
Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

Gé Weijers

On Fri, Jul 6, 2018 at 12:58 AM Pierre Chapuis <[hidden email]> wrote:
This would be a pretty big change to the language, since as of today newlines have no special meaning in Lua.

For reference, the manual documents the ambiguity and how it is resolved ( https://www.lua.org/manual/5.3/manual.html#3.3.1 ).

The (minor) problem is that most people (programmers that are not Lua experts) trying to understand the code would interpret

   a = b + c(print or write)(3)

and

  a = b + c
  (print or write)(3)

to be different things. The BCPL approach does not fix everything either:

  f((a or b)(c)) -- this works
  f((a or b)
    (c)) -- the implicit semicolon would break things.

Compared to the use of 'typename' in C++ to give the parser a hint about "qualified dependent types" it's a minor issue :-)


--
--

Reply | Threaded
Open this post in threaded view
|

Re: Lua has reached a local optimum

szbnwer@gmail.com
Gé Weijers:
the newline stuff is a bit overcomplicated in javascript, therefore i
always use them there, except at the end of scopes, so where are curly
braces. in lua, i think ive never needed any. mostly id like to use
this for ad-hoc strings with :rep, :sub, :gsub and whatever. generally
(in assignment, func argument and concat) they would be totally
innocent and i just feel them redundant. like i cant even find a case
for trouble, an ad-hoc string/table must be used immediately, and in a
function call id write `f(' ':rep(8))`, otherwise its meaning for my
mind and the interpreter `(f' '):rep(5)` (if im right). a semicolon is
fine, and it woulndt complicate nothing as its already the case for
current lua that it is required sometimes... writing stuffs the same
way as currently would still be possible, so compatibility remains.

whats the difference between `a=b+c (' '):rep(8)` and `a=b+c '
':rep(8)`? if `c` is callable then it can be `c(' ')` and `c' '` just
as well...

`f((a or b); (c))` - what?! :D i cant even see the purpose... maybe
for ` f((a or b), c)` but thats a different beast and legit anyhow...


anyway what about `f() or g()` or `option and f()` without using the
value of the expression? i think its nice with lazy evaluation, and
could produce shorter codes with less `if`s.
probably this could take a bit of carefulness:
```
local a, b, c=1, 2 -- `c` probably wont be just an undefined local
f() or g()
--here can come any kinda trouble related to `c`
```
so it can be either `... 1, 2;` or `;f() ...`, but id prefer the 1st,
as that is the dangerous portion, so copypaste will remain safe.



i believe the 1st proposal is in nice harmony with optional braces,
and both match with the mindset of lua users, who already know that
sometimes semicolons are required.

so what else is against these?!



the 3rd proposal with trimmed err messages also wouldnt break nothing.
who never met with it, they didnt prepare nothing related to
`...longpath`, who did, they had to check for `...` and the "didnt
find" branch will handle the new way (if it will be accepted :D ).


thx for your time and all! :)

12