What would you remove from Lua - a case of regression?

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

What would you remove from Lua - a case of regression?

Dibyendu Majumdar
Hi,

One of the difficulties of generating high performance code for Lua in
a _naive_ way is that many op codes have a lot of branching. A simple
add instruction may do:

if (isint(a) && isint(b))
   integer add a, b
else if (isfloat(a) && isfloat(b))
   float add a, b
else if (hasaddmeta(a) || hasaddmeta(b))
   invoke meta add a, b

This is really inefficient when JIT compiled.

Added to this is the need to support coroutines, which can yield in
the middle of a metamethod, so that complicates life further because
after a metamethod call, you cannot assume that the Lua stack is the
same.

So I am thinking of experimenting with a minified Lua - regress back
to 5.1 but on top of that:

* Eliminate metamethods except for __gc.
* No more coroutines

So back to only one number type. As you can imagine this would mean
add can simply be:
a + b

Now, I do not know how often people implement operator overloading -
but given some languages do not have this feature it may not be a big
loss.

I personally can do without coroutines too, I am not sure how widely
they are used; but the trade off is that we eliminate a bunch of
complexity from mini Lua.

I think there are other things I could eliminate such as var args. But
for now, I may start with above. I don't know right now if I should
try to trim down current Ravi code or start with Lua 5.1.

The difficult bit is getting time to do all this.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:

> Hi,
>
> One of the difficulties of generating high performance code for Lua in
> a _naive_ way is that many op codes have a lot of branching. A simple
> add instruction may do:
>
> if (isint(a) && isint(b))
>    integer add a, b
> else if (isfloat(a) && isfloat(b))
>    float add a, b
> else if (hasaddmeta(a) || hasaddmeta(b))
>    invoke meta add a, b
>
> This is really inefficient when JIT compiled.

  There are languages that do type inference, where from the context of the
code it can determine the type of a variable.  So for instance, if all calls
to a function always pass in integers, you can specialize the code for that.
For example, this function:

        -- written for Lua 5.1
        function fromjulianday(jd)
          local a = jd + 32044
          local b = math.floor((4 * a + 3) / 146097)
          local c = a - math.floor((b * 146097) / 4)
          local d = math.floor((4 * c + 3) / 1461)
          local e = c - math.floor((1461 * d) / 4)
          local m = math.floor((5 * e + 2) / 153)
         
          return {
            day   = e - math.floor((153 * m + 2) / 5) + 1,
            month = m + 3 - 12 * math.floor((m / 10)),
            year  = b * 100 + d - 4800 + math.floor(m / 10)
          }
        end

  Given the presence of math.floor(), this indicates the code wants integer
results.  Also, jd is used as if an integer, so this function can be flagged
as taking an integer as its parameter.

  Now, having only heard of type inference (and very rarely used a language
that has it) I don't know how difficult this would be.

> * Eliminate metamethods except for __gc.

  I looked over my own code for uses of metatables.  I use them extensively
for userdata, very rarely for tables, and not at all for the other types.
Furthermore, there are a few methods I've repeatedly used (for both, unless
otherwise noted):

        __mode (table)
        __index
        __newindex
        __tostring (userdata)
        __gc
        __len

  I did find one or two uses of the following in all my code:

        __call (usedata)
        __add (userdata [1])
        __sub (userdata [1])
        __unm (userdata [1])
        __eq (userdata)
        __lt (userdata)
        __le (userdata)
        __concat (userdata)

  But this is just my code.  Take it with a grain of salt.

  At the very least, consider keeping __mode (which is used in conjunction
with the garbage collector), and possibly making a cheap check for __index,
__newindex and __len (if possible).

> * No more coroutines

  This is a deal-breaker for me (more on this below).

> So back to only one number type. As you can imagine this would mean
> add can simply be:
> a + b
>
> Now, I do not know how often people implement operator overloading -
> but given some languages do not have this feature it may not be a big
> loss.

  Personally, almost never.  But I do use LPeg (a lot) and that does use
operator overloading.  And again, for me, this would be a big loss.

> I personally can do without coroutines too, I am not sure how widely
> they are used; but the trade off is that we eliminate a bunch of
> complexity from mini Lua.

  I use them.  Maybe not as often as LPeg, but in the context I use them,
they are invaluable.  I use coroutines to implement servers.  A network
request comes on, I create a coroutine to handle that request.  A simple
echo service:

        local signal    = require "org.conman.signal"
        local nfl       = require "org.conman.nfl"
        local tcp       = require "org.conman.nfl.tcp"
        local addr,port = ...
       
        signal.catch('int')
        signal.catch('term')
       
        tcp.listen(addr or "0.0.0.0",port or 8000,function(ios)
          repeat
            local line = ios:read("*L") -- potential yield spot
            ios:write(line) -- potential yield spot
          until line == ""
          ios:close()
        end)
       
        nfl.eventloop() -- this manages scheduling of coroutines

  It makes the actual code to handle the connection straightforward instead
of chopping it up into an incomprehensible nest of callback functions.

  There are a few other instances were coroutines are nice (such as the
Same Fringe Problem [3]) but for me, it's handling network services.  I
think this would be the common use for coroutines today.

  -spc

[1] One module and that one related to signals [2], used to add and
        remove signals from a signal set (a POSIX thing).

[2] https://github.com/spc476/lua-conmanorg/blob/master/src/signal.c

[3] http://wiki.c2.com/?SameFringeProblem

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Philippe Verdy
Note: math.floor() is not required toi return a native binary integer. It may return a variant type using integers for some ranges, and doubles outside the range. So even in this case the the inference would not help eliminate the necessary tests on the effective type of the variant... The compiler has to know also the value range.
But your existing function fromjulianday(jd) does not check the value range of d, so it's not possible to determine the value range for math.floor() and then reduce it to code using only native integers.

What your code can do however is to favorize the integer case by making branch prediction and avoiding most branches for that case. The CPU will iteself make its own branch prediction anyway and possibly adapt itself using a branch prediction cache.

What is much more useful is to know how instructions are scheduled and pipelined: augment the number of distinct instructions that can handle data in parallel, according to the number of native registers you have and avoid scheduling two costly execution units (like FPUs) in parallel.

Instruction scheduling (and correct understanding of how CPU pipelines are working (and how the central pipeline stage, i.e. execution in a ALU/FPU/VPU port or memory access, can avoid using the same cycles: this also mitigates the time attacks based on dynamic branch predictions and speculative execution).

A good optimizer also can know equivalent patterns of instructions that avoid branch predictions (e.g. it's perfectly possible to implement the function abx(x) without using any conditional branch, by computing the two parallel branches and combining them with a binary or: this uses two parallel pipelines and a final "rendez-vous" where some other independant instructions are scheduled just before computing the binary OR): it is difficult to find other instructions in an isolated abs(x) function, but not if the function is inlined within more complex expressions. Detecting such patterns requires a good database of equivalent binary expressions (e.g. optimizing multipliations by shifts, and optimizing shifts by additions, and avoiding patterns like "store Register[n] to [x]; read Register[m] from [x]" and replacing it by "move Register[n] to [x]; move Register[n] to Register[m]" where the two instructions can run in parallel...). Such rewriting required analyzing the datapaths and which virtual intermediate result is reused later or not, so that you can reduce the number of registers needed. This will then largely reduce the footprint on the datacaches (from the stack or actual native registers).

Note that the ompiler must still be correct semantically (notably for handling exceptions/errors and pcall() correctly).

As well a compiler can autodetect constant expressions like "(a+b)*(c-(a+b))". But this is not easy at all if some elements are actually functions like in "Rand()+Rand()" where not only the two calls to function Rand() is supposed to return different numbers, but also the function must be called twice (the number of calls may be significant is the function uses variables kept across calls in its "closure")


Le dim. 25 nov. 2018 à 23:04, Sean Conner <[hidden email]> a écrit :
It was thus said that the Great Dibyendu Majumdar once stated:
> Hi,
>
> One of the difficulties of generating high performance code for Lua in
> a _naive_ way is that many op codes have a lot of branching. A simple
> add instruction may do:
>
> if (isint(a) && isint(b))
>    integer add a, b
> else if (isfloat(a) && isfloat(b))
>    float add a, b
> else if (hasaddmeta(a) || hasaddmeta(b))
>    invoke meta add a, b
>
> This is really inefficient when JIT compiled.

  There are languages that do type inference, where from the context of the
code it can determine the type of a variable.  So for instance, if all calls
to a function always pass in integers, you can specialize the code for that.
For example, this function:

        -- written for Lua 5.1
        function fromjulianday(jd)
          local a = jd + 32044
          local b = math.floor((4 * a + 3) / 146097)
          local c = a - math.floor((b * 146097) / 4)
          local d = math.floor((4 * c + 3) / 1461)
          local e = c - math.floor((1461 * d) / 4)
          local m = math.floor((5 * e + 2) / 153)

          return {
            day   = e - math.floor((153 * m + 2) / 5) + 1,
            month = m + 3 - 12 * math.floor((m / 10)),
            year  = b * 100 + d - 4800 + math.floor(m / 10)
          }
        end

  Given the presence of math.floor(), this indicates the code wants integer
results.  Also, jd is used as if an integer, so this function can be flagged
as taking an integer as its parameter.

  Now, having only heard of type inference (and very rarely used a language
that has it) I don't know how difficult this would be.

> * Eliminate metamethods except for __gc.

  I looked over my own code for uses of metatables.  I use them extensively
for userdata, very rarely for tables, and not at all for the other types.
Furthermore, there are a few methods I've repeatedly used (for both, unless
otherwise noted):

        __mode          (table)
        __index
        __newindex
        __tostring      (userdata)
        __gc
        __len

  I did find one or two uses of the following in all my code:

        __call          (usedata)
        __add           (userdata [1])
        __sub           (userdata [1])
        __unm           (userdata [1])
        __eq            (userdata)
        __lt            (userdata)
        __le            (userdata)
        __concat        (userdata)

  But this is just my code.  Take it with a grain of salt.

  At the very least, consider keeping __mode (which is used in conjunction
with the garbage collector), and possibly making a cheap check for __index,
__newindex and __len (if possible).

> * No more coroutines

  This is a deal-breaker for me (more on this below).

> So back to only one number type. As you can imagine this would mean
> add can simply be:
> a + b
>
> Now, I do not know how often people implement operator overloading -
> but given some languages do not have this feature it may not be a big
> loss.

  Personally, almost never.  But I do use LPeg (a lot) and that does use
operator overloading.  And again, for me, this would be a big loss.

> I personally can do without coroutines too, I am not sure how widely
> they are used; but the trade off is that we eliminate a bunch of
> complexity from mini Lua.

  I use them.  Maybe not as often as LPeg, but in the context I use them,
they are invaluable.  I use coroutines to implement servers.  A network
request comes on, I create a coroutine to handle that request.  A simple
echo service:

        local signal    = require "org.conman.signal"
        local nfl       = require "org.conman.nfl"
        local tcp       = require "org.conman.nfl.tcp"
        local addr,port = ...

        signal.catch('int')
        signal.catch('term')

        tcp.listen(addr or "0.0.0.0",port or 8000,function(ios)
          repeat
            local line = ios:read("*L") -- potential yield spot
            ios:write(line) -- potential yield spot
          until line == ""
          ios:close()
        end)

        nfl.eventloop() -- this manages scheduling of coroutines

  It makes the actual code to handle the connection straightforward instead
of chopping it up into an incomprehensible nest of callback functions.

  There are a few other instances were coroutines are nice (such as the
Same Fringe Problem [3]) but for me, it's handling network services.  I
think this would be the common use for coroutines today.

  -spc

[1]     One module and that one related to signals [2], used to add and
        remove signals from a signal set (a POSIX thing).

[2]     https://github.com/spc476/lua-conmanorg/blob/master/src/signal.c

[3]     http://wiki.c2.com/?SameFringeProblem

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Gé Weijers
In reply to this post by Dibyendu Majumdar
> On Nov 25, 2018, at 09:40, Dibyendu Majumdar <[hidden email]> wrote:
> So I am thinking of experimenting with a minified Lua - regress back
> to 5.1 but on top of that:
>
> * Eliminate metamethods except for __gc.
> * No more coroutines
>
> So back to only one number type. As you can imagine this would mean
> add can simply be:
> a + b

You would still have to do a type check before adding the values,
because you may be adding a table to nil in stead of two numbers. Why
not call a runtime system routine that implements the full add
functionality if the type check fails?

> Now, I do not know how often people implement operator overloading -
> but given some languages do not have this feature it may not be a big
> loss.

I have used it in a module implementing date and calendar arithmetic,
and in some other cases.

> I personally can do without coroutines too, I am not sure how widely
> they are used; but the trade off is that we eliminate a bunch of
> complexity from mini Lua.

Coroutines are very useful in event driven programs, I use them all
the time to implement protocols/state machines. You can use callbacks
like you see in Javascript programs, but coroutines make the code much
more readable, the code is not littered with lambdas.

If removing complexity from the language adds complexity to the
programs written in the language you may be going down the wrong path.

An implementation of “mini Lua” would not be all that useful to me,
Lua 5.x has just about the right compromise of simplicity and
expressive power for me, and I can add any missing pieces through the
C interface. I’m currently using Lua and some custom C code for
in-house test scripts that interface to prototype hardware, and it’s
using about 1% of 1 CPU, so a JIT compiler would not be particularly
useful to have in this case.



Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Muh Muhten
In reply to this post by Philippe Verdy
On 11/25/18, Philippe Verdy <[hidden email]> wrote:
> Note: math.floor() is not required toi return a native binary integer. It
> may return a variant type using integers for some ranges, and doubles
> outside the range. So even in this case the the inference would not help
> eliminate the necessary tests on the effective type of the variant... The

math.floor can return literally anything. It can be redefined.

In the case where it _isn't_, this is pointless pedantry since it's
not unreasonable to expect the compiler to be aware of (idioms
involving) the stdlib. (And vice versa.)

> Note that the ompiler must still be correct semantically (notably for
> handling exceptions/errors and pcall() correctly).

Pray tell, what does this entail under the pretext of _changing the
semantics_ to make things easier for the compiler?

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Dirk Laurie-2
In reply to this post by Dibyendu Majumdar
Op So. 25 Nov. 2018 om 19:40 het Dibyendu Majumdar
<[hidden email]> geskryf:
> [snip - TLDR]

We've been here before.
    http://lua-users.org/lists/lua-l/2018-07/msg00700.html
I'll just repeat a remark I made then and not play along again this time.

Look at the specs of Lua 4.0 to find a quite decent little language
with far fewer features than Lua 5.3 has.

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Philippe Verdy
In reply to this post by Muh Muhten
Le lun. 26 nov. 2018 à 01:21, Muh Muhten <[hidden email]> a écrit :
On 11/25/18, Philippe Verdy <[hidden email]> wrote:
> Note: math.floor() is not required toi return a native binary integer. It
> may return a variant type using integers for some ranges, and doubles
> outside the range. So even in this case the the inference would not help
> eliminate the necessary tests on the effective type of the variant... The

math.floor can return literally anything. It can be redefined
 
Here I was speaking about the standard library function, without any redefinition: it takes a 'number' (usually a double when compiling Lua itself) and returns a 'number' which is not limited to a 53-bit integer).

So the function computing the Julian day MUST still contain a code testing the value to make sure it is in range. Technically the given sample function was NOT correct as it returns random value if it is not. But it will be correct if "number" is a 64-bit double, and the function first check that the parameter is within +/-2^31 before performing all the rest in the "if" branch (the else branch may just return nil to indicate the error or call error()); then the compiler know that the given number is not a NaN and not infinite; and the standard function math.floor (the compiler can know that it is not overloaded) is warrantied to return a 32-bit integer. Then it can perform optimization of the arithmetic code that follows (it still needs to check that arthmetic operators are also not overloaded).

Operators can be redefined too ! The code "1+1" in Lua is not even warrantied to return 2 if "+" is redefined (in function contexts where metatable contain an overload).

In summary the compiler needs to check the environment (it is easy to generate two compiled codes version depending if the environment uses the standard libary or not, it's not so complex because it is determined only by the standard Lua language semantics), before doing any kind of type inference (which is more complex: tracing the possible value range is much more complex as it depends on preevaluating all possible limits!)

A simple check like "if (d>>0 == d) { ... } else return nil" within the Lua functio nto compiler would do the trick where the compiler can determine that d is effectively using an integer in correct range for integers. but the actual test **in Lua** is a bit more complex because "local a = jd + 32044" can overflow in 32-bit integers, just like the subexpression "(4 * a + 3)" used in the next statement of the sample: a 32-bit check is not enough to ensure the result would be correct : the result of the function is actually correct if "jd" is in range of 29-bit integers so the initial test should be  "if (d>>3<<3 == d) { ... } else return nil"  ...

So the compiler would need to track all constant and variable value ranges to make inference. Type inference is not enough !


In the case where it _isn't_, this is pointless pedantry since it's
not unreasonable to expect the compiler to be aware of (idioms
involving) the stdlib. (And vice versa.)

So you're creating another language which is not Lua, if you remove parts of its core features. The compiler you're creating is not a compiler for Lua...
 

> Note that the ompiler must still be correct semantically (notably for
> handling exceptions/errors and pcall() correctly).

Pray tell, what does this entail under the pretext of _changing the
semantics_ to make things easier for the compiler?

Changing the semantics of the language means you're creating a compiler for another language. Lua cannot work at all without exception/error handling
Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Sean Conner
In reply to this post by Muh Muhten
It was thus said that the Great Muh Muhten once stated:
> On 11/25/18, Philippe Verdy <[hidden email]> wrote:
> > Note: math.floor() is not required toi return a native binary integer. It
> > may return a variant type using integers for some ranges, and doubles
> > outside the range. So even in this case the the inference would not help
> > eliminate the necessary tests on the effective type of the variant... The
>
> math.floor can return literally anything. It can be redefined.

  Of course, one could rewrite the function to remove the call to
math.floor() to apease the Great Pedantic One:

        function fromjulianday(jd)
          local a = jd + 32044
          local b = (4 * a + 3) // 146097
          local c = a - (b * 146097) // 4
          local d = (4 * c + 3) // 1461
          local e = c - (1461 * d) // 4
          local m = (5 * e + 2) // 153
         
          return {
            day   = e - (153 * m + 2) // 5 + 1,
            month = m + 3 - 12 * (m // 10),
            year  = b * 100 + d - 4800 + (m // 10)
          }
        end

  This is Lua 5.3 specific code where '//' is integer division.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Philippe Verdy
You've not understood the "Great Pedantic" (sic! visibly you use that "The Great" expression as a directed insult) at all and what I really wanted to show: your suggestion does not change anything to what I meant (you've removed the call to the library "math.floor" (which does nbot matter at all, I was just speaking about what it meant, not the fact it could be overriden: the fact that even without changing/overridening it it still does not warranties to return a valid 32-bit integer).

Now you've used the "//" operator, which also does not warranty it (even without changing/overriding it): it still operates on Lua "number", which are floating points, and returns a floating point value whose value may STILL be out of range for native integers, so it's still not safe to "optimize" the compilation of this code using native integers, because this is still not what the Lua code effectively does: such compilation would be broken as well (an I insist: this is is true EVEN if there's no override at all for the Lua arithmetic operators).

The "//" operator can return a value like 1e99 which is an "integer", but given as a floatting point value, completely out of range for native integers (even if these native integers are 64 bit!). And the same applies to the math.floor() operation. 

Do you see now what I mean?

The rest of the function can still be computed, but will give "almost random"/"non-meaningful" date results if you don't take into account the maximum precision of floating point values in "Lua number" (which is 53 bit if "number" is compiled as an IEEE 64-bit double), but which successfully returns correct values when "jd" is in the range of a 50-bit signed integer.

Note also that this function is not correct if "jd" is negative: the "a // b" operation in Lua is NOT returning a "floor(a/b)" operation but a "floor(abs(a)/abs(b))*sign(a)*sign(b)" operation. But it is still defined for the whole range of Lua "numbers" (even if their absolute value is higher than 2^52 and then cannot store any fractional bits and can only represent *some* integers by rounding them to some *other* integers)

You are then making false assumptions about what are Lua numbers: they are not a simple mix of native binary integers and native floating points, they only represent floatting point values with limited precision, but much wider value ranges (plus special values for "NaNs", "infinites", "negative zero", plus "denormal values" which are normal finite numbers but with limited precision; all of them do not exist as native binary integers)

The Lua code as it is written is not simply optimizable using 32-bit or 64-bit native integers without taking into account the precision and value range of numbers (independantly of the fact that the function does not really test for the actual Lua type of its parameter).

So the real Lua code actually makes tests everywhere about types to infer which function will be used to make every step (including in the Lua virtual machine which operates only on "numbers"), and will then determine how to handle the value range and precision correctly: there are type checking and bound checking at every step).





Le lun. 26 nov. 2018 à 07:59, Sean Conner <[hidden email]> a écrit :
It was thus said that the Great Muh Muhten once stated:
> On 11/25/18, Philippe Verdy <[hidden email]> wrote:
> > Note: math.floor() is not required toi return a native binary integer. It
> > may return a variant type using integers for some ranges, and doubles
> > outside the range. So even in this case the the inference would not help
> > eliminate the necessary tests on the effective type of the variant... The
>
> math.floor can return literally anything. It can be redefined.

  Of course, one could rewrite the function to remove the call to
math.floor() to apease the Great Pedantic One:

        function fromjulianday(jd)
          local a = jd + 32044
          local b = (4 * a + 3) // 146097
          local c = a - (b * 146097) // 4
          local d = (4 * c + 3) // 1461
          local e = c - (1461 * d) // 4
          local m = (5 * e + 2) // 153

          return {
            day   = e - (153 * m + 2) // 5 + 1,
            month = m + 3 - 12 * (m // 10),
            year  = b * 100 + d - 4800 + (m // 10)
          }
        end

  This is Lua 5.3 specific code where '//' is integer division.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Dibyendu Majumdar
In reply to this post by Sean Conner
On Sun, 25 Nov 2018 at 22:04, Sean Conner <[hidden email]> wrote:
>   There are languages that do type inference, where from the context of the
> code it can determine the type of a variable.

Yes that is an option that I would like to persue. One problem is
whenever you have return value from a function call you don't know
what the result will be ... so at that point you have to go back to
not knowing the type

The ideal solution is to specialize functions by types. I used to
think the function should be specialized upon entry - but it may make
more sense to specialize at call site. But return values are still a
problem.

>
> > * Eliminate metamethods except for __gc.
>
>   I looked over my own code for uses of metatables.  I use them extensively
> for userdata, very rarely for tables, and not at all for the other types.
> Furthermore, there are a few methods I've repeatedly used (for both, unless
> otherwise noted):
>
>         __mode          (table)
>         __index
>         __newindex
>         __tostring      (userdata)
>         __gc
>         __len

Perhaps I should retain these. Certainly I intend to get rid of the
arithmetic operators to start with.

> > Now, I do not know how often people implement operator overloading -
> > but given some languages do not have this feature it may not be a big
> > loss.
>
>   Personally, almost never.  But I do use LPeg (a lot) and that does use
> operator overloading.  And again, for me, this would be a big loss.
>

Yes LPeg is one of the heavy users of operator overloading.

> > I personally can do without coroutines too, I am not sure how widely
> > they are used; but the trade off is that we eliminate a bunch of
> > complexity from mini Lua.
>
>   I use them.  Maybe not as often as LPeg, but in the context I use them,
> they are invaluable.  I use coroutines to implement servers.

I think it would be nice to have two VMs built-in - a full featured
one and a cut-down one, with user being able to choose the one they
want to use. But it is harder to switch between dual number type to a
single number type.


Thanks for the feedback.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Dibyendu Majumdar
In reply to this post by Gé Weijers
On Sun, 25 Nov 2018 at 23:28, Gé Weijers <[hidden email]> wrote:
>
> > On Nov 25, 2018, at 09:40, Dibyendu Majumdar <[hidden email]> wrote:

> > So back to only one number type. As you can imagine this would mean
> > add can simply be:
> > a + b
>
> You would still have to do a type check before adding the values,
> because you may be adding a table to nil in stead of two numbers. Why
> not call a runtime system routine that implements the full add
> functionality if the type check fails?

It seems compilers are happier if your code is transparent, i.e.
compiler can see what is going on. The moment you call a function, it
goes into a blackhole so to speak.
You are right that type checks are still needed, But they can of the form:

if (isnum(a) && isnum(b))
  c = a + b
else goto error;
if (isnum(d) && isnum(f))
  e = d / f
else goto error;

The compiler can see what is happening and having a single error
handling block helps too.

> If removing complexity from the language adds complexity to the
> programs written in the language you may be going down the wrong path.
>

There is always a tradeoff otherwise why use Lua? Why not use the most
powerful language available. I am considering whether it is possible
to have a dual VM system, a full featured VM and a cutdown VM,
selectable by user.

> An implementation of “mini Lua” would not be all that useful to me,
> Lua 5.x has just about the right compromise of simplicity and
> expressive power for me, and I can add any missing pieces through the
> C interface. I’m currently using Lua and some custom C code for
> in-house test scripts that interface to prototype hardware, and it’s
> using about 1% of 1 CPU, so a JIT compiler would not be particularly
> useful to have in this case.
>

To be honest as I have posted before Lua's performance is usually
fine; I have not had any issues. But it does seem that LuaJIT has made
people think performance matters. It does maybe in some restricted use
cases where Lua is used almost like a proxy for C. But looking at
JavaScript as an example, performance and JIT compilation has no doubt
been part of the reason for its success. The problem with Lua is how
to achieve the same level of performance while keeping to the 'spirit
of Lua' - i.e. small and beautiful rather than large and ugly
implementation. LuaJIT is one way - and might have been the only way
if the effort to understand and maintain it were in the realms of what
is human.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Philippe Verdy-2
In reply to this post by Dibyendu Majumdar
Scientific apps are very likely to overload operators to work on less limited numeric systems or microstructure semantics than just plus numbers whose precision and value range are unspecified, but also have undesired values that we want to restrict, or when we need more dimensions than just one. This requires developing a new class where all operators will be overridden, even if internally they are implemented with numbers.

Unfortunately we still cannot use any strict integer type in Lua without the cost of IEEE doubles, and Lua is already inconsistent with some number operators like shifts and binary ops which truncate operands to ints but return the result by reconverting it to doubles, looking their initial restriction of range and precision.

As well Lua still does not standardize (like C or C++, or C#, or JavaScript or Java, or even COBOL, SQL, and RDF) any way to know the limits of its own type system. This creates portability problems. This is a problem also shared by PHP.

Le lun. 26 nov. 2018 à 20:33, Dibyendu Majumdar <[hidden email]> a écrit :
On Sun, 25 Nov 2018 at 22:04, Sean Conner <[hidden email]> wrote:
>   There are languages that do type inference, where from the context of the
> code it can determine the type of a variable.

Yes that is an option that I would like to persue. One problem is
whenever you have return value from a function call you don't know
what the result will be ... so at that point you have to go back to
not knowing the type

The ideal solution is to specialize functions by types. I used to
think the function should be specialized upon entry - but it may make
more sense to specialize at call site. But return values are still a
problem.

>
> > * Eliminate metamethods except for __gc.
>
>   I looked over my own code for uses of metatables.  I use them extensively
> for userdata, very rarely for tables, and not at all for the other types.
> Furthermore, there are a few methods I've repeatedly used (for both, unless
> otherwise noted):
>
>         __mode          (table)
>         __index
>         __newindex
>         __tostring      (userdata)
>         __gc
>         __len

Perhaps I should retain these. Certainly I intend to get rid of the
arithmetic operators to start with.

> > Now, I do not know how often people implement operator overloading -
> > but given some languages do not have this feature it may not be a big
> > loss.
>
>   Personally, almost never.  But I do use LPeg (a lot) and that does use
> operator overloading.  And again, for me, this would be a big loss.
>

Yes LPeg is one of the heavy users of operator overloading.

> > I personally can do without coroutines too, I am not sure how widely
> > they are used; but the trade off is that we eliminate a bunch of
> > complexity from mini Lua.
>
>   I use them.  Maybe not as often as LPeg, but in the context I use them,
> they are invaluable.  I use coroutines to implement servers.

I think it would be nice to have two VMs built-in - a full featured
one and a cut-down one, with user being able to choose the one they
want to use. But it is harder to switch between dual number type to a
single number type.


Thanks for the feedback.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Dibyendu Majumdar
On Mon, 26 Nov 2018 at 21:21, Philippe Verdy <[hidden email]> wrote:
> Unfortunately we still cannot use any strict integer type in Lua without the cost of IEEE doubles, and Lua is already inconsistent with some number operators like shifts and binary ops which truncate operands to ints but return the result by reconverting it to doubles, looking their initial restriction of range and precision.
>

Hi, Lua 5.3 has 64-bit integer types - I was not sure whether you are
thinking of an earlier version of Lua above.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Philippe Verdy-2
That is not true. Lua is designed to allow its number type to be configurable but in its default configuration it just uses what C gives as a double.

Nothing indicates it can store any 64 bit integer and in fact it can almost never store them in a single number, unless Lua is specifically compiled using long double in C and they are represented as 80 bit with a 64 bit mantissa part...

No there's no standard way to know the limits of numbers in lua. You need to test them by trying some operations in test loops during initialization of your code, and then use these dynamic results as if they were constants. But a Lua interpreter or compiler will never know that they are constants after the initialisation when bref to be variables during the initialization where their final value is still not known.

Le lun. 26 nov. 2018 à 22:32, Dibyendu Majumdar <[hidden email]> a écrit :
On Mon, 26 Nov 2018 at 21:21, Philippe Verdy <[hidden email]> wrote:
> Unfortunately we still cannot use any strict integer type in Lua without the cost of IEEE doubles, and Lua is already inconsistent with some number operators like shifts and binary ops which truncate operands to ints but return the result by reconverting it to doubles, looking their initial restriction of range and precision.
>

Hi, Lua 5.3 has 64-bit integer types - I was not sure whether you are
thinking of an earlier version of Lua above.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Sean Conner
In reply to this post by Dibyendu Majumdar
It was thus said that the Great Dibyendu Majumdar once stated:
> On Sun, 25 Nov 2018 at 22:04, Sean Conner <[hidden email]> wrote:
> >   There are languages that do type inference, where from the context of the
> > code it can determine the type of a variable.
>
> Yes that is an option that I would like to persue. One problem is
> whenever you have return value from a function call you don't know
> what the result will be ... so at that point you have to go back to
> not knowing the type

  Only for functions called before their definition.  My gut says that
doesn't happen as often as one may think, but as always, measure.  Perhaps
look at luarocks and luacheck (two sizable Lua applications, probably a good
representation of Lua usage) and check to see how often a function is called
prior to its defintion.  

  Also, isn't Ravi a JIT?  I would think the first time through encountering
a situation, you check the return code, but on subsequent visits you may
have the information to re-JIT that part of the code.

  Again, not doing this I have no idea of the level of effort involed, I'm
just throwing out ideas here.

> > > * Eliminate metamethods except for __gc.
> >
> >   I looked over my own code for uses of metatables.  I use them extensively
> > for userdata, very rarely for tables, and not at all for the other types.
> > Furthermore, there are a few methods I've repeatedly used (for both, unless
> > otherwise noted):
> >
> >         __mode          (table)
> >         __index
> >         __newindex
> >         __tostring      (userdata)
> >         __gc
> >         __len
>
> Perhaps I should retain these. Certainly I intend to get rid of the
> arithmetic operators to start with.

  At the very least, __mode should be retained since it helps guide the GC.

> Thanks for the feedback.

  You're welcome.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Philippe Verdy-2
In reply to this post by Philippe Verdy-2
The same is true for Lua 5.3 when it also adds integers with unspecified limits that are just by default bound to what the C compiler gives in it's int datatype but also allows configuring that type.

Lua numeric types have unpredictable limits

Le lun. 26 nov. 2018 à 22:46, Philippe Verdy <[hidden email]> a écrit :
That is not true. Lua is designed to allow its number type to be configurable but in its default configuration it just uses what C gives as a double.

Nothing indicates it can store any 64 bit integer and in fact it can almost never store them in a single number, unless Lua is specifically compiled using long double in C and they are represented as 80 bit with a 64 bit mantissa part...

No there's no standard way to know the limits of numbers in lua. You need to test them by trying some operations in test loops during initialization of your code, and then use these dynamic results as if they were constants. But a Lua interpreter or compiler will never know that they are constants after the initialisation when bref to be variables during the initialization where their final value is still not known.

Le lun. 26 nov. 2018 à 22:32, Dibyendu Majumdar <[hidden email]> a écrit :
On Mon, 26 Nov 2018 at 21:21, Philippe Verdy <[hidden email]> wrote:
> Unfortunately we still cannot use any strict integer type in Lua without the cost of IEEE doubles, and Lua is already inconsistent with some number operators like shifts and binary ops which truncate operands to ints but return the result by reconverting it to doubles, looking their initial restriction of range and precision.
>

Hi, Lua 5.3 has 64-bit integer types - I was not sure whether you are
thinking of an earlier version of Lua above.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Dibyendu Majumdar
In reply to this post by Philippe Verdy-2
On Mon, 26 Nov 2018 at 21:47, Philippe Verdy <[hidden email]> wrote:
>
> That is not true. Lua is designed to allow its number type to be configurable but in its default configuration it just uses what C gives as a double.
>
> Nothing indicates it can store any 64 bit integer and in fact it can almost never store them in a single number, unless Lua is specifically compiled using long double in C and they are represented as 80 bit with a 64 bit mantissa part...
>

Hi, above is incorrect. I know as I maintain Ravi a derivative of Lua
5.3. Integers are maintained natively as integer values in 5.3.
Conversion to double happens as per C promotion rules - when you do
operations that involve integer and double values. I am a bit
concerned that you may be stating things as if they are true when they
are not.

I would suggest you try out Lua 5.3 and look at its source code.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:

> On Mon, 26 Nov 2018 at 21:47, Philippe Verdy <[hidden email]> wrote:
> >
> > That is not true. Lua is designed to allow its number type to be
> > configurable but in its default configuration it just uses what C gives
> > as a double.
> >
> > Nothing indicates it can store any 64 bit integer and in fact it can
> > almost never store them in a single number, unless Lua is specifically
> > compiled using long double in C and they are represented as 80 bit with
> > a 64 bit mantissa part...
> >
>
> Hi, above is incorrect. I know as I maintain Ravi a derivative of Lua
> 5.3. Integers are maintained natively as integer values in 5.3.
> Conversion to double happens as per C promotion rules - when you do
> operations that involve integer and double values. I am a bit
> concerned that you may be stating things as if they are true when they
> are not.
>
> I would suggest you try out Lua 5.3 and look at its source code.

  It's even stated in the Lua 5.3 Reference Manual, section 2.1:

        Standard Lua uses 64-bit integers and double-precision (64-bit)
        floats, but you can also compile Lua so that it uses 32-bit integers
        and/or single-precision (32-bit) floats.

                        http://lucy/~spc/docs/Lua-5.3/manual.html

  Also, the constants math.maxinteger and math.mininteger are defined to
determine the limits of integers in Lua 5.3.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Gavin Wraith
In reply to this post by Dibyendu Majumdar
In message <[hidden email]>
          Dibyendu Majumdar <[hidden email]> wrote:

>On Sun, 25 Nov 2018 at 22:04, Sean Conner <[hidden email]> wrote:
>>   There are languages that do type inference, where from the context of the
>> code it can determine the type of a variable.

I have always been a fan of Lua, and have great respect for the decisions
made in its development; so a thread like this about what one would like
to remove, add or change is inescapably invidious for me.

I would like to see a reversal of the 'numbers are just numbers' policy;
that is, I would like more of a type barrier between integers and floats
for example. I would like to see
> type (57) --> integer
> type (57.0) --> float
Integers are always needed by the interpreter for its own internal
purposes - the "necessary numbers", but floats, or other number
systems (big numbers, complex numbers, modular arithmetic, ... ) are better suited for libraries, and their suitability is platform dependent - they
are "optional number systems".
I would like to see the type function capable of extension to tables with metatables with a __type index.

With numbers that take arbitrary amount of storage, there is no
avoiding them using heap storage. But with numbers that use a fixed
number of bytes, but more than are required for a value on the stack,
there is a choice: either you bung them in a union type with other
values on the stack, so that a lot of the stack is wasted space, or
you have have an auxiliary stack for them, and every time you push a number
to the auxiliary stack you also push a pointer to it onto the ordinary
stack. There has to be some jargon for this - sorry I do not know it.
There is an obvious space/time tradeoff between the two approaches.
Somebody's thesis somewhere?
The latter has the advantage that it makes for more uniform
handling of optional number systems with fixed size storage.
The auxiliary stack requires no garbage collection, of course.
You might have 32-bit integers internally, and 256-bit bigintegers for cryptographic purposes, for example.

--
Gavin Wraith ([hidden email])
Home page: http://www.wra1th.plus.com/

Reply | Threaded
Open this post in threaded view
|

Re: What would you remove from Lua - a case of regression?

Dibyendu Majumdar
On Mon, 26 Nov 2018 at 22:01, Gavin Wraith <[hidden email]> wrote:

> I would like to see a reversal of the 'numbers are just numbers' policy;
> that is, I would like more of a type barrier between integers and floats
> for example. I would like to see
> > type (57) --> integer
> > type (57.0) --> float

This is already true. Try this:

math.type(57)
math.type(57.0)

12