future of annotations in Lua?

classic Classic list List threaded Threaded
57 messages Options
123
Reply | Threaded
Open this post in threaded view
|

future of annotations in Lua?

Hisham
A pattern in Lua's history is that some features appear first as a
special-case feature that solves a narrow set of problems, and are
later generalized into a broader mechanism.

Lua first had fallbacks, then those became tag methods, which
eventually became metatables.

Lua first had %var upvalues, then eventually gained lexical scoping.

If the same happens to the <toclose> and <const> constructs, the
obvious direction for them as a broad mechanism is to become
decorators/annotations/attributes in the future.

* Python decorators: https://wiki.python.org/moin/PythonDecorators
* Java annotations: https://en.wikipedia.org/wiki/Java_annotation
* C# attributes:
https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/concepts/attributes/
* Swift annotations:
https://docs.swift.org/swift-book/ReferenceManual/Attributes.html

(funny that there doesn't seem to be a standard name for this, but
perhaps the different names are there to signal the variations in
semantics. I'll call them annotations as it wins my sampling by a
narrow margin.)

Python, Java and Swift use the @annotation syntax; C# uses the
[annotation] syntax.

Given that a future generalization of annotations would probably mean
that the annotation name would be user-extensible as any identifier,
Lua 5.4's current choice of syntax means that an eventual
generalization would add `'<' Ident '>'` to the grammar. Given that
such generalization as seen in other languages might include adding
annotations to function declarations, we would end up with `'<' Ident
'>' 'function' Name '(' Args ')'`.

Taking a page from past struggles of the C++ grammar, I'm concerned
that using '<' like that would introduce ambiguites that would
preclude general function annotations in the commonly-adopted
annotation style. The following would be hard to parse without
significant lookahead:

   local x = y

   <my_custom_annotation(with, arguments)>
   function my_global_function()
      -- ...
   end

If the door is to be left open for such a generalized mechanism in
future versions of Lua beyond 5.4, then adopting a syntax based on an
unused token such as @annotation would be advisable. Matching the
annotation syntax of other languages would be a side benefit.

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Dibyendu Majumdar
On Thu, 6 Jun 2019 at 21:42, Hisham <[hidden email]> wrote:

> If the door is to be left open for such a generalized mechanism in
> future versions of Lua beyond 5.4, then adopting a syntax based on an
> unused token such as @annotation would be advisable. Matching the
> annotation syntax of other languages would be a side benefit.
>

+1

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Lorenzo Donati-3
In reply to this post by Hisham
On 06/06/2019 22:41, Hisham wrote:
> A pattern in Lua's history is that some features appear first as a
> special-case feature that solves a narrow set of problems, and are
> later generalized into a broader mechanism.
>

[snip]


> If the door is to be left open for such a generalized mechanism in
> future versions of Lua beyond 5.4, then adopting a syntax based on an
> unused token such as @annotation would be advisable. Matching the
> annotation syntax of other languages would be a side benefit.
>
> -- Hisham
>
>

I fully agree!

As I wrote in another thread (in a sub-sub thread - urgh! this 5.4 thing
is stirring lua-l quite a lot :-) compare:

local <toclose, static, nice, helpful, wonderful> resource = ....

vs.

local @toclose @static @nice @helpful @wonderful resource = ....

and also regarding code hard wrapping:

local < toclose, static,
   nice, helpful, wonderful > resource = ....

vs.

local @toclose @static
   @nice @helpful @wonderful resource = ....

The more I think about it, the more I find the syntax with "@" more
readable and more easily "expandable": parametrized annotations anyone?
Like for example:

local @const myTable = @table(64) {} -- preallocates 64 elements in the
array part


BTW, as for naming, I'd prefer to call them "attributes" more than
"annotations". The latter conveys the idea that the "annotation" doesn't
change the semantics of the code, whereas "attribute" gives a more
general idea of "something that gives additional characterization".

My 2 EURcents! :-)

Cheers!

-- Lorenzo






Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
Le jeu. 6 juin 2019 à 23:35, Lorenzo Donati <[hidden email]> a écrit :
On 06/06/2019 22:41, Hisham wrote:
> A pattern in Lua's history is that some features appear first as a
> special-case feature that solves a narrow set of problems, and are
> later generalized into a broader mechanism.
>

[snip]


> If the door is to be left open for such a generalized mechanism in
> future versions of Lua beyond 5.4, then adopting a syntax based on an
> unused token such as @annotation would be advisable. Matching the
> annotation syntax of other languages would be a side benefit.
>
> -- Hisham
>
>

I fully agree!

As I wrote in another thread (in a sub-sub thread - urgh! this 5.4 thing
is stirring lua-l quite a lot :-) compare:

local <toclose, static, nice, helpful, wonderful> resource = ....

vs.

local @toclose @static @nice @helpful @wonderful resource = ....

and also regarding code hard wrapping:

local < toclose, static,
   nice, helpful, wonderful > resource = ....

vs.

local @toclose @static
   @nice @helpful @wonderful resource = ....

The more I think about it, the more I find the syntax with "@" more
readable and more easily "expandable": parametrized annotations anyone?
Like for example:

local @const myTable = @table(64) {} -- preallocates 64 elements in the
array part

The syntax "@table(64){}" is ambiguous, it can be thought of the annotation "@table" for the expression 64, which is the right handside of a binary operator (a function call).

Consider: "@table(t){}" ...

The only way is to surround annotations with parameters by parentheses:
  (@table(64)) {}
but we can also drop the unnecessary inner parentheses for the parameters of the annotation itself:
  (@table 64) {}

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
And if someone does not like the "@", we can still use the proposed "*":
  (*table 64) {}
Here also the inner parentheses surrounding the parameters of the annotation are removed, exactly and consistantly with the currification of regular function calls.

Le ven. 7 juin 2019 à 10:10, Philippe Verdy <[hidden email]> a écrit :
Le jeu. 6 juin 2019 à 23:35, Lorenzo Donati <[hidden email]> a écrit :
On 06/06/2019 22:41, Hisham wrote:
> A pattern in Lua's history is that some features appear first as a
> special-case feature that solves a narrow set of problems, and are
> later generalized into a broader mechanism.
>

[snip]


> If the door is to be left open for such a generalized mechanism in
> future versions of Lua beyond 5.4, then adopting a syntax based on an
> unused token such as @annotation would be advisable. Matching the
> annotation syntax of other languages would be a side benefit.
>
> -- Hisham
>
>

I fully agree!

As I wrote in another thread (in a sub-sub thread - urgh! this 5.4 thing
is stirring lua-l quite a lot :-) compare:

local <toclose, static, nice, helpful, wonderful> resource = ....

vs.

local @toclose @static @nice @helpful @wonderful resource = ....

and also regarding code hard wrapping:

local < toclose, static,
   nice, helpful, wonderful > resource = ....

vs.

local @toclose @static
   @nice @helpful @wonderful resource = ....

The more I think about it, the more I find the syntax with "@" more
readable and more easily "expandable": parametrized annotations anyone?
Like for example:

local @const myTable = @table(64) {} -- preallocates 64 elements in the
array part

The syntax "@table(64){}" is ambiguous, it can be thought of the annotation "@table" for the expression 64, which is the right handside of a binary operator (a function call).

Consider: "@table(t){}" ...

The only way is to surround annotations with parameters by parentheses:
  (@table(64)) {}
but we can also drop the unnecessary inner parentheses for the parameters of the annotation itself:
  (@table 64) {}

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
with surrounding parentheses around "complex" annotations, we have much more freedom to choose the first token: not only we can use binary operators (that are not also unary operators) like ":", ".", "*", "<", but we can also reuse other existing keywords such as:
  (if expression) {}

which is another kind of annotation similar to some preprocessor condition applied to the unary expression "{}" that follows it, possibly extended to:
  (if expression1, else error) {}

which adds an "else" parameter to the "if" annotation, also as an annotation with parameter "error", so the "if" has two parameters: "expression1" and "else error"

This can be used for example to perform platform-specific optimizations:
  (if target.platform~='i386', else asm 'pushf; pop ax') flags()

This means that the "flags()" expression is annotated by a conditional "if" annotation that checks if the target.platform is 'i386' and replaces in that case the expression by the assembly code, also given by another annotation, "asm" which takes in parameter a string that it will compile...

As well this can be used to add anotation for debuggers by inserting a "break" annotation taking in parameter some possible expressions useful for the debugger
  (break) {}
  (break"my breakpoint") {}


Le ven. 7 juin 2019 à 10:12, Philippe Verdy <[hidden email]> a écrit :
And if someone does not like the "@", we can still use the proposed "*":
  (*table 64) {}
Here also the inner parentheses surrounding the parameters of the annotation are removed, exactly and consistantly with the currification of regular function calls.

Le ven. 7 juin 2019 à 10:10, Philippe Verdy <[hidden email]> a écrit :
Le jeu. 6 juin 2019 à 23:35, Lorenzo Donati <[hidden email]> a écrit :
On 06/06/2019 22:41, Hisham wrote:
> A pattern in Lua's history is that some features appear first as a
> special-case feature that solves a narrow set of problems, and are
> later generalized into a broader mechanism.
>

[snip]


> If the door is to be left open for such a generalized mechanism in
> future versions of Lua beyond 5.4, then adopting a syntax based on an
> unused token such as @annotation would be advisable. Matching the
> annotation syntax of other languages would be a side benefit.
>
> -- Hisham
>
>

I fully agree!

As I wrote in another thread (in a sub-sub thread - urgh! this 5.4 thing
is stirring lua-l quite a lot :-) compare:

local <toclose, static, nice, helpful, wonderful> resource = ....

vs.

local @toclose @static @nice @helpful @wonderful resource = ....

and also regarding code hard wrapping:

local < toclose, static,
   nice, helpful, wonderful > resource = ....

vs.

local @toclose @static
   @nice @helpful @wonderful resource = ....

The more I think about it, the more I find the syntax with "@" more
readable and more easily "expandable": parametrized annotations anyone?
Like for example:

local @const myTable = @table(64) {} -- preallocates 64 elements in the
array part

The syntax "@table(64){}" is ambiguous, it can be thought of the annotation "@table" for the expression 64, which is the right handside of a binary operator (a function call).

Consider: "@table(t){}" ...

The only way is to surround annotations with parameters by parentheses:
  (@table(64)) {}
but we can also drop the unnecessary inner parentheses for the parameters of the annotation itself:
  (@table 64) {}

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Roberto Ierusalimschy
In reply to this post by Hisham
> [...]
> that using '<' like that would introduce ambiguites that would
> preclude general function annotations in the commonly-adopted
> annotation style. The following would be hard to parse without
> significant lookahead:
>
>    local x = y
>
>    <my_custom_annotation(with, arguments)>
>    function my_global_function()
>       -- ...
>    end
>
> If the door is to be left open for such a generalized mechanism in
> future versions of Lua beyond 5.4, then adopting a syntax based on an
> unused token such as @annotation would be advisable. Matching the
> annotation syntax of other languages would be a side benefit.

There is a big difference between all those syntaxes and the one
we are using in Lua, not because of the peculiarities of '<name>'
versus '@name': They are all prefixed to the whole item to which they
apply. Following their syntax, we should write '@toclose local x = 1',
instead of 'local @toclose x = 1'.

For locals it is not a big difference, but the following looks weird to
my eyes, no matter the details:

     @my_custom_annotation(with, arguments)
     function my_global_function()
        -- ...
     end

Maybe it is a consequence of Lua not having explicit statement
termination, but I do not visualy associate the attribute to the
function. They look like two independent statements. (BTW, note that
the C++ [syntax] in Lua would have the same problem that <name>,
assuming prefixed attributes.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Dibyendu Majumdar
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Pierre Chapuis
In reply to this post by Roberto Ierusalimschy
On Fri, Jun 7, 2019, at 16:11, Roberto Ierusalimschy wrote:

> For locals it is not a big difference, but the following looks weird to
> my eyes, no matter the details:
>
>      @my_custom_annotation(with, arguments)
>      function my_global_function()
>         -- ...
>      end

It does not to me because it is the syntax used by TypeScript decorators,
which they also had to tack onto the pre-existing JavaScript syntax:
https://www.typescriptlang.org/docs/handbook/decorators.html

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Hisham
In reply to this post by Roberto Ierusalimschy
On Fri, 7 Jun 2019 at 11:11, Roberto Ierusalimschy
<[hidden email]> wrote:

>
> > [...]
> > that using '<' like that would introduce ambiguites that would
> > preclude general function annotations in the commonly-adopted
> > annotation style. The following would be hard to parse without
> > significant lookahead:
> >
> >    local x = y
> >
> >    <my_custom_annotation(with, arguments)>
> >    function my_global_function()
> >       -- ...
> >    end
> >
> > If the door is to be left open for such a generalized mechanism in
> > future versions of Lua beyond 5.4, then adopting a syntax based on an
> > unused token such as @annotation would be advisable. Matching the
> > annotation syntax of other languages would be a side benefit.
>
> There is a big difference between all those syntaxes and the one
> we are using in Lua, not because of the peculiarities of '<name>'
> versus '@name': They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.

The "whole item" the annotation is applying to is the variable, and to
my eyes that is `x`, and not `local x`, because a `local` declaration
may have multiple variables. Of course, given Lua's general
support for multiple returns and expression lists, I was expecting
something like

   local x, <toclose> y = 0, io.open("hello")

to be valid, but looks like it isn't. (I understand it can be worked around
with an extra line, but it seems like a needless inconsistency, unless I'm
missing something.)

> For locals it is not a big difference, but the following looks weird to
> my eyes, no matter the details:
>
>      @my_custom_annotation(with, arguments)
>      function my_global_function()
>         -- ...
>      end

As of 2019 this idiom with the linebreak is pretty standard-looking
syntax for languages that have annotations. It did look weird to me
as well the first time I had to write programs with Java annotations
years ago, but like all things syntax, it eventually became natural
with use, and I was unsurprised to see other languages adopting it.
I believe it has become an idiom to stack annotations in function
declarations with line breaks like this because otherwise the
function declarations grow too long. (In realistic examples, of course:
that's why I stopped using examples like `@a function f()` or
`t[t+1]` when discussing syntax -- what looks tidy in a discussion
with single-letter variable names breaks down in real world
programs with longer variables.)

For local functions, we have an interesting situation.
I assume for locals the direct Lua adaptation for that would become

   local @my_custom_annotation(with, arguments)
   function my_local_function()
      -- ...
   end

which could end up looking like a local variable followed by a global function
to the uninitiated. Allowing the annotation to be prefixed to the
entire `local` declaration
(made possible by the unambiguous @ syntax), one could do the more natural

   @my_custom_annotation(with, arguments)
   local function my_local_function()
      -- ...
   end

Given that `local function f` is already a syntactic shorthand to
`local f = function`, I guess `@annotation local function f` could be
a syntactic shorthand to `local @annotation f = function`.

This would preserve the notion that annotations apply to the variable
name being declared, which brings back to the point about variable
lists. To reiterate on the issue of annotations in variable lists
(which is a matter of concern for the current applicability of the
upcoming Lua 5.4 features rather than musing about a future
extensibility), the following scenario seems like an obvious and
common application for toclose, if annotations were allowed in
multi-var lists:

   local @toclose connection, err = make_connection(host, port)
   if not connection then
      return nil, err
   end

Requiring people to do

   local connection, err = make_connection(host, port)
   if not connection then
      return nil, err
   end
   local @toclose connection = connection

every time seems really unfortunate. (That's another tricky thing in
syntax discussions: people's tiny examples often omit error checking,
which real world programs need to do. Lots of examples were floated
around in other threads with people doing `local <toclose> f =
io.open("file.txt")`. The moment I try to catch the err in that line,
the pretty example breaks down and we end up with a feature that can't
really be used in practice without a verbose idiom.)

> (BTW, note that
> the C++ [syntax] in Lua would have the same problem that <name>,
> assuming prefixed attributes.)

(C#, but still) yes, I do realize and that's why I did not suggest it.

So, I guess the short version of my rambling is:

1. please consider @ rather than <> in Lua 5.4 if you consider
extending this someday in Lua > 5.4
2. please allow per-variable annotations in multi-variable
declarations in Lua 5.4, no matter what syntax you end up choosing.

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Hisham
In reply to this post by Philippe Verdy
On Fri, 7 Jun 2019 at 05:11, Philippe Verdy <[hidden email]> wrote:

>
> Le jeu. 6 juin 2019 à 23:35, Lorenzo Donati <[hidden email]> a écrit :
>>
>> On 06/06/2019 22:41, Hisham wrote:
>> > A pattern in Lua's history is that some features appear first as a
>> > special-case feature that solves a narrow set of problems, and are
>> > later generalized into a broader mechanism.
>> >
>>
>> [snip]
>>
>>
>> > If the door is to be left open for such a generalized mechanism in
>> > future versions of Lua beyond 5.4, then adopting a syntax based on an
>> > unused token such as @annotation would be advisable. Matching the
>> > annotation syntax of other languages would be a side benefit.
>> >
>> > -- Hisham
>> >
>> >
>>
>> I fully agree!
>>
>> As I wrote in another thread (in a sub-sub thread - urgh! this 5.4 thing
>> is stirring lua-l quite a lot :-) compare:
>>
>> local <toclose, static, nice, helpful, wonderful> resource = ....
>>
>> vs.
>>
>> local @toclose @static @nice @helpful @wonderful resource = ....
>>
>> and also regarding code hard wrapping:
>>
>> local < toclose, static,
>>    nice, helpful, wonderful > resource = ....
>>
>> vs.
>>
>> local @toclose @static
>>    @nice @helpful @wonderful resource = ....
>>
>> The more I think about it, the more I find the syntax with "@" more
>> readable and more easily "expandable": parametrized annotations anyone?
>> Like for example:
>>
>> local @const myTable = @table(64) {} -- preallocates 64 elements in the
>> array part
>
>
> The syntax "@table(64){}" is ambiguous, it can be thought of the annotation "@table" for the expression 64, which is the right handside of a binary operator (a function call).

The ambiguity here comes from attempting to apply an annotation to a
value rather than a variable. Perhaps the confusion arose from the
`@annotation function` example, but even though functions are values,
that annotation is binding to a variable: `function x()...` in Lua is
a variable declaration, shorthand for `x = function()...`. With
annotations applied to variables rather than values, the @ syntax is
not ambiguous, even with parameters. (The <> syntax would be equally
problematic to apply to values.)


> Consider: "@table(t){}" ...
>
> The only way is to surround annotations with parameters by parentheses:
>   (@table(64)) {}
> but we can also drop the unnecessary inner parentheses for the parameters of the annotation itself:
>   (@table 64) {}
>

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Dibyendu Majumdar
In reply to this post by Hisham
On Fri, 7 Jun 2019 at 23:56, Hisham <[hidden email]> wrote:

> Allowing the annotation to be prefixed to the
> entire `local` declaration
> (made possible by the unambiguous @ syntax), one could do the more natural
>
>    @my_custom_annotation(with, arguments)
>    local function my_local_function()
>       -- ...
>    end
>
> Given that `local function f` is already a syntactic shorthand to
> `local f = function`, I guess `@annotation local function f` could be
> a syntactic shorthand to `local @annotation f = function`.
>

Prefixing an entire local statement would imply the annotation applies
to all names declared in that local statement. I would argue that
annotations should be allowed per variable name, hence they should
suffix or prefix a variable name.
We should also think about allowing annotations in function parameters
... although not needed right now, a syntax that supports this in
future would be nice.

The advantage of '@' prefix is that it can introduce an annotation in
any context - even in expressions.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
In reply to this post by Dibyendu Majumdar
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.
There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any expression (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.



Le sam. 8 juin 2019 à 05:44, Philippe Verdy <[hidden email]> a écrit :
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
I see only a single useful case for annotating an empty statement (i.e. just after any single ";" which is always isolated and has no further tokens in any context), it's for emitting some debugger info (or execution tracking logger, or breakpoint) that applies to a reachable point of execution, i.e. between two separate statements that are otherwise not themselves annotated by it.


Le sam. 8 juin 2019 à 05:55, Philippe Verdy <[hidden email]> a écrit :
And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.
There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any expression (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.



Le sam. 8 juin 2019 à 05:44, Philippe Verdy <[hidden email]> a écrit :
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
Also in my view, all annotations are to be considered optional, they create an "intermingled" meta-language which is to be interpreted by another engine (a compiler, some specific runtime, a debugger) but they should all be safely removed and the code should still be valid and interpretable by a core Lua engine.

The metalanguage however can be written in Lua too just like the engine interpreting it.

But a basic annotation "const" or "toclose" are not meaningful isolately as a plain statement for a metalanguage, they make sense only with the core language statement toi which they are directly associated (i.e. always after the statement starting token but before any declared item and any expression). What they annotate can only be used by a compiler (as type hints) but not directly by the runtime engine, a debugger or tracking logger (whose behavior cannot be controled by these annotations, if they use these annotations, it is only to interpret the item in the core language with which they are necessarily associated).



Le sam. 8 juin 2019 à 06:04, Philippe Verdy <[hidden email]> a écrit :
I see only a single useful case for annotating an empty statement (i.e. just after any single ";" which is always isolated and has no further tokens in any context), it's for emitting some debugger info (or execution tracking logger, or breakpoint) that applies to a reachable point of execution, i.e. between two separate statements that are otherwise not themselves annotated by it.


Le sam. 8 juin 2019 à 05:55, Philippe Verdy <[hidden email]> a écrit :
And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.
There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any expression (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.



Le sam. 8 juin 2019 à 05:44, Philippe Verdy <[hidden email]> a écrit :
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
In reply to this post by Philippe Verdy
Example of use of an annotation on an empty statement:
   a = 1
   ;(break 'here')
   b = 2
here "(break 'here')" is the "break" annotation which may have optional parameters possibly currified, entirely written between parentheses because "break" without them is a core langage statement.

If the annotation is ignored (because there's no debugger using it and the code is just run with a core engine) then it is equivalent to the 3 statements:
   a = 1
   ;
   b = 2
with the same empty statement, not necessary here because "1 b" is not a valid currified function call "1(b)" as a constant number is not an object and has no function members, so it is equivalent also to the two separate statements:
  a = 1 b=2


Le sam. 8 juin 2019 à 06:04, Philippe Verdy <[hidden email]> a écrit :
I see only a single useful case for annotating an empty statement (i.e. just after any single ";" which is always isolated and has no further tokens in any context), it's for emitting some debugger info (or execution tracking logger, or breakpoint) that applies to a reachable point of execution, i.e. between two separate statements that are otherwise not themselves annotated by it.


Le sam. 8 juin 2019 à 05:55, Philippe Verdy <[hidden email]> a écrit :
And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.
There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any expression (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.



Le sam. 8 juin 2019 à 05:44, Philippe Verdy <[hidden email]> a écrit :
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
But if I write:
  a = b
  ;(break 'here')
  c = d
it is still functionally equivalent (in core language) to the 3 statements (including the empty statement);
  a = b
  ;
  c = d
but simply not to:
  a = b
  c = d
In that last case, there's an ambiguity because "b c" may potentially be a currified function call "b(c)".
That ambiguity already complexifies the existing Lua parsing because the compiler MUST rollback after parsing "a = b c" when it sees the next "=" (there cannot be two assignments in the same statement).

This is the known case in Lua syntax where there's a "shift/reduce conflict" in Lua, that is not always resolved simply as a "shift" (which still has the priority), but where the compiler MUST support rollback at this point in order to be able to retry with a "reduce", and why the empty statement ";" was necessary to the syntax (because of the priority given to the "shift" which may still be valid up to some later point in the source stream of tokens in which case there will be an inner reduce later, that cannot be safely rollbacked and that cancels any prior candidate shift/reduce rollback point)


Le sam. 8 juin 2019 à 06:30, Philippe Verdy <[hidden email]> a écrit :
Example of use of an annotation on an empty statement:
   a = 1
   ;(break 'here')
   b = 2
here "(break 'here')" is the "break" annotation which may have optional parameters possibly currified, entirely written between parentheses because "break" without them is a core langage statement.

If the annotation is ignored (because there's no debugger using it and the code is just run with a core engine) then it is equivalent to the 3 statements:
   a = 1
   ;
   b = 2
with the same empty statement, not necessary here because "1 b" is not a valid currified function call "1(b)" as a constant number is not an object and has no function members, so it is equivalent also to the two separate statements:
  a = 1 b=2


Le sam. 8 juin 2019 à 06:04, Philippe Verdy <[hidden email]> a écrit :
I see only a single useful case for annotating an empty statement (i.e. just after any single ";" which is always isolated and has no further tokens in any context), it's for emitting some debugger info (or execution tracking logger, or breakpoint) that applies to a reachable point of execution, i.e. between two separate statements that are otherwise not themselves annotated by it.


Le sam. 8 juin 2019 à 05:55, Philippe Verdy <[hidden email]> a écrit :
And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.
There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any expression (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.



Le sam. 8 juin 2019 à 05:44, Philippe Verdy <[hidden email]> a écrit :
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
The syntaxic analysis in Lua is made with the constraint that it can be parsed with a LR(1) parser as much as possible (with a single token of look-ahead). But actually Lua is LALR because it must also support a few rollbacks for resolving a few shift/reduce conflicts.

Implementing these rollbacks adds a cost in the parser because before it cannot be written simply using trailing function calls: these functions have to return a validation status that must be tested, so these functions inserted in the parser will stack up in memory with all their existing local context (unlike trailing function calls that are easily compiled by Lua as simple jumps.

If we add syntaxic features to Lua, we should keep this in mind and not transform Lua into a Fortran-like language (whose parsing is extremely complex because of its optional whitespaces between all keywords and identifiers, and very tricky to write and test with enough coverages) or in a C++-like language (also very tricky).



Le sam. 8 juin 2019 à 06:41, Philippe Verdy <[hidden email]> a écrit :
But if I write:
  a = b
  ;(break 'here')
  c = d
it is still functionally equivalent (in core language) to the 3 statements (including the empty statement);
  a = b
  ;
  c = d
but simply not to:
  a = b
  c = d
In that last case, there's an ambiguity because "b c" may potentially be a currified function call "b(c)".
That ambiguity already complexifies the existing Lua parsing because the compiler MUST rollback after parsing "a = b c" when it sees the next "=" (there cannot be two assignments in the same statement).

This is the known case in Lua syntax where there's a "shift/reduce conflict" in Lua, that is not always resolved simply as a "shift" (which still has the priority), but where the compiler MUST support rollback at this point in order to be able to retry with a "reduce", and why the empty statement ";" was necessary to the syntax (because of the priority given to the "shift" which may still be valid up to some later point in the source stream of tokens in which case there will be an inner reduce later, that cannot be safely rollbacked and that cancels any prior candidate shift/reduce rollback point)


Le sam. 8 juin 2019 à 06:30, Philippe Verdy <[hidden email]> a écrit :
Example of use of an annotation on an empty statement:
   a = 1
   ;(break 'here')
   b = 2
here "(break 'here')" is the "break" annotation which may have optional parameters possibly currified, entirely written between parentheses because "break" without them is a core langage statement.

If the annotation is ignored (because there's no debugger using it and the code is just run with a core engine) then it is equivalent to the 3 statements:
   a = 1
   ;
   b = 2
with the same empty statement, not necessary here because "1 b" is not a valid currified function call "1(b)" as a constant number is not an object and has no function members, so it is equivalent also to the two separate statements:
  a = 1 b=2


Le sam. 8 juin 2019 à 06:04, Philippe Verdy <[hidden email]> a écrit :
I see only a single useful case for annotating an empty statement (i.e. just after any single ";" which is always isolated and has no further tokens in any context), it's for emitting some debugger info (or execution tracking logger, or breakpoint) that applies to a reachable point of execution, i.e. between two separate statements that are otherwise not themselves annotated by it.


Le sam. 8 juin 2019 à 05:55, Philippe Verdy <[hidden email]> a écrit :
And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.
There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any expression (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.



Le sam. 8 juin 2019 à 05:44, Philippe Verdy <[hidden email]> a écrit :
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: future of annotations in Lua?

Philippe Verdy
Note also that Javascript has similar issues, but it does not require ";" only because it differenciates whitespaces (those that include a newline not part of a comment from other whitespaces); you cannot safely remove all newlines in Javascript and merge all whitespaces to a single space, without adding a few ";" where needed. So Javascript's parser defines the newline token!



Le sam. 8 juin 2019 à 06:55, Philippe Verdy <[hidden email]> a écrit :
The syntaxic analysis in Lua is made with the constraint that it can be parsed with a LR(1) parser as much as possible (with a single token of look-ahead). But actually Lua is LALR because it must also support a few rollbacks for resolving a few shift/reduce conflicts.

Implementing these rollbacks adds a cost in the parser because before it cannot be written simply using trailing function calls: these functions have to return a validation status that must be tested, so these functions inserted in the parser will stack up in memory with all their existing local context (unlike trailing function calls that are easily compiled by Lua as simple jumps.

If we add syntaxic features to Lua, we should keep this in mind and not transform Lua into a Fortran-like language (whose parsing is extremely complex because of its optional whitespaces between all keywords and identifiers, and very tricky to write and test with enough coverages) or in a C++-like language (also very tricky).



Le sam. 8 juin 2019 à 06:41, Philippe Verdy <[hidden email]> a écrit :
But if I write:
  a = b
  ;(break 'here')
  c = d
it is still functionally equivalent (in core language) to the 3 statements (including the empty statement);
  a = b
  ;
  c = d
but simply not to:
  a = b
  c = d
In that last case, there's an ambiguity because "b c" may potentially be a currified function call "b(c)".
That ambiguity already complexifies the existing Lua parsing because the compiler MUST rollback after parsing "a = b c" when it sees the next "=" (there cannot be two assignments in the same statement).

This is the known case in Lua syntax where there's a "shift/reduce conflict" in Lua, that is not always resolved simply as a "shift" (which still has the priority), but where the compiler MUST support rollback at this point in order to be able to retry with a "reduce", and why the empty statement ";" was necessary to the syntax (because of the priority given to the "shift" which may still be valid up to some later point in the source stream of tokens in which case there will be an inner reduce later, that cannot be safely rollbacked and that cancels any prior candidate shift/reduce rollback point)


Le sam. 8 juin 2019 à 06:30, Philippe Verdy <[hidden email]> a écrit :
Example of use of an annotation on an empty statement:
   a = 1
   ;(break 'here')
   b = 2
here "(break 'here')" is the "break" annotation which may have optional parameters possibly currified, entirely written between parentheses because "break" without them is a core langage statement.

If the annotation is ignored (because there's no debugger using it and the code is just run with a core engine) then it is equivalent to the 3 statements:
   a = 1
   ;
   b = 2
with the same empty statement, not necessary here because "1 b" is not a valid currified function call "1(b)" as a constant number is not an object and has no function members, so it is equivalent also to the two separate statements:
  a = 1 b=2


Le sam. 8 juin 2019 à 06:04, Philippe Verdy <[hidden email]> a écrit :
I see only a single useful case for annotating an empty statement (i.e. just after any single ";" which is always isolated and has no further tokens in any context), it's for emitting some debugger info (or execution tracking logger, or breakpoint) that applies to a reachable point of execution, i.e. between two separate statements that are otherwise not themselves annotated by it.


Le sam. 8 juin 2019 à 05:55, Philippe Verdy <[hidden email]> a écrit :
And given the way Lua parses the ";" (only as an empty statement, which is a no-op), adding an annotation just after it would make no sense.
There's no way to unambiguously allow any annotation in Lua at *start* of any statement: it must necessarily be in the middle of the statement before any expression (it can follow an "," separator too, which probably makes sense only if there's something after it which is not the end of the comma-separated list) or just at end of the statement.



Le sam. 8 juin 2019 à 05:44, Philippe Verdy <[hidden email]> a écrit :
Yes but Java requires the ";" terminator, so there's no ambiguity when parsing, even if the annotation precedes all the rest of the statement.

In Lua, without the required ";" there will an ambiguity of parsing if the annotation does not follow immediately a statement initial keyword (local, function, for, return, if, then, else, begin...), or a "(" or "[" or "{".


Le ven. 7 juin 2019 à 18:07, Dibyendu Majumdar <[hidden email]> a écrit :
> There is a big difference between all those syntaxes
[snip]
> They are all prefixed to the whole item to which they
> apply. Following their syntax, we should write '@toclose local x = 1',
> instead of 'local @toclose x = 1'.
>

In Java the annotation precedes the type in a declaration; it being
classed as a type modifier in the grammar (this is one of the uses).
Lua of course doesn't have type declarations therefore annotations
cannot be placed in the same way.

Regards

123