Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

classic Classic list List threaded Threaded
44 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Dibyendu Majumdar
On 11 January 2017 at 21:06, Hisham <[hidden email]> wrote:
> On 11 January 2017 at 17:08, Roberto Ierusalimschy
> <[hidden email]> wrote:
>>> Please, do not make such a severely breaking change in a 5.3.x release.
>>
>> We did not realize that. It will be corrected.
>
> Thank you! :)
>

Hi,

I was wondering if it is possible to consider making strong backwards
compatibility guarantees for Lua from 5.3 upwards. I think this will
mean some constraints to Lua's development obviously, but it would
strengthen Lua 's ecosystem.

The strongest argument I could make is that Lua is built on C, and
imagine how difficult it would have been if C had kept changing in
incompatible ways every few years, and all new C compilers stopped
supporting older versions of C.

The other argument for a compatibility promise is to try and bring
everyone up to 5.3 at least. If the language again changes in an
incompatible way that then you will end up with users in 5.1, 5.2, 5.3
and future versions. I think this fragmentation is not beneficial to
Lua.

I was thinking initially of requesting that "Lua be frozen" as a
language ... but that is maybe too radical a thought.

I think we can see from experience that languages that have done this
- i.e. maintained strong backwards compatibility - such as C, C++,
Java, C#, Go, have all benefited from it. We see also the struggle
being faced by Python moving everyone to 3.x, and similarly the
fragmented ecosystem of Lua.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

William Ahern
On Thu, Jan 12, 2017 at 12:02:49AM +0000, Dibyendu Majumdar wrote:

> On 11 January 2017 at 21:06, Hisham <[hidden email]> wrote:
> > On 11 January 2017 at 17:08, Roberto Ierusalimschy
> > <[hidden email]> wrote:
> >>> Please, do not make such a severely breaking change in a 5.3.x release.
> >>
> >> We did not realize that. It will be corrected.
> >
> > Thank you! :)
> >
>
> Hi,
>
> I was wondering if it is possible to consider making strong backwards
> compatibility guarantees for Lua from 5.3 upwards. I think this will
> mean some constraints to Lua's development obviously, but it would
> strengthen Lua 's ecosystem.
>
> The strongest argument I could make is that Lua is built on C, and
> imagine how difficult it would have been if C had kept changing in
> incompatible ways every few years, and all new C compilers stopped
> supporting older versions of C.

OTOH, Lua wouldn't be the language it is today if backwards compatibility
was a higher priority. Features like closures over mutable variables and
stackful coroutines wouldn't have been developed; features which arguably
are now defining characteristics of Lua relative to other languages. Other
than Scheme, I can't think of any other non-academic language with similarly
powerful constructs. And that's largely because before they get there
language designers and implementers decide their language is "good enough"
and switch into long-term maintenance mode where, at best, only pale
imitations of the above can be added.

If Lua were more like Python, where backwards compatibility was sacrificed
for much less expressive, only marginally better constructs, the argument
for more stable changes would be stronger. But are you really prepared to
say that Lua 5.3 is "good enough"? Why wasn't Lua 5.1 good enough? It's an
important question because LuaJIT is fully committed to the semantics of Lua
5.1, and the LuaJIT community is probably larger than that of Lua 5.3.

If Lua 5.3 really is good enough, given Lua's track record I would expect
future versions to naturally maintain strong backwards compatibility,
regardless of any specific intent to do so. Indeed, for most problems it's
trivial to write Lua 5.1, 5.2, and 5.3 compatible code.

But if it turns out that Lua 5.3 isn't good enough--that there are more
powerful constructs or semantics that would work well in the Lua-language
family--it would be a shame to forgo that experiment.

Preserving the ability and motivation for substantial experimentation means
we sometimes get incompatibilities, like with __ipairs[1], that have
arguably poor cost+benefit tradeoffs. But there's no way to avoid those
while also preserving the ability to adopt really greats changes. The true
cost+benefit can only be known in hindsight, and so failure has to remain an
option.


[1] My beef is that __len forecloses lazy evaluation. I'd personally have
liked to see __ipairs preserved with a fallback to __len. But I can totally
understand how that might seem too awkward. And I can appreciate the desire
to circumscibe the complexity of metamethods. __pairs and especially
__ipairs were already concessions in the sense that, unlike __call or __gc,
they're trivial to implement in user code and don't directly supplement the
language semantics. __len makes alot more sense as a language addition, and
I'd have been blissfully ignorant had __len come first. I guess the
evolution of __pairs begetting __ipairs begetting __len is a great example
of how even at a micro-scale Lua iteratively becomes a much better language
by being less strict about backwards compatibility.


Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Dirk Laurie-2
2017-01-12 3:18 GMT+02:00 William Ahern <[hidden email]>:

> If Lua 5.3 really is good enough, given Lua's track record I would expect
> future versions to naturally maintain strong backwards compatibility,
> regardless of any specific intent to do so. Indeed, for most problems it's
> trivial to write Lua 5.1, 5.2, and 5.3 compatible code.

Perhaps someone concerned about these issues could design
a specification of "clean Lua", the intersection of Lua 5.1, 5.2 and
5.3, and provide a tool that checks whether code conforms to it.

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Charles Heywood
I feel a problem would arise with, for example, loadstring and unpack being (re)moved. 

On Wed, Jan 11, 2017, 21:47 Dirk Laurie <[hidden email]> wrote:
2017-01-12 3:18 GMT+02:00 William Ahern <[hidden email]>:

> If Lua 5.3 really is good enough, given Lua's track record I would expect
> future versions to naturally maintain strong backwards compatibility,
> regardless of any specific intent to do so. Indeed, for most problems it's
> trivial to write Lua 5.1, 5.2, and 5.3 compatible code.

Perhaps someone concerned about these issues could design
a specification of "clean Lua", the intersection of Lua 5.1, 5.2 and
5.3, and provide a tool that checks whether code conforms to it.

--

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Tomás Guisasola-2
Hi

I work with Lua since 2.1 days and my team maintain code developed
initialy with CGILua 3, 20 years ago, and were converted/upgraded to
every new Lua version until 5.2 (we are planning to upgrade to Lua 5.3,
but we have to wait for the libraries: CGILua, LuaFileSystem, LuaSQL,
LuaExpat, LuaSOAP, LuaSec, LuaSocket, CGILua LuaZip, lua-iconv, LPEG,
HTK, Dado).  So we did many upgrades and converted our code many times.  
Maybe our case is too particular to generalize, but that task was always
a simple and easy task.

I think the Lua team always made a good judgement when faced with the
problem of introducing a new feature and also an incompatibility.  I
don't think this should be changed.

Regards,
Tomás

On 2017-01-12 01:58, Charles Heywood wrote:

> I feel a problem would arise with, for example, loadstring and unpack
> being (re)moved.
>
> On Wed, Jan 11, 2017, 21:47 Dirk Laurie <[hidden email]> wrote:
>
>> 2017-01-12 3:18 GMT+02:00 William Ahern
>> <[hidden email]>:
>>
>>> If Lua 5.3 really is good enough, given Lua's track record I
>> would expect
>>> future versions to naturally maintain strong backwards
>> compatibility,
>>> regardless of any specific intent to do so. Indeed, for most
>> problems it's
>>> trivial to write Lua 5.1, 5.2, and 5.3 compatible code.
>>
>> Perhaps someone concerned about these issues could design
>> a specification of "clean Lua", the intersection of Lua 5.1, 5.2
>> and
>> 5.3, and provide a tool that checks whether code conforms to it.
>
> --

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Nagaev Boris
In reply to this post by William Ahern
On Thu, Jan 12, 2017 at 1:18 AM, William Ahern
<[hidden email]> wrote:

> On Thu, Jan 12, 2017 at 12:02:49AM +0000, Dibyendu Majumdar wrote:
>> On 11 January 2017 at 21:06, Hisham <[hidden email]> wrote:
>> > On 11 January 2017 at 17:08, Roberto Ierusalimschy
>> > <[hidden email]> wrote:
>> >>> Please, do not make such a severely breaking change in a 5.3.x release.
>> >>
>> >> We did not realize that. It will be corrected.
>> >
>> > Thank you! :)
>> >
>>
>> Hi,
>>
>> I was wondering if it is possible to consider making strong backwards
>> compatibility guarantees for Lua from 5.3 upwards. I think this will
>> mean some constraints to Lua's development obviously, but it would
>> strengthen Lua 's ecosystem.
>>
>> The strongest argument I could make is that Lua is built on C, and
>> imagine how difficult it would have been if C had kept changing in
>> incompatible ways every few years, and all new C compilers stopped
>> supporting older versions of C.
>
> OTOH, Lua wouldn't be the language it is today if backwards compatibility
> was a higher priority. Features like closures over mutable variables and
> stackful coroutines wouldn't have been developed; features which arguably
> are now defining characteristics of Lua relative to other languages. Other
> than Scheme, I can't think of any other non-academic language with similarly
> powerful constructs.

Go seems to have variables mutable from closures.
https://play.golang.org/p/o23Z6bbUTp

> And that's largely because before they get there
> language designers and implementers decide their language is "good enough"
> and switch into long-term maintenance mode where, at best, only pale
> imitations of the above can be added.
>
> If Lua were more like Python, where backwards compatibility was sacrificed
> for much less expressive, only marginally better constructs, the argument
> for more stable changes would be stronger. But are you really prepared to
> say that Lua 5.3 is "good enough"? Why wasn't Lua 5.1 good enough? It's an
> important question because LuaJIT is fully committed to the semantics of Lua
> 5.1, and the LuaJIT community is probably larger than that of Lua 5.3.
>
> If Lua 5.3 really is good enough, given Lua's track record I would expect
> future versions to naturally maintain strong backwards compatibility,
> regardless of any specific intent to do so. Indeed, for most problems it's
> trivial to write Lua 5.1, 5.2, and 5.3 compatible code.
>
> But if it turns out that Lua 5.3 isn't good enough--that there are more
> powerful constructs or semantics that would work well in the Lua-language
> family--it would be a shame to forgo that experiment.
>
> Preserving the ability and motivation for substantial experimentation means
> we sometimes get incompatibilities, like with __ipairs[1], that have
> arguably poor cost+benefit tradeoffs. But there's no way to avoid those
> while also preserving the ability to adopt really greats changes. The true
> cost+benefit can only be known in hindsight, and so failure has to remain an
> option.
>
>
> [1] My beef is that __len forecloses lazy evaluation. I'd personally have
> liked to see __ipairs preserved with a fallback to __len. But I can totally
> understand how that might seem too awkward. And I can appreciate the desire
> to circumscibe the complexity of metamethods. __pairs and especially
> __ipairs were already concessions in the sense that, unlike __call or __gc,
> they're trivial to implement in user code and don't directly supplement the
> language semantics. __len makes alot more sense as a language addition, and
> I'd have been blissfully ignorant had __len come first. I guess the
> evolution of __pairs begetting __ipairs begetting __len is a great example
> of how even at a micro-scale Lua iteratively becomes a much better language
> by being less strict about backwards compatibility.
>
>



--
Best regards,
Boris Nagaev

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Dibyendu Majumdar
In reply to this post by Tomás Guisasola-2
Hi Tomás,

On 12 January 2017 at 14:12, tomas <[hidden email]> wrote:
> I work with Lua since 2.1 days and my team maintain code developed initialy
> with CGILua 3, 20 years ago, and were converted/upgraded to every new Lua
> version until 5.2 (we are planning to upgrade to Lua 5.3, but we have to
> wait for the libraries: CGILua, LuaFileSystem, LuaSQL, LuaExpat, LuaSOAP,
> LuaSec, LuaSocket, CGILua LuaZip, lua-iconv, LPEG, HTK, Dado).  So we did
> many upgrades and converted our code many times.  Maybe our case is too
> particular to generalize, but that task was always a simple and easy task.
>

Would it not be better though if you could upgrade to the next version
without having to change things?

I think there are other problems than just upgrading your code.

Firstly, I see that many libraries are still on 5.1 and so many people
have not bothered to upgrade. Unless you are full time working on
maintaining your Lua code (which it seems you are) then it may be
difficult to keep supporting different versions.

Secondly - and I think this is a bigger pain point - you need to build
multiple versions of your library as one library cannot support
various Lua versions. This issue is quite a tragedy I think. For
instance I get requests to support 5.1 in my VSCode Lua 5.3 debugger.
I cannot do so without creating another extension dedicated to 5.1.
Whereas ideally my debugger should just be able to run 5.1 code.

In my post I suggested providing strong backwards compatibility from
5.3 upwards, only because I think as far as the 5.2 and 5.1
compatibility story is concerned, the 'horse has bolted' so to speak.
In fact, I think there ought to be a concentrated effort to bring back
/ support as much of 5.1 and 5.2 as possible - to enable more people
to upgrade to 5.3.

> I think the Lua team always made a good judgement when faced with the
> problem of introducing a new feature and also an incompatibility.  I don't
> think this should be changed.
>

I think that it is possible to evolve the language without breaking
compatibility. Just look at how much C++ has evolved since 1998, and
yet all the old code still works. It requires more effort and thought
- and sometimes things don't look so nice but that is a price worth
paying.


Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Marc Balmer
In reply to this post by Dibyendu Majumdar


Am 12.01.17 um 01:02 schrieb Dibyendu Majumdar:

> On 11 January 2017 at 21:06, Hisham <[hidden email]> wrote:
>> On 11 January 2017 at 17:08, Roberto Ierusalimschy
>> <[hidden email]> wrote:
>>>> Please, do not make such a severely breaking change in a 5.3.x release.
>>>
>>> We did not realize that. It will be corrected.
>>
>> Thank you! :)
>>
>
> Hi,
>
> I was wondering if it is possible to consider making strong backwards
> compatibility guarantees for Lua from 5.3 upwards. I think this will
> mean some constraints to Lua's development obviously, but it would
> strengthen Lua 's ecosystem.

If the wheel was not constantly being reinvented, we would still be
driving on bumpy rocks...

If you want to be compatible with ancient Lua, why don't you just use
ancient Lua?

> The strongest argument I could make is that Lua is built on C, and
> imagine how difficult it would have been if C had kept changing in
> incompatible ways every few years, and all new C compilers stopped
> supporting older versions of C.

C has changed a lot in incompatible ways.  Todays C is not the C I used
decades ago.

> The other argument for a compatibility promise is to try and bring
> everyone up to 5.3 at least. If the language again changes in an
> incompatible way that then you will end up with users in 5.1, 5.2, 5.3
> and future versions. I think this fragmentation is not beneficial to
> Lua.

How is that Lua's fault it lazy people are not going to update their stuff?

> I was thinking initially of requesting that "Lua be frozen" as a
> language ... but that is maybe too radical a thought.

Yes.

> I think we can see from experience that languages that have done this
> - i.e. maintained strong backwards compatibility - such as C, C++,
> Java, C#, Go, have all benefited from it. We see also the struggle
> being faced by Python moving everyone to 3.x, and similarly the
> fragmented ecosystem of Lua.

Oh and then there is more fragmentation going on, I remember one guy
wrote a thing called Ravi we added, well, to the fragmentation...


Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Marc Balmer
In reply to this post by Dibyendu Majumdar


Am 12.01.17 um 15:54 schrieb Dibyendu Majumdar:

> Hi Tomás,
>
> On 12 January 2017 at 14:12, tomas <[hidden email]> wrote:
>> I work with Lua since 2.1 days and my team maintain code developed initialy
>> with CGILua 3, 20 years ago, and were converted/upgraded to every new Lua
>> version until 5.2 (we are planning to upgrade to Lua 5.3, but we have to
>> wait for the libraries: CGILua, LuaFileSystem, LuaSQL, LuaExpat, LuaSOAP,
>> LuaSec, LuaSocket, CGILua LuaZip, lua-iconv, LPEG, HTK, Dado).  So we did
>> many upgrades and converted our code many times.  Maybe our case is too
>> particular to generalize, but that task was always a simple and easy task.
>>
>
> Would it not be better though if you could upgrade to the next version
> without having to change things?

Why?  Having to do this also mean reviewing you code, which is not bad.

> I think there are other problems than just upgrading your code.
>
> Firstly, I see that many libraries are still on 5.1 and so many people
> have not bothered to upgrade. Unless you are full time working on
> maintaining your Lua code (which it seems you are) then it may be
> difficult to keep supporting different versions.

We only support the latest version in our stuff.  To be frank, we don't
even test compile for older versions.  Our "#if LUA_VERSION < 503" lines
are an outright lie and a courtesy to those folks that did not yet have
the time to upgrade their Lua.

> Secondly - and I think this is a bigger pain point - you need to build
> multiple versions of your library as one library cannot support
> various Lua versions. This issue is quite a tragedy I think. For
> instance I get requests to support 5.1 in my VSCode Lua 5.3 debugger.
> I cannot do so without creating another extension dedicated to 5.1.
> Whereas ideally my debugger should just be able to run 5.1 code.
>
> In my post I suggested providing strong backwards compatibility from
> 5.3 upwards, only because I think as far as the 5.2 and 5.1
> compatibility story is concerned, the 'horse has bolted' so to speak.
> In fact, I think there ought to be a concentrated effort to bring back
> / support as much of 5.1 and 5.2 as possible - to enable more people
> to upgrade to 5.3.

Using third party software of course means that you have to be vigilant,
it is your responsability, not the third party software author's.

>> I think the Lua team always made a good judgement when faced with the
>> problem of introducing a new feature and also an incompatibility.  I don't
>> think this should be changed.
>>
>
> I think that it is possible to evolve the language without breaking
> compatibility. Just look at how much C++ has evolved since 1998, and
> yet all the old code still works. It requires more effort and thought
> - and sometimes things don't look so nice but that is a price worth
> paying.

Have I already mentioned that I like Lua for what it is?  An slowly
evolving language without batteries included, without to much care for
backwards compatability?  I hope it stays so.

- marc

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Peter Aronoff
In reply to this post by Dibyendu Majumdar
Dibyendu Majumdar <[hidden email]> wrote:
> Would it not be better though if you could upgrade to the next version
> without having to change things?

I think you mean this to be a rhetorical question: the answer to you seems
to be an obvious yes. I can understand some reasons why you might think
that. Less work is better. Stability is nice.

But I don’t think it’s a rhetorical question (really), and I don’t think
the answer is obvious. There are always trade offs—you hint at this below
when you talk about C++. If a guarantee of backwards compatibility hinders
the progress of a language or forces significant compromises in how the
language evolves, then I think the answer to your question tilts towards
“No.” Of course, opinions will differ.

I’d also add, as others have said, that Lua’s current approach to change
seems to me a very healthy mix of respect for current code and a desire to
improve the language. Also, and others have said this too, it’s not
especially hard to use older versions of Lua on projects where upgrading
might be more work than a team wants or can handle.

Just my two cents.

Peter

--
We have not been faced with the need to satisfy someone else's
requirements, and for this freedom we are grateful.
    Dennis Ritchie and Ken Thompson, The UNIX Time-Sharing System

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Dibyendu Majumdar
Hi Peter,

On 12 January 2017 at 15:23, Peter Aronoff <[hidden email]> wrote:

> Dibyendu Majumdar <[hidden email]> wrote:
>> Would it not be better though if you could upgrade to the next version
>> without having to change things?
>
> I think you mean this to be a rhetorical question: the answer to you seems
> to be an obvious yes. I can understand some reasons why you might think
> that. Less work is better. Stability is nice.
>
> But I don’t think it’s a rhetorical question (really), and I don’t think
> the answer is obvious. There are always trade offs—you hint at this below
> when you talk about C++. If a guarantee of backwards compatibility hinders
> the progress of a language or forces significant compromises in how the
> language evolves, then I think the answer to your question tilts towards
> “No.” Of course, opinions will differ.
>

Sure there are trade offs, and I accept that. I think Lua is a mature
language; for a young language, it may be necessary to make breaking
changes to correct serious issues. I hope that serious issues with Lua
have already been fixed. Hence the trade off towards a better
compatibility guarantee seems worth making.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Scott Morgan
In reply to this post by Marc Balmer
On 01/12/2017 02:56 PM, Marc Balmer wrote:
> C has changed a lot in incompatible ways.  Todays C is not the C I used
> decades ago.

2014 Lua 5.2 built with 1990 Turbo C compiler:
http://lua-users.org/lists/lua-l/2014-01/msg00685.html

Scott


Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Coda Highland
In reply to this post by Nagaev Boris
On Thu, Jan 12, 2017 at 6:44 AM, Nagaev Boris <[hidden email]> wrote:
>> OTOH, Lua wouldn't be the language it is today if backwards compatibility
>> was a higher priority. Features like closures over mutable variables and
>> stackful coroutines wouldn't have been developed; features which arguably
>> are now defining characteristics of Lua relative to other languages. Other
>> than Scheme, I can't think of any other non-academic language with similarly
>> powerful constructs.
>
> Go seems to have variables mutable from closures.
> https://play.golang.org/p/o23Z6bbUTp

C++, Obj-C, and Javascript have them too. It's not all THAT rare.

Python and C/C++ (and maybe Obj-C?) have stackful coroutines.

/s/ Adam

On Thu, Jan 12, 2017 at 6:44 AM, Nagaev Boris <[hidden email]> wrote:

> On Thu, Jan 12, 2017 at 1:18 AM, William Ahern
> <[hidden email]> wrote:
>> On Thu, Jan 12, 2017 at 12:02:49AM +0000, Dibyendu Majumdar wrote:
>>> On 11 January 2017 at 21:06, Hisham <[hidden email]> wrote:
>>> > On 11 January 2017 at 17:08, Roberto Ierusalimschy
>>> > <[hidden email]> wrote:
>>> >>> Please, do not make such a severely breaking change in a 5.3.x release.
>>> >>
>>> >> We did not realize that. It will be corrected.
>>> >
>>> > Thank you! :)
>>> >
>>>
>>> Hi,
>>>
>>> I was wondering if it is possible to consider making strong backwards
>>> compatibility guarantees for Lua from 5.3 upwards. I think this will
>>> mean some constraints to Lua's development obviously, but it would
>>> strengthen Lua 's ecosystem.
>>>
>>> The strongest argument I could make is that Lua is built on C, and
>>> imagine how difficult it would have been if C had kept changing in
>>> incompatible ways every few years, and all new C compilers stopped
>>> supporting older versions of C.
>>
>> OTOH, Lua wouldn't be the language it is today if backwards compatibility
>> was a higher priority. Features like closures over mutable variables and
>> stackful coroutines wouldn't have been developed; features which arguably
>> are now defining characteristics of Lua relative to other languages. Other
>> than Scheme, I can't think of any other non-academic language with similarly
>> powerful constructs.
>
> Go seems to have variables mutable from closures.
> https://play.golang.org/p/o23Z6bbUTp
>
>> And that's largely because before they get there
>> language designers and implementers decide their language is "good enough"
>> and switch into long-term maintenance mode where, at best, only pale
>> imitations of the above can be added.
>>
>> If Lua were more like Python, where backwards compatibility was sacrificed
>> for much less expressive, only marginally better constructs, the argument
>> for more stable changes would be stronger. But are you really prepared to
>> say that Lua 5.3 is "good enough"? Why wasn't Lua 5.1 good enough? It's an
>> important question because LuaJIT is fully committed to the semantics of Lua
>> 5.1, and the LuaJIT community is probably larger than that of Lua 5.3.
>>
>> If Lua 5.3 really is good enough, given Lua's track record I would expect
>> future versions to naturally maintain strong backwards compatibility,
>> regardless of any specific intent to do so. Indeed, for most problems it's
>> trivial to write Lua 5.1, 5.2, and 5.3 compatible code.
>>
>> But if it turns out that Lua 5.3 isn't good enough--that there are more
>> powerful constructs or semantics that would work well in the Lua-language
>> family--it would be a shame to forgo that experiment.
>>
>> Preserving the ability and motivation for substantial experimentation means
>> we sometimes get incompatibilities, like with __ipairs[1], that have
>> arguably poor cost+benefit tradeoffs. But there's no way to avoid those
>> while also preserving the ability to adopt really greats changes. The true
>> cost+benefit can only be known in hindsight, and so failure has to remain an
>> option.
>>
>>
>> [1] My beef is that __len forecloses lazy evaluation. I'd personally have
>> liked to see __ipairs preserved with a fallback to __len. But I can totally
>> understand how that might seem too awkward. And I can appreciate the desire
>> to circumscibe the complexity of metamethods. __pairs and especially
>> __ipairs were already concessions in the sense that, unlike __call or __gc,
>> they're trivial to implement in user code and don't directly supplement the
>> language semantics. __len makes alot more sense as a language addition, and
>> I'd have been blissfully ignorant had __len come first. I guess the
>> evolution of __pairs begetting __ipairs begetting __len is a great example
>> of how even at a micro-scale Lua iteratively becomes a much better language
>> by being less strict about backwards compatibility.
>>
>>
>
>
>
> --
> Best regards,
> Boris Nagaev
>

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Tomás Guisasola-2
In reply to this post by Marc Balmer
Hi

On 2017-01-12 13:02, Marc Balmer wrote:

> Am 12.01.17 um 15:54 schrieb Dibyendu Majumdar:
>> Hi Tomás,
>>
>> On 12 January 2017 at 14:12, tomas <[hidden email]> wrote:
>>> I work with Lua since 2.1 days and my team maintain code developed
>>> initialy
>>> with CGILua 3, 20 years ago, and were converted/upgraded to every new
>>> Lua
>>> version until 5.2 (we are planning to upgrade to Lua 5.3, but we have
>>> to
>>> wait for the libraries: CGILua, LuaFileSystem, LuaSQL, LuaExpat,
>>> LuaSOAP,
>>> LuaSec, LuaSocket, CGILua LuaZip, lua-iconv, LPEG, HTK, Dado).  So we
>>> did
>>> many upgrades and converted our code many times.  Maybe our case is
>>> too
>>> particular to generalize, but that task was always a simple and easy
>>> task.
>>>
>>
>> Would it not be better though if you could upgrade to the next version
>> without having to change things?
>
> Why?  Having to do this also mean reviewing you code, which is not bad.
>
>> I think there are other problems than just upgrading your code.
>>
>> Firstly, I see that many libraries are still on 5.1 and so many people
>> have not bothered to upgrade. Unless you are full time working on
>> maintaining your Lua code (which it seems you are) then it may be
>> difficult to keep supporting different versions.
>
> We only support the latest version in our stuff.  To be frank, we don't
> even test compile for older versions.  Our "#if LUA_VERSION < 503"
> lines
> are an outright lie and a courtesy to those folks that did not yet have
> the time to upgrade their Lua.
>
>> Secondly - and I think this is a bigger pain point - you need to build
>> multiple versions of your library as one library cannot support
>> various Lua versions. This issue is quite a tragedy I think. For
>> instance I get requests to support 5.1 in my VSCode Lua 5.3 debugger.
>> I cannot do so without creating another extension dedicated to 5.1.
>> Whereas ideally my debugger should just be able to run 5.1 code.
>>
>> In my post I suggested providing strong backwards compatibility from
>> 5.3 upwards, only because I think as far as the 5.2 and 5.1
>> compatibility story is concerned, the 'horse has bolted' so to speak.
>> In fact, I think there ought to be a concentrated effort to bring back
>> / support as much of 5.1 and 5.2 as possible - to enable more people
>> to upgrade to 5.3.
>
> Using third party software of course means that you have to be
> vigilant,
> it is your responsability, not the third party software author's.
>
>>> I think the Lua team always made a good judgement when faced with the
>>> problem of introducing a new feature and also an incompatibility.  I
>>> don't
>>> think this should be changed.
>>>
>>
>> I think that it is possible to evolve the language without breaking
>> compatibility. Just look at how much C++ has evolved since 1998, and
>> yet all the old code still works. It requires more effort and thought
>> - and sometimes things don't look so nice but that is a price worth
>> paying.
>
> Have I already mentioned that I like Lua for what it is?  An slowly
> evolving language without batteries included, without to much care for
> backwards compatability?  I hope it stays so.
>
> - marc
Thanks Marc.  You have answered him for me :-)

Regards,
Tomás


Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Dibyendu Majumdar
In reply to this post by Marc Balmer
Hi Marc,

On 12 January 2017 at 14:56, Marc Balmer <[hidden email]> wrote:

>> I think we can see from experience that languages that have done this
>> - i.e. maintained strong backwards compatibility - such as C, C++,
>> Java, C#, Go, have all benefited from it. We see also the struggle
>> being faced by Python moving everyone to 3.x, and similarly the
>> fragmented ecosystem of Lua.
>
> Oh and then there is more fragmentation going on, I remember one guy
> wrote a thing called Ravi we added, well, to the fragmentation...
>

I think you are missing the point. Ravi is superset of Lua 5.3; Lua
5.3 programs are valid Ravi programs. I go to some pains to ensure
that. And this is what I am suggesting - that Lua 5.3 upwards, later
versions of Lua should be a superset of Lua 5.3.

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Andrew Starks-2
In reply to this post by Dibyendu Majumdar

On Wed, Jan 11, 2017 at 18:03 Dibyendu Majumdar <[hidden email]> wrote:
On 11 January 2017 at 21:06, Hisham <[hidden email]> wrote:

> On 11 January 2017 at 17:08, Roberto Ierusalimschy

> <[hidden email]> wrote:

>>> Please, do not make such a severely breaking change in a 5.3.x release.

>>

>> We did not realize that. It will be corrected.

>

> Thank you! :)

>



Hi,



I was wondering if it is possible to consider making strong backwards

compatibility guarantees for Lua from 5.3 upwards. I think this will

mean some constraints to Lua's development obviously, but it would

strengthen Lua 's ecosystem.



The strongest argument I could make is that Lua is built on C, and

imagine how difficult it would have been if C had kept changing in

incompatible ways every few years, and all new C compilers stopped

supporting older versions of C.



The other argument for a compatibility promise is to try and bring

everyone up to 5.3 at least. If the language again changes in an

incompatible way that then you will end up with users in 5.1, 5.2, 5.3

and future versions. I think this fragmentation is not beneficial to

Lua.



I was thinking initially of requesting that "Lua be frozen" as a

language ... but that is maybe too radical a thought.



I think we can see from experience that languages that have done this

- i.e. maintained strong backwards compatibility - such as C, C++,

Java, C#, Go, have all benefited from it. We see also the struggle

being faced by Python moving everyone to 3.x, and similarly the

fragmented ecosystem of Lua.



Regards

Dibyendu


I believe that the mission of simplicity would come into conflict many attempts to maintain strict backwards compatibility, when progress in the language is the goal. In other languages where size is and simplicity are further down the list of priorities, progress and compatibility are possible.

I think that the authors have adjusted the balance of disruption as Lua has gained popularity, but the above choices probably preclude them from drawing a line in the sand. Also, Lua's primary use case is as an embedded library, not as a standalone language. Given that all versions of Lua are supported and patched when needed, there is no risk to using older versions and there for the need to update is never critical. 

For a while it has seemed to me that Lua is reaching perfection. But that is maybe always a silly thought when applied to software. Good ideas are still out there. 

-Andrew
Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

William Ahern
In reply to this post by Coda Highland
On Thu, Jan 12, 2017 at 07:42:49AM -0800, Coda Highland wrote:

> On Thu, Jan 12, 2017 at 6:44 AM, Nagaev Boris <[hidden email]> wrote:
> >> OTOH, Lua wouldn't be the language it is today if backwards compatibility
> >> was a higher priority. Features like closures over mutable variables and
> >> stackful coroutines wouldn't have been developed; features which arguably
> >> are now defining characteristics of Lua relative to other languages. Other
> >> than Scheme, I can't think of any other non-academic language with similarly
> >> powerful constructs.
> >
> > Go seems to have variables mutable from closures.
> > https://play.golang.org/p/o23Z6bbUTp
>
> C++, Obj-C, and Javascript have them too. It's not all THAT rare.

Javascript does, but C++ and Obj-C do not. C++ lambdas and Obj-C blocks
capture by value fundamentally. It's why a lambda can't close over an
automatic (stack allocated) variable and then access that variable after the
scope of that variable has exited. Being able to access variables after
their scope has ended is pretty much the definition of lexically bound
closures.

C and C++ are fundamentally pass-by-value languages. In C++ and Obj-C you
have to manually heap-allocate the variable and close over the pointer
(literally or implicitly via syntactic sugar), as the lambdas/block are
really only capturing the value of the pointer.

> Python and C/C++ (and maybe Obj-C?) have stackful coroutines.

Python doesn't have stackful coroutines. Quite the opposite. Same for C++. A
stackful coroutine means you can yield from nested function invocations
_without_ having to specifically invoke the nested function as a coroutine.

As for C and C++, multiple stacks don't really count, otherwise you could
argue that any language with threads has coroutines. But coroutines aren't
just about maintaining state of function invocations, but about how control
flow is passed.


Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

William Ahern
In reply to this post by Nagaev Boris
On Thu, Jan 12, 2017 at 02:44:15PM +0000, Nagaev Boris wrote:

> On Thu, Jan 12, 2017 at 1:18 AM, William Ahern
> <[hidden email]> wrote:
> > On Thu, Jan 12, 2017 at 12:02:49AM +0000, Dibyendu Majumdar wrote:
> >> On 11 January 2017 at 21:06, Hisham <[hidden email]> wrote:
> >> > On 11 January 2017 at 17:08, Roberto Ierusalimschy
> >> > <[hidden email]> wrote:
> >> >>> Please, do not make such a severely breaking change in a 5.3.x release.
> >> >>
> >> >> We did not realize that. It will be corrected.
> >> >
> >> > Thank you! :)
> >> >
> >>
> >> Hi,
> >>
> >> I was wondering if it is possible to consider making strong backwards
> >> compatibility guarantees for Lua from 5.3 upwards. I think this will
> >> mean some constraints to Lua's development obviously, but it would
> >> strengthen Lua 's ecosystem.
> >>
> >> The strongest argument I could make is that Lua is built on C, and
> >> imagine how difficult it would have been if C had kept changing in
> >> incompatible ways every few years, and all new C compilers stopped
> >> supporting older versions of C.
> >
> > OTOH, Lua wouldn't be the language it is today if backwards compatibility
> > was a higher priority. Features like closures over mutable variables and
> > stackful coroutines wouldn't have been developed; features which arguably
> > are now defining characteristics of Lua relative to other languages. Other
> > than Scheme, I can't think of any other non-academic language with similarly
> > powerful constructs.
>
> Go seems to have variables mutable from closures.
> https://play.golang.org/p/o23Z6bbUTp

Ah, right. And Go effectively has stackful coroutines. It totally slipped my
mind.
 
FWIW, I meant to say that few languages have _both_ lexically-bound closures
_and_ stackful coroutines.


Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Dibyendu Majumdar
In reply to this post by William Ahern
Hi William,

On 12 January 2017 at 01:18, William Ahern <[hidden email]> wrote:

> OTOH, Lua wouldn't be the language it is today if backwards compatibility
> was a higher priority. Features like closures over mutable variables and
> stackful coroutines wouldn't have been developed; features which arguably
> are now defining characteristics of Lua relative to other languages. Other
> than Scheme, I can't think of any other non-academic language with similarly
> powerful constructs. And that's largely because before they get there
> language designers and implementers decide their language is "good enough"
> and switch into long-term maintenance mode where, at best, only pale
> imitations of the above can be added.
>

I am relatively new to Lua so my knowledge of Lua history may be
flaky, but I thought these features you mention were introduced in Lua
5.0 in 2003? So they can hardly be justification for incompatibilities
between 5.1, 5.2 and 5.3. Or am I missing something?

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: Lua and backwards compatibility - was Re: [ANN] Lua 5.3.4 (rc2) now available,

Coda Highland
In reply to this post by William Ahern
On Thu, Jan 12, 2017 at 10:33 AM, William Ahern
<[hidden email]> wrote:

> On Thu, Jan 12, 2017 at 07:42:49AM -0800, Coda Highland wrote:
>> On Thu, Jan 12, 2017 at 6:44 AM, Nagaev Boris <[hidden email]> wrote:
>> >> OTOH, Lua wouldn't be the language it is today if backwards compatibility
>> >> was a higher priority. Features like closures over mutable variables and
>> >> stackful coroutines wouldn't have been developed; features which arguably
>> >> are now defining characteristics of Lua relative to other languages. Other
>> >> than Scheme, I can't think of any other non-academic language with similarly
>> >> powerful constructs.
>> >
>> > Go seems to have variables mutable from closures.
>> > https://play.golang.org/p/o23Z6bbUTp
>>
>> C++, Obj-C, and Javascript have them too. It's not all THAT rare.
>
> Javascript does, but C++ and Obj-C do not. C++ lambdas and Obj-C blocks
> capture by value fundamentally. It's why a lambda can't close over an
> automatic (stack allocated) variable and then access that variable after the
> scope of that variable has exited. Being able to access variables after
> their scope has ended is pretty much the definition of lexically bound
> closures.

That C++ stack variables go away is really a sign that it IS in fact
shared, and one consumer has explicitly destroyed it. You'd have the
same problem with any other data-sharing construct in C++, so this
specific limitation is just a matter of language best practices rather
than "C++ can't do it".

Obj-C blocks can capture boxed values that are refcounted/garbage-collected.

>> Python and C/C++ (and maybe Obj-C?) have stackful coroutines.
>
> Python doesn't have stackful coroutines. Quite the opposite. Same for C++. A
> stackful coroutine means you can yield from nested function invocations
> _without_ having to specifically invoke the nested function as a coroutine.
>
> As for C and C++, multiple stacks don't really count, otherwise you could
> argue that any language with threads has coroutines. But coroutines aren't
> just about maintaining state of function invocations, but about how control
> flow is passed.

Okay, so maybe I'm wrong about Python, but I'm not wrong about the C
family. You're right that multiple stacks on its own aren't equivalent
to coroutines, but they provide the essential structure necessary to
implement them. You can't just look at the core language definition
and stop there. I've used a coroutine library before. It's really not
bad at all.

Also, ES7 has them.

/s/ Adam

123