A rant about backward-incompatible changes

classic Classic list List threaded Threaded
42 messages Options
123
Reply | Threaded
Open this post in threaded view
|

A rant about backward-incompatible changes

Simon Cozens
Hello!

I maintain SILE, which is quite a large, complex application written in
Lua. Because it's an application, I can't control which versions of Lua
its end-users will be using, so we have to support a range of
environments. At the moment we have support for 5.1, 5.2 and I have
recently added support for 5.3.

If I was starting again, though, I would probably not use Lua.

When 5.3 came out, SILE stopped working for users who had upgraded. Not
only SILE's code directly, but code that SILE was using indirectly
through libraries and luarocks simply stopped working and started giving
errors. I don't think this is a great state of affairs.

In particular, one of the changes in 5.3 was that the core "unpack()"
function for handling variable arguments was removed. You might argue
that it was moved to table.unpack, but that's irrelevant: the end result
from a user's perspective was "attempt to call a nil value (global
'unpack')".

A backwards-incompatible change that breaks end-user code is the sort of
thing that probably should be documented, perhaps in the changelog. The
changelog says that "the reference manual lists the incompatibilities
that had to be introduced". Perhaps it was an oversight, but there's no
mention in chapter 8 of the reference manual that a core function was
incompatibly removed. In the changelog, it lists "functions for packing
and unpacking values" under "main changes", but that's as much detail as
you get.

I had to hunt around to find out what was going on, because I didn't
expect built-in functions to silently disappear, and passing variable
arguments to a function is a pretty key thing to suddenly vanish without
trace or documentation.

One more thing: from an application writer's perspective, having
backward-compatibility options available in luaconf.h makes things a
whole lot worse, not better. Because now you have to write code for two
possible platforms, both calling themselves Lua 5.3, but actually with
different and incompatible functionality. What could possibly go wrong?

I know that Lua is an evolving language and I don't think you should
never be able to make any core changes ever. But at the least, could we
please have a deprecation cycle with warnings that code needs to be
updated, rather than unceremoniously breaking end-user code?

Could we also please have a commitment to full documentation of
incompatible changes, with explanation of what needs to be done to older
code to make it work?

And please, can we think about the impact on third-party code of
shifting things around in the core?  To make a Lua application
compatible across Lua versions, I have to stuff my code with lines like

        unpack = unpack or table.unpack

That's not a hard thing. Lua could actually do that for me, but now I
have to do it. In other words, making code backwards-compatible in Lua
is a burden, and the burden is being placed on the developer, not on the
language. I don't think that's not a good state of affairs either.

Knowing that, when 5.4 comes out, my code is probably going to stop
working again, through no fault of my own, does not really fill me with
confidence or joy.

Simon

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Luiz Henrique de Figueiredo
> I maintain SILE, which is quite a large, complex application written in
> Lua. Because it's an application, I can't control which versions of Lua
> its end-users will be using, so we have to support a range of
> environments.

Why does a complex application depend on the version of Lua that end-users have?

If you control the application, why not embed a fixed version of Lua?

Many applications do, for instance VLC.

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Simon Cozens
On 14/07/2015 23:32, Luiz Henrique de Figueiredo wrote:
> Why does a complex application depend on the version of Lua that end-users have?

Because the application is written in Lua, and uses the Lua interpreter.


Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Rob Kendrick-2
On Tue, Jul 14, 2015 at 11:35:07PM +0900, Simon Cozens wrote:
> On 14/07/2015 23:32, Luiz Henrique de Figueiredo wrote:
> > Why does a complex application depend on the version of Lua that end-users have?
>
> Because the application is written in Lua, and uses the Lua interpreter.

#!/usr/bin/env lua5.1

And then mock anyone who isn't using a Debian derivative.

To be honest, exactly the same thing happens with Python, Ruby, and
Perl.  What does $(which python) expand to, Python 1, 2, or 3?  Nobody
knows.

B.

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Simon Cozens
On 14/07/2015 23:40, Rob Kendrick wrote:
> To be honest, exactly the same thing happens with Python, Ruby, and
> Perl.

No, it doesn't, because these other languages take backwards
compatibility seriously.

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Rob Kendrick-2
On Tue, Jul 14, 2015 at 11:43:33PM +0900, Simon Cozens wrote:
> On 14/07/2015 23:40, Rob Kendrick wrote:
> > To be honest, exactly the same thing happens with Python, Ruby, and
> > Perl.
>
> No, it doesn't, because these other languages take backwards
> compatibility seriously.

Python 3 is not backwards compatible with Python 2.  And Perl 5 is not
with 4.  In fact, 5 has broken stuff in "point releases" for me in the
past.

B.

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Coda Highland
In reply to this post by Simon Cozens
On Tue, Jul 14, 2015 at 7:43 AM, Simon Cozens <[hidden email]> wrote:
> On 14/07/2015 23:40, Rob Kendrick wrote:
>> To be honest, exactly the same thing happens with Python, Ruby, and
>> Perl.
>
> No, it doesn't, because these other languages take backwards
> compatibility seriously.
>

Lua does, too. The feature in question spent the full duration of 5.2
clearly marked as "deprecated" and was accordingly removed from the
next major revision.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Simon Cozens
In reply to this post by Rob Kendrick-2
On 14/07/2015 23:45, Rob Kendrick wrote:
> Python 3 is not backwards compatible with Python 2.

Right, but it has a deprecation/future process. ("from __future__
import...")

> And Perl 5 is not with 4.

Actually, it is.
https://en.wikibooks.org/wiki/Perl_6_Programming/Perl_History says,
correctly, "Through all these developments, Perl has remained backwards
compatible with previous versions. The Perl 5 interpreter can read,
understand and execute (for the most part) programs that were written in
Perl 1, Perl 2, Perl 3 and Perl 4." And there is a test suite for this,
and a policy:
http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION



Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Björn Kalkbrenner
In reply to this post by Simon Cozens
Hi!

On 14.07.2015 16:35, Simon Cozens wrote:
> On 14/07/2015 23:32, Luiz Henrique de Figueiredo wrote:
>> Why does a complex application depend on the version of Lua that end-users have?
>
> Because the application is written in Lua, and uses the Lua interpreter.

Some things i have learned while using Lua is that i can:

- trust Lua's major-minor releases regarding revisions. If the revision
changes, nothing breaks. Not more or less (better than on other languages).

- find or create compatibility modules (compat-x.y.lua) to be compatible
to a major/minor jump if i REALLY need this

- can check the version number

- there are few versions as RC which i can try or test if i am
maintaining a software using Lua

- there are changelogs where such changes are documented

- distributions have slots for the different Lua versions (such as with
PHP4/PHP5/PHP6) or Python 2.7/3.0/3.2). Comparing, i thing that much
will break if you are using python2.7 and jumping higher (but i am not a
python coder). Same with PHP, i had that problem with PHP4 regarding
references to variables. You are not objective if you are saying "other
languages take backwards compatibilty seriously". I had that problems in
bigger scale with PHP really really often.

- write application/user documentation that makes supported versions clear

- it is really nice with Lua to implement dependency startup stuff (take
a look at prosody as example).

Bye
Björn


Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Luiz Henrique de Figueiredo
In reply to this post by Coda Highland
> Lua does, too. The feature in question spent the full duration of 5.2
> clearly marked as "deprecated" and was accordingly removed from the
> next major revision.

And if you really need compatibilty with Lua 5.1, you can even build
Lua 5.3 with LUA_COMPAT_5_1 on or just LUA_COMPAT_UNPACK.

See luaconf.h for all compatibility options.
Search for the section "Compatibility with previous versions".

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Rob Kendrick-2
In reply to this post by Simon Cozens
On Tue, Jul 14, 2015 at 11:50:44PM +0900, Simon Cozens wrote:
> On 14/07/2015 23:45, Rob Kendrick wrote:
> > Python 3 is not backwards compatible with Python 2.
>
> Right, but it has a deprecation/future process. ("from __future__
> import...")

That's forwards compatibility.

And Python 3 includes new keywords, so any Python 2 program that uses
them as variable names is shafted.

> > And Perl 5 is not with 4.
>
> Actually, it is.
> https://en.wikibooks.org/wiki/Perl_6_Programming/Perl_History says,
> correctly, "Through all these developments, Perl has remained backwards
> compatible with previous versions. The Perl 5 interpreter can read,
> understand and execute (for the most part) programs that were written in
> Perl 1, Perl 2, Perl 3 and Perl 4." And there is a test suite for this,
> and a policy:
> http://perldoc.perl.org/perlpolicy.html#BACKWARD-COMPATIBILITY-AND-DEPRECATION

"for the most part".

B.

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Simon Cozens
In reply to this post by Luiz Henrique de Figueiredo
On 14/07/2015 23:51, Luiz Henrique de Figueiredo wrote:
> And if you really need compatibilty with Lua 5.1, you can even build
> Lua 5.3 with LUA_COMPAT_5_1 on or just LUA_COMPAT_UNPACK.

I don't know what end-users are going to build with. Having multiple
versions of a version doesn't help here.


Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Daniel Silverstone
On Tue, Jul 14, 2015 at 23:59:22 +0900, Simon Cozens wrote:
> I don't know what end-users are going to build with. Having multiple
> versions of a version doesn't help here.

The one Lua-based project I have which really doesn't play nicely with multiple
lua versions simply refuses to work if the version is not the one it is written
for.

Operating systems which blithely jump forwards from 5.1 to 5.2 to 5.3 without
providing co-installability for those projects which have not been brought
forward should be larted until they behave properly.

D.

--
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 3CCE BABE 206C 3B69

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Simon Cozens
On 15/07/2015 00:03, Daniel Silverstone wrote:
> The one Lua-based project I have which really doesn't play nicely with multiple
> lua versions simply refuses to work if the version is not the one it is written
> for.

I think the take-away here is that each minor revision of Lua should be
regarded as a different language. If that's the policy, then, fine, I'll
run with it.


Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Coda Highland
On Tue, Jul 14, 2015 at 8:05 AM, Simon Cozens <[hidden email]> wrote:
> On 15/07/2015 00:03, Daniel Silverstone wrote:
>> The one Lua-based project I have which really doesn't play nicely with multiple
>> lua versions simply refuses to work if the version is not the one it is written
>> for.
>
> I think the take-away here is that each minor revision of Lua should be
> regarded as a different language. If that's the policy, then, fine, I'll
> run with it.
>

As a different dialect of the language, at least, yes, akin to the
difference between Python 2 and Python 3. (LuaJIT is also a different
dialect of the language.) A good distro will install them as slotted
packages that can coexist.

As Lua's primary goal is to be embedded, applications that run from
the system-installed Lua interpreter are a fairly recent development.
For most deployments you're expected to roll your own Lua.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Daniel Silverstone
In reply to this post by Simon Cozens
On Wed, Jul 15, 2015 at 00:05:23 +0900, Simon Cozens wrote:
> On 15/07/2015 00:03, Daniel Silverstone wrote:
> > The one Lua-based project I have which really doesn't play nicely with multiple
> > lua versions simply refuses to work if the version is not the one it is written
> > for.
>
> I think the take-away here is that each minor revision of Lua should be
> regarded as a different language. If that's the policy, then, fine, I'll
> run with it.

It's certainly how I run with it.  Some of my work can be used in more than
one of these sub-languages, but not always.

D.

--
Daniel Silverstone                         http://www.digital-scurf.org/
PGP mail accepted and encouraged.            Key Id: 3CCE BABE 206C 3B69

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Rob Kendrick-2
In reply to this post by Simon Cozens
On Wed, Jul 15, 2015 at 12:05:23AM +0900, Simon Cozens wrote:
> On 15/07/2015 00:03, Daniel Silverstone wrote:
> > The one Lua-based project I have which really doesn't play nicely with multiple
> > lua versions simply refuses to work if the version is not the one it is written
> > for.
>
> I think the take-away here is that each minor revision of Lua should be
> regarded as a different language. If that's the policy, then, fine, I'll
> run with it.

The thing to remember here is that in Lua's versioning scheme x.y.z, the
y is the major version, and the z the minor...

B.

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Staven
In reply to this post by Simon Cozens
On Tue, Jul 14, 2015 at 11:35:07PM +0900, Simon Cozens wrote:
> On 14/07/2015 23:32, Luiz Henrique de Figueiredo wrote:
> > Why does a complex application depend on the version of Lua that end-users have?
>
> Because the application is written in Lua, and uses the Lua interpreter.
>

So, don't do that.


Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Dirk Laurie-2
In reply to this post by Luiz Henrique de Figueiredo
2015-07-14 16:17 GMT+02:00 Simon Cozens <[hidden email]>:
>> I maintain SILE, which is quite a large, complex application written in
>> Lua. Because it's an application, I can't control which versions of Lua
>> its end-users will be using, so we have to support a range of
>> environments.
2015-07-14 16:32 GMT+02:00 Luiz Henrique de Figueiredo <[hidden email]>:
> If you control the application, why not embed a fixed version of Lua?

That was my first reaction too. You already embed over 1MB of libtexpdf
C source, so less than 0.4KB of Lua won't hurt.

LuaTeX, one of your main competitors in the typesetting world, does just
that.

Reply | Threaded
Open this post in threaded view
|

Re: A rant about backward-incompatible changes

Coda Highland
On Tue, Jul 14, 2015 at 8:14 AM, Dirk Laurie <[hidden email]> wrote:
> That was my first reaction too. You already embed over 1MB of libtexpdf
> C source, so less than 0.4KB of Lua won't hurt.

Lua's not THAT small. ;)

/s/ Adam

123