[OFF] 5 years to find a typo in a config script

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

[OFF] 5 years to find a typo in a config script

Egor Skriptunoff-2
Hi!

This post is a bit off-topic.

A typo in an identifier inside a config script of one unreal-engine-based game
worsened the game AI significantly.
But it tooks five years to find the typo.
(Lua was not the scripting language that the config was written in.)

https://www.pcgamer.com/all-this-time-aliens-colonial-marines-stupid-ai-may-have-been-caused-by-a-single-typo/

BTW, there were lawsuits against game's producer.
I guess the typo was one of the reasons of why "the game stinks".

https://arstechnica.com/gaming/2014/07/gearbox-responds-to-aliens-class-action-lawsuit-throws-sega-under-bus/

The conclusions from this story:
1) Typos are hard-to-find bugs.
2) IMO, Lua have to somehow solve the problem of absence of warnings about misspelled identifiers.

-- Egor
Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Enrico Colombini
On 02-Sep-18 13:14, Egor Skriptunoff wrote:
> The conclusions from this story:
> 1) Typos are hard-to-find bugs.

True. A couple of weeks ago I spent half an hour trying to understand
why a C function I was testing did not work as expected.
It turned out I accidentally assigned a value to the wrong local variable.

> 2) IMO, Lua have to somehow solve the problem of absence of warnings
> about misspelled identifiers.

I heard a mysterious sect performs dark rituals involving a
little-known, exoteric technique revealed only to a few selected
believers. It's called "testing".

Seriously, having used my share of static and non-static languages over
many years, I didn't notice much difference in the frequence of this
sort of problems. But, again, I may be part of the above-mentioned sect.

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Dirk Laurie-2
Op So., 2 Sep. 2018 om 15:17 het Enrico Colombini <[hidden email]> geskryf:

> > 2) IMO, Lua have to somehow solve the problem of absence of warnings
> > about misspelled identifiers.

It has solved the problem of retrieving unassigned globals. It's
called 'strict.lua'. It's not in the distribution. Maybe it should be.
But Penlight and stdlib both supply it.

It has not solved the problem of mind-reading the programmer. If you
have variables x1 and xi, and by accident type xi when you mean x1,
not even strict.lua will complain.

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Enrico Colombini
On 02-Sep-18 20:18, Dirk Laurie wrote:
> Op So., 2 Sep. 2018 om 15:17 het Enrico Colombini<[hidden email]>  geskryf:
>
>>> 2) IMO, Lua have to somehow solve the problem of absence of warnings
>>> about misspelled identifiers.

A small note: that quotation is not from me, as it may appear.

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Dirk Laurie-2
Op So., 2 Sep. 2018 om 20:30 het Enrico Colombini <[hidden email]> geskryf:
>
> On 02-Sep-18 20:18, Dirk Laurie wrote:
> > Op So., 2 Sep. 2018 om 15:17 het Enrico Colombini<[hidden email]>  geskryf:
> >
> >>> 2) IMO, Lua have to somehow solve the problem of absence of warnings
> >>> about misspelled identifiers.
>
> A small note: that quotation is not from me, as it may appear.

Sorry about that. One can count the number of > signs and come to that
conclusion, but  the attribution should have read:

Op So., 2 Sep. 2018 om 15:17 het Enrico Colombini <[hidden email]> geskryf:
>
> On 02-Sep-18 13:14, Egor Skriptunoff wrote:
> > The conclusions from this story:
> > .2) IMO, Lua have to somehow solve the problem of absence of warnings
> >>> about misspelled identifiers.

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Roberto Ierusalimschy
In reply to this post by Dirk Laurie-2
> It has solved the problem of retrieving unassigned globals. It's
> called 'strict.lua'. It's not in the distribution. Maybe it should be.
> But Penlight and stdlib both supply it.
>
> It has not solved the problem of mind-reading the programmer. If you
> have variables x1 and xi, and by accident type xi when you mean x1,
> not even strict.lua will complain.

strict.lua will also not complain if you write '<' instead of '>', or
'<' instead '<=', or '+1' instead of nothing, or nothing istead of '+1',
etc. In my view, they are all in the same category of typos in
indentifiers, and very few languages detect them (coq?). OTOH,
Enrico's sect seems to cure that, too.

(Actually, there is a difference. A main issue here is what kind
of errors people expect, and that depends heavly from where those
people come from. For instance, some people look after non-intialized
variables when seeing random values in a program, some people do not.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Dirk Laurie-2
Op So., 2 Sep. 2018 om 21:15 het Roberto Ierusalimschy
<[hidden email]> geskryf:

>
> > It has solved the problem of retrieving unassigned globals. It's
> > called 'strict.lua'. It's not in the distribution. Maybe it should be.
> > But Penlight and stdlib both supply it.
> >
> > It has not solved the problem of mind-reading the programmer. If you
> > have variables x1 and xi, and by accident type xi when you mean x1,
> > not even strict.lua will complain.
>
> strict.lua will also not complain if you write '<' instead of '>', or
> '<' instead '<=', or '+1' instead of nothing, or nothing istead of '+1',
> etc. In my view, they are all in the same category of typos in
> indentifiers, and very few languages detect them (coq?). OTOH,
> Enrico's sect seems to cure that, too.

The so-called London sect [1] indulges in an even darker and less-known ritual.

> The often asked question, "How can one demonstrate that a computer program does what it is supposed to do?" has at least three answers:
> (i) use the standard debugging technique of testing "representative" data and checking the results,
> (ii) read the program and mysteriously discern that it works, thereby convincing oneself that the program is right or
> (iii) give a rigorous mathematical proof of the correctness of the program.

-- Dirk

[1] https://slideheaven.com/proving-programs-correct-some-techniques-and-examples.html

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Sean Conner
In reply to this post by Egor Skriptunoff-2
It was thus said that the Great Egor Skriptunoff once stated:
>
> The conclusions from this story:
> 1) Typos are hard-to-find bugs.
> 2) IMO, Lua have to somehow solve the problem of absence of warnings about
> misspelled identifiers.

  I use luacheck for that.  

  More specifically, Lua would have to be changed to require all variables
decleared before use.  Even luacheck requires this with special comments.
For example, luacheck does:  "Hmm ... this is the first instance of 'baa'
I've seen.  It has not been declared as a local, nor is it a standard
global, nor has it been defined as a global to me.  Therefore, it must be a
misspelled identifier [1]."

  -spc

[1] Calling Gregg Reynolds!  Calling Gregg Reynolds! [2]

[2] Email thread starting here:
        http://lua-users.org/lists/lua-l/2018-07/msg00094.html


Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Luiz Henrique de Figueiredo
In reply to this post by Dirk Laurie-2
> > The often asked question, "How can one demonstrate that a computer program does what it is supposed to do?" has at least three answers:
> > (i) use the standard debugging technique of testing "representative" data and checking the results,
> > (ii) read the program and mysteriously discern that it works, thereby convincing oneself that the program is right or
> > (iii) give a rigorous mathematical proof of the correctness of the program.

Which reminds me of this famous quote by Donald Knuth:

"Beware of bugs in the above code; I have only proved it correct, not tried it."
https://en.wikiquote.org/wiki/Donald_Knuth

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Lorenzo Donati-3
In reply to this post by Egor Skriptunoff-2
On 02/09/2018 13:14, Egor Skriptunoff wrote:

> Hi!
>
> This post is a bit off-topic.
>
> A typo in an identifier inside a config script of one unreal-engine-based
> game
> worsened the game AI significantly.
> But it tooks five years to find the typo.
> (Lua was not the scripting language that the config was written in.)
>

On a related note, I was recently browsing source files for a game in
"The Battle for Wesnoth" open-source game engine (strategic, turn-based,
fantasy-oriented). It is scripted in Lua, but it uses a DSL (WML -
Wesnoth Markup Language) for campaign/scenarios definitions.

In one of the WML files of a user-defined campaign, inside a side AI
definition block I read the following line (and it was repeated in 2
other spots in the code):

aggresion = 1

That, of course, is a blatant spelling mistake, but was buried deep into
code written (quite well) by a probably non-English-speaking developer.

Although, as others in this thread have said, it is not possible to
prevent any sort of typos, it would help, IMO, running the code through
a spell-checker.

This would help significantly, especially if one sticks to identifiers
written in plain english, with component words spelled out clearly
(underscores as spaces or CamelCase - of course we would need a
specially configured spell-checker for this).

Although C tradition goes against this, I think that in scopes longer
than about 10-15 lines using very short (or abbreviated) identifiers is
usually a very bad practice.

Anyway, a properly trained/configured spell-checker could also cope
effectively with abbreviated words if care is taken to stick to commonly
used abbreviations and not to invent a new abbreviation for every term
longer than 10 chars (or worse, different abbreviations for the same
term in different part of the code)!

So, to link to what Dirk Laurie said in another message, if you mistake
x1 for xi, the programmer who chose those names is at fault. Of course I
know that in math world many algorithms contain names that mimick math
notations (short and not resembling words of any spoken language), but
these cases are relatively rare in the SW world in general.

And anyway, you sometimes can do better in math code spanning longer
scopes. For example: "root_i" is often better than "xi" (yes, I know the
mathematically inclined could barf at it :-)

My 2 EUR-cent

[snip]



> -- Egor
>


Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Dirk Laurie-2
Op Ma., 3 Sep. 2018 om 10:52 het Lorenzo Donati
<[hidden email]> geskryf:

> On a related note, I was recently browsing source files for a game in
> "The Battle for Wesnoth" open-source game engine (strategic, turn-based,
> fantasy-oriented). It is scripted in Lua, but it uses a DSL (WML -
> Wesnoth Markup Language) for campaign/scenarios definitions.
>
> In one of the WML files of a user-defined campaign, inside a side AI
> definition block I read the following line (and it was repeated in 2
> other spots in the code):
>
> aggresion = 1
>
> That, of course, is a blatant spelling mistake, but was buried deep into
> code written (quite well) by a probably non-English-speaking developer.

If a programmer uses the word "aggresion", it means precisely what the
Humpty Dumpty inside them chooses it to mean — neither more nor less.
There is no right or wrong here. The question is, which is to be
master — that's all.

It's no more eccentric than the British insistence on the French
spelling "colour" and and the pseudo-Greek spelling "sulphur" despite
the fact that, as all good Americans know, those words come from Latin
and should therefore be spelled "color" and "sulfur".

Come to think of it, "aggresion" may even be a word, formed from a
verb "aggrede" like "collision" is formed from "collide". Where's my
copy of the collected works of Sir Edmund Spenser?

-- Dirk

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Lorenzo Donati-3
On 03/09/2018 12:07, Dirk Laurie wrote:

> Op Ma., 3 Sep. 2018 om 10:52 het Lorenzo Donati
> <[hidden email]> geskryf:
>
>> On a related note, I was recently browsing source files for a game in
>> "The Battle for Wesnoth" open-source game engine (strategic, turn-based,
>> fantasy-oriented). It is scripted in Lua, but it uses a DSL (WML -
>> Wesnoth Markup Language) for campaign/scenarios definitions.
>>
>> In one of the WML files of a user-defined campaign, inside a side AI
>> definition block I read the following line (and it was repeated in 2
>> other spots in the code):
>>
>> aggresion = 1
>>
>> That, of course, is a blatant spelling mistake, but was buried deep into
>> code written (quite well) by a probably non-English-speaking developer.
>
> If a programmer uses the word "aggresion", it means precisely what the
> Humpty Dumpty inside them chooses it to mean — neither more nor less.
> There is no right or wrong here. The question is, which is to be
> master — that's all.
>

Sorry for not being clearer: that configuration parameter was not a
choice of the programmer, it was meant to be "aggression" as defined by
the WML language (it is a parameter of the behavior of the AI).

So /it is/ a spelling mistake (that's why I used it as an example).

Here we have a difference between the programmer of the WML system (who
chooses "keywords", i.e. field names in this case) vs. the programmer of
the specific adventure game (who programs /in/ WML in order to define
scenario settings and behavior).

BTW, actually "sulphur" is also a correct, alternative, Latin spelling.
You just tickled my rusty "Latin brain" and so I run for my old and
dusty Latin-Italian dictionary (Castiglioni-Mariotti).

I read from it that it was used in that form by Catullo and Lucrezio.

Since we are digressing, the origin of the dual spelling are intriguing.
This document I just found might be interesting:

https://europepmc.org/backend/ptpmcrender.fcgi?accid=PMC1838885&blobtype=pdf


> It's no more eccentric than the British insistence on the French
> spelling "colour" and and the pseudo-Greek spelling "sulphur" despite
> the fact that, as all good Americans know, those words come from Latin
> and should therefore be spelled "color" and "sulfur".
>

Well, that's extreme nitpicking. :-)

As I hinted in my post, I dare say most programmers worldwide nowadays
are not native English speakers, and I doubt many of them are aware of
etymology of English words.


> Come to think of it, "aggresion" may even be a word, formed from a
> verb "aggrede" like "collision" is formed from "collide". Where's my
> copy of the collected works of Sir Edmund Spenser?
>

In my experience reading code written by others, cases where the
programmer was so sophisticated in choosing his identifiers versus the
case it was a mere spelling mistake are 1 out of 20 (upper bound).


> -- Dirk
>
>



Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Egor Skriptunoff-2
In reply to this post by Dirk Laurie-2
On Sun, Sep 2, 2018 at 9:18 PM, Dirk Laurie wrote:
If you have variables x1 and xi, and by accident type xi when you mean x1


Yes, the typo "x1" -> "xi" is easy to make and hard to detect.
As Lorenzo already said, variables naming like "x1, x2, x3, ..., xi, xj" is a bad programming style.
In most use cases (under "good style"), the Levenshtein distance between existing identifiers is rather big.
(the Levenshtein distance is the minimal number of typos required to transform one string into another)
So, the probability of typos accidentally converting one existing identifier into another, is almost zero.

Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Egor Skriptunoff-2
In reply to this post by Roberto Ierusalimschy
On Sun, Sep 2, 2018 at 10:15 PM, Roberto Ierusalimschy wrote:
A main issue here is what kind of errors people expect,
and that depends heavily from where those people come from.


Even Lua-only developer (without previous "toxic" experience with other programming languages) would stumble upon misspelled variables problem.
Detecting mistakes in variable names would help all of Lua users.

 
...if you write '<' instead of '>', or '<' instead '<=', or '+1' instead of nothing,...
In my view, they are all in the same category of typos in identifiers

 
Yes, Lua can't warn user about typos like "+1" -> "+11", "<" -> ">" and so on, because Lua is unable to determine whether they are mistakes or not.
But nevertheless detecting the most of the typos is achievable, because typos in identifiers are the most frequent typos and they usually are mistakes.
 
I'd suggest to perform an experiment to calculate (by Monte Carlo simulation) the probability Lua is unable to detect a random typo at compile time.
1) Take a big corpus of Lua scripts (Roberto, you've mentioned you have such one)
2) Emulate random typos: insert a symbol (Lua->Luna), delete a symbol (Lua->La), replace a symbol with one of its keyboard-neighbors (Lua->Lia), swap two adjacent symbols (Lua->Lau).
A typo which don't introduce changes in the bytecode should not be counted as a typo (for example, inserting an extra space in Lua program might be harmless).
3) Calculate the probability a typo in Lua 5.3 script would stay undetected ("detected" means "fails to compile").
4) Calculate the probability a typo in Lua script would stay undetected with "global-by-default" feature disabled.
5) Publish the results in this mailing list (how big would be the quotient of these two numbers?).


Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Oliver Kroth
In reply to this post by Egor Skriptunoff-2


Am 03.09.2018 um 21:06 schrieb Egor Skriptunoff:
On Sun, Sep 2, 2018 at 9:18 PM, Dirk Laurie wrote:
If you have variables x1 and xi, and by accident type xi when you mean x1


Yes, the typo "x1" -> "xi" is easy to make and hard to detect.
As Lorenzo already said, variables naming like "x1, x2, x3, ..., xi, xj" is a bad programming style.
In most use cases (under "good style"), the Levenshtein distance between existing identifiers is rather big.
(the Levenshtein distance is the minimal number of typos required to transform one string into another)
So, the probability of typos accidentally converting one existing identifier into another, is almost zero.

x1 <-> xi is relatively safe, compared with x1 <-> xl, or X0 <-> XO
"ex" "one" <-> "ex" "el", or "EX" "zero" <-> "EX" "OH"

I never use small L or capital O for variable names:
C:    for (i=l; i>O; i++)
Lua; for i=O,l do ... end


--
01iver
Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Tom Sutcliffe
In reply to this post by Enrico Colombini
> On 2 Sep 2018, at 2:17 pm, Enrico Colombini <[hidden email]> wrote:
>
> On 02-Sep-18 13:14, Egor Skriptunoff wrote:
>> The conclusions from this story:
>> 1) Typos are hard-to-find bugs.
>
> True. A couple of weeks ago I spent half an hour trying to understand why a C function I was testing did not work as expected.
> It turned out I accidentally assigned a value to the wrong local variable.
>
>> 2) IMO, Lua have to somehow solve the problem of absence of warnings about misspelled identifiers.
>
> I heard a mysterious sect performs dark rituals involving a little-known, exoteric technique revealed only to a few selected believers. It's called "testing".
>
> Seriously, having used my share of static and non-static languages over many years, I didn't notice much difference in the frequence of this sort of problems. But, again, I may be part of the above-mentioned sect.

For one of the Lua-based DSL projects I work on, I added a validator which requires that every line of code in your script gets hit when you throw the test data at it, and that the calculated results are as expected for each set of inputs. (The test data is also defined in the script). It makes heavy use of the debug library and line hooks as you'd expect, and has been surprisingly useful to detect not only bugs in the scripts, but also in the modules they rely on ("if all my scripts still validate as expected, the other libraries probably aren't too broken") and highlighting unhandled or untested code ("how exactly *would* I get to the situation where I would hit this line of code?"). If the validator doesn't pass, the build fails meaning no-one ever merges broken scripts assuming they built it at least once beforehand, and if they don't even build their code before merging then I put my Build Team hat on and give them a polite explanation of why I've reverted all their commits.

Obviously it's only practical in a tightly-controlled DSL environment, but it's proved quite a nice hybrid of static analyser and test harness for thousands of lines of conditional-heavy scripts which examine a lot of global state. And requiring every line to be hit is only a rough approximation of testing every possible code path and combination of input data, but it works pretty well.

I really should add strict.lua-esque global variable checks too.

Cheers,

Tom
Reply | Threaded
Open this post in threaded view
|

Re: [OFF] 5 years to find a typo in a config script

Russell Haley
In reply to this post by Egor Skriptunoff-2


On Mon, Sep 3, 2018 at 12:07 PM Egor Skriptunoff <[hidden email]> wrote:
On Sun, Sep 2, 2018 at 9:18 PM, Dirk Laurie wrote:
If you have variables x1 and xi, and by accident type xi when you mean x1


Yes, the typo "x1" -> "xi" is easy to make and hard to detect.
As Lorenzo already said, variables naming like "x1, x2, x3, ..., xi, xj" is a bad programming style.
In most use cases (under "good style"), the Levenshtein distance between existing identifiers is rather big.
(the Levenshtein distance is the minimal number of typos required to transform one string into another)
So, the probability of typos accidentally converting one existing identifier into another, is almost zero.

Thanks Egor, I figured there was already an algorithm for this.  A check script that outputs some statistics based on Levenshtein distance would be a neat little tool. 

Russ