Lua 5.4.2 crashes where Lua 5.3.6 does not

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Lua 5.4.2 crashes where Lua 5.3.6 does not

Sean Conner

  I've been spending the past few days updating from Lua 5.3.6 (with all the
patches) to Lua 5.4.2.  So far, I've encounted one issue where Lua 5.4.2
will segfault where Lua 5.3.6 does not (I have not tested Lua 5.4.0 or
5.4.1).  I have tried with Lua 5.4.2 with LUA_USE_APICHECK, and it still
segfaults without any diagnostics.

  I've isolated the problem:

(gdb) where
#0  0x0806a7bc in resizebox (L=0x8d9900c, idx=-1, newsize=2048) at lauxlib.c:477
#1  0x0806a96e in prepbuffsize (B=0xbfa1d448, sz=1, boxidx=-1) at lauxlib.c:546
#2  0x0806aa41 in luaL_addlstring (B=0xbfa1d448, s=0x8dbd1b0 "\n", l=1) at lauxlib.c:572
#3  0xb7de62ca in yy_5_CHAR (yy=0xbfa1d400, yytext=0x8dbd1b0 "\n", yyleng=1) at html.i:3
#4  0xb7de600c in yyDone (yy=0xbfa1d400) at html.i:226
#5  0xb7e1b7c8 in yyparsefrom (yyctx=0xbfa1d400, yystart=0xb7e1b5e0 <yy_BODY>) at html.i:11517
#6  0xb7e1b809 in yyparse (yyctx=0xbfa1d400) at html.i:11524
#7  0xb7e1bd11 in parse (L=0x8d9900c) at html.c:161
        [ rest of stack trace snipped ]
(gdb) p box
$1 = (UBox *) 0x0
(gdb)

  Now, some context.  I originally wrote an HTML parser in LPEG, but found
that it consumed way too much memory for my liking.  I then found a PEG
library for C, which I used to generate a replacement module that uses way
less memory (and is faster, but that's just a nice benefit).  The module
works find under Lua 5.3.6.  I've also included both the PEG file and the
output from the PEG library so no one has to install it [1].

  I'm including a tarball of everything needed to reproduce the issue as I'm
not wise in the internals of Lua.  I checked section 8 of the Lua manual,
and I didn't see anything that applied.

  Thanks for any help.

  -spc

[1] https://www.piumarta.com/software/peg/
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 crashes where Lua 5.3.6 does not

Sean Conner
It was thus said that the Great Sean Conner once stated:
>
>   I'm including a tarball of everything needed to reproduce the issue as I'm
> not wise in the internals of Lua.  I checked section 8 of the Lua manual,
> and I didn't see anything that applied.

  The tarball seems to be a bit larger than what the list will allow by
default, So until that message is approved, I've put the tarball here:

        http://www.flummux.org/crash.tar.gz

  -spc
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 crashes where Lua 5.3.6 does not

Andrew Gierth
In reply to this post by Sean Conner
>>>>> "Sean" == Sean Conner <[hidden email]> writes:

 Sean> (gdb) where
 Sean> #0  0x0806a7bc in resizebox (L=0x8d9900c, idx=-1, newsize=2048) at lauxlib.c:477
 Sean> #1  0x0806a96e in prepbuffsize (B=0xbfa1d448, sz=1, boxidx=-1) at lauxlib.c:546
 Sean> #2  0x0806aa41 in luaL_addlstring (B=0xbfa1d448, s=0x8dbd1b0 "\n", l=1) at lauxlib.c:572
 Sean> #3  0xb7de62ca in yy_5_CHAR (yy=0xbfa1d400, yytext=0x8dbd1b0 "\n", yyleng=1) at html.i:3

The overwhelmingly most likely cause for the crash here is that you
changed the Lua stack in an unbalanced way between a previous add* call
and this one. resizebox is only hit if the buffer has already overflowed
its static allocation and been moved to a udata on the Lua stack; but
that the value of "box" is NULL implies that something other than a
udata was found at the expected stack index.

This isn't any different between 5.3 and 5.4, but one thing that _did_
change was the default size for the static buffer (in 5.4 it's much
smaller - 512/1024 bytes for 32/64 bit, vs. 4096/8192 bytes for 5.3), so
it's possible that you simply didn't reach the limit on 5.3.

Your issue didn't reproduce for me when I tried it. But I was on a
64-bit system, and the values in your stacktrace suggest a 32-bit
system?

--
Andrew.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 crashes where Lua 5.3.6 does not

Andrew Gierth
>>>>> "Andrew" == Andrew Gierth <[hidden email]> writes:

 Andrew> The overwhelmingly most likely cause for the crash here is that
 Andrew> you changed the Lua stack in an unbalanced way between a
 Andrew> previous add* call and this one.

And indeed, what you do to the stack in the deentify / deentifyn
functions is a classic example of violating the luaL_Buffer protocol -
you're assuming you can push something, do an addstring, and then pop
the thing that you pushed - but this is explicitly forbidden, since
addstring is allowed to push new entries on the stack.

You should be able to fix it fairly simply by using luaL_addvalue
instead of lua_tolstring / luaL_addlstring.

--
Andrew.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 crashes where Lua 5.3.6 does not

Viacheslav Usov
On Sat, Dec 19, 2020 at 11:25 AM Andrew Gierth
<[hidden email]> wrote:

> And indeed, what you do to the stack in the deentify / deentifyn
> functions is a classic example of violating the luaL_Buffer protocol -
> you're assuming you can push something, do an addstring, and then pop
> the thing that you pushed - but this is explicitly forbidden, since
> addstring is allowed to push new entries on the stack.

Is this the same thing as in
http://lua-users.org/lists/lua-l/2017-11/msg00004.html?

If this is, it is somewhat ironic that Sean participated in that thread.

It is a pity that three years later, this facility is neither
deprecated nor better specified as unsafe.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 crashes where Lua 5.3.6 does not

Andrew Gierth
>>>>> "Viacheslav" == Viacheslav Usov <[hidden email]> writes:

 >> And indeed, what you do to the stack in the deentify / deentifyn
 >> functions is a classic example of violating the luaL_Buffer protocol
 >> - you're assuming you can push something, do an addstring, and then
 >> pop the thing that you pushed - but this is explicitly forbidden,
 >> since addstring is allowed to push new entries on the stack.

 Viacheslav> Is this the same thing as in
 Viacheslav> http://lua-users.org/lists/lua-l/2017-11/msg00004.html?

Seems so.

--
Andrew.
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 crashes where Lua 5.3.6 does not

Sean Conner
In reply to this post by Viacheslav Usov
It was thus said that the Great Viacheslav Usov once stated:

> On Sat, Dec 19, 2020 at 11:25 AM Andrew Gierth
> <[hidden email]> wrote:
>
> > And indeed, what you do to the stack in the deentify / deentifyn
> > functions is a classic example of violating the luaL_Buffer protocol -
> > you're assuming you can push something, do an addstring, and then pop
> > the thing that you pushed - but this is explicitly forbidden, since
> > addstring is allowed to push new entries on the stack.
>
> Is this the same thing as in
> http://lua-users.org/lists/lua-l/2017-11/msg00004.html?
>
> If this is, it is somewhat ironic that Sean participated in that thread.

  Now *that's* funny.  I completely forgot about that thread when I wrote
the code that's crashing earlier this year.

> It is a pity that three years later, this facility is neither
> deprecated nor better specified as unsafe.

  Or at least a way for LUA_USE_APICHECK to detect the issue.  I did try
that when debugging to no avail.

  -spc
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.4.2 crashes where Lua 5.3.6 does not

Sean Conner
In reply to this post by Andrew Gierth
It was thus said that the Great Andrew Gierth once stated:

> >>>>> "Viacheslav" == Viacheslav Usov <[hidden email]> writes:
>
>  >> And indeed, what you do to the stack in the deentify / deentifyn
>  >> functions is a classic example of violating the luaL_Buffer protocol
>  >> - you're assuming you can push something, do an addstring, and then
>  >> pop the thing that you pushed - but this is explicitly forbidden,
>  >> since addstring is allowed to push new entries on the stack.
>
>  Viacheslav> Is this the same thing as in
>  Viacheslav> http://lua-users.org/lists/lua-l/2017-11/msg00004.html?
>
> Seems so.

  It was.  Philipp Janda replied directly to me (late last night/very early
this morning) pointing out the issue and when I made the appropriate change
(to use luaL_addvalue()) the crash went away.

  Thank you Philipp, Andrew and Usov for your help.  I was too close to the
code to see the problem (and completely forgot about the stack balance when
using a luaL_Buffer).

  -spc