Problem with closures, nCcalls on Lua 5.4.0 (alpha)

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Problem with closures, nCcalls on Lua 5.4.0 (alpha)

Tom Sutcliffe
Hi list,

I tried upgrading a 5.3 based project to 5.4.0 (alpha) today, and hit some issues. After some minimal adaptations to the native code (this is a project that embeds Lua in a Windows exe), everything looked fine but some of the project's unit tests started failing in odd and non-deterministic ways, whereas on 5.3.5 they ran fine. Tests that ran fine in isolation would fail in weird ways when run one after another. This is code that gets hammered fairly heavily (dozens of automation runs of the 5.3-based code per day), so I'm reasonably sure it's not doing anything too egregiously stupid (although it's always possible...).

I narrowed it down to something going wrong with the value of nCcalls in a coroutine's lua_State. I managed to get it to underflow and wrap around, and once it has a value like 0xFFFFFFFF all sorts of interesting things go wrong (spurious warnings about yielding across a C boundary, out of stack errors, etc).

When the issue occurs, the main lua_State is calling into a previously-yielded coroutine via a call to lua_resume(). The coroutine was previously yielded from a C function with a call to lua_yieldk() with a continuation function specified. The C callstack looks like this:

my.exe!luaE_shrinkCI(lua_State * L)
my.exe!luaD_shrinkstack(lua_State * L)
my.exe!recover(lua_State * L, int status)
my.exe!lua_resume(lua_State * L, lua_State * from, int nargs, int * nresults)
my.exe!my_call_into_coroutine(lua_State * L, int nargs, int nret)
my.exe!main(int argc, char * * argv)

An error() has just happened (inside a pcall, in the coroutine) and is being handled, by the looks of it. On entry to luaE_shrinkCI, L->nCcalls is 1 and L->nci is 14. At the end of the fn, nci is 12 and nCcalls is 0xffffffff. After that point, it appears that the damage is done and exactly what goes wrong varies. I haven't worked through just what is happening in luaE_shrinkCI() to achieve this, but it seems likely to be related to how I'm using the coroutine APIs in particular yielding and resuming into cfunctions. The Lua stack (both the current frame, and the CallInfo::previous links in L->ci) look sane, and the same code has been running fine with 5.3 for months.

I haven't yet worked out a minimal reproduction of this problem (although it reproduces reliably for me locally, I just can't share that code plus it's enormous), and I definitely haven't ruled out the possibility that it's my native code that's the problem, but I wanted to let people know now in case anyone else is seeing something similar. I'll report back if I get a minimal repro working, or indeed if I find it's a bug in my native code :-)

Cheers,

Tom
Reply | Threaded
Open this post in threaded view
|

Re: Problem with closures, nCcalls on Lua 5.4.0 (alpha)

Roberto Ierusalimschy
> I narrowed it down to something going wrong with the value of nCcalls in a coroutine's lua_State. I managed to get it to underflow and wrap around, and once it has a value like 0xFFFFFFFF all sorts of interesting things go wrong (spurious warnings about yielding across a C boundary, out of stack errors, etc).

If it is not asking to much, do you think you could repeat your tests
with the current version in github? Commit 3cd9b56ae6 is titled
"Revamp around 'L->nCcalls' count". Maybe this bug has already been
fixed there.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Problem with closures, nCcalls on Lua 5.4.0 (alpha)

Tom Sutcliffe
In reply to this post by Tom Sutcliffe
Hi Roberto,

I keep forgetting latest master is on github, I realise I should have checked there first.

I've rebuilt from https://github.com/lua/lua/commit/05ba2880491fa1ecfe78d8a3542edcd6220e4022 and the entire test suite is now passing except for one test relating to GCing of userdatas, which I'll look in to further when the weekend isn't rapidly approaching/overdue.

In short updating to latest github master has fixed the nCcalls problem I was seeing. Many thanks for the quick response!

Cheers,

Tom

On June 28, 2019 at 4:45 PM, Roberto Ierusalimschy <[hidden email]> wrote:

I narrowed it down to something going wrong with the value of nCcalls in a coroutine's lua_State. I managed to get it to underflow and wrap around, and once it has a value like 0xFFFFFFFF all sorts of interesting things go wrong (spurious warnings about yielding across a C boundary, out of stack errors, etc).

If it is not asking to much, do you think you could repeat your tests
with the current version in github? Commit 3cd9b56ae6 is titled
"Revamp around 'L->nCcalls' count". Maybe this bug has already been
fixed there.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

tostring userdata (was: Problem with closures, nCcalls on Lua 5.4.0 (alpha))

Tom Sutcliffe
On 28 Jun 2019, at 8:58 pm, Tom Sutcliffe <[hidden email]> wrote:
>
> I've rebuilt from https://github.com/lua/lua/commit/05ba2880491fa1ecfe78d8a3542edcd6220e4022 and the entire test suite is now passing except for one test relating to GCing of userdatas, which I'll look in to further when the weekend isn't rapidly approaching/overdue.

I've tracked this down to the fact that the test - not the production code :) - was relying on the tostring representation of a full userdata 0xABCD being the same as the lightuserdata representation of the same pointer - they were both "userdata: 0x1234" on 5.3 but on 5.4 the userdata is now formatted as "MyUserData: 0x1234" (where the userdata's metatable's __name is "MyUserData")

I couldn't find anywhere in the manual that this was mentioned - in particular __name is mentioned solely in the docs for luaL_newmetatable which states "The entry __name is used by some error-reporting functions" - and this change has broadened the scope of what __name is used for. I realise that the exact formatting of tostring is not defined in the API but previously changes like that of tostring(2.0) in 5.3 were at least mentioned in the "Changes in the Language" section - could we do the same here?

The good news is my unit tests are now passing 100% on 5.4 (at commit 05ba2880491fa1ecfe78d8a3542edcd6220e4022).

Cheers,

Tom
Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Andrew Gierth
>>>>> "Tom" == Tom Sutcliffe <[hidden email]> writes:

 Tom> I've tracked this down to the fact that the test - not the
 Tom> production code :) - was relying on the tostring representation of
 Tom> a full userdata 0xABCD being the same as the lightuserdata
 Tom> representation of the same pointer - they were both "userdata:
 Tom> 0x1234" on 5.3

er... no?

In 5.3, the tostring representation of something that's not a number,
string, boolean or nil and which has no __tostring metamethod is:

    lua_pushfstring(L, "%s: %p", kind, lua_topointer(L, idx));

where "kind" is the value of the __name metafield if it exists and is a
string, and is the luaL_typename of the original value otherwise.

I checked my own code and I definitely get "somename: 0xblah" when
printing full userdatas that have __name metafields.

 Tom> I couldn't find anywhere in the manual that this was mentioned -
 Tom> in particular __name is mentioned solely in the docs for
 Tom> luaL_newmetatable which states "The entry __name is used by some
 Tom> error-reporting functions" - and this change has broadened the
 Tom> scope of what __name is used for. I realise that the exact
 Tom> formatting of tostring is not defined in the API but previously
 Tom> changes like that of tostring(2.0) in 5.3 were at least mentioned
 Tom> in the "Changes in the Language" section - could we do the same
 Tom> here?

It's presumably not documented as a change because it didn't actually
change.

Most likely, what you're seeing is resulting from some other change that
is causing either the metatable to be (still?) set when it previously
wasn't, or __name to be set in the metatable when it previously wasn't.

--
Andrew.

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Daurnimator
On Mon, 1 Jul 2019 at 02:42, Andrew Gierth <[hidden email]> wrote:

>
> >>>>> "Tom" == Tom Sutcliffe <[hidden email]> writes:
>
>  Tom> I've tracked this down to the fact that the test - not the
>  Tom> production code :) - was relying on the tostring representation of
>  Tom> a full userdata 0xABCD being the same as the lightuserdata
>  Tom> representation of the same pointer - they were both "userdata:
>  Tom> 0x1234" on 5.3
>
> er... no?
>
> In 5.3, the tostring representation of something that's not a number,
> string, boolean or nil and which has no __tostring metamethod is:
>
>     lua_pushfstring(L, "%s: %p", kind, lua_topointer(L, idx));
>
> where "kind" is the value of the __name metafield if it exists and is a
> string, and is the luaL_typename of the original value otherwise.
>
> I checked my own code and I definitely get "somename: 0xblah" when
> printing full userdatas that have __name metafields.
>
>  Tom> I couldn't find anywhere in the manual that this was mentioned -
>  Tom> in particular __name is mentioned solely in the docs for
>  Tom> luaL_newmetatable which states "The entry __name is used by some
>  Tom> error-reporting functions" - and this change has broadened the
>  Tom> scope of what __name is used for. I realise that the exact
>  Tom> formatting of tostring is not defined in the API but previously
>  Tom> changes like that of tostring(2.0) in 5.3 were at least mentioned
>  Tom> in the "Changes in the Language" section - could we do the same
>  Tom> here?
>
> It's presumably not documented as a change because it didn't actually
> change.
>
> Most likely, what you're seeing is resulting from some other change that
> is causing either the metatable to be (still?) set when it previously
> wasn't, or __name to be set in the metatable when it previously wasn't.
>
> --
> Andrew.
>

IIRC __name in tostring output was introduced between 5.3.3 and 5.3.4.
It was allowed as the output of tostring on objects without a
__tostring has always been undefined.

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Tom Sutcliffe
In reply to this post by Andrew Gierth

On June 30, 2019 at 5:42 PM, Andrew Gierth <[hidden email]> wrote:

>>>>>> "Tom" == Tom Sutcliffe <[hidden email]> writes:
>
> Tom> I've tracked this down to the fact that the test - not the
> Tom> production code :) - was relying on the tostring representation of
> Tom> a full userdata 0xABCD being the same as the lightuserdata
> Tom> representation of the same pointer - they were both "userdata:
> Tom> 0x1234" on 5.3
>
> er... no?

>
> In 5.3, the tostring representation of something that's not a number,
> string, boolean or nil and which has no __tostring metamethod is:


Well, this is interesting. The project I was porting originally used 5.3.3 (plus keeping an eye on subsequent security bugs). In 5.3.3 the behaviour very definitely was as I described. Looking through the commit history, it appears this changed in 5.3.4.
 

>
> Tom> I couldn't find anywhere in the manual that this was mentioned -
> Tom> in particular __name is mentioned solely in the docs for
> Tom> luaL_newmetatable which states "The entry __name is used by some
> Tom> error-reporting functions" - and this change has broadened the
> Tom> scope of what __name is used for. I realise that the exact
> Tom> formatting of tostring is not defined in the API but previously
> Tom> changes like that of tostring(2.0) in 5.3 were at least mentioned
> Tom> in the "Changes in the Language" section - could we do the same
> Tom> here?
>
> It's presumably not documented as a change because it didn't actually
> change.


Not when I originally thought, but it did change :) It's not a huge issue, but the precedent of mentioning things which aren't technically breaks but nevertheless could trip people up, was established with the float string formatting notes in the 5.3 docs, and it would've been nice to have had a similar note here. Of course, since it changed during the 5.3 era that ship has now sailed.

To put it another way, everything else I had to change in this project to go from 5.3.x to current master branch was something mentioned in section 8 of the work manual - so it seemed worthwhile to raise it on the list.

Besides, how/when __name is used by the Lua core is somewhat inadequately documented, whenever it’s current behaviour was introduced.

Edit: ninja’d by Daurnimator

Regards,

Tom



Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Patrick Donnelly
In reply to this post by Andrew Gierth


On Sun, Jun 30, 2019, 9:42 AM Andrew Gierth <[hidden email]> wrote:
>>>>> "Tom" == Tom Sutcliffe <[hidden email]> writes:

 Tom> I've tracked this down to the fact that the test - not the
 Tom> production code :) - was relying on the tostring representation of
 Tom> a full userdata 0xABCD being the same as the lightuserdata
 Tom> representation of the same pointer - they were both "userdata:
 Tom> 0x1234" on 5.3

er... no?

In 5.3, the tostring representation of something that's not a number,
string, boolean or nil and which has no __tostring metamethod is:

    lua_pushfstring(L, "%s: %p", kind, lua_topointer(L, idx));

where "kind" is the value of the __name metafield if it exists and is a
string, and is the luaL_typename of the original value otherwise.

While we are on this subject, giving Lua scripts access to the pointer address to something in memory is a needless footstool to breaking out of a sandbox and potentially taking over an application. It'd be more secure if Lua assigned an instance counter to userdata and printed that. A new __instance metamethod could complement the __name one to allow customizing.

[One challenge is a light userdata has no instance other than the pointer. Maybe just don't print an instance string or hash the pointer with a random number.]

- Patrick
Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Sean Conner
It was thus said that the Great Patrick Donnelly once stated:

> On Sun, Jun 30, 2019, 9:42 AM Andrew Gierth <[hidden email]>
> wrote:
>
> > >>>>> "Tom" == Tom Sutcliffe <[hidden email]> writes:
> >
> >  Tom> I've tracked this down to the fact that the test - not the
> >  Tom> production code :) - was relying on the tostring representation of
> >  Tom> a full userdata 0xABCD being the same as the lightuserdata
> >  Tom> representation of the same pointer - they were both "userdata:
> >  Tom> 0x1234" on 5.3
> >
> > er... no?
> >
> > In 5.3, the tostring representation of something that's not a number,
> > string, boolean or nil and which has no __tostring metamethod is:
> >
> >     lua_pushfstring(L, "%s: %p", kind, lua_topointer(L, idx));
> >
> > where "kind" is the value of the __name metafield if it exists and is a
> > string, and is the luaL_typename of the original value otherwise.
>
>
> While we are on this subject, giving Lua scripts access to the pointer
> address to something in memory is a needless footstool to breaking out of a
> sandbox and potentially taking over an application.

  I'd like to see a proof-of-concept before I worry about that.  I mean, I
can always do

        x = 0xcbc5c0

which *is* a valid address on a running instance of Lua on my system.  Or
0xb7d7f000 or 0x00cbe040 or any number of other values.

  -spc (And no, loading a special C module to exploit this won't cut it)

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Patrick Donnelly
On Tue, Jul 2, 2019, 6:03 PM Sean Conner <[hidden email]> wrote:
It was thus said that the Great Patrick Donnelly once stated:
> On Sun, Jun 30, 2019, 9:42 AM Andrew Gierth <[hidden email]>
> wrote:
>
> > >>>>> "Tom" == Tom Sutcliffe <[hidden email]> writes:
> >
> >  Tom> I've tracked this down to the fact that the test - not the
> >  Tom> production code :) - was relying on the tostring representation of
> >  Tom> a full userdata 0xABCD being the same as the lightuserdata
> >  Tom> representation of the same pointer - they were both "userdata:
> >  Tom> 0x1234" on 5.3
> >
> > er... no?
> >
> > In 5.3, the tostring representation of something that's not a number,
> > string, boolean or nil and which has no __tostring metamethod is:
> >
> >     lua_pushfstring(L, "%s: %p", kind, lua_topointer(L, idx));
> >
> > where "kind" is the value of the __name metafield if it exists and is a
> > string, and is the luaL_typename of the original value otherwise.
>
>
> While we are on this subject, giving Lua scripts access to the pointer
> address to something in memory is a needless footstool to breaking out of a
> sandbox and potentially taking over an application.

  I'd like to see a proof-of-concept before I worry about that.  I mean, I
can always do

        x = 0xcbc5c0

which *is* a valid address on a running instance of Lua on my system.  Or
0xb7d7f000 or 0x00cbe040 or any number of other values.

I'm not talking about numbers of course. If you have knowledge of the addresses to userdata then you can use that to write assembly code referencing that address to execute code. That's assuming you can write arbitrary data in the userdata (not at all uncommon) and that you have an attack vector to cause that code to be executed (maybe possible with poorly written libraries). I nearly got far enough to do this in WoW back in the day when I was breaking any sandbox I could find. At the time, I was trying to exploit getting access to the Lua registry [1] which gave access to some interesting WoW internals. I don't recall exact details.


  -spc (And no, loading a special C module to exploit this won't cut it)

Why not? A Lua sandbox in some application presumably has some C modules which may be quite... special. :)
Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Sean Conner
It was thus said that the Great Patrick Donnelly once stated:

> On Tue, Jul 2, 2019, 6:03 PM Sean Conner <[hidden email]> wrote:
>
> >   I'd like to see a proof-of-concept before I worry about that.  I mean, I
> > can always do
> >
> >         x = 0xcbc5c0
> >
> > which *is* a valid address on a running instance of Lua on my system.  Or
> > 0xb7d7f000 or 0x00cbe040 or any number of other values.
>
> I'm not talking about numbers of course. If you have knowledge ... then
> you can use that to write assembly code ... to execute code. That's
> assuming you can write arbitrary data ... and that you have an attack
> vector to cause that code to be executed (maybe possible with poorly
> written libraries).

  I personally don't believe that just knowing an address is dangerous in
and of itself.  Just like a virus can't spread via images [2][3].

> I nearly got far enough to do this in WoW back in the
> day when I was breaking any sandbox I could find. At the time, I was trying
> to exploit getting access to the Lua registry [1] which gave access to some
> interesting WoW internals. I don't recall exact details.
>
> [1] https://www.lua.org/bugs.html#5.1.3-1

  Wow!  You were busy.

>   -spc (And no, loading a special C module to exploit this won't cut it)
>
> Why not? A Lua sandbox in some application presumably has some C modules
> which may be quite... special. :)

  Yeah, but a module specifically written to be exploited is not the same  
thing as exploiting a module *not* written to be exploited (or expected to  
be exploited).  It's like shooting fish in a barrel---not much sport in it,

  -spc (Now, the commonly used method of mixing parameters with return
        addresses on the same stack is a dumb idea, but I can see why
        it was done ... )

[2] Oh wait ... you can on Windows, becaue MICROSOFT EXPLICITELY CHECKED
        FOR CODE IN IMAGES TO EXECUTE!  There's little hope when MBAs
        override engineers.

[3] Did I just counter my own argument?  I don't know.

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Roberto Ierusalimschy
In reply to this post by Patrick Donnelly
> While we are on this subject, giving Lua scripts access to the pointer
> address to something in memory is a needless footstool to breaking out of a
> sandbox and potentially taking over an application.

1) I disagree this is "needless". I find it useful.

2) After your explanations, I still fail to see how this is a
"footstool". If we assume poorly written libraries, anything
can be dangerous.

3) If you really think this is dangerous, it just got worse :-)

  $ lua
  Lua 5.4.0  Copyright (C) 1994-2019 Lua.org, PUC-Rio
  > string.format("%p", {})
  0x2108f70

(After several requests for a "rawtostring".)


> It'd be more secure if
> Lua assigned an instance counter to userdata and printed that. A new
> __instance metamethod could complement the __name one to allow customizing.

It'd would also be slower. It'd would also increase the memory overhead
for userdata. And why only userdata? Any memory address might be
in danger (strings spring to mind), once we assume poorly written
libraries.

Moreover, there are some problems with such global counter. (We just
had this discussion on this list, about numbering string instances.)
Not to mention your own argument regarding light userdata.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Patrick Donnelly
On Wed, Jul 3, 2019 at 6:51 AM Roberto Ierusalimschy
<[hidden email]> wrote:
>
> > While we are on this subject, giving Lua scripts access to the pointer
> > address to something in memory is a needless footstool to breaking out of a
> > sandbox and potentially taking over an application.
>
> 1) I disagree this is "needless". I find it useful.

Right, wrong adjective. It is a handy tool. :)

> 2) After your explanations, I still fail to see how this is a
> "footstool". If we assume poorly written libraries, anything
> can be dangerous.

It's dangerous for the same reasons that a static starting address for
the stack is. There are classic hacks that utilize the known address
location of an environment variable at the beginning of a stack to
implement return-to-libc attacks with shellcode in the environment
variable. Since then, mitigations like ASLR have made this more
difficult. But, ASLR can't help if the scripting language voluntarily
gives away this information (the address location of the heap or
strings) to scripts.

I'm not a security researcher so I recommend talking to others more
familiar with these exploits. Maybe it's no longer considered useful
information in today's world where applications are frequently built
with various buffer-overflow protections (I doubt it though!).

> 3) If you really think this is dangerous, it just got worse :-)
>
>   $ lua
>   Lua 5.4.0  Copyright (C) 1994-2019 Lua.org, PUC-Rio
>   > string.format("%p", {})
>   0x2108f70

Or more pointedly:

lua -e 'function f(s) print(string.format("%p", s)) end f"" f{}
f(("1"):rep(1e3))'
0x5606b34f1fb0
0x5606b34f4190
0x5606b34f47b0

As I said, I'm not an expert in this area so maybe I'm just raising
alarm about something not considered a serious problem.

With that said, consider that print is frequently removed from
sandboxes but string.format almost never is. Is it really desirable to
build that into a function that is frequently left in sandboxes
without any thoughtful consideration?

--
Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Sean Conner
It was thus said that the Great Patrick Donnelly once stated:
> On Wed, Jul 3, 2019 at 6:51 AM Roberto Ierusalimschy
> <[hidden email]> wrote:
> >
> > 2) After your explanations, I still fail to see how this is a
> > "footstool". If we assume poorly written libraries, anything
> > can be dangerous.
>
> It's dangerous for the same reasons that a static starting address for
> the stack is.

  Most C compilers will use the system stack for both return addresses
(because that's inherent to a CALL instruction) and parameters to a
function.  Separate the two---have a separate "return stack" and "data
stack" and this becomes harder to exploit (Forth does this).  There's
nothing in the C standard that mandates only one stack.

> There are classic hacks that utilize the known address
> location of an environment variable

  If an attacker can modify the environment variables (like $HOME) then you
have more problems than just the attack itself.

> at the beginning of a stack

  An implementation detail.  There are operating systems out there that have
environment variables elsewhere.

> to
> implement return-to-libc attacks with shellcode in the environment
> variable. Since then, mitigations like ASLR have made this more
> difficult.

  And it has made debugging production issues harder as well, because stack
dumps are now worthless, and of *course* you are running stripped binaries
because *not* doing so leaks information and ...

> But, ASLR can't help if the scripting language voluntarily
> gives away this information (the address location of the heap or
> strings) to scripts.

  Please, I implore you, *give a proof-of-concept* here, because otherwise
it's a pointless change because of cargo cult understanding of attacks.

> > 3) If you really think this is dangerous, it just got worse :-)
> >
> >   $ lua
> >   Lua 5.4.0  Copyright (C) 1994-2019 Lua.org, PUC-Rio
> >   > string.format("%p", {})
> >   0x2108f70
>
> Or more pointedly:
>
> lua -e 'function f(s) print(string.format("%p", s)) end f"" f{}
> f(("1"):rep(1e3))'
> 0x5606b34f1fb0
> 0x5606b34f4190
> 0x5606b34f47b0
>
> As I said, I'm not an expert in this area so maybe I'm just raising
> alarm about something not considered a serious problem.

  I think you are.

  -spc

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Coda Highland
On Fri, Jul 5, 2019 at 4:56 PM Sean Conner <[hidden email]> wrote:
It was thus said that the Great Patrick Donnelly once stated:

> But, ASLR can't help if the scripting language voluntarily
> gives away this information (the address location of the heap or
> strings) to scripts.

  Please, I implore you, *give a proof-of-concept* here, because otherwise
it's a pointless change because of cargo cult understanding of attacks.

You need look no further than any of the various Webkit exploits that have been used for breaking game console security. The hacking community sometimes does interesting write-ups describing the steps necessary to get code running on a given platform.
 
If you want to run native code without permission, you have to trick the CPU into jumping to memory you control. The typical procedure these days is to use what's called "return-oriented programming" because the return stack is the only thing that can arbitrarily impact program flow that's also writable memory. It's easy enough to trigger a crash this way, but if you want a ROP to do anything useful, you need to know the address of useful code, and ASLR thwarts the obvious ways of doing this.

This means that one of the first things that an exploit is going to need to do is discover an address to a known data structure, and then it can start computing offsets from there. In Lua, one of the best options available would be the string metatable, because it contains pointers to native-code functions baked into the binary.

On any reasonably secure platform, this is difficult. Usually you have to exploit a buffer overflow or a use-after-free or some similar technique in order to trick the program into writing the address somewhere you can access. But if you can just say "hey, interpreter, could you please tell me the address of string.concat?" and it comes back with "you seem like a fine gentleprogram, here you go!" then you've just skipped the hardest part of kicking off an exploit chain.

/s/ Adam
v
Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

v
On Fri, 2019-07-05 at 17:28 -0500, Coda Highland wrote:

> If you want to run native code without permission, you have to trick
> the CPU into jumping to memory you control. The typical procedure
> these days is to use what's called "return-oriented programming"
> because the return stack is the only thing that can arbitrarily
> impact program flow that's also writable memory. It's easy enough to
> trigger a crash this way, but if you want a ROP to do anything
> useful, you need to know the address of useful code, and ASLR thwarts
> the obvious ways of doing this.
>
> /s/ Adam

Crashing is also major problem and according to your idea is possible
even without known address (thing you're complaining about). Sounds
like part which allows to either trigger crash or exploit vulnerability
is a real problem here.

So go on and show POC exploit which can crash Lua this way (we'll thank
you). Only plaintext does count as bytecode is unprotected by design
and already have been proven to be able to be able to do arbitrary
jumps: https://saelo.github.io/posts/pwning-lua-through-load.html.

If I'm not mistaken, that exploit is based on what you're talking
about. But good luck doing so in non-precompiled Lua.

--
v <[hidden email]>


Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Sean Conner
In reply to this post by Coda Highland
It was thus said that the Great Coda Highland once stated:

> On Fri, Jul 5, 2019 at 4:56 PM Sean Conner <[hidden email]> wrote:
>
> > It was thus said that the Great Patrick Donnelly once stated:
> >
> > > But, ASLR can't help if the scripting language voluntarily
> > > gives away this information (the address location of the heap or
> > > strings) to scripts.
> >
> >   Please, I implore you, *give a proof-of-concept* here, because otherwise
> > it's a pointless change because of cargo cult understanding of attacks.

  [ Mention of Webkit exploits deleted because it's Webkit, NOT Lua]

  [ Mention of ROP and ASLR deleted because it's been mentioned before]
 

> This means that one of the first things that an exploit is going to need to
> do is discover an address to a known data structure, and then it can start
> computing offsets from there. In Lua, one of the best options available
> would be the string metatable, because it contains pointers to native-code
> functions baked into the binary.
>
> On any reasonably secure platform, this is difficult. Usually you have to
> exploit a buffer overflow or a use-after-free or some similar technique in
> order to trick the program into writing the address somewhere you can
> access. But if you can just say "hey, interpreter, could you please tell me
> the address of string.concat?" and it comes back with "you seem like a fine
> gentleprogram, here you go!" then you've just skipped the hardest part of
> kicking off an exploit chain.

  Sigh.

  Okay, how about a proof-of-concept *written in Lua*, text only (because
precompiled Lua code is dangerous, there are proof-of-concepts of that) that
will execute arbitrary code, just from an address one can get from
tostring() or string.format("%p") (both will return an address, so obviously
tostring() must go, right?).

  -spc (I keep asking for a "proof-of-concept" but I keep getting "here's
        how it *could* be done".  I'm sorry, I want to "see it" being done.
        It should be trivial in Lua given an actual address, right?)

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Coda Highland


On Fri, Jul 5, 2019 at 6:51 PM Sean Conner <[hidden email]> wrote:
It was thus said that the Great Coda Highland once stated:
> On Fri, Jul 5, 2019 at 4:56 PM Sean Conner <[hidden email]> wrote:
>
> > It was thus said that the Great Patrick Donnelly once stated:
> >
> > > But, ASLR can't help if the scripting language voluntarily
> > > gives away this information (the address location of the heap or
> > > strings) to scripts.
> >
> >   Please, I implore you, *give a proof-of-concept* here, because otherwise
> > it's a pointless change because of cargo cult understanding of attacks.

  [ Mention of Webkit exploits deleted because it's Webkit, NOT Lua]

  [ Mention of ROP and ASLR deleted because it's been mentioned before]

> This means that one of the first things that an exploit is going to need to
> do is discover an address to a known data structure, and then it can start
> computing offsets from there. In Lua, one of the best options available
> would be the string metatable, because it contains pointers to native-code
> functions baked into the binary.
>
> On any reasonably secure platform, this is difficult. Usually you have to
> exploit a buffer overflow or a use-after-free or some similar technique in
> order to trick the program into writing the address somewhere you can
> access. But if you can just say "hey, interpreter, could you please tell me
> the address of string.concat?" and it comes back with "you seem like a fine
> gentleprogram, here you go!" then you've just skipped the hardest part of
> kicking off an exploit chain.

  Sigh.

  Okay, how about a proof-of-concept *written in Lua*, text only (because
precompiled Lua code is dangerous, there are proof-of-concepts of that) that
will execute arbitrary code, just from an address one can get from
tostring() or string.format("%p") (both will return an address, so obviously
tostring() must go, right?).

  -spc (I keep asking for a "proof-of-concept" but I keep getting "here's
        how it *could* be done".  I'm sorry, I want to "see it" being done.
        It should be trivial in Lua given an actual address, right?)


You asked for a proof of concept so I referenced one illustrating the general technique that's beyond just proof-of-concept level to full productionization. You didn't explicitly specify "in Lua" before now.

If I had a proof of concept of a critical security flaw in Lua, I would be submitting a patch for it. The concern isn't about the bugs that the good guys can find. It's a practice of defense-in-depth, so that if a heretofore unknown bug is discovered then the damage it can deal is limited. There is of course the cost-benefit analysis to consider, so it wouldn't be worth hobbling the entire Lua VM for an obscure hypothetical vulnerability, but disabling a trivial way to crack ASLR has a pretty tiny cost.

And as an aside to the previous poster: To be honest, while crashing is a nasty opportunity for denial of service, it's the bugs that DON'T crash the program that are far more damaging.

/s/ Adam
Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Sean Conner
It was thus said that the Great Coda Highland once stated:

> On Fri, Jul 5, 2019 at 6:51 PM Sean Conner <[hidden email]> wrote:
>
> >   Okay, how about a proof-of-concept *written in Lua*, text only (because
> > precompiled Lua code is dangerous, there are proof-of-concepts of that)
> > that
> > will execute arbitrary code, just from an address one can get from
> > tostring() or string.format("%p") (both will return an address, so
> > obviously
> > tostring() must go, right?).
> >
> >   -spc (I keep asking for a "proof-of-concept" but I keep getting "here's
> >         how it *could* be done".  I'm sorry, I want to "see it" being done.
> >         It should be trivial in Lua given an actual address, right?)
> >
> You asked for a proof of concept so I referenced one illustrating the
> general technique that's beyond just proof-of-concept level to full
> productionization. You didn't explicitly specify "in Lua" before now.

  I thought that was clear from the context (this mailing list being about
Lua, and the topic of getting address information from Lua leading to OMG
Armageddon! but I probably should have been more explicit about it).

> If I had a proof of concept of a critical security flaw in Lua, I would be
> submitting a patch for it. The concern isn't about the bugs that the good
> guys can find. It's a practice of defense-in-depth, so that if a heretofore
> unknown bug is discovered then the damage it can deal is limited.

  Okay, so was this *ever* brought up about tostring()?  Because:

[spc]lucy:/tmp>lua-51
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> print(print)
function: 0x8ff9020
>
[spc]lucy:/tmp>lua-52
Lua 5.2.4  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> print(print)
function: 0x805c804
>
[spc]lucy:/tmp>lua-53
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> print(print)
function: 0x805f3f8
>
[spc]lucy:/tmp>lua-54
Lua 5.4.0  Copyright (C) 1994-2019 Lua.org, PUC-Rio
> print(print)
function: 0x8063954
>
[spc]lucy:/tmp>

  It goes back a @#%@!#$@#$ long way, and only NOW are people freaking out
about it because string.format() got a "%p" format specifier?  I would
expect that if there was an issue about obtaining an address, it would have
shown itself in the past, oh, thirteen years or so.  It's not like ASLR
wasn't known back then (hint---it was).

  But so far, all I've seen is "OMG! ASLR is VIOLATED!  Burn the feature!"
which to me comes across as cargo cult security, of which I seem to be in
the minority.  So let's neuter both tostring() and string.format() to save
ASLR!

  -spc (And request, nay!  Demand!  That all modules in C avoid printing an
        address as part of the __tostring() metamethod ...)

Reply | Threaded
Open this post in threaded view
|

Re: tostring userdata

Coda Highland


On Fri, Jul 5, 2019 at 10:56 PM Sean Conner <[hidden email]> wrote:
It was thus said that the Great Coda Highland once stated:
> On Fri, Jul 5, 2019 at 6:51 PM Sean Conner <[hidden email]> wrote:
> If I had a proof of concept of a critical security flaw in Lua, I would be
> submitting a patch for it. The concern isn't about the bugs that the good
> guys can find. It's a practice of defense-in-depth, so that if a heretofore
> unknown bug is discovered then the damage it can deal is limited.

  Okay, so was this *ever* brought up about tostring()?  Because:

[spc]lucy:/tmp>lua-51
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> print(print)
function: 0x8ff9020
>
[spc]lucy:/tmp>lua-52
Lua 5.2.4  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> print(print)
function: 0x805c804
>
[spc]lucy:/tmp>lua-53
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
> print(print)
function: 0x805f3f8
>
[spc]lucy:/tmp>lua-54
Lua 5.4.0  Copyright (C) 1994-2019 Lua.org, PUC-Rio
> print(print)
function: 0x8063954
>
[spc]lucy:/tmp>

  It goes back a @#%@!#$@#$ long way, and only NOW are people freaking out
about it because string.format() got a "%p" format specifier?  I would
expect that if there was an issue about obtaining an address, it would have
shown itself in the past, oh, thirteen years or so.  It's not like ASLR
wasn't known back then (hint---it was).

  But so far, all I've seen is "OMG! ASLR is VIOLATED!  Burn the feature!"
which to me comes across as cargo cult security, of which I seem to be in
the minority.  So let's neuter both tostring() and string.format() to save
ASLR!

  -spc (And request, nay!  Demand!  That all modules in C avoid printing an
        address as part of the __tostring() metamethod ...)


It had been mentioned before, in passing, but it had never really become a focus of conversation. Then again, security in Lua has rarely been a focus of conversation except for a few discussions about sandboxing.

And the difference has already been pointed out in this thread: most sandboxes exclude print(), and tostring() can be overridden easily, but dealing with string.format isn't quite as simple.

Also, I'm not making any demands. My own use case really doesn't care; I don't have need to run untrusted scripts. I'm making an academic discussion: trying to provide information about the subject at hand so that informed decisions can be made instead of jumping straight on either the "kill it!" or "keep it!" bandwagons. In my personal opinion, the most straightforward solution to this problem would be an option to disable pointer rendering for the cases where it's a problem but leave it enabled by default because it's usually more helpful than hurtful. I wouldn't want to completely remove it because its usefulness in debugging C extensions is far too valuable.

(Also, where do you draw the line between "cargo cult" and "best practices"? Because I've certainly seen you describing the "right" way to do things before.)

/s/ Adam


12