Can lua_next() error?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

Can lua_next() error?

Tom Sutcliffe
Hi list,

I've been reviewing some of my native code for safety and noticed that I've been assuming lua_next() will not raise arbitrary errors. In the 5.3 documentation however, lua_next is listed as 'e' meaning it can.

I couldn't think of any obvious reason how it could error, since there is no metamethod to override a next iteration, and I couldn't see anything in the implementation that would trigger an __index. Am I missing something or is the documentation incorrect?

Thanks,

Tom
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

云风 Cloud Wu
Tom Sutcliffe <[hidden email]>于2018年4月16日周一 下午8:22写道:
Hi list,

I've been reviewing some of my native code for safety and noticed that I've been assuming lua_next() will not raise arbitrary errors. In the 5.3 documentation however, lua_next is listed as 'e' meaning it can.

I couldn't think of any obvious reason how it could error, since there is no metamethod to override a next iteration, and I couldn't see anything in the implementation that would trigger an __index. Am I missing something or is the documentation incorrect?


`lua_next`  may raise error when you pass a key that is not exist in the table.
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Tom Sutcliffe
In reply to this post by Tom Sutcliffe

On 16 Apr, 2018, at 02:08 PM, 云风 Cloud Wu <[hidden email]> wrote:

`lua_next`  may raise error when you pass a key that is not exist in the table.

Good point, but that would qualify it for a "v" ("the function may raise an error on purpose") rather than an "e" ("the function may raise any errors (it can run arbitrary Lua code, either directly or through metamethods)" wouldn't it?

Is there any way in which a call to lua_next() (with a valid table index and a valid key on the stack) could result in arbitrary code being run?

Thanks,

Tom
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Viacheslav Usov
In reply to this post by Tom Sutcliffe
On Mon, Apr 16, 2018 at 2:22 PM, Tom Sutcliffe <[hidden email]> wrote:

> I've been reviewing some of my native code for safety and noticed that I've been assuming lua_next() will not raise arbitrary errors.

Not only can it raise an error, it can also crash.


I am not sure whether the thread above should mean it is OK for it to crash in the given circumstances, but since you are checking for safety, this is definitely something to beware.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Roberto Ierusalimschy
> On Mon, Apr 16, 2018 at 2:22 PM, Tom Sutcliffe <[hidden email]> wrote:
>
> > I've been reviewing some of my native code for safety and noticed that
> I've been assuming lua_next() will not raise arbitrary errors.
>
> Not only can it raise an error, it can also crash.
>
> http://lua-users.org/lists/lua-l/2017-12/msg00057.html

Practically all functions in the Lua-C API can crash, when called with
invalid arguments. It would be expensive, and often impossible, to check
for that kind of errors. (I believe this happens with most C libraries.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Roberto Ierusalimschy
In reply to this post by Tom Sutcliffe
> `lua_next`  may raise error when you pass a key that is not exist in the table.
>
> Good point, but that would qualify it for a "v" ("the function may raise an error on purpose") rather than an "e" ("the function may raise any errors (it can run arbitrary Lua code, either directly or through metamethods)" wouldn't it?

I have to recognize that the distinction between "e" and "v" here is not
very clear. By that comment, that "it can run arbitrary Lua code, either
directly or through metamethods", you are right that this one should
have a "v".


> Is there any way in which a call to lua_next() (with a valid table index and a valid key on the stack) could result in arbitrary code being run?

No.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Viacheslav Usov
In reply to this post by Roberto Ierusalimschy
On Mon, Apr 16, 2018 at 6:04 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> Practically all functions in the Lua-C API can crash, when called with invalid arguments.

In the referenced thread, the C-language arguments given to the function were not invalid. It was the content of the Lua stack that was problematic for that function.

In the "checking for safety" context, that means the stack's content should be validated, if it is accepted from another party.

> I believe this happens with most C libraries.

No, not really. It is one thing if a parameter in a C function is a pointer, then the argument simply has to point to a valid memory location. It is quite another when it is a kind of a "handle", then it is rather typical for a function to return an error result if the argument has an invalid or unsuitable value. See POSIX/read(), for example. Lua's stack indices look more like handles than pointers to me, but that, of course, can be just me.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Tom Sutcliffe
In reply to this post by Roberto Ierusalimschy
On 16 Apr 2018, at 17:12, Roberto Ierusalimschy <[hidden email]> wrote:

>
> I have to recognize that the distinction between "e" and "v" here is not
> very clear. By that comment, that "it can run arbitrary Lua code, either
> directly or through metamethods", you are right that this one should
> have a "v".
>
>
>> Is there any way in which a call to lua_next() (with a valid table index and a valid key on the stack) could result in arbitrary code being run?
>
> No.
>

Many thanks Roberto for clarifying! Panic averted :-)

Cheers,

Tom
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Dirk Laurie-2
In reply to this post by Viacheslav Usov
2018-04-16 18:49 GMT+02:00 Viacheslav Usov <[hidden email]>:

> No, not really. It is one thing if a parameter in a C function is a pointer,
> then the argument simply has to point to a valid memory location. It is
> quite another when it is a kind of a "handle", then it is rather typical for
> a function to return an error result if the argument has an invalid or
> unsuitable value. See POSIX/read(), for example. Lua's stack indices look
> more like handles than pointers to me, but that, of course, can be just me.

The C API functions themselves do no type checking. The standard
library, wriiten using only the API, does it all the time. That should
serve as a model of what user functions based on the API should do. If
lua_next(), or any other API function, causes a crash, it is a bug in
the user program.

Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Soni "They/Them" L.
In reply to this post by Viacheslav Usov


On 2018-04-16 01:49 PM, Viacheslav Usov wrote:
> On Mon, Apr 16, 2018 at 6:04 PM, Roberto Ierusalimschy
> <[hidden email] <mailto:[hidden email]>> wrote:
>
> > Practically all functions in the Lua-C API can crash, when called
> with invalid arguments.
>
> In the referenced thread, the C-language arguments given to the
> function were not invalid. It was the content of the Lua stack that
> was problematic for that function.

As far as the Lua C API is concerned, stack values are function
arguments. How else would you pass tables to functions? They don't exist
outside the Lua state.

>
> In the "checking for safety" context, that means the stack's content
> should be validated, if it is accepted from another party.
>
> > I believe this happens with most C libraries.
>
> No, not really. It is one thing if a parameter in a C function is a
> pointer, then the argument simply has to point to a valid memory
> location. It is quite another when it is a kind of a "handle", then it
> is rather typical for a function to return an error result if the
> argument has an invalid or unsuitable value. See POSIX/read(), for
> example. Lua's stack indices look more like handles than pointers to
> me, but that, of course, can be just me.
>
> Cheers,
> V.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Viacheslav Usov
On Tue, Apr 17, 2018 at 1:51 AM, Soni "They/Them" L. <[hidden email]> wrote:

> As far as the Lua C API is concerned, stack values are function arguments.

In the C standard, the definition of 'argument' is certainly not dependent on Lua's stack values.

In the standard C library API, an effort is made to warn the user of undefined behavior when an argument is valid at the basic C level, but semantically it is not. For example, fopen(): "The argument mode points to a string. If the string is one of the following, the file is open in the indicated mode. Otherwise, the behavior is undefined."

Compare that with the current note for lua_tointegerx(): "The Lua value must be an integer, or a number or string convertible to an integer (see §3.4.3); otherwise, lua_tointegerx returns 0", and then compare with an imaginary note for lua_next(): "The value at the given index must be a table; otherwise, the behavior is undefined".

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Roberto Ierusalimschy
> Compare that with the current note for lua_tointegerx(): "The Lua value
> must be an integer, or a number or string convertible to an integer (see
> §3.4.3); otherwise, lua_tointegerx returns 0", and then compare with an
> *imaginary* note for lua_next(): "The value at the given index must be a
> table; otherwise, the behavior is undefined".

As far as my English goes, "undefined" means "not defined". So, if it
is not defined, it is undefined. We some times do it, but to say that
the behavior is undefined reminds me of "This page intentionally left
blank".

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Viacheslav Usov
On Tue, Apr 17, 2018 at 5:34 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> As far as my English goes, "undefined" means "not defined". So, if it not defined, it is undefined. We some times do it, but to say that behavior is undefined reminds me of "This page intentionally left blank".

In C and C++ circles, "undefined behavior", frequently shortened to just UB, is a term defined in the language standard. In C:

1 undefined behavior
behavior, upon use of a nonportable or erroneous program construct or of erroneous data,
for which this International Standard imposes no requirements
2 NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable
results, to behaving during translation or program execution in a documented manner characteristic of the
environment (with or without the issuance of a diagnostic message), to terminating a translation or
execution (with the issuance of a diagnostic message).
3 EXAMPLE An example of undefined behavior is the behavior on integer overflow

When we read an API spec that says "this is undefined behavior", we mentally substitute that with "it can crash or format your hard drive; do not do this". When we read a spec that does not say that, we do not always understand that some combinations of input will result in THAT kind of UB; for example, I expected a Lua error to be raised by lua_next() when given a non-table input (why? perhaps because it had the 'e" error category?). So even though it feels like stating the obvious, it helps.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

云风 Cloud Wu


> 在 2018年4月18日,上午2:04,Viacheslav Usov <[hidden email]> 写道:
>
> input will result in THAT kind of UB; for example, I expected a Lua error to be raised by lua_next() when given a non-table input (why? perhaps because it had the 'e" error category?). So even though it feels like stating the obvious, it helps.

You can turn on LUA_USE_APICHECK to do more checks on the C API .
Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Vaughan McAlley
In reply to this post by Roberto Ierusalimschy
On Wed, 18 Apr 2018, 01:34 Roberto Ierusalimschy, <[hidden email]> wrote:
> Compare that with the current note for lua_tointegerx(): "The Lua value
> must be an integer, or a number or string convertible to an integer (see
> §3.4.3); otherwise, lua_tointegerx returns 0", and then compare with an
> *imaginary* note for lua_next(): "The value at the given index must be a
> table; otherwise, the behavior is undefined".

As far as my English goes, "undefined" means "not defined". So, if it
is not defined, it is undefined. We some times do it, but to say that
the behavior is undefined reminds me of "This page intentionally left
blank".

-- Roberto

I need to put undefined behaviour into my sequences.

;-)

Vaughan

Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Dirk Laurie-2
In reply to this post by Viacheslav Usov
2018-04-17 20:04 GMT+02:00 Viacheslav Usov <[hidden email]>:

> When we read a spec that does not say that, we do not always
> understand that some combinations of input will result in THAT kind of UB;
> for example, I expected a Lua error to be raised by lua_next() when given a
> non-table input (why? perhaps because it had the 'e" error category?). So
> even though it feels like stating the obvious, it helps.

How about this: put the third paragraph of "4 – The Application Program Interface" in technicolor?

As in most C libraries, the Lua API functions do not check their arguments for validity or consistency.
However, you can change this behavior by compiling Lua with the macro LUA_USE_APICHECK defined.


Reply | Threaded
Open this post in threaded view
|

Re: Can lua_next() error?

Viacheslav Usov
On Wed, Apr 18, 2018 at 6:51 AM, Dirk Laurie <[hidden email]> wrote:

> As in most C libraries, the Lua API functions do not check their arguments for validity or consistency.

That makes perfect sense, once "validity" and "consistency" are unambiguously defined for each API call. As you might have inferred from this and the earlier discussion, I suspect this may not always be the case today.

> However, you can change this behavior by compiling Lua with the macro LUA_USE_APICHECK defined.

I addressed this in the previous discussion.

That said, the fact that the macro exists and the corresponding verification macros are in the code base, it is probably not very painful to identify all the existing API expectations and document them as such. That will probably not cover everything, but it is a good starting point, and it will certainly cover the lua_next() case. Something a student could do, perhaps :)

Looking at the source of Lua 5.3.3, api_check() is used in 30 places directly, in 23 places via api_checknelems(), in 2 places via api_checkstackindex(), 
in 1 place via api_checkvalidindex(), and  in 2 places via checkresults()

Having inspected all that, I have the following list of API functions that have validation code:

lua_arith *
lua_callk
lua_checkstack
lua_compare *
lua_concat
lua_copy
lua_dump
lua_error
lua_getinfo -
lua_getuservalue
lua_next
lua_pcallk
lua_pushcclosure
lua_rawget
lua_rawgeti
lua_rawgetp
lua_rawset
lua_rawseti
lua_rawsetp
lua_resume
lua_rotate *
lua_setfield
lua_setglobal
lua_seti
lua_setmetatable
lua_settable
lua_settop -
lua_setupvalue
lua_setuservalue
lua_typename *
lua_upvalueid
lua_upvaluejoin
lua_xmove
lua_yieldk

The list may be incomplete, because the following internal functions, apparently related to Lua/C transitions and coroutines, also do API checks:

luaD_precall
finishCcall
resume

These would need to be investigated by someone more knowledgeable about Lua's internals.

In the list above, the functions marked with * have some wording in their descriptions that could be interpreted as a specification of "validity", such as "The value of op must be one of the following constants" for lua_arith. This is great but the specs are incomplete and, unfortunately, very rare overall.

The functions marked with - have descriptions that may be interpreted as more permissive than their internal expectations.

lua_getinfo: "This function returns 0 on error (for instance, an invalid option in what)." Yet certain combinations of inputs are fatal. Unless I had looked at the implementation, I would most likely have assumed that invalid input is signalled by the zero return value, not by a crash or some other misbehavior.

lua_settop: "Accepts any index, or 0, and sets the stack top to this index. If the new top is larger than the old one, then the new elements are filled with nil." Yet an attempt to specify a new top value greater that the current stack's size can be fatal. Changing this to "Accepts any acceptable index..." could resolve the ambiguity. I am aware of the statement in the manual that "Except when noted otherwise, functions in the API work with acceptable indices.", but "any index" could be interpreted as "noted otherwise".

In the entire list above, perhaps only lua_callk, lua_getinfo, and lua_pcallk would need an elaborate specification of validity, in all the others, like lua_next(), not much more than the following needs to be said: "The value at the given index must be a table; otherwise, the behavior is undefined".

For the record, api_check() is also used in 34 places via api_incr_top() and 47 places in index2addr(), but I think that is already adequately covered by the manual that documents the API stack discipline.

Cheers,
V.