Can lua_next() error?

[Tom Sutcliffe]

Hi list,

I've been reviewing some of my native code for safety and noticed that I've been assuming lua_next() will not raise arbitrary errors. In the 5.3 documentation however, lua_next is listed as 'e' meaning it can.

I couldn't think of any obvious reason how it could error, since there is no metamethod to override a next iteration, and I couldn't see anything in the implementation that would trigger an __index. Am I missing something or is the documentation incorrect?

Thanks,

Tom

[云风 Cloud Wu]

Tom Sutcliffe <[hidden email]>于2018年4月16日周一 下午8:22写道：

Hi list,

I've been reviewing some of my native code for safety and noticed that I've been assuming lua_next() will not raise arbitrary errors. In the 5.3 documentation however, lua_next is listed as 'e' meaning it can.

I couldn't think of any obvious reason how it could error, since there is no metamethod to override a next iteration, and I couldn't see anything in the implementation that would trigger an __index. Am I missing something or is the documentation incorrect?

`lua_next`  may raise error when you pass a key that is not exist in the table.

[Tom Sutcliffe]

On 16 Apr, 2018, at 02:08 PM, 云风 Cloud Wu <[hidden email]> wrote:

`lua_next`  may raise error when you pass a key that is not exist in the table.

Good point, but that would qualify it for a "v" ("the function may raise an error on purpose") rather than an "e" ("the function may raise any errors (it can run arbitrary Lua code, either directly or through metamethods)" wouldn't it?

Is there any way in which a call to lua_next() (with a valid table index and a valid key on the stack) could result in arbitrary code being run?

Thanks,

Tom

[Viacheslav Usov]

On Mon, Apr 16, 2018 at 2:22 PM, Tom Sutcliffe <[hidden email]> wrote:

> I've been reviewing some of my native code for safety and noticed that I've been assuming lua_next() will not raise arbitrary errors.

Not only can it raise an error, it can also crash.

http://lua-users.org/lists/lua-l/2017-12/msg00057.html

I am not sure whether the thread above should mean it is OK for it to crash in the given circumstances, but since you are checking for safety, this is definitely something to beware.

Cheers,

V.

[Roberto Ierusalimschy]

> On Mon, Apr 16, 2018 at 2:22 PM, Tom Sutcliffe <[hidden email]> wrote:
>
> > I've been reviewing some of my native code for safety and noticed that
> I've been assuming lua_next() will not raise arbitrary errors.
>
> Not only can it raise an error, it can also crash.
>
> http://lua-users.org/lists/lua-l/2017-12/msg00057.html

Practically all functions in the Lua-C API can crash, when called with
invalid arguments. It would be expensive, and often impossible, to check
for that kind of errors. (I believe this happens with most C libraries.)

-- Roberto

[Roberto Ierusalimschy]

> `lua_next`  may raise error when you pass a key that is not exist in the table.
>
> Good point, but that would qualify it for a "v" ("the function may raise an error on purpose") rather than an "e" ("the function may raise any errors (it can run arbitrary Lua code, either directly or through metamethods)" wouldn't it?

I have to recognize that the distinction between "e" and "v" here is not
very clear. By that comment, that "it can run arbitrary Lua code, either
directly or through metamethods", you are right that this one should
have a "v".


> Is there any way in which a call to lua_next() (with a valid table index and a valid key on the stack) could result in arbitrary code being run?

No.

-- Roberto

[Viacheslav Usov]

On Mon, Apr 16, 2018 at 6:04 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> Practically all functions in the Lua-C API can crash, when called with invalid arguments.

In the referenced thread, the C-language arguments given to the function were not invalid. It was the content of the Lua stack that was problematic for that function.

In the "checking for safety" context, that means the stack's content should be validated, if it is accepted from another party.

> I believe this happens with most C libraries.

No, not really. It is one thing if a parameter in a C function is a pointer, then the argument simply has to point to a valid memory location. It is quite another when it is a kind of a "handle", then it is rather typical for a function to return an error result if the argument has an invalid or unsuitable value. See POSIX/read(), for example. Lua's stack indices look more like handles than pointers to me, but that, of course, can be just me.

Cheers,

V.

[Tom Sutcliffe]

On 16 Apr 2018, at 17:12, Roberto Ierusalimschy <[hidden email]> wrote:

>
> I have to recognize that the distinction between "e" and "v" here is not
> very clear. By that comment, that "it can run arbitrary Lua code, either
> directly or through metamethods", you are right that this one should
> have a "v".
>
>
>> Is there any way in which a call to lua_next() (with a valid table index and a valid key on the stack) could result in arbitrary code being run?
>
> No.
>

Many thanks Roberto for clarifying! Panic averted :-)

Cheers,

Tom

[Dirk Laurie-2]

2018-04-16 18:49 GMT+02:00 Viacheslav Usov <[hidden email]>:

> No, not really. It is one thing if a parameter in a C function is a pointer,
> then the argument simply has to point to a valid memory location. It is
> quite another when it is a kind of a "handle", then it is rather typical for
> a function to return an error result if the argument has an invalid or
> unsuitable value. See POSIX/read(), for example. Lua's stack indices look
> more like handles than pointers to me, but that, of course, can be just me.

The C API functions themselves do no type checking. The standard
library, wriiten using only the API, does it all the time. That should
serve as a model of what user functions based on the API should do. If
lua_next(), or any other API function, causes a crash, it is a bug in
the user program.

[Soni "They/Them" L.]

On 2018-04-16 01:49 PM, Viacheslav Usov wrote:
> On Mon, Apr 16, 2018 at 6:04 PM, Roberto Ierusalimschy
> <[hidden email] <mailto:[hidden email]>> wrote:
>
> > Practically all functions in the Lua-C API can crash, when called
> with invalid arguments.
>
> In the referenced thread, the C-language arguments given to the
> function were not invalid. It was the content of the Lua stack that
> was problematic for that function.

As far as the Lua C API is concerned, stack values are function
arguments. How else would you pass tables to functions? They don't exist
outside the Lua state.

>
> In the "checking for safety" context, that means the stack's content
> should be validated, if it is accepted from another party.
>
> > I believe this happens with most C libraries.
>
> No, not really. It is one thing if a parameter in a C function is a
> pointer, then the argument simply has to point to a valid memory
> location. It is quite another when it is a kind of a "handle", then it
> is rather typical for a function to return an error result if the
> argument has an invalid or unsuitable value. See POSIX/read(), for
> example. Lua's stack indices look more like handles than pointers to
> me, but that, of course, can be just me.
>
> Cheers,
> V.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.

[Viacheslav Usov]

On Tue, Apr 17, 2018 at 1:51 AM, Soni "They/Them" L. <[hidden email]> wrote:

> As far as the Lua C API is concerned, stack values are function arguments.

In the C standard, the definition of 'argument' is certainly not dependent on Lua's stack values.

In the standard C library API, an effort is made to warn the user of undefined behavior when an argument is valid at the basic C level, but semantically it is not. For example, fopen(): "The argument mode points to a string. If the string is one of the following, the file is open in the indicated mode. Otherwise, the behavior is undefined."

Compare that with the current note for lua_tointegerx(): "The Lua value must be an integer, or a number or string convertible to an integer (see §3.4.3); otherwise, lua_tointegerx returns 0", and then compare with an imaginary note for lua_next(): "The value at the given index must be a table; otherwise, the behavior is undefined".

Cheers,

V.

[Roberto Ierusalimschy]

> Compare that with the current note for lua_tointegerx(): "The Lua value
> must be an integer, or a number or string convertible to an integer (see
> §3.4.3); otherwise, lua_tointegerx returns 0", and then compare with an
> *imaginary* note for lua_next(): "The value at the given index must be a
> table; otherwise, the behavior is undefined".

As far as my English goes, "undefined" means "not defined". So, if it
is not defined, it is undefined. We some times do it, but to say that
the behavior is undefined reminds me of "This page intentionally left
blank".

-- Roberto

[Viacheslav Usov]

On Tue, Apr 17, 2018 at 5:34 PM, Roberto Ierusalimschy <[hidden email]> wrote:

> As far as my English goes, "undefined" means "not defined". So, if it not defined, it is undefined. We some times do it, but to say that behavior is undefined reminds me of "This page intentionally left blank".

In C and C++ circles, "undefined behavior", frequently shortened to just UB, is a term defined in the language standard. In C:

1 undefined behavior

behavior, upon use of a nonportable or erroneous program construct or of erroneous data,

for which this International Standard imposes no requirements

2 NOTE Possible undefined behavior ranges from ignoring the situation completely with unpredictable

results, to behaving during translation or program execution in a documented manner characteristic of the

environment (with or without the issuance of a diagnostic message), to terminating a translation or

execution (with the issuance of a diagnostic message).

3 EXAMPLE An example of undefined behavior is the behavior on integer overflow

When we read an API spec that says "this is undefined behavior", we mentally substitute that with "it can crash or format your hard drive; do not do this". When we read a spec that does not say that, we do not always understand that some combinations of input will result in THAT kind of UB; for example, I expected a Lua error to be raised by lua_next() when given a non-table input (why? perhaps because it had the 'e" error category?). So even though it feels like stating the obvious, it helps.

Cheers,

V.

[云风 Cloud Wu]

> 在 2018年4月18日，上午2:04，Viacheslav Usov <[hidden email]> 写道：
>
> input will result in THAT kind of UB; for example, I expected a Lua error to be raised by lua_next() when given a non-table input (why? perhaps because it had the 'e" error category?). So even though it feels like stating the obvious, it helps.

You can turn on LUA_USE_APICHECK to do more checks on the C API .

[Vaughan McAlley]

On Wed, 18 Apr 2018, 01:34 Roberto Ierusalimschy, <[hidden email]> wrote:

> Compare that with the current note for lua_tointegerx(): "The Lua value
> must be an integer, or a number or string convertible to an integer (see
> §3.4.3); otherwise, lua_tointegerx returns 0", and then compare with an
> *imaginary* note for lua_next(): "The value at the given index must be a
> table; otherwise, the behavior is undefined".

As far as my English goes, "undefined" means "not defined". So, if it
is not defined, it is undefined. We some times do it, but to say that
the behavior is undefined reminds me of "This page intentionally left
blank".

-- Roberto

I need to put undefined behaviour into my sequences.

;-)

Vaughan

[Dirk Laurie-2]

2018-04-17 20:04 GMT+02:00 Viacheslav Usov <[hidden email]>:

> When we read a spec that does not say that, we do not always
> understand that some combinations of input will result in THAT kind of UB;
> for example, I expected a Lua error to be raised by lua_next() when given a
> non-table input (why? perhaps because it had the 'e" error category?). So
> even though it feels like stating the obvious, it helps.

How about this: put the third paragraph of "4 – The Application Program Interface" in technicolor?

As in most C libraries, the Lua API functions do not check their arguments for validity or consistency.
However, you can change this behavior by compiling Lua with the macro LUA_USE_APICHECK defined.

[Viacheslav Usov]

On Wed, Apr 18, 2018 at 6:51 AM, Dirk Laurie <[hidden email]> wrote:

> As in most C libraries, the Lua API functions do not check their arguments for validity or consistency.

That makes perfect sense, once "validity" and "consistency" are unambiguously defined for each API call. As you might have inferred from this and the earlier discussion, I suspect this may not always be the case today.

> However, you can change this behavior by compiling Lua with the macro LUA_USE_APICHECK defined.

I addressed this in the previous discussion.

That said, the fact that the macro exists and the corresponding verification macros are in the code base, it is probably not very painful to identify all the existing API expectations and document them as such. That will probably not cover everything, but it is a good starting point, and it will certainly cover the lua_next() case. Something a student could do, perhaps :)

Looking at the source of Lua 5.3.3, api_check() is used in 30 places directly, in 23 places via api_checknelems(), in 2 places via api_checkstackindex(), in 1 place via api_checkvalidindex(), and  in 2 places via checkresults()

Having inspected all that, I have the following list of API functions that have validation code:

lua_arith *

lua_callk

lua_checkstack

lua_compare *

lua_concat

lua_copy

lua_dump

lua_error

lua_getinfo -

lua_getuservalue

lua_next

lua_pcallk

lua_pushcclosure

lua_rawget

lua_rawgeti

lua_rawgetp

lua_rawset

lua_rawseti

lua_rawsetp

lua_resume

lua_rotate *

lua_setfield

lua_setglobal

lua_seti

lua_setmetatable

lua_settable

lua_settop -

lua_setupvalue

lua_setuservalue

lua_typename *

lua_upvalueid

lua_upvaluejoin

lua_xmove

lua_yieldk

The list may be incomplete, because the following internal functions, apparently related to Lua/C transitions and coroutines, also do API checks:

luaD_precall

finishCcall

resume

These would need to be investigated by someone more knowledgeable about Lua's internals.

In the list above, the functions marked with * have some wording in their descriptions that could be interpreted as a specification of "validity", such as "The value of op must be one of the following constants" for lua_arith. This is great but the specs are incomplete and, unfortunately, very rare overall.

The functions marked with - have descriptions that may be interpreted as more permissive than their internal expectations.

lua_getinfo: "This function returns 0 on error (for instance, an invalid option in what)." Yet certain combinations of inputs are fatal. Unless I had looked at the implementation, I would most likely have assumed that invalid input is signalled by the zero return value, not by a crash or some other misbehavior.

lua_settop: "Accepts any index, or 0, and sets the stack top to this index. If the new top is larger than the old one, then the new elements are filled with nil." Yet an attempt to specify a new top value greater that the current stack's size can be fatal. Changing this to "Accepts any acceptable index..." could resolve the ambiguity. I am aware of the statement in the manual that "Except when noted otherwise, functions in the API work with acceptable indices.", but "any index" could be interpreted as "noted otherwise".

In the entire list above, perhaps only lua_callk, lua_getinfo, and lua_pcallk would need an elaborate specification of validity, in all the others, like lua_next(), not much more than the following needs to be said: "The value at the given index must be a table; otherwise, the behavior is undefined".

For the record, api_check() is also used in 34 places via api_incr_top() and 47 places in index2addr(), but I think that is already adequately covered by the manual that documents the API stack discipline.

Cheers,

V.