metatables and == nil

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

metatables and == nil

ThePhD
I have a question about working around the behavior of metatables when it comes to invoking the "__eq" or "==" abstraction for a userdata and `nil`.

From the docs (emphasis mine):

Lua will try a metamethod only when the values being compared are either both tables or both full userdata and they are not primitively equal. The result of the call is always converted to a boolean.

This prevents me from doing "some_userdata == nil" and having it return true in the case of working with something like a nullptr. I want to keep the ease-of-use/"C-like" style of comparing against Nil, but I cannot invoke a metamethod to check. Does anyone have any suggestions? I'm open to just about anything, and obviously the more elegant/syntax-clean suggestion the better.

Background:
Previously, I used to return an actual "nil" when I detected "nullptr" in the connected C code, but I can't do that anymore due to complicated typing issues that can be somewhat explained in these github issues:

https://github.com/ThePhD/sol2/issues/434#issue-239098943
https://github.com/ThePhD/sol2/issues/419#issue-234207101

I might revert the change back and tell the users to use a higher-level abstraction that's not std::shared_ptr/std::unique_ptr, or crash if I can't convert rather than returning a nullptr-containing version of these C++ abstractions, but.... well, it's hard figuring out which is the right way to go.
Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

Duane Leslie
> On 30 Jun 2017, at 02:51, ThePhD <[hidden email]> wrote:
>
> This prevents me from doing "some_userdata == nil" and having it return true in the case of working with something like a nullptr. I want to keep the ease-of-use/"C-like" style of comparing against Nil, but I cannot invoke a metamethod to check. Does anyone have any suggestions? I'm open to just about anything, and obviously the more elegant/syntax-clean suggestion the better.

I needed the same, and so I run a modified version of the equality test function, attached.  If you control the Lua interpreter you can always tweak it, you just need to be careful to vet any third party packages you use to make sure that your modified behaviour doesn't confuse it.

Regards,

Duane.


luaV_equalobj.c (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

Eric Man-3
What about writing your own null checking function?

function isnull(value)
  if value is nullptr: return nil
  if value is nil: return nil
  if not value: return value
  return true
end

Then use it everytime you want to check it.

if isnull(myvaluefromfunction) then ... end

I suppose in Lua convention you can name it "isnil" instead of "isnull".

On Fri, Jun 30, 2017 at 9:24 AM, Duane Leslie <[hidden email]> wrote:
> On 30 Jun 2017, at 02:51, ThePhD <[hidden email]> wrote:
>
> This prevents me from doing "some_userdata == nil" and having it return true in the case of working with something like a nullptr. I want to keep the ease-of-use/"C-like" style of comparing against Nil, but I cannot invoke a metamethod to check. Does anyone have any suggestions? I'm open to just about anything, and obviously the more elegant/syntax-clean suggestion the better.

I needed the same, and so I run a modified version of the equality test function, attached.  If you control the Lua interpreter you can always tweak it, you just need to be careful to vet any third party packages you use to make sure that your modified behaviour doesn't confuse it.

Regards,

Duane.


Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

Jonathan Goble
On Jun 29, 2017 8:02 PM, "Eric Man" <[hidden email]> wrote:
What about writing your own null checking function?

function isnull(value)
  if value is nullptr: return nil
  if value is nil: return nil
  if not value: return value
  return true
end

I read this and got confused as to whether I was reading the lua-l list or one of the several Python lists that I'm also subscribed to. :P
Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

szbnwer@gmail.com
if you stick to the == operator, then maybe you can use a table as
NULL={} with a metatable, or a userdata, maybe even with a NULL
pointer, but ive got no idea if these could work or not, because ive
never played with metatables and userdata... just i think if these can
work, then these could be good ways to go if there isnt a better one

Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

Duane Leslie
In reply to this post by ThePhD
> On 30 Jun 2017, at 10:01, Eric Man <[hidden email]> wrote:
>
> What about writing your own null checking function?

If the target is just truthy/false you could overload the binary not operator.  To get consistent behaviour with `nil` you'd also have to modify its metatable as follows:

```
debug.setmetatable(nil,(debug.getmetatable(nil) or {}).__bnot = function () return true end)
```

Then you'd be able to say:

```
if ~userdata then
        print("null/nil userdata")
else
        print("valid userdata")
end
```

And it would work for both your userdata type and the nil type.

Regards,

Duane.


Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

Viacheslav Usov
In reply to this post by ThePhD
On Thu, Jun 29, 2017 at 6:51 PM, ThePhD <[hidden email]> wrote:


I have tried to understand your problem, but something does not click.

What I have understood is that when the user code wants to push a shared_ptr holding a nullptr, your code pushes sol::nil, which is not Lua's nil. Correct so far?

Then the user pops something assuming it is a shared_ptr pushing earlier, and your code also assumes it is shared_ptr, just casting it as such. But in this case it casts sol::nil as shared_ptr, and that is bad. Is that correct?

The disconnect I have here is why, since you know that in certain cases you push shared_ptr as sol::nil, the popping code has no check for this case. That sounds like a very silly bug, easily fixable, so I must be missing something.

Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

ThePhD
Thanks for all the suggestions. Because I do not want to modify metatables for core types or modify the interpreter, it is likely that I will have to revert the behavior some.

To answer your question, Viacheslav Usov, I can indeed protect from the user making that mistake (and I did with my original iteration of type-checker). But it becomes a a problem of violating user expectations in different scenarios. Here are the expectations I'm trying to meet:

User Expectation 1: If something is a nullptr from C or C++, I can check it against "nil" in Lua or put it by itself in an if statement and have it coerce to false.
User Expectation 2: If I push a std::shared_ptr (or std::unique_ptr or boost::shared_ptr), I should be able to retrieve that "std::shared_ptr" out from Lua, even if its a nullptr.

Note that I have no examples of the above expectations showing, in the tutorials or documentation of my library. It is just an assumed expectation that users share. Expectation 1 is far more prevalent than Expectation 2, although both make sense.

Below, I will try to explain as best as I can and muse a bit about the solutions. I will start with a demonstration:

std::shared_ptr<int> giving_function ( ) {
     return nullptr;
}

void receiving_function (std::shared_ptr<int>& s) {
     // do something with s
}

lua_State* L = luaL_newstate();
// binding magic to insert giving_function into the state
// binding magic to insert receiving_function into the state
const char* code = "
     v = giving_function() 
     print(v == nil)
     receiving_function(v) 
";

luaL_dostring(L, code);

The way I handled it at first with the "binding magic" was this:
1. check if the pointer is null
2a. Yes -> push Lua's nil with `lua_pushnil(L)` (what sol::nil represents in C++ land)
2b. No -> push userdata

With this way, doing a check for "v == nil" works and the code prints "true". But, when "receiving_function" is called and it inspects "v"s type, there's a bit of a problem. What's being asked for is a reference to a std::shared_ptr. This does not work because lua_touserdata returns `nullptr`, and thusly there's no std::shared_ptr inside I can pass a reference to. The type checker did fail this code. Users complained that this code should "still work", because it violated their expectations. Someone filed an issue, where I gave them a tiny amount of advice to just take a raw pointer T*, where I could convert Lua's nil to a "nullptr" and not have to find a "std::shared_ptr" inside of the userdata.

Still, I tried a second way to handle it with "binding magic":
1. push userdata of std::shared_ptr, regardless of whether it contained a nullptr or not

With this way, doing a check for "v == nil" does not work and prints "false". This violated user expectations and once more an issue was filed with my library. When "receiving_function" is called and it inspects "v"s type, it works because there's a userdata, and thusly I can hand the user a reference to the std::shared_ptr, even if that std::shared_ptr is just a container for "nullptr". Note: A fringe benefit of doing it the secondary way is if these userdata are put into a table as a "sequence", then there are no "nil" holes. (This does not affect collections and data structures returned from C++, as we override the metatable for them and let Lua 5.2/5.3 proper iterator or pairs functions, where we use the well-designed Lua iteration hooks to provide safe iteration over the entire sequence or data structure regardless of what it contains.)

"But why does someone need a reference to the std::shared_ptr and not just a value?"
When we store non-primitive types in Lua with our "binding magic" (sol2), they are stored as userdata of some sort. All userdata carry with it the expectation that you can take a reference to the original data that you pushed: this allows people to do things like work with "std::unique_ptr"s and other types which cannot be copied (and thusly, we cannot take them by-value in a function because "by-value" in a function signature means "take a copy"). It also allows someone to directly manipulate the data held in Lua, without needing to perform a special dance or copy, swap or move their data (all operations can be done in-place on Lua's memory). I also cannot specialize the behavior for checking if someone takes a shared_ptr by-value versus if they ask for a reference: I always have to return a reference from the function that retrieves a specific type.

"What about writing a null-checking function?"
I have spent a long-time curating the Binding Magic (sol2) Library's documentation. Nobody will read it unless something goes wrong, and even then I need to find a way to allow them to quickly find out that working with sol2's types would require me to find the perfect spots to place this information so nobody makes any mistakes. I want the library to provide good defaults that make the most sense, which is why I'm trying these ways and exploring options for handling it. If I write a null-checking function, I also need to make it opt-in. I have a mechanism for including "built-in" libraries, and I could very well write a "built in" lib that comes with my framework where people have to use "sol.is_null( some_userdata )" to check. This, introduces a piece of sol2-specific code into people's code-bases and inevitably ties them to me. The goal of this library is that all of the code looks and feels like Lua code, with no special library-internal libs or strings attached. The Lua code is supposed to look like plain old Lua code, and be replaceable with Lua metatables and other stuff so it works as seamlessly as possible.

"What about Tamás Kiss explanation of using a NULL userdata or a table to do == comparisons with?"
Again, I would need to introduce this special NULL type to the user and make it something they have to opt-into to use (like with the null-checking function). I can do it and it gets closer to elegant by enabling the syntax "if some_userdata == sol.null", but once more I'm injecting Library-specific types that other people's code now depends on (and many would not know specifically how to implement). I would like to avoid features that lay-users would have a hard time replicating on their own.

"So then maybe returning actual Lua nil is the best solution, and you just need to warn people who use unique pointer-types to be careful?"
I am beginning to think this is the best way to go. People have depended on me doing things the first-way for quite a long while now, and this recent change was an experiment to see if things would work out properly. The "there's a hole in my sequence table" problem is not really a problem I care for either, because -- as I mentioned before -- with C++ containers as userdata I can simply override the default iteration ability. If someone is deliberately sticking with Lua 5.1 -- for LuaJIT or other reasons -- then they expect/deserve the pain of ipairs/pairs and its non-overridable behavior for old, crufty 5.1. I only dream that someday LuaJIT will release a 5.3-complaint version of itself, with all the squeaky, shiny metatable wheels and the lack of table-enforcing checks in pairs/ipairs, and summarily kill of the usage of Lua 5.1 except for a few niche users.

But that is just a dream. :)

I apologize for the long-winded explanation, but I wanted to explain everything in-full. Thanks to everyone who helped me bikeshed this: if there's no other keen suggestions I will likely revert to just doing `lua_pushnil` when I detect a "nullptr", and warn users accordingly that not taking a raw pointer (T*) or not taking a raw reference (T&) of the underlying type is a dangerous game with Lua, since Lua owns the memory and directly modifying the internally-stored unique pointer types is bad.

Sincerely,
ThePhD


On Fri, Jun 30, 2017 at 6:35 AM, Viacheslav Usov <[hidden email]> wrote:
On Thu, Jun 29, 2017 at 6:51 PM, ThePhD <[hidden email]> wrote:


I have tried to understand your problem, but something does not click.

What I have understood is that when the user code wants to push a shared_ptr holding a nullptr, your code pushes sol::nil, which is not Lua's nil. Correct so far?

Then the user pops something assuming it is a shared_ptr pushing earlier, and your code also assumes it is shared_ptr, just casting it as such. But in this case it casts sol::nil as shared_ptr, and that is bad. Is that correct?

The disconnect I have here is why, since you know that in certain cases you push shared_ptr as sol::nil, the popping code has no check for this case. That sounds like a very silly bug, easily fixable, so I must be missing something.

Cheers,
V.


Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

Viacheslav Usov
On Fri, Jun 30, 2017 at 4:09 PM, ThePhD <[hidden email]> wrote:

The way I handled it at first with the "binding magic" was this:
> 1. check if the pointer is null
> 2a. Yes -> push Lua's nil with `lua_pushnil(L)` (what sol::nil represents in C++ land)
> 2b. No -> push userdata

So I did misunderstood things a bit, but now I think I get the picture.

I'd say there are two questions for you to answer:

Q1. What should I do if the user wants to pop a smart pointer, but the value is incompatible? E.g., a number where a shared_ptr is expected? I am not a user of your library, so I just do not know. An exception that bubbles up as a Lua error would not be unreasonable here.

Q2. Do I need to handle the special nil case differently from the general case #1?

#2 can be considered independently of #1. Inside your library pop code, you know that it might be nil, because that is how it is pushed in case 2a. You can therefore construct an empty smart pointer (or, if your code is more generic than this, default-construct an object) and give it to the user code. Perhaps you can just construct it on stack, because, if the user code changes that object, it will probably be impossible for you to inject it into where the nil came from.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

ThePhD
Thanks for getting back to me, I hope I'm explaining this right! Here goes...

If you pop a number from something that is not a number, a few things can happen depending on what you ask the library for.
1a. "int v = binding_magic.get("not_a_number");" -- luaL_error is called when safeties are on.
1b. "sol::optional<int> maybe_v = binding_magic.get("not_a_number");" -- no error is called, simply return something that presents "this didn't work"

This can also be used with smart pointers. For #2, the "general case" because different because the binding magic library does not return a value in the case of a userdata: it returns a reference/pointer. Let's say we have "v", which is a shared pointer. The library will be okay with the following:

2a. some_type* v_ptr = binding_magic["v"]; // ok
2b. some_type& v_ref = binding_magic["v"]; // ok
2c. std::shared_ptr<some_type>& v_shared_ref = binding_magic["v"]; // ok
2d. std::shared_ptr<some_type> v_shared_value = binding_magic["v"]; // ok
2e. sol::optional<std::shared_ptr<some_type>> maybe_v_shared_value = binding_magic["v"]; // ok

Now, let's assume "v" is "nil". Of those 5 forms:

[ 2a ] We can handle the "nil" case, and return a nullptr safely
[ 2b ] We cannot convert "nil" to a reference: errors
[ 2c ] We cannot convert "nil" to a shared_ptr (lua_touserdata returns nullptr): errors
[ 2d ] Same error as 2c (even though it is a value, the conversion function returns a reference and the user makes a copy when it reaches the top-level and gets converted to a value by the user specifying a type)
[ 2e ] We can handle the "nil" case, and return a "this optional does not have stuff"

When "nil" is present, only the forms [ 2a ] and [ 2e ] are valid ways of working with it. I am a little disappointed about [ 2d ], because it's a value and we should _reasonably_ be able to create a nullptr-containing object (default-constructed, as you say), but I cannot differentiate between someone asking for a reference std::shared_ptr, versus a value std::shared_ptr (it's a C++ quirk about conversions that would probably not be useful to fully discuss on the list). That means I can't construct something on the stack and then return it, because I would ultimately be returning a dangling reference when this conversion is done.

To reiterate what I said before, 2d confused some people because when a function return a "std::shared_ptr<some_type>(nullptr)" and then they retrieved a "std::shared_ptr" as an argument or something later, they expected a shared_ptr value to be made that was a nullptr. My advice to them was not to take the shared_ptr, just take a raw pointer, but sometimes APIs are fixed and some people do not want to write wrapper functions just to communicate through my library.

Hencewhy, I had a Second Idea, where I would NOT "lua_pushnil" but instead construct a shared_ptr that was filled with nothing (nullptr) and store that as a userdata in Lua. This made all the conversions work, at the cost of "my_userdata == nil" no longer evaluating to true (hence the reason I started this thread).

Does that make sense?


On Fri, Jun 30, 2017 at 10:48 AM, Viacheslav Usov <[hidden email]> wrote:
On Fri, Jun 30, 2017 at 4:09 PM, ThePhD <[hidden email]> wrote:

The way I handled it at first with the "binding magic" was this:
> 1. check if the pointer is null
> 2a. Yes -> push Lua's nil with `lua_pushnil(L)` (what sol::nil represents in C++ land)
> 2b. No -> push userdata

So I did misunderstood things a bit, but now I think I get the picture.

I'd say there are two questions for you to answer:

Q1. What should I do if the user wants to pop a smart pointer, but the value is incompatible? E.g., a number where a shared_ptr is expected? I am not a user of your library, so I just do not know. An exception that bubbles up as a Lua error would not be unreasonable here.

Q2. Do I need to handle the special nil case differently from the general case #1?

#2 can be considered independently of #1. Inside your library pop code, you know that it might be nil, because that is how it is pushed in case 2a. You can therefore construct an empty smart pointer (or, if your code is more generic than this, default-construct an object) and give it to the user code. Perhaps you can just construct it on stack, because, if the user code changes that object, it will probably be impossible for you to inject it into where the nil came from.

Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: metatables and == nil

Viacheslav Usov
On Fri, Jun 30, 2017 at 5:31 PM, ThePhD <[hidden email]> wrote:

> When "nil" is present, only the forms [ 2a ] and [ 2e ] are valid ways of working with it. I am a little disappointed about [ 2d ], because it's a value and we should _reasonably_ be able to create a nullptr-containing object (default-constructed, as you say), but I cannot differentiate between someone asking for a reference std::shared_ptr, versus a value std::shared_ptr (it's a C++ quirk about conversions that would probably not be useful to fully discuss on the list). That means I can't construct something on the stack and then return it, because I would ultimately be returning a dangling reference when this conversion is done.

It sounds like the difficulty in instantiating a temporary smart pointer to represent the nil stands in the way of a good solution. I am afraid this is a C++ issue rather than a Lua issue. I had a brief look at your code, and I understand why a stack-based temporary would be difficult to inject.

I think, however, that you could still allocate a temporary on the heap in the stack_detail::call/eval chain. The record/tracking structure can keep track of those temporaries and deallocate them when done. You could probably optimise that by having a stack-based buffer (established in call()) for the first few temporaries, thus avoiding the heap overhead in the common case. I am less sure here, but your code seems to know all the C++ parameters expected at that point, which makes the max size of the stack based buffer known, thus allowing to dispense with heap allocation completely.

One interesting issue is how to construct the temporary. I said earlier that perhaps this could done using the default constructor. But it is also reasonable to believe that a constructor taking a nullptr could be used. Either way will break some user code that used parameter types non constructible in those ways, but such code was unsafe to begin with, while code using std pointers types will not be affected.

Anyway, I think that pushing nil is the right approach, and the above makes it possible to have automatic conversion from nil to empty smart pointers.

Cheers,
V.