luaclose_mylib function (inverse of "luaopen_mylib" C function)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
bel
Reply | Threaded
Open this post in threaded view
|

luaclose_mylib function (inverse of "luaopen_mylib" C function)

bel
Hello,

What would be the appropriate way to implement a "luaclose_mylib" function in C, that would work as an inverse of "luaopen_mylib"?

For some C libraries, initializations need to be done before any function of the library can be used. This can safely be done in the "luaopen_*" callback function, together with the registration of new functions to a Lua state (the first call in the operating system process).
However, there should be some de-initialization, once the library is no longer required ... at the very latest when the Lua state is closed. But there is no "luaclose_*" callback mechanism.

Is there some recommended way to implement it in a C library?

Some boundary conditions:
- It is not possible to do de-initialisation in a static destructor, DllMain, ... any operation system "unload library" hook (due to OS restrictions of these hooks being incompatible with the underlying library).
- It is not possible to call a "luaclose_mylib" function from the same code calling lua_close (since they are different, independent modules).
- It must work with multithreaded applications using multiple independent Lua states, there is no 1:1 relation between threads and Lua states, but there are appropriate locks (this should not add any complexity for a clean solution, but it might be incompatible with some simple hacks/workarounds)
- There is no "init" or "open" function exposed to the Lua programmer (the C programmer has to handle the complexity of initializing and freeing C dependencies)
- It should work for all Lua 5.x versions (if it is not possible without some `if version == xy then` ... so be it)

Some solutions I could imagine:
- Add some "key=lwuserdada, value=userdata with __gc metatable" to the library table ... tested, works ... but seems like an inelegant workaround and "what is this undocumented userdata in your libary"
- Find some library with a similar requirement (like luasocket : https://github.com/diegonehab/luasocket/blob/5b18e475f38fcf28429b1cc4b17baee3b9793a62/src/luasocket.c#L49 ), and realize they are using the undocumented __unload method ... does NOT work in my test
- Patch "ll_unloadlib" / loadlib.c in Lua C code, to add my own call to "luaclose_*" ... ending up with my private branch :-(

So, what is the official way to find out when a Lua state is done with your library?

BR,
 B

Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

Roberto Ierusalimschy
> Some solutions I could imagine:
> - Add some "key=lwuserdada, value=userdata with __gc metatable" to the
> library table ... tested, works ... but seems like an inelegant workaround
> and "what is this undocumented userdata in your libary"

I would suggest this approach, but adding this entry into the registry
instead of in the library.

-- Roberto
bel
Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

bel
Is there any way the library could ever be unloaded, before the Lua state is destroyed?
If not, the registry is certainly a good solution.

If there is some way to "unrequire" (http://lua-users.org/lists/lua-l/2012-12/msg00883.html ... does not seem to work),
and dlclose/FreeLibrary (ll_unloadlib) is called before the luaclose_lib call, a call into an unloaded lib will crash the entire application when the Lua state is destroyed.

On Mon, Aug 24, 2020 at 8:54 PM Roberto Ierusalimschy <[hidden email]> wrote:
> Some solutions I could imagine:
> - Add some "key=lwuserdada, value=userdata with __gc metatable" to the
> library table ... tested, works ... but seems like an inelegant workaround
> and "what is this undocumented userdata in your libary"

I would suggest this approach, but adding this entry into the registry
instead of in the library.

-- Roberto
Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

Viacheslav Usov
In reply to this post by bel
On Mon, Aug 24, 2020 at 8:46 PM bel <[hidden email]> wrote:

> What would be the appropriate way to implement a "luaclose_mylib" function in C, that would work as an inverse of "luaopen_mylib"?

Have you considered that your library might be used by a process with
multiple Lua states, which come and go dynamically in parallel
threads?

My advice is NOT to have any global state in a library. Instead, all
the state should be linked to the object that the require function
returns. Doing otherwise is a source of persistent problems,
especially in the scenario above.

Cheers,
V.
bel
Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

bel
> Have you considered that your library might be used by a process with
> multiple Lua states, which come and go dynamically in parallel
> threads?

I *know* my library is used in exactly this way.
Even worse: The Lua states may communicate with each other (which is a topic in itself), and all must be "strictly sandboxed" so they don't harm the execution environment and influence each other as little as possible.

Yet, there are restrictions in the backend (the library used by my library) that do require one global state.
So the solution must look something like:

luaopen_mylib(lua_State * L)
{
    mutex_lock();
    if (refcount == 0) {
        my_global_open(); // <-- heavy weight operation
    }
    refcount++;
    mutex_unlock();

    init_lua_state_specific_stuff() // <-- can be stored in L
    lua_new..stuff(L);  (luaL_requiref or whatever)
}


luaclose_mylib(L)
{
    mutex_lock();
    refcount--;
    if (refcount == 0) {
       my_global_close(); // <-- this is the essential operation here
    }
    mutex_unlock();

   free_lua_state_specific_stuff(L);

}

and some library functions require locks too, or serialization through message queues.
(Yes, it's a full scale distributed multithreaded system with many manyears on code that is supposed to run unattended for months/years ... I think I know *exactly* what kind of problems you have in mind ...)

A bug in reference counting may either cause a memory leak, or a complete crash by calling into an unloaded module.
--> This is why I'm asking here.
And yes, of course I will test it extensively ;-)


On Mon, Aug 24, 2020 at 10:04 PM Viacheslav Usov <[hidden email]> wrote:
On Mon, Aug 24, 2020 at 8:46 PM bel <[hidden email]> wrote:

> What would be the appropriate way to implement a "luaclose_mylib" function in C, that would work as an inverse of "luaopen_mylib"?

Have you considered that your library might be used by a process with
multiple Lua states, which come and go dynamically in parallel
threads?

My advice is NOT to have any global state in a library. Instead, all
the state should be linked to the object that the require function
returns. Doing otherwise is a source of persistent problems,
especially in the scenario above.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

Andrew Gierth
In reply to this post by bel
>>>>> "bel" == bel  <[hidden email]> writes:

 bel> Is there any way the library could ever be unloaded, before the
 bel> Lua state is destroyed?

What Lua does with dynamically loaded libraries is this: there is a
table stored in the registry which contains the pathnames and handles of
all dynamically opened objects, and this table has a __gc metamethod
that closes all of those handles. Since this is in the registry and has
had its finalizer set before loading any library, and since finalizers
are called in reverse order of setting, this means that libraries won't
be unloaded until after any finalizers that they set themselves have
run, if I understand things correctly.

There is no way to "unrequire" a module short of closing the state
(unless you do really dangerous things in C code, like replacing the
registry table or the CLIBS table in the registry, or calling its __gc
method explicitly).

If there are multiple Lua states, it's up to the OS facilities being
used to load dynamic libraries to do the necessary reference-counting so
that a library opened multiple times is not unloaded until all
references are closed.

--
Andrew.
bel
Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

bel
Thank you for your explanations.
Using the registry works well in my tests, and it seems there is no way to break the sandbox from inside.
In case someone is interested in the code, this worked for me:

static int lua_regkey_dtor;

lua_pushlightuserdata(L, &lua_regkey_dtor);
lua_newuserdata(L, 0);
lua_newtable(L);
lua_pushliteral(L, "__gc");
lua_pushlightuserdata(L, my_luaclose_argument);
lua_pushcclosure(L, luaclose_mylib, 1);
lua_rawset(L, -3);
lua_setmetatable(L, -2);
lua_settable(L, LUA_REGISTRYINDEX);

(Combined with reference counting in luaopen_mylib and my luaclose_mylib function.)

On Tue, Aug 25, 2020 at 1:43 PM Andrew Gierth <[hidden email]> wrote:
>>>>> "bel" == bel  <[hidden email]> writes:

 bel> Is there any way the library could ever be unloaded, before the
 bel> Lua state is destroyed?

What Lua does with dynamically loaded libraries is this: there is a
table stored in the registry which contains the pathnames and handles of
all dynamically opened objects, and this table has a __gc metamethod
that closes all of those handles. Since this is in the registry and has
had its finalizer set before loading any library, and since finalizers
are called in reverse order of setting, this means that libraries won't
be unloaded until after any finalizers that they set themselves have
run, if I understand things correctly.

There is no way to "unrequire" a module short of closing the state
(unless you do really dangerous things in C code, like replacing the
registry table or the CLIBS table in the registry, or calling its __gc
method explicitly).

If there are multiple Lua states, it's up to the OS facilities being
used to load dynamic libraries to do the necessary reference-counting so
that a library opened multiple times is not unloaded until all
references are closed.

--
Andrew.
Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

Andrew Gierth
>>>>> "bel" == bel  <[hidden email]> writes:

 bel> Thank you for your explanations. Using the registry works well in
 bel> my tests, and it seems there is no way to break the sandbox from
 bel> inside. In case someone is interested in the code, this worked for
 bel> me:

 bel> static int lua_regkey_dtor;

 bel> lua_pushlightuserdata(L, &lua_regkey_dtor);
 bel> lua_newuserdata(L, 0);
 bel> lua_newtable(L);
 bel> lua_pushliteral(L, "__gc");
 bel> lua_pushlightuserdata(L, my_luaclose_argument);
 bel> lua_pushcclosure(L, luaclose_mylib, 1);
 bel> lua_rawset(L, -3);
 bel> lua_setmetatable(L, -2);
 bel> lua_settable(L, LUA_REGISTRYINDEX);

You can simplify:

    lua_newuserdata(L, 0);
    lua_newtable(L);
    lua_pushlightuserdata(L, my_luaclose_argument);
    lua_pushcclosure(L, luaclose_mylib, 1);
    lua_setfield(L, -2, "__gc");
    lua_setmetatable(L, -2);
    lua_rawsetp(L, LUA_REGISTRYINDEX, &lua_regkey_dtor);

--
Andrew.
Reply | Threaded
Open this post in threaded view
|

Re: luaclose_mylib function (inverse of "luaopen_mylib" C function)

Philippe Verdy-2
In reply to this post by Andrew Gierth
The bad thing is that the registry maintains the finalizer only as a single ordered list and not according to a dependency tree. This means that libraries cannot be unloaded if another unrelated library was registered after it and actually does not depend on the library we'd liek to unload.

How can we maintain a dependency tree (and no longer depend on the finalizer ordered list) ?

Using finalizers seems the wrong way to do that. a single finalizer should be used for the whole dependency tree (when terminating the whole Lua instance itself), but there's a need to develop better registry maintaining a dependency graph (possibly with reference counters, but this is not necessarily the best approach as it can be broken easily, causing a library to be unloaded prematurely when it is still in use): this should better be part of the GC, possibly using "weak references" for libraries that can be easily unloaded when needed and reloaded later possibly with their own persistant external storage (some DB interface or external network REST API?) for some data needed to restore its previous state.


Le mar. 25 août 2020 à 13:43, Andrew Gierth <[hidden email]> a écrit :
>>>>> "bel" == bel  <[hidden email]> writes:

 bel> Is there any way the library could ever be unloaded, before the
 bel> Lua state is destroyed?

What Lua does with dynamically loaded libraries is this: there is a
table stored in the registry which contains the pathnames and handles of
all dynamically opened objects, and this table has a __gc metamethod
that closes all of those handles. Since this is in the registry and has
had its finalizer set before loading any library, and since finalizers
are called in reverse order of setting, this means that libraries won't
be unloaded until after any finalizers that they set themselves have
run, if I understand things correctly.

There is no way to "unrequire" a module short of closing the state
(unless you do really dangerous things in C code, like replacing the
registry table or the CLIBS table in the registry, or calling its __gc
method explicitly).

If there are multiple Lua states, it's up to the OS facilities being
used to load dynamic libraries to do the necessary reference-counting so
that a library opened multiple times is not unloaded until all
references are closed.

--
Andrew.