Best way to save/re-load an lua state

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Best way to save/re-load an lua state

kathrin_69@gmx.de
Hi,

I want to save the state of a Lua VM to a file to re-load and restart execution later.

Three options come to my mind:

1. Serializing the global index table of the VM. I even found a library called "Pluto" which promises to do this.

2. Trying to make a deep copy of the lua_State* pointing to a VM and save/load the binary data.

3. Trying to overwrite malloc/free for the Lua.dll, place all memory the VM holds in an pre-allocated chunk. Save and re-load that whole chunk. I'm not sure If I'm yet experienced (in C/C++) enough to get this done. Probably not.



My Questions are:

1. Which of that options will work or which one are used and which not?
2. Are there any other options I missed?
3. Which option is most promising in terms of time and complexity?
4. Are there any other helper libraries (beside Pluto) which helps doing one of the options?

Thank you!

Regards
Joerg


Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Ben Sunshine-Hill
I wouldn't suggest #2. There's a lot of types of objects referenced by
the lua_State, with some moderately complicated connectivity. It's
certainly possible, but it's a pain in the tuckus.

If you want to serialize *everything* in the Lua state, the easiest
approach is a Lua variant called lper. That basically does your #3.
It's unix-specific, though... I'm not sure how much work would be
involved in porting it to win32.

Ben

On Fri, Mar 20, 2009 at 7:10 AM, [hidden email] <[hidden email]> wrote:
> Hi,
>
> I want to save the state of a Lua VM to a file to re-load and restart
> execution later.
>
> Three options come to my mind:
>
> 1. Serializing the global index table of the VM. I even found a library
> called "Pluto" which promises to do this.
>
> 2. Trying to make a deep copy of the lua_State* pointing to a VM and
> save/load the binary data.
>
> 3. Trying to overwrite malloc/free for the Lua.dll, place all memory the VM
> holds in an pre-allocated chunk. Save and re-load that whole chunk. I'm not
> sure If I'm yet experienced (in C/C++) enough to get this done. Probably
> not.
>
>
>
> My Questions are:
>
> 1. Which of that options will work or which one are used and which not?
> 2. Are there any other options I missed?
> 3. Which option is most promising in terms of time and complexity?
> 4. Are there any other helper libraries (beside Pluto) which helps doing one
> of the options?
>
> Thank you!
>
> Regards
> Joerg
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

kathrin_69@gmx.de
Hello Ben and every buddy else,

I investigated a bit and it seems much simpler: If I call lua_Newstate instead of luaL_Newstate as I did until now, I am able to pass an allocator function which is used exclusively for memory management by Lua regarding the lua_State* it is assigned to.


The default allocator function used by luaL_Newstate() is this:

-----------------------------
static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
 (void)ud;
 (void)osize;
 if (nsize == 0) {
   free(ptr);
   return NULL;
 }
 else
   return realloc(ptr, nsize);
}
-----------------------------

If I would overwrite it in a way so that all memory used by my lua_State* is inside my own (pre-allocated) buffer, I could just save and re-load that buffer to/from disc and resume execution. What do you think, will this work? It sounds pretty easy so far.

Two questions to the l_alloc function shown above:
1. What is void* ud for? The src code says something about "auxiliary data to the allocation function". Why is it not used by that function?

2. What are the first two lines for?
 (void)ud;
 (void)osize;
They do absolutely nothing, right? Are they there to let the compiler check something?

Thank you + Regards
Joeerg


Ben Sunshine-Hill schrieb:
If you want to serialize *everything* in the Lua state, the easiest
approach is a Lua variant called lper. That basically does your #3.
It's unix-specific, though... I'm not sure how much work would be
involved in porting it to win32.

Ben

On Fri, Mar 20, 2009 at 7:10 AM, [hidden email] <[hidden email]> wrote:
Hi,

I want to save the state of a Lua VM to a file to re-load and restart
execution later.

Three options come to my mind:

1. Serializing the global index table of the VM. I even found a library
called "Pluto" which promises to do this.

2. Trying to make a deep copy of the lua_State* pointing to a VM and
save/load the binary data.

3. Trying to overwrite malloc/free for the Lua.dll, place all memory the VM
holds in an pre-allocated chunk. Save and re-load that whole chunk. I'm not
sure If I'm yet experienced (in C/C++) enough to get this done. Probably
not.



My Questions are:

1. Which of that options will work or which one are used and which not?
2. Are there any other options I missed?
3. Which option is most promising in terms of time and complexity?
4. Are there any other helper libraries (beside Pluto) which helps doing one
of the options?

Thank you!

Regards
Joerg





Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Timothy Hunter
[hidden email] wrote:
Hello Ben and every buddy else,

I investigated a bit and it seems much simpler: If I call lua_Newstate instead of luaL_Newstate as I did until now, I am able to pass an allocator function which is used exclusively for memory management by Lua regarding the lua_State* it is assigned to.


The default allocator function used by luaL_Newstate() is this:

-----------------------------
static void *l_alloc (void *ud, void *ptr, size_t osize, size_t nsize) {
 (void)ud;
 (void)osize;
 if (nsize == 0) {
   free(ptr);
   return NULL;
 }
 else
   return realloc(ptr, nsize);
}
-----------------------------

If I would overwrite it in a way so that all memory used by my lua_State* is inside my own (pre-allocated) buffer, I could just save and re-load that buffer to/from disc and resume execution. What do you think, will this work? It sounds pretty easy so far.

Two questions to the l_alloc function shown above:
1. What is void* ud for? The src code says something about "auxiliary data to the allocation function". Why is it not used by that function?
It's there for custom allocation functions that need to keep extra state. That state can be stored in memory addressed by ud.

2. What are the first two lines for?
 (void)ud;
 (void)osize;
They do absolutely nothing, right? Are they there to let the compiler check something?

I explained ud above. The osize argument is necessary if the custom memory manager doesn't track the size of each piece of allocated memory the way the standard C malloc function does. In that case, the memory manager needs to be told how much memory to free as well as the address of the memory.

The default allocation function doesn't need to use those two arguments, but a custom allocation function may find them useful.

If you intend to write your own memory manager you may find those arguments useful as well.



Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

David Given
In reply to this post by kathrin_69@gmx.de
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[hidden email] wrote:
[...]
> If I would overwrite it in a way so that all memory used by  my
> lua_State* is inside my own (pre-allocated) buffer, I could just save
> and re-load that buffer to/from disc and resume execution. What do you
> think, will this work? It sounds pretty easy so far.

This will only work if you new buffer is loaded at the same address as
the old one, as there'll be pointers embedded all throughout the data.
And if there happen to be pointers anywhere else --- such as on the
stack --- then you will be in for what is technically known as A World
Of Pain! Personally I'd go for something like Pluto, which
saves/restores the state at a much higher level.

[...]
> 2. What are the first two lines for?
>  (void)ud;
>  (void)osize;
> They do absolutely nothing, right? Are they there to let the compiler
> check something?

They're to avoid 'unused parameter' warnings.

- --
ââââ ïïïïïïïïïïïïïï âââââ http://www.cowlark.com âââââ
â
â "All power corrupts, but we need electricity." --- Diana Wynne Jones,
â _Archer's Goon_
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJxQc3f9E0noFvlzgRAjDjAJ0fl5SJ/m7pbLiiwWKDnnmNWAVmMgCglnwj
2wCSn7sQk+5PQuD81tHrlJI=
=rpgU
-----END PGP SIGNATURE-----


Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Asko Kauppi
In reply to this post by kathrin_69@gmx.de

I would recommend another (simpler) kind of mechanism:

- have all of your application state in a separate table (or in the _G)
- store that table when exiting
- read that table when re-launching, and if it already contains stuff, rebuild your application's visual appearance accordingly - in regular exit (truly quitting the software) erase the state table so the next launch will be "virgin"

At least Maemo had this kind of a framework, though very roughly implemented. With Lua, it can be simple. It requires special attention from the scripts you are running, though. And if it's a GUI, you need to make some assumptions s.a. not "hybernating" when a dialog is active. Otherwise there's waaaay too much state to be restored (had this dialog open, with this field selected, with these entries edited....) Just consider dialogs invocation-temporal.

Lua Lanes supports this kind of a mechanism in the thread cancellation (you can get the globals of a cancelled thread). But you can implement it also on bare Lua, or with Pluto.

Things s.a. userdata cannot be serialized, of course. They can have a lifespan within each invocation only.

- asko



[hidden email] kirjoitti 20.3.2009 kello 13:10:

Hi,

I want to save the state of a Lua VM to a file to re-load and restart execution later.

Three options come to my mind:

1. Serializing the global index table of the VM. I even found a library called "Pluto" which promises to do this.

2. Trying to make a deep copy of the lua_State* pointing to a VM and save/load the binary data.

3. Trying to overwrite malloc/free for the Lua.dll, place all memory the VM holds in an pre-allocated chunk. Save and re-load that whole chunk. I'm not sure If I'm yet experienced (in C/C++) enough to get this done. Probably not.



My Questions are:

1. Which of that options will work or which one are used and which not?
2. Are there any other options I missed?
3. Which option is most promising in terms of time and complexity?
4. Are there any other helper libraries (beside Pluto) which helps doing one of the options?

Thank you!

Regards
Joerg



Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Ketmar Dark-2
On Mon, 23 Mar 2009 07:09:56 +0200
Asko Kauppi <[hidden email]> wrote:

> I would recommend another (simpler) kind of mechanism:
> 
> - have all of your application state in a separate table (or in the
> _G)
> - store that table when exiting
> - read that table when re-launching, and if it already contains
> stuff, rebuild your application's visual appearance accordingly
> - in regular exit (truly quitting the software) erase the state
> table so the next launch will be "virgin"
it may be good for GUI apps, but it's inappropriate, for example, in
complex rogue-like game. the power of Lua closures allows to write
simple callbacks as anonymous functions, and those will not be
serialized properly. so I must either drop the idea of using anonymous
functions, or write alot of additional code to serialize/deserialize
such things. the both ways are bad, 'cause turn simple things to
overcomplicated ones.

i wonder if Lua 6 can include some built-ins for easy
serialization/deserialization of the interpreter state (with
metamethods for userdatas). as Lua is good suited for game scripting,
it can be much friendlier to game developers (and not only for them
%-). just ask Lua: "serialize yourself with this chunk writing
function" and "deserialize yourself with this chunk reading function".

maybe something like Pluto, but tightly integrated with Lua and with
deep knowledge about internal interpreter structures.

Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Asko Kauppi

You might want to study the source code of Lanes. It does copy closures stack-to-stack and that could most likely be modified to serialize them as well.

But even then, you won't be able to serialize userdata. I think even the "serialize yourself" feature would need to leave userdata out of the equation. And the rest can be done already. If you have any questions of the Lanes internals, pls. send mail to me directly.

-asko


ketmar kirjoitti 23.3.2009 kello 7:23:

On Mon, 23 Mar 2009 07:09:56 +0200
Asko Kauppi <[hidden email]> wrote:

I would recommend another (simpler) kind of mechanism:

- have all of your application state in a separate table (or in the
_G)
- store that table when exiting
- read that table when re-launching, and if it already contains
stuff, rebuild your application's visual appearance accordingly
- in regular exit (truly quitting the software) erase the state
table so the next launch will be "virgin"
it may be good for GUI apps, but it's inappropriate, for example, in
complex rogue-like game. the power of Lua closures allows to write
simple callbacks as anonymous functions, and those will not be
serialized properly. so I must either drop the idea of using anonymous
functions, or write alot of additional code to serialize/deserialize
such things. the both ways are bad, 'cause turn simple things to
overcomplicated ones.

i wonder if Lua 6 can include some built-ins for easy
serialization/deserialization of the interpreter state (with
metamethods for userdatas). as Lua is good suited for game scripting,
it can be much friendlier to game developers (and not only for them
%-). just ask Lua: "serialize yourself with this chunk writing
function" and "deserialize yourself with this chunk reading function".

maybe something like Pluto, but tightly integrated with Lua and with
deep knowledge about internal interpreter structures.


Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Ketmar Dark-2
On Mon, 23 Mar 2009 07:36:58 +0200
Asko Kauppi <[hidden email]> wrote:

> You might want to study the source code of Lanes. It does copy  
> closures stack-to-stack and that could most likely be modified to  
> serialize them as well.
i'm pretty sure it *can* be done in some or another way (never looked
at Lanes, tnx for hint), 'cause at least Pluto did it. what i mean is
that such thing should be a part of Lua itself (may be in a form of
standard library). for now writing such thing requires some sort of
deep knowledge of Lua internals. sure that a good C programmer can dive
in Lua sources and find all the necessary info, but it's not so
friendly as it can be.

> But even then, you won't be able to serialize userdata.  I think
yes, userdata needs some sort of methamethods or other machinery.

so what i mean is that it can be done, but it's not so easy as a lazy
person like me want it to be. %-)

Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

kathrin_69@gmx.de
In reply to this post by David Given
David Given wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[hidden email] wrote:
[...]
If I would overwrite it in a way so that all memory used by  my
lua_State* is inside my own (pre-allocated) buffer, I could just save
and re-load that buffer to/from disc and resume execution. What do you
think, will this work? It sounds pretty easy so far.

This will only work if you new buffer is loaded at the same address as
the old one, as there'll be pointers embedded all throughout the data.
Yes. But how can this be achieved? Is there any windows API to get a fixed address? Or do I have to use a DLL with an "shared" part and load that DLL on a fixed address into process space?

And if there happen to be pointers anywhere else --- such as on the
stack --- then you will be in for what is technically known as A World
Of Pain!
I'm not the most experienced C-programmer, but if I save the lua_State* between two calls to lua_Resume(), there shouldn't be any process stack memory in use by Lua functions, because all Lua-functions are already returned and therefore all stack is un-winded. Am I wrong with this?

Personally I'd go for something like Pluto, which
saves/restores the state at a much higher level.
I have not tried it yet, but I fear serialization will be to slow for my needs, because I'll have to save the lua_State on every cycle of a simulation loop to allow precise/granular rollback.


Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Ketmar Dark-2
On Mon, 23 Mar 2009 12:01:35 +0100
"[hidden email]" <[hidden email]> wrote:

> > This will only work if you new buffer is loaded at the same address
> > as the old one, as there'll be pointers embedded all throughout the
> > data. 
> Yes. But how can this be achieved? Is there any windows API to get a 
> fixed address? Or do I have to use a DLL with an "shared" part and
> load that DLL on a fixed address into process space?
see VirtualAllocEx() function.

Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Peter Cawley
In reply to this post by David Given
On Sat, Mar 21, 2009 at 3:26 PM, David Given <[hidden email]> wrote:
> This will only work if you new buffer is loaded at the same address as
> the old one, as there'll be pointers embedded all throughout the data.
> And if there happen to be pointers anywhere else --- such as on the
> stack --- then you will be in for what is technically known as A World
> Of Pain!

A much more interesting (read: technically challenging) solution would
be to try and use x86 segment registers to allow the buffer to be
loaded at different addresses and have an entry in the LDT pointing at
the base of the buffer. (I've been reading too much of Google's
NativeClient codebase and it's usage of segments).

Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Sam Roberts
In reply to this post by kathrin_69@gmx.de
On Mon, Mar 23, 2009 at 4:01 AM, [hidden email] <[hidden email]> wrote:
> I'm not the most experienced C-programmer, but if I save the lua_State*
> between two calls to lua_Resume(), there shouldn't be any process stack
> memory in use by Lua functions, because all Lua-functions are already
> returned and therefore all stack is un-winded. Am I wrong with this?

Kathrin, this won't work. There will be address inter-dependencies
between the chunks of memory.

> I have not tried it yet, but I fear serialization will be to slow for my
> needs, because I'll have to save the lua_State on every cycle of a
> simulation loop to allow precise/granular rollback.

Saving and restoring complex dynamically allocated C datastructres is
quite difficult, and the approaches that do work, like Pluto?, are
heavy-weight enough that they are very unlikely to be fast enough to
use in an inner loop.

I think you're on the wrong path, why don't you do state snapshot and
rollback in pure lua?

You might get more useful advice from the list if you described what
you want to achieve.

Sam

Reply | Threaded
Open this post in threaded view
|

Re: Best way to save/re-load an lua state

Andre Leiradella
In reply to this post by kathrin_69@gmx.de
>> This will only work if you new buffer is loaded at the same address as
>> the old one, as there'll be pointers embedded all throughout the data.
>> And if there happen to be pointers anywhere else --- such as on the
>> stack --- then you will be in for what is technically known as A World
>> Of Pain!

> A much more interesting (read: technically challenging) solution would
> be to try and use x86 segment registers to allow the buffer to be
> loaded at different addresses and have an entry in the LDT pointing at
> the base of the buffer. (I've been reading too much of Google's
> NativeClient codebase and it's usage of segments).

With lper you don't have to worry with any of this. Usually (all times in my tests), the OS can remap the file to the same memory address, and you Lua state with everything else will be in place. lper even has a facility to create a new state if the given file name doesn't exist, or to use the state in the file if it does.

What you need to worry about here though are C functions used in the Lua world. Your executable will probably be loaded at a different address every time, so pointers to C functions change at every run. There are some workarounds, such as making all your C functions heavy userdata with a __call metamethod. The metamethod would get the userdata pointer (that could even be an int casted to void* identifying the function) and lookup the function pointer in an array and then make the call.

If you pass C pointers to Lua as userdata you can also have this kind of problem, unless you make sure they are unserialized at the same address. One way to make this is to use the same allocator as lper to build C-side objects, but then you don't have control over this if you use third-party libraries. If it's the case, you could apply the same userdata trick described above, adding one level of indirection to all your C pointers when they come from Lua.

Cheers,

Andre