Time Travel Debugging

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Time Travel Debugging

Abhijit Nandy
Hi,

Has anyone tried to implement a time travelling debugger for Lua, like in Elm and other languages: https://youtu.be/PUv66718DII?t=153

It seems there was an attempt here: https://news.ycombinator.com/item?id=10704065 

So a useful case maybe to hit a breakpoint and change a variable's value. Then resume execution to see what happens. If it doesn't work out then reverting the virtual machine to the same state as it was when the machine hit the breakpoint. And again changing the value to something more suitable and executing forward.

Is it possible to save the Lua VM state and restore it or can it be done with some changes to lvm.c? It seems saving the stack and global state should be enough. Any userdata may be tricky.

I guess variables changes can be tracked by inserting local _ENVs within all functions and their entire value set saved. But executing forward and being able to return to a previous point requires snapshotting the LVM state.

Thanks,
Abhijit

Reply | Threaded
Open this post in threaded view
|

Re: Time Travel Debugging

Luiz Henrique de Figueiredo
> Is it possible to save the Lua VM state and restore it?

In principle, it can be done since the Lua API accepts a memory
allocator. Like you've said, it is tricky to handle userdata.

I've experimented with LPSM to make a Lua state persistent across
invocations of the interpreter. It worked reasonably well for a while.
The LPSM library is available at http://freshmeat.net/projects/lpsm/
but it does not seem to work on modern systems. I'd love to hear about
a modern replacement.

Reply | Threaded
Open this post in threaded view
|

Re: Time Travel Debugging

Abhijit Nandy
Hi Luiz,

Thanks for your reply. Maybe I can focus on Lua code only, so ignoring user values for now. So using the custom memory allocator, I could work off a C array for the memory contents. If I save the array to a binary file and then let the interpreter proceed, then it continues to make changes to the array. Now maybe I pause execution somehow by using a co-routine or the debug library.

Then later can I simply restore the array contents from the file and then resume the interpretor, and the interpretor would continue executing from the saved point? And I could repeat this as many times as I want maybe after changing a value of a variable using the debug library :) ?

Thanks,
Abhijit


On Fri, Nov 1, 2019 at 4:15 PM Luiz Henrique de Figueiredo <[hidden email]> wrote:
> Is it possible to save the Lua VM state and restore it?

In principle, it can be done since the Lua API accepts a memory
allocator. Like you've said, it is tricky to handle userdata.

I've experimented with LPSM to make a Lua state persistent across
invocations of the interpreter. It worked reasonably well for a while.
The LPSM library is available at http://freshmeat.net/projects/lpsm/
but it does not seem to work on modern systems. I'd love to hear about
a modern replacement.

Reply | Threaded
Open this post in threaded view
|

Re: Time Travel Debugging

Abhijit Nandy
Hi,

So I tried a simple test to work off a char array and back it up:

#include <string>

extern "C" {
    #include <lua.h>
    #include <lauxlib.h>
    #include <lualib.h>
}

char buffer[4096];
char backupBuffer[4096];
char *pTop = buffer;

void* customLuaAlloc(void* userData, void* ptr, size_t oldSize, size_t newSize)
{
    if (newSize == 0)
    {
        //free(ptr);
        return 0;
    }
    else {
        //return realloc(ptr, newSize);
        if (ptr != NULL) {
            // Copy contents
            memcpy(pTop, ptr, oldSize);
        }
        ptr = NULL;
        char *pOldTop = pTop;
        pTop += newSize;
        return (void*)pOldTop;
    }
}

int main()
{
    lua_State* L = luaL_newstate();
    lua_setallocf(L, customLuaAlloc, NULL);

    luaL_openlibs(L);

    luaL_dostring(L, "a = 1");
    memcpy(backupBuffer, buffer, 4096);   // this causes a crash

    luaL_dostring(L, "a = 2");

    //memcpy(buffer, backupBuffer, 4096);  // try restoring 'a' here once back up works
    luaL_dostring(L, "print(a)");

    lua_close(L);
}

If I comment out:  memcpy(backupBuffer, buffer, 4096)    then things are fine. Otherwise a crash at:
> LuaTTD.exe!luaH_getshortstr(Table * t, TString * key) Line 544 C
  LuaTTD.exe!luaT_gettmbyobj(lua_State * L, const lua_TValue * o, TMS event) Line 82 C
  LuaTTD.exe!GCTM(lua_State * L, int propagateerrors) Line 812 C
  LuaTTD.exe!callallpendingfinalizers(lua_State * L) Line 862 C
  LuaTTD.exe!luaC_freeallobjects(lua_State * L) Line 971 C
  LuaTTD.exe!close_state(lua_State * L) Line 245 C
  LuaTTD.exe!lua_close(lua_State * L) Line 344 C
  LuaTTD.exe!main() Line 50 C++
  LuaTTD.exe!invoke_main() Line 78 C++
  LuaTTD.exe!__scrt_common_main_seh() Line 288 C++
  LuaTTD.exe!__scrt_common_main() Line 331 C++
  LuaTTD.exe!mainCRTStartup() Line 17 C++

Trying to figure out why.

Thanks,
Abhijit


On Fri, Nov 1, 2019 at 5:09 PM Abhijit Nandy <[hidden email]> wrote:
Hi Luiz,

Thanks for your reply. Maybe I can focus on Lua code only, so ignoring user values for now. So using the custom memory allocator, I could work off a C array for the memory contents. If I save the array to a binary file and then let the interpreter proceed, then it continues to make changes to the array. Now maybe I pause execution somehow by using a co-routine or the debug library.

Then later can I simply restore the array contents from the file and then resume the interpretor, and the interpretor would continue executing from the saved point? And I could repeat this as many times as I want maybe after changing a value of a variable using the debug library :) ?

Thanks,
Abhijit


On Fri, Nov 1, 2019 at 4:15 PM Luiz Henrique de Figueiredo <[hidden email]> wrote:
> Is it possible to save the Lua VM state and restore it?

In principle, it can be done since the Lua API accepts a memory
allocator. Like you've said, it is tricky to handle userdata.

I've experimented with LPSM to make a Lua state persistent across
invocations of the interpreter. It worked reasonably well for a while.
The LPSM library is available at http://freshmeat.net/projects/lpsm/
but it does not seem to work on modern systems. I'd love to hear about
a modern replacement.

Reply | Threaded
Open this post in threaded view
|

Re: Time Travel Debugging

Viacheslav Usov
On Sat, Nov 2, 2019 at 2:10 PM Abhijit Nandy <[hidden email]> wrote:

> If I comment out:  memcpy(backupBuffer, buffer, 4096)    then things are fine.

No they are not. Your buffers are too small and you have a buffer overrun in any case. However, with the memcpy in place, it modifies the the content of the overrun memory so you see the problem earlier.

Another problem is that you call luaL_newstate() and then lua_setallocf(). The first call will already have allocated some memory using the default allocator. This is benign for now, but may bite you in future. Use lua_newstate() instead.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Time Travel Debugging

Abhijit Nandy
Thank you Viacheslav :)
Yes, I will implement the free() part as well to keep the memory under control.

Works now with output:  1 
Need to try out non-trivial cases now.

#include <string>

extern "C" {
    #include <lua.h>
    #include <lauxlib.h>
    #include <lualib.h>
}

char buffer[1048576];
char backupBuffer[1048576];
char *pTop = buffer;

void* customLuaAlloc(void* userData, void* ptr, size_t oldSize, size_t newSize)
{
    if (newSize == 0)
    {
        //free(ptr);
        return 0;
    }
    else {
        //return realloc(ptr, newSize);
        if (ptr != NULL) {
            // Copy contents
            memcpy(pTop, ptr, oldSize);
        }
        ptr = NULL;
        char *pOldTop = pTop;
        pTop += newSize;
        return (void*)pOldTop;
    }
}

int main()
{
    lua_State* L = lua_newstate(customLuaAlloc, NULL);

    luaL_openlibs(L);

    luaL_dostring(L, "a = 1");
    memcpy(backupBuffer, buffer, 1048576);

    luaL_dostring(L, "a = 2");

    memcpy(buffer, backupBuffer, 1048576);
    luaL_dostring(L, "print(a)");

    lua_close(L);
}


Thanks,
Abhijit

On Sat, Nov 2, 2019 at 11:57 PM Viacheslav Usov <[hidden email]> wrote:
On Sat, Nov 2, 2019 at 2:10 PM Abhijit Nandy <[hidden email]> wrote:

> If I comment out:  memcpy(backupBuffer, buffer, 4096)    then things are fine.

No they are not. Your buffers are too small and you have a buffer overrun in any case. However, with the memcpy in place, it modifies the the content of the overrun memory so you see the problem earlier.

Another problem is that you call luaL_newstate() and then lua_setallocf(). The first call will already have allocated some memory using the default allocator. This is benign for now, but may bite you in future. Use lua_newstate() instead.

Cheers,
V.
Reply | Threaded
Open this post in threaded view
|

Re: Time Travel Debugging

Francisco Olarte
On Sun, Nov 3, 2019 at 8:36 PM Abhijit Nandy <[hidden email]> wrote:

As you are using C++:
> extern "C" {

You could, very easily, make a class for the allocator, with a save
function, it would probably save you some headaches.

Something like... ( using a couple things from std, assuming c++11 or
better )...

UNTESTED!

class lua_allocator {
   unique_ptr<char> buffer;
   char * base;
   char * top;
   char * end;
   lua_allocator(int size)
     : buf_holder(new char[size])
     , base(buffer.get())
     , top(base)
     , end(base+size)
  {}
  // Writing inline, you may need to define them outlines, depending
on the standard used.

  // Reflect to a method for easy debugging.
  static void * lua_alloc(void* userData, void* ptr, size_t oldSize,
size_t newSize)  {
    return static_cast<lua_allocator>(userData)->lua_alloc(ptr,
oldSize, newSize);
  }
  // Condensing a bit...
  void * lua_alloc (void* ptr, size_t oldSize, size_t newSize) {
    if (newSize == 0) {
        //free(ptr);
        return 0;
    }  else {
        //return realloc(ptr, newSize);
      if (ptr != NULL) {
    if (newSize>oldSize) { // Bigger.
      // Copy contents
      if (newSize > end - top)
        return nullptr;
      auto res = top;
      memcpy(top, ptr, oldSize);
      top += newSize;
      return res;
    } else {
      // Shrinking or equal.
      return ptr;
    }
      } else { // Ptr == null
    if (newSize > end - top)
      return nullptr;
    auto res = top;
    top += newSize;
    return res;
      }
    }
  }
  // Now you have it packed, you have it easier to do things like:
  void save_to(vector<char> & holder) {
    holder.resize(top-base);
    copy(base, top, holder.begin()); // Or memcpy to holder.data()
  }
  bool restore_from(const vector char & holder) {
    auto l = holder.size();
    if (l>end-base) {
      return false; // Bad save vector.
    }
    copy(holder.begin(), holder.end(), base);
    top = base + l;
    returnm true.
  }
  // And even do things like this to avoid typos
  luaState * new_state() {
    return lua_newstate(lua_alloc, this);
  }
  // The thing is C++ gives you some minimum overhead ways to save/restore it,
  // and packing things let's you experiment easily.
  // i.e., once you have this you can experiment on doing things like
delta-coding the
  // copys so you can easily go back to different places in your program.

};

Francisco Olarte.