Memory limits

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory limits

Mark Pulford
Hi,

I've recently been toying with Lua for use in a system where 3rd
parties can submit scripts to be executed. I need to be able to catch
scripts that accidentally run away. CPU usage can be monitored through
HOOKCOUNT (which works for my purposes), however there doesn't appear
to be any way to limit memory usage easily under the interpreter.

I've attached a simple patch which creates the function
"lua_setmemlimit()".

Perhaps this patch or something like it could be integrated into the
main Lua distro?

While I'm here I'd also like to thank the authors for creating Lua :).
Its been a long time since I've been this impressed by some software.

Now I just need to find more time to work with it ;).

Regards,
Mark
diff -urN lua-5.0/include/lua.h lua-5.0-mem/include/lua.h
--- lua-5.0/include/lua.h	Tue Mar 18 23:01:39 2003
+++ lua-5.0-mem/include/lua.h	Mon Oct  6 19:09:35 2003
@@ -104,6 +104,7 @@
 LUA_API lua_State *lua_newthread (lua_State *L);
 
 LUA_API lua_CFunction lua_atpanic (lua_State *L, lua_CFunction panicf);
+LUA_API unsigned long lua_setmemlimit (lua_State *L, unsigned long limit);
 
 
 /*
diff -urN lua-5.0/src/lapi.c lua-5.0-mem/src/lapi.c
--- lua-5.0/src/lapi.c	Tue Apr  8 00:06:08 2003
+++ lua-5.0-mem/src/lapi.c	Mon Oct  6 19:13:19 2003
@@ -135,6 +135,16 @@
 }
 
 
+LUA_API unsigned long lua_setmemlimit(lua_State *L, unsigned long limit) {
+  unsigned long old;
+  lua_lock(L);
+  old = G(L)->maxblocks;
+  G(L)->maxblocks = limit;
+  lua_unlock(L);
+  return old;
+}
+
+
 LUA_API lua_State *lua_newthread (lua_State *L) {
   lua_State *L1;
   lua_lock(L);
diff -urN lua-5.0/src/lmem.c lua-5.0-mem/src/lmem.c
--- lua-5.0/src/lmem.c	Thu Dec  5 04:08:31 2002
+++ lua-5.0-mem/src/lmem.c	Wed Oct  8 00:03:45 2003
@@ -73,6 +73,8 @@
   }
   else if (size >= MAX_SIZET)
     luaG_runerror(L, "memory allocation error: block too big");
+  else if (L && G(L)->nblocks - oldsize + size > G(L)->maxblocks)
+    luaD_throw(L, LUA_ERRMEM);
   else {
     block = l_realloc(block, oldsize, size);
     if (block == NULL) {
diff -urN lua-5.0/src/lstate.c lua-5.0-mem/src/lstate.c
--- lua-5.0/src/lstate.c	Thu Apr  3 23:05:34 2003
+++ lua-5.0-mem/src/lstate.c	Mon Oct  6 19:12:57 2003
@@ -108,6 +108,7 @@
   setnilvalue(gval(g->dummynode));
   g->dummynode->next = NULL;
   g->nblocks = sizeof(lua_State) + sizeof(global_State);
+  g->maxblocks = MAX_SIZET;
   stack_init(L, L);  /* init stack */
   /* create default meta table with a dummy table, and then close the loop */
   defaultmeta(L)->tt = LUA_TTABLE;
diff -urN lua-5.0/src/lstate.h lua-5.0-mem/src/lstate.h
--- lua-5.0/src/lstate.h	Thu Feb 27 22:22:30 2003
+++ lua-5.0-mem/src/lstate.h	Mon Oct  6 18:43:50 2003
@@ -116,6 +116,7 @@
   Mbuffer buff;  /* temporary buffer for string concatentation */
   lu_mem GCthreshold;
   lu_mem nblocks;  /* number of `bytes' currently allocated */
+  lu_mem maxblocks;  /* maximum `bytes' that can be allocated */
   lua_CFunction panic;  /* to be called in unprotected errors */
   TObject _registry;
   TObject _defaultmeta;
Reply | Threaded
Open this post in threaded view
|

Re: Memory limits

Luiz Henrique de Figueiredo
>CPU usage can be monitored through HOOKCOUNT (which works for my
>purposes), however there doesn't appear to be any way to limit memory
>usage easily under the interpreter.

I see two ways of doing this without patching Lua:

1. In your count (or other) hook, call lua_getgccount and raise an error
   if too much memory is being used.

2. Recompile lmem.c to use your own memory allocator that refuses to allocate
   too much memory.

--lhf

Reply | Threaded
Open this post in threaded view
|

RE: Memory limits

Virgil Smith
In reply to this post by Mark Pulford
Script "security" seems to be a typical issue.

This just came up as "I want to be able to safely share the base libraries
between scripts", and frequently comes up from the other side of the fence
as "I want to sandbox individual scripts", and can even come up as
complaints about the time needed to initialize a Lua state (as you can't
trust scripts to leave the libraries and other globals unmolested).

I have to admit that I very quickly came to feel competent enough with Lua
to figure out the work arounds to these issues (Lua's just so darn well laid
out ;), but I never trust that whatever time I spend thinking about security
"what ifs" is sufficient, so.....

It would be very nice if some people donated their ideas and someone
compiled them into a simple "issues to consider / recommended practices
document".  This may already exist on the Wiki, but it would be very nice to
see this boiled down and either included with the manual as an appendix or
placed right along side the manual on the Lua sight. (um so what do you
think about that idea LHF et al?)



-----Original Message-----
From: [hidden email]
[[hidden email] Behalf Of Mark Pulford
Sent: Thursday, October 09, 2003 9:33 AM
To: [hidden email]
Subject: Memory limits


Hi,

I've recently been toying with Lua for use in a system where 3rd
parties can submit scripts to be executed. I need to be able to catch
scripts that accidentally run away. CPU usage can be monitored through
HOOKCOUNT (which works for my purposes), however there doesn't appear
to be any way to limit memory usage easily under the interpreter.

I've attached a simple patch which creates the function
"lua_setmemlimit()".

Perhaps this patch or something like it could be integrated into the
main Lua distro?

While I'm here I'd also like to thank the authors for creating Lua :).
Its been a long time since I've been this impressed by some software.

Now I just need to find more time to work with it ;).

Regards,
Mark


Reply | Threaded
Open this post in threaded view
|

A problem implementing standard oop construction, which shares C++ and Lua classes

Dmitriy Iassenev
In reply to this post by Mark Pulford
hi all,

1. I have a manager class, which deals with the instances of some interface
class
2. I exported this interface class to Lua
3. I derived a class in Lua from this interface and defined needed virtual
functions
4. When manager class calls interface method, it doesn't work properly since
manager class surely doesn't see Lua classes in the virtual table

Is there any other algorithm to implement such a construction?

Many thanks in advance,
Dmitriy Iassenev

P.S. I use LuaBind


Reply | Threaded
Open this post in threaded view
|

RE: Security (was RE: Memory limits)

Kevin Baca-2
In reply to this post by Virgil Smith
I haven't seen anything like this on the Wiki, but I am very interested
to see what others have done, so I took the liberty of adding a page:

http://lua-users.org/wiki/ScriptSecurity

and an article on making tables read-only.

I'm eager to see the techniques others are using to make their scripts
more robust and secure.

-Kevin

> 
> Script "security" seems to be a typical issue.
> 
> This just came up as "I want to be able to safely share the 
> base libraries between scripts", and frequently comes up from 
> the other side of the fence as "I want to sandbox individual 
> scripts", and can even come up as complaints about the time 
> needed to initialize a Lua state (as you can't trust scripts 
> to leave the libraries and other globals unmolested).
> 
> I have to admit that I very quickly came to feel competent 
> enough with Lua to figure out the work arounds to these 
> issues (Lua's just so darn well laid out ;), but I never 
> trust that whatever time I spend thinking about security 
> "what ifs" is sufficient, so.....
> 
> It would be very nice if some people donated their ideas and 
> someone compiled them into a simple "issues to consider / 
> recommended practices document".  This may already exist on 
> the Wiki, but it would be very nice to see this boiled down 
> and either included with the manual as an appendix or placed 
> right along side the manual on the Lua sight. (um so what do 
> you think about that idea LHF et al?)
> 


Reply | Threaded
Open this post in threaded view
|

Re: Memory limits

Taj Khattra-2
In reply to this post by Mark Pulford
On Fri, Oct 10, 2003 at 12:03:02AM +0930, Mark Pulford wrote:
> I've recently been toying with Lua for use in a system where 3rd
> parties can submit scripts to be executed. I need to be able to catch
> scripts that accidentally run away. CPU usage can be monitored through
> HOOKCOUNT (which works for my purposes), however there doesn't appear
> to be any way to limit memory usage easily under the interpreter.

if you run each 3rd party script in a separate process, then you may
be able to use your operating system's resource limits api (if it has
one) as a coarse sandbox without having to modify lua.

e.g. under unix, if i want to limit a lua script to approx 8MB of
address space and approx 30 seconds of cpu time, i do

	% softlimit -a 8000000 -t 30 lua ...

softlimit is at http://cr.yp.to/daemontools/softlimit.html.  it uses
setrlimit() to set resource limits before running a program.  of course,
you could also call setrlimit() directly (either in C, or via exporting
it to lua).

this doesn't give you as fine-grained control as modifying the
interpreter, using HOOKCOUNT etc., but it may be "good enough" if it
fits your application model.

-taj

Reply | Threaded
Open this post in threaded view
|

Re: Memory limits

Roberto Ierusalimschy
In reply to this post by Mark Pulford
Another option is to use a GC metamethod. Everytime the memory use
crosses a threshold Lua runs the garbage collector, which calls the __gc
metamethods for collectable userdata. This is a perfect time to check
for memory overflow.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: A problem implementing standard oop construction, which shares C++ and Lua classes

Daniel Wallin-2
In reply to this post by Dmitriy Iassenev
At 17:39 2003-10-09, Dmitriy Iassenev wrote:
hi all,

1. I have a manager class, which deals with the instances of some interface
class
2. I exported this interface class to Lua
3. I derived a class in Lua from this interface and defined needed virtual
functions
4. When manager class calls interface method, it doesn't work properly since
manager class surely doesn't see Lua classes in the virtual table

Is there any other algorithm to implement such a construction?

Yes, there is. Inheritance from C++ classes is demonstrated in the
luabind docs. Did you create a wrapper type for the C++ class so that
virtual function dispatch will work correctly?

You'll have to show me some code of what you are doing to get more help.

Many thanks in advance,
Dmitriy Iassenev

P.S. I use LuaBind

BTW. luabind has it's own mailing list:

  http://lists.sourceforge.net/lists/listinfo/luabind-user.

Please redirect future questions there.

---
Daniel Wallin


Reply | Threaded
Open this post in threaded view
|

Re: Memory limits

Mark Pulford
In reply to this post by Luiz Henrique de Figueiredo
On Thu, Oct 09, 2003 at 11:39:45AM -0300, Luiz Henrique de Figueiredo wrote:
> I see two ways of doing this without patching Lua:
> 
> 1. In your count (or other) hook, call lua_getgccount and raise an error
>    if too much memory is being used.

I considered this, but if I track memory usage as well the hook needs
to be called every few instructions (instead of once for CPU limiting)
and maintain its own instruction count in the registry. This just
seems to complicate things and give a performance hit.

> 2. Recompile lmem.c to use your own memory allocator that refuses to allocate
>    too much memory.

I need to limit memory on a per lua_State basis. My own allocator
wouldn't easily know which lua_State it was called from unless lmem.c
was patched. Since lua_State already tracks memory usage the most
elegant way to provide memory limit functionality appears to be
doing the check in lmem.c.

Regards,
Mark

-- 
Mark Pulford <[hidden email]>    I'd give my right arm to be ambidextrous

Reply | Threaded
Open this post in threaded view
|

Re: A problem implementing standard oop construction, which shares C++ and Lua classes

Dmitriy Iassenev
In reply to this post by Daniel Wallin-2
hi Daniel,

> Yes, there is. Inheritance from C++ classes is demonstrated in the
> luabind docs. Did you create a wrapper type for the C++ class so that
> virtual function dispatch will work correctly?

oh, right you are, thank you for attention

> You'll have to show me some code of what you are doing to get more help.

thank you, I'll try to do it by myself first

> >P.S. I use LuaBind
> BTW. luabind has it's own mailing list:
>    http://lists.sourceforge.net/lists/listinfo/luabind-user.
> Please redirect future questions there.
thank you again

Best regards,
Dmitriy Iassenev