CPU/Memory Limiting

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

CPU/Memory Limiting

TongKe Xue
Hey Everyone,

  I'm working on a massive simulation project (like worldforge). I need to
be able to allow users to script objects in the world. I am thinking of
ussing Ruby as my scripting language of choice. However:

  (1) I need to be able to limit the memory usage of the object.
  (2) I need to be able to limit the CPU cycle usage of the object.

  I.e. I don't want users to use my game to calculate digits of PI.

  Is there a way to let me create these light-weight tasklets/threadlets/processlets (there's too many objects to have a UNIX process for each object) such that I can limit the memory/cpu usage on each?


Thanks,

Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Irayo
TongKe Xue wrote:

ussing Ruby as my scripting language of choice.
Might try mailing the Ruby list then... :P

  (1) I need to be able to limit the memory usage of the object.
  (2) I need to be able to limit the CPU cycle usage of the object.

This can be done fairly simply. Try a Google search for 'Lua sandbox'. For a simple implementation, checking the os.clock(); function result in a debug hook function will tell you how long a script has been running, and then you can decide whether to terminate it. collectgarbage("count") will tell you how much memory (in kilobytes) a script is using. In C, setting a memory alloc function will allow you full control over memory and this would be better and more effective. CPU is more difficult, but if you use threads (might check LuaTask, LuaThreads, etc. but I don't have experience with these), then each script will be given (roughly) equal processing time so it won't matter too much anyway.

--
Irayo


Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Leonardo Maciel-2
To address (2) you can also have a coroutine for each object and write your own coroutine scheduler that keeps track of the cpu time taken by each coroutine.

 Sat, 07 Jul 2007 20:39:10 -0500, Irayo <[hidden email]> escreveu:

> TongKe Xue wrote:
> >
> > ussing Ruby as my scripting language of choice.
> Might try mailing the Ruby list then... :P
> >
> >   (1) I need to be able to limit the memory usage of the object.
> >   (2) I need to be able to limit the CPU cycle usage of the object.
> >
> This can be done fairly simply.  Try a Google search for 'Lua sandbox'.  
> For a simple implementation, checking the os.clock(); function result in 
> a debug hook function will tell you how long a script has been running, 
> and then you can decide whether to terminate it.  
> collectgarbage("count") will tell you how much memory (in kilobytes) a 
> script is using.  In C, setting a memory alloc function will allow you 
> full control over memory and this would be better and more effective.  
> CPU is more difficult, but if you use threads (might check LuaTask, 
> LuaThreads, etc. but I don't have experience with these), then each 
> script will be given (roughly) equal processing time so it won't matter 
> too much anyway.
> 
> -- 
> Irayo
> 
> 
> 
> 

Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Ketmar Dark-2
In reply to this post by TongKe Xue
On Sat, 7 Jul 2007 17:24:06 -0700
"TongKe Xue" <[hidden email]> wrote:

> I am thinking of ussing Ruby as my scripting language of choice.
we are glad to know this. but this is mailing list about Lua... %-)

Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Asko Kauppi
In reply to this post by TongKe Xue

Lua Lanes allows you to give specific OS level priorities to the Lua states you are running, and/or aborting them completely if necessary. It kind of helps you do the kinds of sandboxing that Lua is good for, without going "inside".

http://luaforge.net/projects/lanes/

The tips Irayo gave seem very valid, though, and using debug features + os.clock() is maybe enough for your tasking needs?

If running potentially on multicore machines, you might consider Lua Lanes for fully utilizing them, though. In an easy way.

-asko


TongKe Xue kirjoitti 8.7.2007 kello 3:24:

Hey Everyone,

I'm working on a massive simulation project (like worldforge). I need to be able to allow users to script objects in the world. I am thinking of
ussing Ruby as my scripting language of choice. However:

  (1) I need to be able to limit the memory usage of the object.
  (2) I need to be able to limit the CPU cycle usage of the object.

  I.e. I don't want users to use my game to calculate digits of PI.

Is there a way to let me create these light-weight tasklets/ threadlets/processlets (there's too many objects to have a UNIX process for each object) such that I can limit the memory/cpu usage on each?


Thanks,



Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Eike Decker-2
In reply to this post by TongKe Xue
Hi

(1) Could be done using the C api in combination with a special allocator
function that tracks the memory use of the running coroutines. A simple query
function could return the current memory usage
(2) CPU cycles can be watched using the debug interfaces - it allows to call a
hook function (either C or Lua) in case of a certain condition, i.e. after a
certain number of instructions. However, this might slow down Lua. For
Competions etc. however, this behavior would be quite fair, as each user can
only use a limited amount of instructions.

Of course you can always limit the functions that the user can use by modifying
the environments of the the functions using setfenv. 

Eike

> Hey Everyone,
> 
>   I'm working on a massive simulation project (like worldforge). I need to
> be able to allow users to script objects in the world. I am thinking of
> ussing Ruby as my scripting language of choice. However:
> 
>   (1) I need to be able to limit the memory usage of the object.
>   (2) I need to be able to limit the CPU cycle usage of the object.
> 
>   I.e. I don't want users to use my game to calculate digits of PI.
> 
>   Is there a way to let me create these light-weight
> tasklets/threadlets/processlets (there's too many objects to have a UNIX
> process for each object) such that I can limit the memory/cpu usage on each?
> 
> 
> Thanks,
> 




Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Ross Berteig
In reply to this post by TongKe Xue
Asko,

Sometime on Sun, 8 Jul 2007, Asko wrote:
>Lua Lanes allows you to give specific OS level priorities to the
>Lua states you are running, and/or aborting them completely if
>necessary. It kind of helps you do the kinds of sandboxing that
>Lua is good for, without going "inside".

I noticed a while back, and was reminded by your post today that
there is something wrong with your project's configuration at
Luaforge. At (http://luaforge.net/projects/lanes/) there is a
link to the project home page (http://lanes.luaforge.net/).
Attempting to follow that link, firefox says:

   "You have chosen to open
	<<<big blank space here>>>
    which is a: application/x-httpd-php ..."

rather than displaying a useful page. I'm guessing that luaforge
isn't configured right to support PHP pages?

Of course, given that index.php (saved to disk and inspected)
does not contain strings like either lua or lanes might indicate
that there is something else going on entirely ;-)


Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Asko Kauppi

There is no project home page, although LuaForge seems to have come up with the link. I will remove it, thanks for informing.

Important pages are the Files tab (http://luaforge.net/frs/? group_id=265) and the manual page, also included in the tgz (http:// luaforge.net/docman/index.php?group_id=265&selected_doc_group_id=292)

-asko


Ross Berteig kirjoitti 9.7.2007 kello 1:27:

Asko,

Sometime on Sun, 8 Jul 2007, Asko wrote:
>Lua Lanes allows you to give specific OS level priorities to the
>Lua states you are running, and/or aborting them completely if
>necessary. It kind of helps you do the kinds of sandboxing that
>Lua is good for, without going "inside".

I noticed a while back, and was reminded by your post today that
there is something wrong with your project's configuration at
Luaforge. At (http://luaforge.net/projects/lanes/) there is a
link to the project home page (http://lanes.luaforge.net/).
Attempting to follow that link, firefox says:

   "You have chosen to open
	<<<big blank space here>>>
    which is a: application/x-httpd-php ..."

rather than displaying a useful page. I'm guessing that luaforge
isn't configured right to support PHP pages?

Of course, given that index.php (saved to disk and inspected)
does not contain strings like either lua or lanes might indicate
that there is something else going on entirely ;-)



Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

gary ng
I see that lanes is in one way similar to rings in
that each thread would have a seperate lua_State(where
I assume it can also load modules of its own), though
it can run in parallel.

My experience with rings however has one critical
issue(which is not actually related to rings but lua
and certain C library like the mysql bindings) in that
if I load the mysql library inside a ring, there is no
way to properly clean up the memory used inside that
ring(not necessary because of lua but more likely
mysql).

At the moment, my work around for it is to load these
offending C libraries modules inside the main
thread(or the hosting environment of rings) even it is
not used. Then I can start and stop rings multiple
times without seeing the RAM usage going up.

--- Asko Kauppi <[hidden email]> wrote:

> 
> There is no project home page, although LuaForge
> seems to have come  
> up with the link.  I will remove it, thanks for
> informing.
> 
> Important pages are the Files tab
> (http://luaforge.net/frs/? 
> group_id=265) and the manual page, also included in
> the tgz (http:// 
>
luaforge.net/docman/index.php?group_id=265&selected_doc_group_id=292)
> 



       
____________________________________________________________________________________
Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
http://farechase.yahoo.com/

Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Asko Kauppi

It is true that some addon modules may have issues when used in multithreaded scenarios. I believe this is mostly just, since this usage hasn't been so wide-spread in the past, and will be gradually fixed by the module authors now that multicore is becoming a norm (at least in the x86 world).

The way Lua Lanes helps make modules multithreaded capable is it protects calls to 'require' so that only one happens at a time. This allows the module author to do things s.a. "did we get this initialized already" checks without resorting to critical sections.

More information at the Lanes manual:

http://kotisivu.dnainternet.net/askok/bin/lanes/

-asko


gary ng kirjoitti 9.7.2007 kello 6:02:

I see that lanes is in one way similar to rings in
that each thread would have a seperate lua_State(where
I assume it can also load modules of its own), though
it can run in parallel.

My experience with rings however has one critical
issue(which is not actually related to rings but lua
and certain C library like the mysql bindings) in that
if I load the mysql library inside a ring, there is no
way to properly clean up the memory used inside that
ring(not necessary because of lua but more likely
mysql).

At the moment, my work around for it is to load these
offending C libraries modules inside the main
thread(or the hosting environment of rings) even it is
not used. Then I can start and stop rings multiple
times without seeing the RAM usage going up.

--- Asko Kauppi <[hidden email]> wrote:


There is no project home page, although LuaForge
seems to have come
up with the link.  I will remove it, thanks for
informing.

Important pages are the Files tab
(http://luaforge.net/frs/?
group_id=265) and the manual page, also included in
the tgz (http://

luaforge.net/docman/index.php?group_id=265&selected_doc_group_id=292)





______________________________________________________________________ ______________ Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
http://farechase.yahoo.com/


Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

gary ng
--- Asko Kauppi <[hidden email]> wrote:
> The way Lua Lanes helps make modules multithreaded
> capable is it protects calls to 'require' so that 
> only one happens at a time. This  allows the module 
> author to do things s.a. "did we get this  
> initialized already" checks without resorting to
> critical sections.
> 
I believe we are talking about different things. What
I was talking about is the scenario of :

single process, multiple lua_State where each state
has its own 'require' of some modules(.so). And
strictly speaking it has nothing to do with the module
author or critical sections but more about how how the
.so is loaded. It seems to be some problem deeper than
that. So even each module make the proper calls to
free the .so things(as far as I can trace to for
example the luasql-mysql), there is something prevents
the lua_State from properly clean up.

This usually is not an issue for other language where
when we are talking about multi-threading, it is
usually more like sharing single VM between threads.

The net result is that if  I am not aware of this, a
few hundred/thousand(depends on memory configuration)
thread creation/termination would result in out of
memory situation.

It is just a "user beware" about this "single process,
multiple lua_State" contineous usage, not in
particular to lua lanes or rings.



       
____________________________________________________________________________________
Got a little couch potato? 
Check out fun summer activities for kids.
http://search.yahoo.com/search?fr=oni_on_mail&p=summer+activities+for+kids&cs=bz 

Reply | Threaded
Open this post in threaded view
|

Re: CPU/Memory Limiting

Philippe Lhoste
In reply to this post by TongKe Xue
On 08/07/2007 02:24, TongKe Xue wrote:
be able to allow users to script objects in the world. I am thinking of
ussing Ruby as my scripting language of choice. However:

Uh? Wrong mailing list or slip of the tongue? :P

--
Philippe Lhoste
--  (near) Paris -- France
--  http://Phi.Lho.free.fr
--  --  --  --  --  --  --  --  --  --  --  --  --  --