Looking for Allocator-Library

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for Allocator-Library

Richter, Jörg
Hi,
 
I want to optimize a program that has a lot of Lua states. If the process size gets
around 10-20G a lot of time is spend in system-malloc. I think it is because of the
sheer amount of small memory objects and the frequent allocation/deallocation
that comes with a dynamic language.
 
To reduce memory fragmentation I want to create every Lua state with an own
allocator instance. Every instance should get its memory from one or more large
mmap’ed blocks. This blocks should not be shared between different Lua states.
If one state gets closed all corresponding blocks can be freed very fast and the
process-malloc is not bothered to clean up so many small objects.
 
Now I am looking for a library that does exactly that. This means it must be able to
create different allocation-contexts with corresponding malloc/free interface.
Does anybody know of such a library?
 
- Jörg
 
 
Reply | Threaded
Open this post in threaded view
|

Re: Looking for Allocator-Library

Luiz Henrique de Figueiredo
> Now I am looking for a library that does exactly that. This means it must be able to
> create different allocation-contexts with corresponding malloc/free interface.
> Does anybody know of such a library?

I think mmalloc does what you need:
        http://www.math.utah.edu/docs/info/mmalloc_toc.html

mmalloc used to be part of gdb. See for instance
        http://www.opensource.apple.com/source/gdb/gdb-413/src/mmalloc/

Apparently it was removed in gdb 6.1 after being introduced in gdb 4.5.

Reply | Threaded
Open this post in threaded view
|

Re: Looking for Allocator-Library

Konstantin Osipov-2
In reply to this post by Richter, Jörg
* Richter, Jörg <[hidden email]> [15/04/15 13:37]:

> I want to optimize a program that has a lot of Lua states. If
> the process size gets around 10-20G a lot of time is spend in
> system-malloc. I think it is because of the
> sheer amount of small memory objects and the frequent
> allocation/deallocation that comes with a dynamic language.
>
> To reduce memory fragmentation I want to create every Lua state
> with an own allocator instance. Every instance should get its
> memory from one or more large
> mmap’ed blocks. This blocks should not be shared between
> different Lua states.  If one state gets closed all
> corresponding blocks can be freed very fast and the
> process-malloc is not bothered to clean up so many small
> objects.
>
> Now I am looking for a library that does exactly that. This
> means it must be able to create different allocation-contexts
> with corresponding malloc/free interface.  Does anybody know of
> such a library?

https://github.com/tarantool/tarantool/blob/master/src/lib/small/README

Never intended as a separate project, but is completely standalone
and does what you need. It has an option to return 32-bit pointers
on 64-bit architecture.

--
http://tarantool.org - a NoSQL database in a Lua script

Reply | Threaded
Open this post in threaded view
|

Re: Looking for Allocator-Library

Roberto Waltman-3
In reply to this post by Richter, Jörg


On 4/15/2015 5:43 AM, Richter, Jörg wrote:
...
 
Now I am looking for a library that does exactly that. This means it must be able to
create different allocation-contexts with corresponding malloc/free interface.
Does anybody know of such a library?
 
- Jörg
 
 
libarena:
http://www.25thandclement.com/~william/projects/libarena.html

-- 
Roberto Waltman