Ziplists

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Ziplists

Glenn edgar
Redis.io uses lua as a scripting language.  For small lists and hashes they store the entire object in a ziplist.

A ziplist might be usefull in the small processor model and for readonly tables.  At run time a ziplist read only table would require no initialization.
Reply | Threaded
Open this post in threaded view
|

Re: Ziplists

Ką Mykolas

...and the ziplist itself is? Did tar(gz) in memory container myself some time ago. I should publish it later.


On Sep 25, 2016 2:23 PM, "Glenn edgar" <[hidden email]> wrote:
Redis.io uses lua as a scripting language.  For small lists and hashes they
store the entire object in a ziplist.

A ziplist might be usefull in the small processor model and for readonly
tables.  At run time a ziplist read only table would require no
initialization.



--
View this message in context: http://lua.2524044.n2.nabble.com/Ziplists-tp7675065.html
Sent from the Lua-l mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Ziplists

Glenn edgar

Ziplist is not a tar.  It is a compact data structure where all the elements of a redis data structure are stored in a compact data structure.  The data structure is described in there documentation of saving memory.  Just google "redis ziplist".

The motivation is that the memory consumption of pointers dominates small objects.  Redis is that the redis database has to fit in Ram.

About 10 years ago I had to fit lua on set top boxes and analog devices dsp's.  I found that the average size of memory block was around 20 bytes.  I had to develop my own private memory manager to reduce the heap overhead.

The embedded lua project introduced the concept of rom tables to reduce memory consumption.  The redis ziplist concept is a way to implement rom tables.


On Sep 26, 2016 5:24 AM, "kamicc olo [via Lua]" <[hidden email]> wrote:

...and the ziplist itself is? Did tar(gz) in memory container myself some time ago. I should publish it later.


On Sep 25, 2016 2:23 PM, "Glenn edgar" <[hidden email]> wrote:
Redis.io uses lua as a scripting language.  For small lists and hashes they
store the entire object in a ziplist.

A ziplist might be usefull in the small processor model and for readonly
tables.  At run time a ziplist read only table would require no
initialization.



--
View this message in context: http://lua.2524044.n2.nabble.com/Ziplists-tp7675065.html
Sent from the Lua-l mailing list archive at Nabble.com.




If you reply to this email, your message will be added to the discussion below:
http://lua.2524044.n2.nabble.com/Ziplists-tp7675065p7675071.html
To unsubscribe from Ziplists, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Ziplists

Mason Bogue
A ziplist is an array of concatenated short memory blocks (<128 B,
i.e. unsigned char) with the length of the block used as the
delimiter, i.e.:

[0x03, f, o, o, 0x03, 0x04, q, u, u, x, 0x04, ...]

This has some dramatic advantages over a linked list for Redis'
use-case, but it can already be implemented in Lua as it stands,
although you might have to write a bit of C code to copy the output of
multiple `string.byte()` calls into a single array efficiently. I
think it's too specific to become part of the language, though.
Read-only zip-lists can also be constructed by concatenating strings,
although I think this is slow in Lua.

On Mon, Sep 26, 2016 at 7:42 AM, Glenn edgar <[hidden email]> wrote:

> Ziplist is not a tar.  It is a compact data structure where all the elements
> of a redis data structure are stored in a compact data structure.  The data
> structure is described in there documentation of saving memory.  Just google
> "redis ziplist".
>
> The motivation is that the memory consumption of pointers dominates small
> objects.  Redis is that the redis database has to fit in Ram.
>
> About 10 years ago I had to fit lua on set top boxes and analog devices
> dsp's.  I found that the average size of memory block was around 20 bytes.
> I had to develop my own private memory manager to reduce the heap overhead.
>
> The embedded lua project introduced the concept of rom tables to reduce
> memory consumption.  The redis ziplist concept is a way to implement rom
> tables.
>
>
> On Sep 26, 2016 5:24 AM, "kamicc olo [via Lua]" <[hidden email]> wrote:
>>
>> ...and the ziplist itself is? Did tar(gz) in memory container myself some
>> time ago. I should publish it later.
>>
>>
>> On Sep 25, 2016 2:23 PM, "Glenn edgar" <[hidden email]> wrote:
>>>
>>> Redis.io uses lua as a scripting language.  For small lists and hashes
>>> they
>>> store the entire object in a ziplist.
>>>
>>> A ziplist might be usefull in the small processor model and for readonly
>>> tables.  At run time a ziplist read only table would require no
>>> initialization.
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://lua.2524044.n2.nabble.com/Ziplists-tp7675065.html
>>> Sent from the Lua-l mailing list archive at Nabble.com.
>>>
>>
>>
>> ________________________________
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://lua.2524044.n2.nabble.com/Ziplists-tp7675065p7675071.html
>> To unsubscribe from Ziplists, click here.
>> NAML
>
>
> ________________________________
> View this message in context: Re: Ziplists
>
> Sent from the Lua-l mailing list archive at Nabble.com.