Lua 5.1-5.3 serialization library

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Lua 5.1-5.3 serialization library

Timm S. Mueller
Hi,

before I'm starting to write my own, I'm looking for a serialization
and deserialization library. Requirements:

- must cover Lua 5.1, 5.2, 5.3 and LuaJIT
- in the MIT/BSD/CC/PD license department
- must cover cyclic references and binary data
- preferrably written in C for performance, Lua would be fine also
- should only use official Lua APIs, no outside dependencies, no state
internals
- The transport format should be legal Lua, but this isn't strictly a
requirement and may be sacrificed if the result can be transported,
preferrably also between different architectures and Lua versions. I'm
not objecting to using JSON format either.

Not required: Userdata, metatables, functions and upvalues. A small
footprint and the ability to operate on a writer/readers would be nice.
Any ideas, recommendations?

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Dirk Laurie-2
2015-01-14 14:51 GMT+02:00 Timm S. Mueller <[hidden email]>:

> a serialization and deserialization library. Requirements:
>
> - must cover Lua 5.1, 5.2, 5.3 and LuaJIT

You mean, Lua 5.3's string.pack/unpack would be taboo?

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Timm S. Mueller
On Wed, 14 Jan 2015 15:03:49 +0200
Dirk Laurie <[hidden email]> wrote:

> 2015-01-14 14:51 GMT+02:00 Timm S. Mueller <[hidden email]>:
>
> > a serialization and deserialization library. Requirements:
> >
> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
>
> You mean, Lua 5.3's string.pack/unpack would be taboo?

No. What I mean is that it must cover Lua 5.1, 5.2, 5.3 and LuaJIT.

- Timm

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Valerio
In reply to this post by Timm S. Mueller


On Wed, Jan 14, 2015 at 1:51 PM, Timm S. Mueller <[hidden email]> wrote:
Hi,

before I'm starting to write my own, I'm looking for a serialization
and deserialization library. Requirements:

- must cover Lua 5.1, 5.2, 5.3 and LuaJIT
...
- preferrably written in C for performance, Lua would be fine also

Is it even possible to have it cover Lua 5.1/2/3 and LuaJIT and being written in C ? 

In the pure-Lua department, we implemented this:

Hope it helps,
Valerio
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Timm S. Mueller
On Wed, 14 Jan 2015 14:23:44 +0100
Valerio Schiavoni <[hidden email]> wrote:

> On Wed, Jan 14, 2015 at 1:51 PM, Timm S. Mueller <
> [hidden email]> wrote:
>
> > Hi,
> >
> > before I'm starting to write my own, I'm looking for a serialization
> > and deserialization library. Requirements:
> >
> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> > ...
> > - preferrably written in C for performance, Lua would be fine also
> >
>
> Is it even possible to have it cover Lua 5.1/2/3 and LuaJIT and being
> written in C ?

Yes, by having checks for LUA_VERSION_NUM in the code.

> In the pure-Lua department, we implemented this:
> https://github.com/splay-project/splay/blob/lua52/src/daemon/modules/splay/benc.lua

> Hope it helps,
> Valerio

Thank you, but it's GPL. This is ruled out.

- Timm

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/

Reply | Threaded
Open this post in threaded view
|

RE: Lua 5.1-5.3 serialization library

James Urso-2
In reply to this post by Timm S. Mueller
> > >On Wed, 14 Jan 2015 15:03:49 +0200
> > >Dirk Laurie <[hidden email]> wrote:

> > 2015-01-14 14:51 GMT+02:00 Timm S. Mueller <[hidden email]>:
> >
> > > a serialization and deserialization library. Requirements:
> > >
> > > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> >
> > You mean, Lua 5.3's string.pack/unpack would be taboo?

> No. What I mean is that it must cover Lua 5.1, 5.2, 5.3 and LuaJIT.

> - Timm

We used Luiz' lpack library http://webserver2.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack for a Lua 5.1 project and when I recently upgraded the project to Lua 5.3, it continued to work fine with no modifications.  I haven't used LuaJIT, so I can't speak for that.  

Based upon the pack / unpack functionality in Lua 5.3, I'm guessing that it is based upon Luiz' original library.

- Jim

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Luiz Henrique de Figueiredo
> Based upon the pack / unpack functionality in Lua 5.3, I'm guessing that it is based upon Luiz' original library.

The pack/unpack functions in 5.3 are based mainly on Roberto's struct library,
but there was some overlap.

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Timm S. Mueller
In reply to this post by James Urso-2
On Wed, 14 Jan 2015 13:32:59 +0000
James Urso <[hidden email]> wrote:

> > > >On Wed, 14 Jan 2015 15:03:49 +0200
> > > >Dirk Laurie <[hidden email]> wrote:
>
> > > 2015-01-14 14:51 GMT+02:00 Timm S. Mueller <[hidden email]>:
> > >
> > > > a serialization and deserialization library. Requirements:
> > > >
> > > > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> > >
> > > You mean, Lua 5.3's string.pack/unpack would be taboo?
>
> > No. What I mean is that it must cover Lua 5.1, 5.2, 5.3 and LuaJIT.
>
> > - Timm
>
> We used Luiz' lpack library http://webserver2.tecgraf.puc-rio.br/~lhf/ftp/lua/#lpack for a Lua 5.1 project and when I recently upgraded the project to Lua 5.3, it continued to work fine with no modifications.  I haven't used LuaJIT, so I can't speak for that.  
>
> Based upon the pack / unpack functionality in Lua 5.3, I'm guessing that it is based upon Luiz' original library.

Thank you, but this one doesn't seem to handle tables:

$ make
a=bpack({hello="world"},"Ab8","\027Lua",5*16+1,0,1,4,4,4,8,0)
test.lua:11: bad argument #1 to 'bpack' (string expected, got table)

Sorry, I did not mention tables (and recursion) exeplicitely, although
I meant to imply this by cyclic references.

Also I would have problems with reliance on binary data representation
between different Lua versions and architectures. This doesn't rule out
a solution entirely for now, but in the medium to long term.

Addendum to the requirements list:
- Must handle tables recursively

- Timm

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Philipp Janda
In reply to this post by Timm S. Mueller
Am 14.01.2015 um 13:51 schröbte Timm S. Mueller:
> Hi,

Hi!

>
> before I'm starting to write my own, I'm looking for a serialization
> and deserialization library. Requirements:
>
> - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> - in the MIT/BSD/CC/PD license department
> - must cover cyclic references and binary data
> - preferrably written in C for performance, Lua would be fine also
> - should only use official Lua APIs, no outside dependencies, no state
> internals
> - The transport format should be legal Lua, but this isn't strictly a
> requirement and may be sacrificed if the result can be transported,
> preferrably also between different architectures and Lua versions. I'm
> not objecting to using JSON format either.
>
> Not required: Userdata, metatables, functions and upvalues. A small
> footprint and the ability to operate on a writer/readers would be nice.
> Any ideas, recommendations?
>

Have you ruled out those[1]?

Philipp

   [1]: http://lua-users.org/wiki/TableSerialization




Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Ignacio Burgueño-2

On Wed, Jan 14, 2015 at 12:08 PM, Philipp Janda <[hidden email]> wrote:

Have you ruled out those[1]?

Philipp

  [1]: http://lua-users.org/wiki/TableSerialization




Right, I think that serpent in particular may suit your needs:

 


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Timm S. Mueller
In reply to this post by Philipp Janda
On Wed, 14 Jan 2015 15:08:31 +0100
Philipp Janda <[hidden email]> wrote:

> Am 14.01.2015 um 13:51 schröbte Timm S. Mueller:
> > Hi,
>
> Hi!
>
> >
> > before I'm starting to write my own, I'm looking for a serialization
> > and deserialization library. Requirements:
> >
> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> > - in the MIT/BSD/CC/PD license department
> > - must cover cyclic references and binary data
> > - preferrably written in C for performance, Lua would be fine also
> > - should only use official Lua APIs, no outside dependencies, no state
> > internals
> > - The transport format should be legal Lua, but this isn't strictly a
> > requirement and may be sacrificed if the result can be transported,
> > preferrably also between different architectures and Lua versions. I'm
> > not objecting to using JSON format either.
> >
> > Not required: Userdata, metatables, functions and upvalues. A small
> > footprint and the ability to operate on a writer/readers would be nice.
> > Any ideas, recommendations?
> >
>
> Have you ruled out those[1]?

Not all, I'm still working through that list, but I found not a single
mention of 5.2, and 5.3 was only recently released. Many entries depend
on old versions, do write to a file, miss handling for cyclic
references. Eris and Pluto are ruled out for version specificy and
looking into state internals. Data Dumper and Serpent are close to my
requirements, but there may be others which I am not aware of or
achieve the same in C or use JSON as a transport format, for example.

> Philipp
>
>    [1]: http://lua-users.org/wiki/TableSerialization

- Timm

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Patrick Donnelly
In reply to this post by Timm S. Mueller
Hi Timm,

On Wed, Jan 14, 2015 at 7:51 AM, Timm S. Mueller
<[hidden email]> wrote:

> Hi,
>
> before I'm starting to write my own, I'm looking for a serialization
> and deserialization library. Requirements:
>
> - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> - in the MIT/BSD/CC/PD license department
> - must cover cyclic references and binary data
> - preferrably written in C for performance, Lua would be fine also
> - should only use official Lua APIs, no outside dependencies, no state
> internals
> - The transport format should be legal Lua, but this isn't strictly a
> requirement and may be sacrificed if the result can be transported,
> preferrably also between different architectures and Lua versions. I'm
> not objecting to using JSON format either.

I just uploaded a serialization library [1] I wrote a few years ago
for an IRC bot. It's pure Lua and handles all of your requirements,
except perhaps 5.2, 5.3, and LuaJIT. I haven't tested on those
platforms. The output is Lua code.

> Not required: Userdata, metatables, functions and upvalues. A small
> footprint and the ability to operate on a writer/readers would be nice.
> Any ideas, recommendations?

It also completely handles metatables and Lua functions with shared upvalues.

[1] https://github.com/batrick/lua-serialize

--
Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Hisham
In reply to this post by Timm S. Mueller
On 14 January 2015 at 12:34, Timm S. Mueller
<[hidden email]> wrote:

> On Wed, 14 Jan 2015 15:08:31 +0100
> Philipp Janda <[hidden email]> wrote:
>
>> Am 14.01.2015 um 13:51 schröbte Timm S. Mueller:
>> > Hi,
>>
>> Hi!
>>
>> >
>> > before I'm starting to write my own, I'm looking for a serialization
>> > and deserialization library. Requirements:
>> >
>> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
>> > - in the MIT/BSD/CC/PD license department
>> > - must cover cyclic references and binary data
>> > - preferrably written in C for performance, Lua would be fine also
>> > - should only use official Lua APIs, no outside dependencies, no state
>> > internals
>> > - The transport format should be legal Lua, but this isn't strictly a
>> > requirement and may be sacrificed if the result can be transported,
>> > preferrably also between different architectures and Lua versions. I'm
>> > not objecting to using JSON format either.
>> >
>> > Not required: Userdata, metatables, functions and upvalues. A small
>> > footprint and the ability to operate on a writer/readers would be nice.
>> > Any ideas, recommendations?
>> >
>>
>> Have you ruled out those[1]?
>
> Not all, I'm still working through that list, but I found not a single
> mention of 5.2, and 5.3 was only recently released. Many entries depend
> on old versions, do write to a file, miss handling for cyclic
> references. Eris and Pluto are ruled out for version specificy and
> looking into state internals. Data Dumper and Serpent are close to my
> requirements, but there may be others which I am not aware of or
> achieve the same in C or use JSON as a transport format, for example.

Make sure to let us know the results of your research, and once you
pick your favorite, mark it as "endorsed" at:

https://lua-toolbox.com/

In the "serialization" tag, Serpent is currently the most endorsed:

https://lua-toolbox.com/label/9

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Timm S. Mueller
In reply to this post by Patrick Donnelly
On Wed, 14 Jan 2015 10:34:29 -0500
Patrick Donnelly <[hidden email]> wrote:

> Hi Timm,
>
> On Wed, Jan 14, 2015 at 7:51 AM, Timm S. Mueller
> <[hidden email]> wrote:
> > Hi,
> >
> > before I'm starting to write my own, I'm looking for a serialization
> > and deserialization library. Requirements:
> >
> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> > - in the MIT/BSD/CC/PD license department
> > - must cover cyclic references and binary data
> > - preferrably written in C for performance, Lua would be fine also
> > - should only use official Lua APIs, no outside dependencies, no state
> > internals
> > - The transport format should be legal Lua, but this isn't strictly a
> > requirement and may be sacrificed if the result can be transported,
> > preferrably also between different architectures and Lua versions. I'm
> > not objecting to using JSON format either.
>
> I just uploaded a serialization library [1] I wrote a few years ago
> for an IRC bot. It's pure Lua and handles all of your requirements,
> except perhaps 5.2, 5.3, and LuaJIT. I haven't tested on those
> platforms. The output is Lua code.

Thanks, added to my testbed. (although it's GPL... :-)

> > Not required: Userdata, metatables, functions and upvalues. A small
> > footprint and the ability to operate on a writer/readers would be nice.
> > Any ideas, recommendations?
>
> It also completely handles metatables and Lua functions with shared upvalues.
>
> [1] https://github.com/batrick/lua-serialize

- Timm

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Timm S. Mueller
In reply to this post by Hisham
On Wed, 14 Jan 2015 13:36:00 -0200
Hisham <[hidden email]> wrote:

> On 14 January 2015 at 12:34, Timm S. Mueller
> <[hidden email]> wrote:
> > On Wed, 14 Jan 2015 15:08:31 +0100
> > Philipp Janda <[hidden email]> wrote:
> >
> >> Am 14.01.2015 um 13:51 schröbte Timm S. Mueller:
> >> > Hi,
> >>
> >> Hi!
> >>
> >> >
> >> > before I'm starting to write my own, I'm looking for a serialization
> >> > and deserialization library. Requirements:
> >> >
> >> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> >> > - in the MIT/BSD/CC/PD license department
> >> > - must cover cyclic references and binary data
> >> > - preferrably written in C for performance, Lua would be fine also
> >> > - should only use official Lua APIs, no outside dependencies, no state
> >> > internals
> >> > - The transport format should be legal Lua, but this isn't strictly a
> >> > requirement and may be sacrificed if the result can be transported,
> >> > preferrably also between different architectures and Lua versions. I'm
> >> > not objecting to using JSON format either.
> >> >
> >> > Not required: Userdata, metatables, functions and upvalues. A small
> >> > footprint and the ability to operate on a writer/readers would be nice.
> >> > Any ideas, recommendations?
> >> >
> >>
> >> Have you ruled out those[1]?
> >
> > Not all, I'm still working through that list, but I found not a single
> > mention of 5.2, and 5.3 was only recently released. Many entries depend
> > on old versions, do write to a file, miss handling for cyclic
> > references. Eris and Pluto are ruled out for version specificy and
> > looking into state internals. Data Dumper and Serpent are close to my
> > requirements, but there may be others which I am not aware of or
> > achieve the same in C or use JSON as a transport format, for example.
>
> Make sure to let us know the results of your research, and once you
> pick your favorite, mark it as "endorsed" at:
>
> https://lua-toolbox.com/
>
> In the "serialization" tag, Serpent is currently the most endorsed:
>
> https://lua-toolbox.com/label/9

Ok, will do.

I'm through with the list [1].

My current favourites are MessagePack [2] and cmsgpack [3], because
this has the prospect of offering both a pure Lua and a C library, it's
for 5.1 to 5.3, and has excellent support in a wide variety of
languages. Need to get into testing and see if msgpack doesn't impose
unacceptable limits for me to sacrifice the readability of Lua.

[1] http://lua-users.org/wiki/TableSerialization
[2] https://github.com/fperrad/lua-MessagePack/
[3] https://github.com/antirez/lua-cmsgpack

- Timm

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Patrick Donnelly
In reply to this post by Timm S. Mueller
On Wed, Jan 14, 2015 at 10:45 AM, Timm S. Mueller
<[hidden email]> wrote:
> Thanks, added to my testbed. (although it's GPL... :-)

Whoops... I'll fix that. I meant to distribute it under the MIT license.

--
Patrick Donnelly

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Andrew Starks
In reply to this post by Timm S. Mueller


On Wednesday, January 14, 2015, Timm S. Mueller <[hidden email]> wrote:
On Wed, 14 Jan 2015 13:36:00 -0200
Hisham <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;h@hisham.hm&#39;)">h@...> wrote:

> On 14 January 2015 at 12:34, Timm S. Mueller
> <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;tmueller@schulze-mueller.de&#39;)">tmueller@...> wrote:
> > On Wed, 14 Jan 2015 15:08:31 +0100
> > Philipp Janda <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;siffiejoe@gmx.net&#39;)">siffiejoe@...> wrote:
> >
> >> Am 14.01.2015 um 13:51 schröbte Timm S. Mueller:
> >> > Hi,
> >>
> >> Hi!
> >>
> >> >
> >> > before I'm starting to write my own, I'm looking for a serialization
> >> > and deserialization library. Requirements:
> >> >
> >> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> >> > - in the MIT/BSD/CC/PD license department
> >> > - must cover cyclic references and binary data
> >> > - preferrably written in C for performance, Lua would be fine also
> >> > - should only use official Lua APIs, no outside dependencies, no state
> >> > internals
> >> > - The transport format should be legal Lua, but this isn't strictly a
> >> > requirement and may be sacrificed if the result can be transported,
> >> > preferrably also between different architectures and Lua versions. I'm
> >> > not objecting to using JSON format either.
> >> >
> >> > Not required: Userdata, metatables, functions and upvalues. A small
> >> > footprint and the ability to operate on a writer/readers would be nice.
> >> > Any ideas, recommendations?
> >> >
> >>
> >> Have you ruled out those[1]?
> >
> > Not all, I'm still working through that list, but I found not a single
> > mention of 5.2, and 5.3 was only recently released. Many entries depend
> > on old versions, do write to a file, miss handling for cyclic
> > references. Eris and Pluto are ruled out for version specificy and
> > looking into state internals. Data Dumper and Serpent are close to my
> > requirements, but there may be others which I am not aware of or
> > achieve the same in C or use JSON as a transport format, for example.
>
> Make sure to let us know the results of your research, and once you
> pick your favorite, mark it as "endorsed" at:
>
> https://lua-toolbox.com/
>
> In the "serialization" tag, Serpent is currently the most endorsed:
>
> https://lua-toolbox.com/label/9

Ok, will do.

I'm through with the list [1].

My current favourites are MessagePack [2] and cmsgpack [3], because
this has the prospect of offering both a pure Lua and a C library, it's
for 5.1 to 5.3, and has excellent support in a wide variety of
languages. Need to get into testing and see if msgpack doesn't impose
unacceptable limits for me to sacrifice the readability of Lua.

[1] http://lua-users.org/wiki/TableSerialization
[2] https://github.com/fperrad/lua-MessagePack/
[3] https://github.com/antirez/lua-cmsgpack

- Timm

--
Timm S. Mueller <<a href="javascript:;" onclick="_e(event, &#39;cvml&#39;, &#39;tmueller@schulze-mueller.de&#39;)">tmueller@...>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. +49 30 85610000, http://www.schulze-mueller.de/


Timm,

This is what we decided as well. However, JSON does not handle mixed type hashes (number and string keys). Also, I'm not sure that negative indexes are supported either.

We were willing to limit what we sent over these messages, but it was something that we had to deal with. 

-Andrew 
Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

François Perrad
In reply to this post by Timm S. Mueller
2015-01-14 16:52 GMT+01:00 Timm S. Mueller <[hidden email]>:

> On Wed, 14 Jan 2015 13:36:00 -0200
> Hisham <[hidden email]> wrote:
>
>> On 14 January 2015 at 12:34, Timm S. Mueller
>> <[hidden email]> wrote:
>> > On Wed, 14 Jan 2015 15:08:31 +0100
>> > Philipp Janda <[hidden email]> wrote:
>> >
>> >> Am 14.01.2015 um 13:51 schröbte Timm S. Mueller:
>> >> > Hi,
>> >>
>> >> Hi!
>> >>
>> >> >
>> >> > before I'm starting to write my own, I'm looking for a serialization
>> >> > and deserialization library. Requirements:
>> >> >
>> >> > - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
>> >> > - in the MIT/BSD/CC/PD license department
>> >> > - must cover cyclic references and binary data
>> >> > - preferrably written in C for performance, Lua would be fine also
>> >> > - should only use official Lua APIs, no outside dependencies, no state
>> >> > internals
>> >> > - The transport format should be legal Lua, but this isn't strictly a
>> >> > requirement and may be sacrificed if the result can be transported,
>> >> > preferrably also between different architectures and Lua versions. I'm
>> >> > not objecting to using JSON format either.
>> >> >
>> >> > Not required: Userdata, metatables, functions and upvalues. A small
>> >> > footprint and the ability to operate on a writer/readers would be nice.
>> >> > Any ideas, recommendations?
>> >> >
>> >>
>> >> Have you ruled out those[1]?
>> >
>> > Not all, I'm still working through that list, but I found not a single
>> > mention of 5.2, and 5.3 was only recently released. Many entries depend
>> > on old versions, do write to a file, miss handling for cyclic
>> > references. Eris and Pluto are ruled out for version specificy and
>> > looking into state internals. Data Dumper and Serpent are close to my
>> > requirements, but there may be others which I am not aware of or
>> > achieve the same in C or use JSON as a transport format, for example.
>>
>> Make sure to let us know the results of your research, and once you
>> pick your favorite, mark it as "endorsed" at:
>>
>> https://lua-toolbox.com/
>>
>> In the "serialization" tag, Serpent is currently the most endorsed:
>>
>> https://lua-toolbox.com/label/9
>
> Ok, will do.
>
> I'm through with the list [1].
>
> My current favourites are MessagePack [2] and cmsgpack [3], because
> this has the prospect of offering both a pure Lua and a C library, it's
> for 5.1 to 5.3, and has excellent support in a wide variety of
> languages. Need to get into testing and see if msgpack doesn't impose
> unacceptable limits for me to sacrifice the readability of Lua.
>
> [1] http://lua-users.org/wiki/TableSerialization
> [2] https://github.com/fperrad/lua-MessagePack/
> [3] https://github.com/antirez/lua-cmsgpack
>

MsgPack cannot handle data with cyclic reference.
This is a limitation from the specification, not from a specific implementation.

François

> - Timm
>
> --
> Timm S. Mueller <[hidden email]>
> Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
> Gesellschafter: Franciska Schulze, Timm S. Mueller,
> Tel. +49 30 85610000, http://www.schulze-mueller.de/
>

Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Tim Channon
In reply to this post by Timm S. Mueller
On 14/01/2015 12:51, Timm S. Mueller wrote:

> Hi,
>
> before I'm starting to write my own, I'm looking for a serialization
> and deserialization library. Requirements:
>
> - must cover Lua 5.1, 5.2, 5.3 and LuaJIT
> - in the MIT/BSD/CC/PD license department
> - must cover cyclic references and binary data
> - preferrably written in C for performance, Lua would be fine also
> - should only use official Lua APIs, no outside dependencies, no state
> internals
> - The transport format should be legal Lua, but this isn't strictly a
> requirement and may be sacrificed if the result can be transported,
> preferrably also between different architectures and Lua versions. I'm
> not objecting to using JSON format either.
>
> Not required: Userdata, metatables, functions and upvalues. A small
> footprint and the ability to operate on a writer/readers would be nice.
> Any ideas, recommendations?
>

After a lot of hassle I settled on Lua Marshal which might be a starting
point.

Plain C.
MIT
No table size limit unlike almost all Lua high level based.
No funny code I've found: many of the mentioned solutions I found didn't
actually work.

5.3? No idea.
Portable? No, is marked as such, why is unclear. If this is binary
endian you have the source, fix it. Lua version, not so simple.

This handles > 100MB serialised on disk tables. Actually used for backup
storage, yes a daft thing to do but reality is this is how it ended up
and works for what I am doing.

https://github.com/richardhundt/lua-marshal


Reply | Threaded
Open this post in threaded view
|

Re: Lua 5.1-5.3 serialization library

Valerio
In reply to this post by Timm S. Mueller
You can add this to your list:


We used it as well, it's MIT-licensed, in C, tested at least with 5.1 and 5.2.
Not sure that it complies with all your requirements.

best,
Valerio


On Wed, Jan 14, 2015 at 1:51 PM, Timm S. Mueller <[hidden email]> wrote:
Hi,

before I'm starting to write my own, I'm looking for a serialization
and deserialization library. Requirements:

- must cover Lua 5.1, 5.2, 5.3 and LuaJIT
- in the MIT/BSD/CC/PD license department
- must cover cyclic references and binary data
- preferrably written in C for performance, Lua would be fine also
- should only use official Lua APIs, no outside dependencies, no state
internals
- The transport format should be legal Lua, but this isn't strictly a
requirement and may be sacrificed if the result can be transported,
preferrably also between different architectures and Lua versions. I'm
not objecting to using JSON format either.

Not required: Userdata, metatables, functions and upvalues. A small
footprint and the ability to operate on a writer/readers would be nice.
Any ideas, recommendations?

--
Timm S. Mueller <[hidden email]>
Schulze & Mueller GbR, Jungstr. 2, 10247 Berlin,
Gesellschafter: Franciska Schulze, Timm S. Mueller,
Tel. <a href="tel:%2B49%2030%2085610000" value="+493085610000">+49 30 85610000, http://www.schulze-mueller.de/


12