Fully Lua YAML library?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Fully Lua YAML library?

D. Matt Placek

Does anybody know if a YAML library implemented entirely in lua?  Everything I've found is a lua binding to C code, which is tolerable but I'd prefer a pure lua implementation if one is available.

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Pierre Chapuis
March 11, 2017 7:32 PM, "D. Matt Placek" <[hidden email]> wrote:
> Does anybody know if a YAML library implemented entirely in lua? Everything I've found is a lua
> binding to C code, which is tolerable but I'd prefer a pure lua implementation if one is available.

I don't think there is one. I looked some years ago, because I wanted one for haricot [1],
but I could not find any. The problem (and the reason why I didn't write my own) is probably
that YAML is (unlike what it looks) a terribly complicated [2] format, and parsing it correctly
is hard [3]. If you have a choice I would suggest you avoid using YAML. Otherwise, lyaml [4] is
probably the best binding.

[1] https://github.com/catwell/haricot
[2] http://yaml.org/spec/1.2/spec.html
[3] https://github.com/cblp/yaml-sucks
[4] https://github.com/gvvaughan/lyaml

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Tony Papadimitriou
In reply to this post by D. Matt Placek
Unfortunately, one major downside of Lua (vs Python, for usual example) is the lack of libraries written in the language itself.
 
Pretty much everything out there (and it’s not all that much, anyway) seems to be written in C as if those whose use Lua enjoy writing C more than writing Lua!
There is, of course, the C speed advantage, but somehow the Python community – unlike the Lua one – seems  to favor libraries written in Python, and as a result they have created an enormous ‘ecosystem’ for pretty much anything – an achievement which in itself is hard to beat.
(We shouldn’t overlook that in today’s world, a language’s popularity and longevity is largely proportional to the amount of publically available source code written in it.)
 
I can only hope the Lua community would start doing the same at some point – but I think, unfortunately, both I and Lua will die hoping.
 
Don’t get me wrong.  I do prefer Lua over Python as a language (or I wouldn’t be here), but the Python library advantage is too big to ignore for many tasks when Lua proves inadequate in that regard.
 
To get back to your request, maybe you’d be interested to clone PyYAML (or some other – many choices) from Python, and thus add one little stone to the (faint) pure Lua library ‘dogma’ by sharing it. :)
 
Sent: Saturday, March 11, 2017 8:31 PM
Subject: Fully Lua YAML library?
 

Does anybody know if a YAML library implemented entirely in lua?  Everything I've found is a lua binding to C code, which is tolerable but I'd prefer a pure lua implementation if one is available.

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

D. Matt Placek

On Sat, Mar 11, 2017 at 5:26 PM, Tony Papadimitriou <[hidden email]> wrote:
Unfortunately, one major downside of Lua (vs Python, for usual example) is the lack of libraries written in the language itself. 
Pretty much everything out there (and it’s not all that much, anyway) seems to be written in C as if those whose use Lua enjoy writing C more than writing Lua!

I do think partly that's because of the ease of interfacing Lua with C/C++ and the frequency that lua is used embedded in either a C/C++ environment.  If you already have the C environment and a C library, it's often quicker to write a binding to an existing C library than to write a pure Lua implementation from scratch.  But it does add extra complexity to have a dependency on building a C extension if you're just looking for something to use quickly and "off the shelf".
 
To get back to your request, maybe you’d be interested to clone PyYAML (or some other – many choices) from Python, and thus add one little stone to the (faint) pure Lua library ‘dogma’ by sharing it. :)

I think Pierre was right that part of the problem is the complexity of the YAML spec and it may just be more than I need at this point.  YAML's not a hard requirement for me, but I'd like a way to embed structured data in free-form text that's a bit more human friendly than JSON.  I'd probably look at other formats - HOCON or HJSON or OGDL or something over YAML if I'm going to implement it from scratch.

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Coda Highland
On Sat, Mar 11, 2017 at 3:32 PM, D. Matt Placek <[hidden email]> wrote:

>
> On Sat, Mar 11, 2017 at 5:26 PM, Tony Papadimitriou <[hidden email]> wrote:
>>
>> Unfortunately, one major downside of Lua (vs Python, for usual example) is
>> the lack of libraries written in the language itself.
>> Pretty much everything out there (and it’s not all that much, anyway)
>> seems to be written in C as if those whose use Lua enjoy writing C more than
>> writing Lua!
>
>
> I do think partly that's because of the ease of interfacing Lua with C/C++
> and the frequency that lua is used embedded in either a C/C++ environment.
> If you already have the C environment and a C library, it's often quicker to
> write a binding to an existing C library than to write a pure Lua
> implementation from scratch.  But it does add extra complexity to have a
> dependency on building a C extension if you're just looking for something to
> use quickly and "off the shelf".
>
>>
>> To get back to your request, maybe you’d be interested to clone PyYAML (or
>> some other – many choices) from Python, and thus add one little stone to the
>> (faint) pure Lua library ‘dogma’ by sharing it. :)
>
>
> I think Pierre was right that part of the problem is the complexity of the
> YAML spec and it may just be more than I need at this point.  YAML's not a
> hard requirement for me, but I'd like a way to embed structured data in
> free-form text that's a bit more human friendly than JSON.  I'd probably
> look at other formats - HOCON or HJSON or OGDL or something over YAML if I'm
> going to implement it from scratch.
>

Why not use Lua source code? That's actually one of the original
design concepts of the language.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

D. Matt Placek


On Mar 11, 2017 6:45 PM, "Coda Highland" <[hidden email]> wrote:
> Why not use Lua source code? That's actually one of the original
> design concepts of the language.

That's what I'm leaning towards right now.  I like YAML slightly better for the application, but not so much it's worth adding a new C library dependency, or writing a buggy halfway implementation of a subset of yaml in lua.

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Daurnimator
In reply to this post by Tony Papadimitriou
On 12 March 2017 at 09:26, Tony Papadimitriou <[hidden email]> wrote:

> Unfortunately, one major downside of Lua (vs Python, for usual example) is
> the lack of libraries written in the language itself.
>
> Pretty much everything out there (and it’s not all that much, anyway) seems
> to be written in C as if those whose use Lua enjoy writing C more than
> writing Lua!
> There is, of course, the C speed advantage, but somehow the Python community
> – unlike the Lua one – seems  to favor libraries written in Python, and as a
> result they have created an enormous ‘ecosystem’ for pretty much anything –
> an achievement which in itself is hard to beat.
> (We shouldn’t overlook that in today’s world, a language’s popularity and
> longevity is largely proportional to the amount of publically available
> source code written in it.)
>
> I can only hope the Lua community would start doing the same at some point –
> but I think, unfortunately, both I and Lua will die hoping.


Why would you want all lua libraries to be written in lua?
Leveraging existing libraries, whether they be written in C, fortran,
rust or whatever is a great choice.
Using developer time to reimplement things in another language "just
because" is a huge waste of time (and money).

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Soni "They/Them" L.


On 11/03/17 11:46 PM, Daurnimator wrote:

> On 12 March 2017 at 09:26, Tony Papadimitriou <[hidden email]> wrote:
>> Unfortunately, one major downside of Lua (vs Python, for usual example) is
>> the lack of libraries written in the language itself.
>>
>> Pretty much everything out there (and it’s not all that much, anyway) seems
>> to be written in C as if those whose use Lua enjoy writing C more than
>> writing Lua!
>> There is, of course, the C speed advantage, but somehow the Python community
>> – unlike the Lua one – seems  to favor libraries written in Python, and as a
>> result they have created an enormous ‘ecosystem’ for pretty much anything –
>> an achievement which in itself is hard to beat.
>> (We shouldn’t overlook that in today’s world, a language’s popularity and
>> longevity is largely proportional to the amount of publically available
>> source code written in it.)
>>
>> I can only hope the Lua community would start doing the same at some point –
>> but I think, unfortunately, both I and Lua will die hoping.
>
> Why would you want all lua libraries to be written in lua?
> Leveraging existing libraries, whether they be written in C, fortran,
> rust or whatever is a great choice.
> Using developer time to reimplement things in another language "just
> because" is a huge waste of time (and money).
>

I want most Lua libraries to be written in Lua because I use sandboxed
environments a lot. Having the libraries written in Lua seems to help
with that.

--
Disclaimer: these emails may be made public at any given time, with or without reason. If you don't agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Thijs Schreijer
In reply to this post by D. Matt Placek

On 12 Mar 2017, at 00:55, D. Matt Placek <[hidden email]> wrote:


On Mar 11, 2017 6:45 PM, "Coda Highland" <[hidden email]> wrote:
> Why not use Lua source code? That's actually one of the original
> design concepts of the language.

That's what I'm leaning towards right now.  I like YAML slightly better for the application, but not so much it's worth adding a new C library dependency, or writing a buggy halfway implementation of a subset of yaml in lua.

my 2cts: stay away from YAML. Especially if users are supposed to edit it for settings etc. small (easily overlooked) changes in formatting will break it, and you’ll have a support problem in teaching all those users how to write proper YAML. Don’t go there.

Have a look at LuaRocks rockspec files for example on how you could leverage Lua for configuration files. You might even steal some code there.

Thijs
Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Sean Conner
It was thus said that the Great Thijs Schreijer once stated:
>
> my 2cts: stay away from YAML. Especially if users are supposed to edit it
> for settings etc. small (easily overlooked) changes in formatting will
> break it, and you’ll have a support problem in teaching all those users
> how to write proper YAML. Don’t go there.
>
> Have a look at LuaRocks rockspec files for example on how you could
> leverage Lua for configuration files. You might even steal some code
> there.

  What's there to steal?  

        CONF = {}
        f = loadfile("config.lua","t",CONF)
        f()

  And CONF contains the contents of the configuration file.  In C, this
would be:

        lua_createtable(L,0,0);
        lua_setglobal(L,"CONF");

        rc = luaL_loadfile(L,"config.lua","t");
        if (rc == LUA_OK)
        {
          lua_getglobal(L,"CONF");
          lua_setupvalue(L,-2,1);
          rc = lua_pcall(L,0,0,0); /* we just loaded our config */
        }

  -spc (BTW:  The code is for Lua 5.3)


Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Pierre Chapuis
In reply to this post by Soni "They/Them" L.
March 12, 2017 4:12 AM, "Soni L." <[hidden email]> wrote:

> I want most Lua libraries to be written in Lua because I use sandboxed environments a lot. Having
> the libraries written in Lua seems to help with that.

I prefer pure Lua libraries too, when possible, for two reasons.

The first one is portability to sandboxed environments, or to environments where the C compiler is very
limited, or to environments where hot code reloading is necessary...

The second one is for teaching / human accessibility purpose. I want some developers who do not know C
code to be able to read and understand the source code of the libraries.

Finally, pure Lua libraries can be faster with LuaJIT sometimes. I don't really care about that
since I use 5.3 myself though.

The best solution is probably to have modules which support both: a pure Lua implementation and a compatible,
faster C implementation with the same API, which can be used if available. That's what I did with my
lua-chacha module, for instance: https://github.com/catwell/lua-chacha

I actually have a semi-stealth project to write a collection of pure Lua 5.3 libraries for network protocols,
file formats and modern cryptographic schemes. I do not intend to include YAML in it for now, though.

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

D. Matt Placek
In reply to this post by Daurnimator
On Sat, Mar 11, 2017 at 9:46 PM, Daurnimator <[hidden email]> wrote:
Why would you want all lua libraries to be written in lua?
Leveraging existing libraries, whether they be written in C, fortran,
rust or whatever is a great choice.
Using developer time to reimplement things in another language "just
because" is a huge waste of time (and money).

I don't want them all written in lua, just the ones that I'd like to use :)

Yes, re-implementing can be a waste of time, but if I have a choice between something with a dependency that requires a separate package in another language and something that doesn't I'll often prefer something that doesn't because my applications are cross platform and I haven't been entirely happy with the ecosystem for getting packages compiled, especially in a Windows environment.  Maybe luarocks or luadist have gotten better in the last few years but the last time I used them it was either not straightforward or impossible to get the native modules that I wanted installed through those tools.  In the calculus of "which package can I get started with right away" the answer is almost always the pure Lua package, and the native extension always involves some amount of additional frustration just getting the darn thing built, and that always happens again the next time you move your app to a new environment.  It adds complexity to your build and installation.

So there is a lot of value in having a pure Lua module, for me at least.  In general I'd say that if you are already working in a C/C++ environment and there's not a pre-existing module, it's often easier to make your own binding to a C/C++ library because of the ease of interfacing Lua and C/C++.  That's usually my first inclination when there's no existing Lua package.

But if I'm looking at different options for an off-the-shelve solution, I'll prefer the pure lua module everytime unless e.g., there's a reason to think the native package may offer a critical performance advantage or something.  In terms of the complexity of using the component a pure Lua implementation always wins on build and packaging requirements.  

I don't think I'm alone in not being entirely happy with the ecosystem for native extensions.  Compared to ruby gems or something it's a much less consistent experience.  Regardless of what the reasons are for that, it certainly factors into the preference for a pure Lua implementation vs. a compiled extension.
Reply | Threaded
Open this post in threaded view
|

Re: Fully Lua YAML library?

Dirk Laurie-2
2017-03-12 20:16 GMT+02:00 D. Matt Placek <[hidden email]>:
> On Sat, Mar 11, 2017 at 9:46 PM, Daurnimator <[hidden email]> wrote:
>>

> I don't think I'm alone in not being entirely happy with the ecosystem for
> native extensions.  Compared to ruby gems or something it's a much less
> consistent experience.  Regardless of what the reasons are for that, it
> certainly factors into the preference for a pure Lua implementation vs. a
> compiled extension.

Lua is designed for the following typical situation:

   1. Big complicated C program wants to provide customization:
   default setting of user options, user-supplied extra features etc,
   2. Instead of inventing a new DSL for that purpose, Lua is provided,
   preferably by including all its source code rather than relying on
   a user's own installation.

As done by e.g. Love2D (via callbacks) and LuaTeX (by allowing direct
Lua instructions).

Lua is also pretty good for little standalone utilities that extend the
experience of command-line programming. Shebang programs,

Lua is not particularly brilliant at taking over from Bash as one's main
scripting language for system applications. It is not in the running with
Ruby and Python as the final solution for those who want to do what
Perl can but just can't come to grips with Perl.

Unfortunately the Hammer Principlie makes us want to use Lua for
everything. And it sure is a Swiss Army hammer. But it's still a hammer.