Is LPeg production ready yet?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Is LPeg production ready yet?

Han Sen
First of all, many thanks to roberto for the great jobs on LPeg and LPeg is a pretty awesome project :)

I want to use it to implement a DSL parser, but have not find out the status of this project yet.
Is LPeg production ready now (v1.0) ? Or have some friends already used it in production environment and how is it going?

Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Sean Conner
It was thus said that the Great Han Sen once stated:
> First of all, many thanks to roberto for the great jobs on LPeg and LPeg is
> a pretty awesome project :)
>
> I want to use it to implement a DSL parser, but have not find out the
> status of this project yet.
> Is LPeg production ready now (v1.0) ? Or have some friends already used it
> in production environment and how is it going?

  I'm using it in production right now (parsing SIP messages [1] in real
time) and there's been no problem with it.

  -spc (In fact, we have more problems with the stuff written in C than in
        Lua/LPeg ...)

[1] Cell phones.


Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Luiz Henrique de Figueiredo
In reply to this post by Han Sen
> I want to use it to implement a DSL parser

It'd help if you could share a sample of how your DSL looks.

Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Jakub Jirutka
Hi,

I’ve implemented complete parser of POSIX Shell Command Language in LPeg [1]. The project is not finished yet, so not used anywhere in production. However, it’s quite complex grammar with complicated rules and I haven’t run into any issue with LPeg yet, even when running it on large shell scripts. So I’d not worry about quality of LPeg, it’s really awesome library.

Jakub

[1]: https://github.com/jirutka/sh-parser

> On 13. Jun 2017, at 3:15, Luiz Henrique de Figueiredo <[hidden email]> wrote:
>
>> I want to use it to implement a DSL parser
>
> It'd help if you could share a sample of how your DSL looks.
>


Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Han Sen
Many thanks to Sean and Jakub :-)

For your kindly information, it gives a lot of encouragement. 

And I also noticed that MoonScript and lua-parser also used LPeg to implement their parser.

On Tue, Jun 13, 2017 at 9:48 PM, Jakub Jirutka <[hidden email]> wrote:
Hi,

I’ve implemented complete parser of POSIX Shell Command Language in LPeg [1]. The project is not finished yet, so not used anywhere in production. However, it’s quite complex grammar with complicated rules and I haven’t run into any issue with LPeg yet, even when running it on large shell scripts. So I’d not worry about quality of LPeg, it’s really awesome library.

Jakub

[1]: https://github.com/jirutka/sh-parser

> On 13. Jun 2017, at 3:15, Luiz Henrique de Figueiredo <[hidden email]> wrote:
>
>> I want to use it to implement a DSL parser
>
> It'd help if you could share a sample of how your DSL looks.
>



Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Charles Heywood
Yep. I don't think MoonScript has had too many issues with version incompatibly either, and is used quite a bit in production.

On Tue, Jun 13, 2017, 15:50 Sen Han <[hidden email]> wrote:
Many thanks to Sean and Jakub :-)

For your kindly information, it gives a lot of encouragement. 

And I also noticed that MoonScript and lua-parser also used LPeg to implement their parser.

On Tue, Jun 13, 2017 at 9:48 PM, Jakub Jirutka <[hidden email]> wrote:
Hi,

I’ve implemented complete parser of POSIX Shell Command Language in LPeg [1]. The project is not finished yet, so not used anywhere in production. However, it’s quite complex grammar with complicated rules and I haven’t run into any issue with LPeg yet, even when running it on large shell scripts. So I’d not worry about quality of LPeg, it’s really awesome library.

Jakub

[1]: https://github.com/jirutka/sh-parser

> On 13. Jun 2017, at 3:15, Luiz Henrique de Figueiredo <[hidden email]> wrote:
>
>> I want to use it to implement a DSL parser
>
> It'd help if you could share a sample of how your DSL looks.
>



--
--

Software Developer / System Administrator
Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Han Sen
In reply to this post by Luiz Henrique de Figueiredo
>> I want to use it to implement a DSL parser

> It'd help if you could share a sample of how your DSL looks.

Hi lhf, the DSL I want to parse is just a subset of lua language, 
and the goal is to implement a source-to-source translator.
Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Luiz Henrique de Figueiredo
> the DSL I want to parse is just a subset of lua language,
> and the goal is to implement a source-to-source translator.

 From Lua to what?

Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Parke
In reply to this post by Han Sen
On Tue, Jun 13, 2017 at 1:59 PM, Sen Han <[hidden email]> wrote:
> the DSL I want to parse is just a subset of lua language,
> and the goal is to implement a source-to-source translator.

One of my hobby projects has been creating an alternative syntax for
Lua.  I use LPeg to transpile from my custom syntax into Lua.

I started by getting LPeg to parse the entire Lua grammar, and then I
started making adjustments to the grammar and syntax.  Many
adjustments can be made with substitution captures.  I also use table
captures, so that LPeg outputs an abstract syntax tree.  I can then
make additional modifications to the tree, and then flatten the tree
to a string of Lua source code.  Comments are preserved.  Line numbers
are preserved.  Source code layout and formatting is preserved as much
as is possible given the inherent differences between my language and
Lua.  Many of the changes I am making are cosmetic.  My language is
much closer to Lua than is Moonscript, for example.

Your goal (parsing "just a subset of lua") should be simpler than
mine, possibly much simpler.

Problems you may encounter:

1.  The Lua grammar is left-recursive.  LPeg cannot parse
left-recursive grammars, so I needed to refactor Lua's grammar to
remove the left recursion.  If you have never refactored a grammar
before, you might find it challenging.

Aside:  There are known techniques to extend PEG parsers so that they
can handle left-recursion.  Here is a fork of LPeg that may do
left-recursion for you.  I have not tried to use it.
https://github.com/sacek/LPeg

2.  My grammar also uses back-references.  Back references (in the re
module) are implemented via LPeg back-captures in combination with
LPeg match-time captures that call a Lua function which may create a
unique string.  Recently, I approximately tripled the number of
back-references in the grammar, and performance dropped by roughly a
factor of 3.  This leads me to suspect that my heavy use of
back-references is the primary bottleneck in my parser.  I have been
pondering implementing back-references directly in LPeg to avoid the
overhead of the match-time capture, the call to Lua, and (for failed
matches) the likely creation of a string in Lua.  I would, of course,
have to extend the C language LPeg source code to implement
back-references inside LPeg itself.

Aside:  If anyone has any tips for profiling LPeg patterns, I would be
happy to hear them.  It would be nice to confirm that my suspicions
are correct before investing time and energy modifying the LPeg C
language source code.

3.  At one point, I needed to increase LPeg's internal stack limit.
This is trivially done via lpeg.setmaxstack, but I did have to guess
as to what value to use, as the default value was not documented at
that time.  (The docs now say the default is 400.)

In general, LPeg has worked very well and has performed as documented
without crashes.  Perhaps best of all, working with LPeg has been fun!

Cheers,

Parke

Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Charles Heywood
In reply to this post by Luiz Henrique de Figueiredo
I think that, because of "a subset of the Lua language", it's like `tokenf` in that it just barely modifies Lua syntax.

On Tue, Jun 13, 2017 at 6:19 PM Luiz Henrique de Figueiredo <[hidden email]> wrote:
> the DSL I want to parse is just a subset of lua language,
> and the goal is to implement a source-to-source translator.

 From Lua to what?

--
--

Software Developer / System Administrator
Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Han Sen
In reply to this post by Luiz Henrique de Figueiredo
>> the DSL I want to parse is just a subset of lua language,
>> and the goal is to implement a source-to-source translator.

> From Lua to what?

From 'lua'(DSL) to lua honestly :p

The reason of that first sight may peculiar goal is because:

    This DSL which I want to implement has the totally different 
    computation model with lua's original one. I have to manipulate 
    and reconstruct the AST(such as insert some codes at each 
    function call and some other more complex transformations) and 
    regenerate the new AST's corresponding lua codes. 


On Wed, Jun 14, 2017 at 7:19 AM, Luiz Henrique de Figueiredo <[hidden email]> wrote:
> the DSL I want to parse is just a subset of lua language,
> and the goal is to implement a source-to-source translator.

 From Lua to what?


Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Han Sen
In reply to this post by Parke
Thanks Parke for your so detailed help :D

> Your goal (parsing "just a subset of lua") should be simpler than
> mine, possibly much simpler.

Yes, it is indeed much easier :-)

> The Lua grammar is left-recursive.  LPeg cannot parse
> left-recursive grammars, so I needed to refactor Lua's grammar to
> remove the left recursion.  If you have never refactored a grammar
> before, you might find it challenging.

Actually, I decide just to modify some open source projects' parser which 
had this enjoyable but effort-costing job already done to fulfill my goal quicker. 
And I already found some good implementations and materials:


What is your opinion? ( And please let me know if you have some advices :)

Thank you so much and all best.

On Wed, Jun 14, 2017 at 7:23 AM, Parke <[hidden email]> wrote:
On Tue, Jun 13, 2017 at 1:59 PM, Sen Han <[hidden email]> wrote:
> the DSL I want to parse is just a subset of lua language,
> and the goal is to implement a source-to-source translator.

One of my hobby projects has been creating an alternative syntax for
Lua.  I use LPeg to transpile from my custom syntax into Lua.

I started by getting LPeg to parse the entire Lua grammar, and then I
started making adjustments to the grammar and syntax.  Many
adjustments can be made with substitution captures.  I also use table
captures, so that LPeg outputs an abstract syntax tree.  I can then
make additional modifications to the tree, and then flatten the tree
to a string of Lua source code.  Comments are preserved.  Line numbers
are preserved.  Source code layout and formatting is preserved as much
as is possible given the inherent differences between my language and
Lua.  Many of the changes I am making are cosmetic.  My language is
much closer to Lua than is Moonscript, for example.

Your goal (parsing "just a subset of lua") should be simpler than
mine, possibly much simpler.

Problems you may encounter:

1.  The Lua grammar is left-recursive.  LPeg cannot parse
left-recursive grammars, so I needed to refactor Lua's grammar to
remove the left recursion.  If you have never refactored a grammar
before, you might find it challenging.

Aside:  There are known techniques to extend PEG parsers so that they
can handle left-recursion.  Here is a fork of LPeg that may do
left-recursion for you.  I have not tried to use it.
https://github.com/sacek/LPeg

2.  My grammar also uses back-references.  Back references (in the re
module) are implemented via LPeg back-captures in combination with
LPeg match-time captures that call a Lua function which may create a
unique string.  Recently, I approximately tripled the number of
back-references in the grammar, and performance dropped by roughly a
factor of 3.  This leads me to suspect that my heavy use of
back-references is the primary bottleneck in my parser.  I have been
pondering implementing back-references directly in LPeg to avoid the
overhead of the match-time capture, the call to Lua, and (for failed
matches) the likely creation of a string in Lua.  I would, of course,
have to extend the C language LPeg source code to implement
back-references inside LPeg itself.

Aside:  If anyone has any tips for profiling LPeg patterns, I would be
happy to hear them.  It would be nice to confirm that my suspicions
are correct before investing time and energy modifying the LPeg C
language source code.

3.  At one point, I needed to increase LPeg's internal stack limit.
This is trivially done via lpeg.setmaxstack, but I did have to guess
as to what value to use, as the default value was not documented at
that time.  (The docs now say the default is 400.)

In general, LPeg has worked very well and has performed as documented
without crashes.  Perhaps best of all, working with LPeg has been fun!

Cheers,

Parke


Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Han Sen
In reply to this post by Luiz Henrique de Figueiredo
Hi lhf, a suddenly idea appears to me. As we already had some lua functions 
like "loadstring" which could compile the lua codes to byte codes directly 
on the fly. But why don't we also provide some c level functions further like:

    load_lua_code_to_ast: return corresponding ast table (the ast described by
    linked lua tables, and users could modify it as their wishes)

    translate_ast_to_lua_code: translate the ast table to lua source code again

    execute_ast: execute the ast directly

Of course some other library already done the equivalent job, but I think this job 
should be done by lua itself as many nowadays language provide their own parser 
which could parse themself to certain AST(like golang) and it would be more stable 
and performant.

We could say:

    Lua: The code is data, the data is code (: we could go even further than lisp :)

On Wed, Jun 14, 2017 at 7:19 AM, Luiz Henrique de Figueiredo <[hidden email]> wrote:
> the DSL I want to parse is just a subset of lua language,
> and the goal is to implement a source-to-source translator.

 From Lua to what?


Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Luiz Henrique de Figueiredo
> But why don't we also provide some c level functions further like:
>
>     *load_lua_code_to_ast*: return corresponding ast table (the ast
> described by
>     linked lua tables, and users could modify it as their wishes)
>
>     *translate_ast_to_lua_code*: translate the ast table to lua source code
> again
>
>     *execute_ast*: execute the ast directly

Because the Lua parser does not build a AST.

Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Parke
In reply to this post by Han Sen
On Wed, Jun 14, 2017 at 7:43 AM, Sen Han <[hidden email]> wrote:
> Actually, I decide just to modify some open source projects' parser which
> had this enjoyable but effort-costing job already done to fulfill my goal
> quicker.

> And I already found some good implementations and materials:
>
>     https://github.com/andremm/lua-parser
>     http://lua-users.org/wiki/MetaLuaAbstractSyntaxTree
>     http://lua-users.org/wiki/LpegRecipes
>     http://lua-users.org/wiki/LuaFish
>
> What is your opinion? ( And please let me know if you have some advices :)

If they work acceptably, then use them.  I have no other advice.

Cheers,

Parke

Reply | Threaded
Open this post in threaded view
|

Re: Is LPeg production ready yet?

Francisco Sant'anna-2
In reply to this post by Han Sen


On Tue, Jun 13, 2017 at 5:50 PM, Sen Han <[hidden email]> wrote:
Many thanks to Sean and Jakub :-)

For your kindly information, it gives a lot of encouragement. 

And I also noticed that MoonScript and lua-parser also used LPeg to implement their parser.

Céu also uses LPeg: www.ceu-lang.org