noob LPEG question

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

noob LPEG question

joy mondal
Hi !

This is the first time posting to a mailing list so I hope I do not get it wrong.

I have trying to understand LPEG to help me build a full programming language, I really like moonscript and want to build something similar, I have a very basic question involving LPEG.

https://gist.github.com/randrews/5eab368f35ab8e774433#gistcomment-2692302

(Posting it in email would make it harder to read due to not having markdown support.)

Are there other good resources like playwithlua.com for the specific purpose of building programming languages or would Andrew's series (Part 1,2,3,4) be enough.

best wishes.
Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Hugo Musso Gualandi
> Are there other good resources like playwithlua.com for the specific
> purpose of building programming languages or would Andrew's series
> (Part 1,2,3,4) be enough.

"Be enough" depends on what you are trying to do :)

One book that might be interesting for you is Bob Nystrom's "Crafting
Interpreters", which is available for free online at
http://www.craftinginterpreters.com/

For parsing specifically, it might also be interesting to learn a bit
about top-down parsing and recursive-descent-parsers.
LPeg fits in the broad category of top-down parsing, so many of the
tricks for top-down-parsing also apply to LPeg grammars. Knowing how to
create a hand-written recursive descent parser is also a very useful
skill, and it might come in handy when you try to write a larger LPeg
grammars.


Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Albert Chan
In reply to this post by joy mondal

On Aug 29, 2018, at 1:21 PM, joy mondal <[hidden email]> wrote:

I have trying to understand LPEG to help me build a full programming language, I really like moonscript and want to build something similar, I have a very basic question involving LPEG.

https://gist.github.com/randrews/5eab368f35ab8e774433#gistcomment-2692302

Regarding the gist question of what it meant that LPeg does no backtracking,
this thread might help:


The thread discuss how to turn Lua pattern "(.*)and(.*)", with backtracking
ability, into LPeg, which does not.

First post also had a link of Nick Gammon's LPeg tutorial.
Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

joy mondal
In reply to this post by Hugo Musso Gualandi
Hi Hugo and Andrew,

Thanks for the info, I am going to mull over it and see what I can do, I just wished their was more LPEG specific resources just because of how easy it is to use, compared to BISON or ANTLR.

best wishes.





On Wed, Aug 29, 2018 at 11:53 PM Hugo Musso Gualandi <[hidden email]> wrote:
> Are there other good resources like playwithlua.com for the specific
> purpose of building programming languages or would Andrew's series
> (Part 1,2,3,4) be enough.

"Be enough" depends on what you are trying to do :)

One book that might be interesting for you is Bob Nystrom's "Crafting
Interpreters", which is available for free online at
http://www.craftinginterpreters.com/

For parsing specifically, it might also be interesting to learn a bit
about top-down parsing and recursive-descent-parsers.
LPeg fits in the broad category of top-down parsing, so many of the
tricks for top-down-parsing also apply to LPeg grammars. Knowing how to
create a hand-written recursive descent parser is also a very useful
skill, and it might come in handy when you try to write a larger LPeg
grammars.


Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Sean Conner
In reply to this post by joy mondal
It was thus said that the Great joy mondal once stated:

> Hi !
>
> This is the first time posting to a mailing list so I hope I do not get it
> wrong.
>
> I have trying to understand LPEG to help me build a full programming
> language, I really like moonscript and want to build something similar, I
> have a very basic question involving LPEG.
>
> https://gist.github.com/randrews/5eab368f35ab8e774433#gistcomment-2692302
>
> (Posting it in email would make it harder to read due to not having
> markdown support.)
>
> Are there other good resources like playwithlua.com for the specific
> purpose of building programming languages or would Andrew's series (Part
> 1,2,3,4) be enough.

  You can read these two mailing list threads that go into LPeg:

        http://lua-users.org/lists/lua-l/2017-10/msg00106.html
        http://lua-users.org/lists/lua-l/2018-01/msg00298.html

  There's also several Lua modules I wrote that use LPeg:

        http://lua-users.org/lists/lua-l/2017-10/msg00126.html

  As for the particular issue you are having, it appears you want to parse
text like:

        3 * -foo + 2

Yes, LPeg can do this.  Sorry to do this, but I'm about to dump some LPeg.
This is from an assembler I wrote [1] that does deal with identifiers in
expressions.

local function knownvalue(n)
  return { value = n % 65536 , unknownpass1 = false , defined = true }
end

local WS     = S" \t"^1
local BIN    = R"01"
local OCT    = R"07"
local DEC    = R"09"
local HEX    = R("09","AF","af")
local OPAREN = P"(" * WS^-1
local CPAREN = P")" * WS^-1
local COMMA  = WS^-1 * P"," * WS^-1

local value_bin = BIN^1              / function(c) return tonumber(c, 2) end
local value_oct = OCT^1              / function(c) return tonumber(c, 8) end
local value_dec = (S"+-"^-1 * DEC^1) / function(c) return tonumber(c,10) end
local value_hex = HEX^1              / function(c) return tonumber(c,16) end
local value_id  = (id * Carg(1))
                / function(name,info)
                    if not info.symbols[name] then
                      info.symbols[name]              = {}
                      info.symbols[name].type         = 'undef'
                      info.symbols[name].value        = 0
                      info.symbols[name].linenum      = 0
                      info.symbols[name].unknownpass1 = true
                    end

                    return {
                        value        = info.symbols[name].value,
                        unknownpass1 = info.symbols[name].unknownpass1,
                        defined      = info.symbols[name].type ~= 'undef',
                    }
                  end

local value = P"$" * value_hex / knownvalue
            + P"&" * value_oct / knownvalue
            + P"%" * value_bin / knownvalue
            +        value_dec / knownvalue
            + P"*" * Carg(1)   / function(info) return knownvalue(info.PC) end
            +        value_id
            + P"-" * value_id  / function(v) v.value = -v.value return v end
            + P"~" * value_id  / function(v) v.value = -v.value - 1 return v end
            + P"'" * C(1)      / function(c) return
knownvalue(string.byte(c)) end

local term_op   = C(S"+-")  * WS^-1
local factor_op = C(S "*/") * WS^-1

local function eval(v1,op,v2)
  if     op == '+' then v1.value = v1.value + v2.value
  elseif op == '-' then v1.value = v1.value - v2.value
  elseif op == '*' then v1.value = v1.value * v2.value
  elseif op == '/' then v1.value = v1.value / v2.value end

  v1.unknownpass1 = v1.unknownpass1 or  v2.unknownpass1
  v1.defined      = v1.defined      and v2.defined

  v1.value = v1.value % 65536
  return v1
end

local expr = P {
  "exp",
  exp    = Cf(V"term"   * Cg(term_op   * V"term"  )^0,eval),
  term   = Cf(V"factor" * Cg(factor_op * V"factor")^0,eval),
  factor = value * WS^-1
         + OPAREN * V"exp" * CPAREN
}

Start at the bottom with "expr" and work your way back up.  The code is
expecting to be called like

        symbol_table = { ... }
        result = parser:match(input,1,symbol_table)

  The key part is the 'value' production, but I'm including the full bit for
context (and remember this is part of an assembler so keep that in mind).

  -spc (It's a two-pass assembler by the way ... )

[1] A Motorola 6809 Assembler in case anyone cares.

Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

joy mondal

Would it make sense to move from mailing list to something like github / gitlab ?

Its really hard to search old answers on the mailing list compared to github issues.

The way mailing lists works ... I just cannot find the right tool to make use of it efficiently.

Are you guys using the terminal or through your own mail servers ?

On Thu, Aug 30, 2018 at 1:33 AM Sean Conner <[hidden email]> wrote:
It was thus said that the Great joy mondal once stated:
> Hi !
>
> This is the first time posting to a mailing list so I hope I do not get it
> wrong.
>
> I have trying to understand LPEG to help me build a full programming
> language, I really like moonscript and want to build something similar, I
> have a very basic question involving LPEG.
>
> https://gist.github.com/randrews/5eab368f35ab8e774433#gistcomment-2692302
>
> (Posting it in email would make it harder to read due to not having
> markdown support.)
>
> Are there other good resources like playwithlua.com for the specific
> purpose of building programming languages or would Andrew's series (Part
> 1,2,3,4) be enough.

  You can read these two mailing list threads that go into LPeg:

        http://lua-users.org/lists/lua-l/2017-10/msg00106.html
        http://lua-users.org/lists/lua-l/2018-01/msg00298.html

  There's also several Lua modules I wrote that use LPeg:

        http://lua-users.org/lists/lua-l/2017-10/msg00126.html

  As for the particular issue you are having, it appears you want to parse
text like:

        3 * -foo + 2

Yes, LPeg can do this.  Sorry to do this, but I'm about to dump some LPeg.
This is from an assembler I wrote [1] that does deal with identifiers in
expressions.

local function knownvalue(n)
  return { value = n % 65536 , unknownpass1 = false , defined = true }
end

local WS     = S" \t"^1
local BIN    = R"01"
local OCT    = R"07"
local DEC    = R"09"
local HEX    = R("09","AF","af")
local OPAREN = P"(" * WS^-1
local CPAREN = P")" * WS^-1
local COMMA  = WS^-1 * P"," * WS^-1

local value_bin = BIN^1              / function(c) return tonumber(c, 2) end
local value_oct = OCT^1              / function(c) return tonumber(c, 8) end
local value_dec = (S"+-"^-1 * DEC^1) / function(c) return tonumber(c,10) end
local value_hex = HEX^1              / function(c) return tonumber(c,16) end
local value_id  = (id * Carg(1))
                / function(name,info)
                    if not info.symbols[name] then
                      info.symbols[name]              = {}
                      info.symbols[name].type         = 'undef'
                      info.symbols[name].value        = 0
                      info.symbols[name].linenum      = 0
                      info.symbols[name].unknownpass1 = true
                    end

                    return {
                        value        = info.symbols[name].value,
                        unknownpass1 = info.symbols[name].unknownpass1,
                        defined      = info.symbols[name].type ~= 'undef',
                    }
                  end

local value = P"$" * value_hex / knownvalue
            + P"&" * value_oct / knownvalue
            + P"%" * value_bin / knownvalue
            +        value_dec / knownvalue
            + P"*" * Carg(1)   / function(info) return knownvalue(info.PC) end
            +        value_id
            + P"-" * value_id  / function(v) v.value = -v.value return v end
            + P"~" * value_id  / function(v) v.value = -v.value - 1 return v end
            + P"'" * C(1)      / function(c) return
knownvalue(string.byte(c)) end

local term_op   = C(S"+-")  * WS^-1
local factor_op = C(S "*/") * WS^-1

local function eval(v1,op,v2)
  if     op == '+' then v1.value = v1.value + v2.value
  elseif op == '-' then v1.value = v1.value - v2.value
  elseif op == '*' then v1.value = v1.value * v2.value
  elseif op == '/' then v1.value = v1.value / v2.value end

  v1.unknownpass1 = v1.unknownpass1 or  v2.unknownpass1
  v1.defined      = v1.defined      and v2.defined

  v1.value = v1.value % 65536
  return v1
end

local expr = P {
  "exp",
  exp    = Cf(V"term"   * Cg(term_op   * V"term"  )^0,eval),
  term   = Cf(V"factor" * Cg(factor_op * V"factor")^0,eval),
  factor = value * WS^-1
         + OPAREN * V"exp" * CPAREN
}

Start at the bottom with "expr" and work your way back up.  The code is
expecting to be called like

        symbol_table = { ... }
        result = parser:match(input,1,symbol_table)

  The key part is the 'value' production, but I'm including the full bit for
context (and remember this is part of an assembler so keep that in mind).

  -spc (It's a two-pass assembler by the way ... )

[1]     A Motorola 6809 Assembler in case anyone cares.

Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Albert Chan

> On Aug 29, 2018, at 6:03 PM, joy mondal <[hidden email]> wrote
>
> The way mailing lists works ... I just cannot find the right tool to make use of it efficiently.

Try search the archive: https://marc.info/?l=lua-l



Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Sean Conner
In reply to this post by joy mondal
It was thus said that the Great joy mondal once stated:
> Would it make sense to move from mailing list to something like github /
> gitlab ?

  For me, no.  I prefer mailing lists.

> Its really hard to search old answers on the mailing list compared to
> github issues.
>
> The way mailing lists works ... I just cannot find the right tool to make
> use of it efficiently.

  Google.  Duck Duck Go.  Bing.  I know for Google and Duck, you can search
for (example):

        lpeg site:lua-users.org

and get results from the mailing list archive.

  Also, I have a copy of every email on this list since Oct 2009 and often
will search my own archive and then post links to the archive on
lua-users.org.

> Are you guys using the terminal or through your own mail servers ?

  I can only speak for myself, but I use mutt, a text-based mail user agent
on my own mail server (I use SSH to log into the command line on the server
and run mutt directly on the mail server).

  -spc (Wouldn't mind USENET, but comp.lang.lua doesn't exist ... )

Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

joy mondal
Alright,

It would be nice if there was a tool to make mailing list conversations a mix of reddit style threads with gtihub style markdown.

Anyway thanks for all the help


Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Albert Chan
In reply to this post by Sean Conner

> I have a copy of every email on this list since Oct 2009 and often
> will search my own archive and then post links to the archive on
> lua-users.org
>
> -- Sean Conner

Wow, saving all emails !  Must be 10,000+ by now !

I still prefer https://marc.info/?l=lua-l
It is not as pretty formatted as http://lua-users.org/lists/lua-l/
But, I noticed Lua own archive have a delayed update, maybe days.

And, thanks for the Googling Tip: lpeg site:lua-users.org

For me, +1 for the mailing list


Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Sean Conner
It was thus said that the Great Albert Chan once stated:
>
> > I have a copy of every email on this list since Oct 2009 and often
> > will search my own archive and then post links to the archive on
> > lua-users.org
> >
> > -- Sean Conner
>
> Wow, saving all emails !  Must be 10,000+ by now !

  Just counted---74,623 emails since October 2nd, 2009 (give or take one or
two that I've mistakenly deleted).  Space is cheap.

  -spc



Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Sean Conner
In reply to this post by joy mondal
It was thus said that the Great joy mondal once stated:
> Alright,
>
> It would be nice if there was a tool to make mailing list conversations a
> mix of reddit style threads with gtihub style markdown.

  I would say use a real email program and not Gmail.  I mean, here's this
mailing list in mutt:

        http://www.flummux.org/mutt.gif

The view is threaded, sorted by date within each thread.  It can even handle
HTML based emails [1].

  -spc (But I think we're straying from Lua here ... )

> Anyway thanks for all the help

[1] As long as a plain text alternative is included.  If not, I can
        still read the message, but it's a bit of a pain as mutt has to
        shell out to Lynx.

Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Dirk Laurie-2
In reply to this post by Sean Conner
Op Do., 30 Aug. 2018 om 00:26 het Sean Conner <[hidden email]> geskryf:

>   I can only speak for myself, but I use mutt, a text-based mail user agent

"All mail clients suck. This one just sucks less."

I used it for a long time. I switched over to GMail only because the
spam filter is so good.

Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Philipp Janda
In reply to this post by Sean Conner
Am 30.08.2018 um 00:26 schröbte Sean Conner:
>
>    -spc (Wouldn't mind USENET, but comp.lang.lua doesn't exist ... )

But there is news://news.gmane.org/gmane.comp.lang.lua.general which is
a mirror of the Lua mailing list.


Philipp



Reply | Threaded
Open this post in threaded view
|

Re: noob LPEG question

Pierre Chapuis
In reply to this post by joy mondal

 I just wished their was more LPEG specific resources just because of how easy it is to use, compared to BISON or ANTLR.
best wishes.

A good starting point is [1] (from the author of Moonscript).

There is also a talk by Roberto at an old Lua workshop (dated but still useful) [2].

Then you probably want working examples. Here are a few suggestions (some use LPegLabel): [3] [4] [5] [6] [7].

Reply | Threaded
Open this post in threaded view
|

re: noob LPEG question

Foster Schucker
In reply to this post by joy mondal
I like reading the digests.  I would like reading the digests more if 1/2 the items were not UTF-8 coded and show up as a large block of characters.    I'm sure there is just amazing stuff in those posts.

Thanks!
Foster