[ANN] Development announcement - Ravi/Lua Compiler project

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] Development announcement - Ravi/Lua Compiler project

Dibyendu Majumdar
Lua's inbuilt parser and code generator is a work of art, very compact
and low overhead but extremely fast. It uses minimal memory and
produces bytecodes as it parses the source code (single pass
compiler). This is great for Lua and Ravi given the use cases of these
languages, but makes the parser and code generator quite hard to
understand, play with, or reuse in tools such as IDEs. It also makes
it harder to perform any advanced type checking or performance
optimizations.

I am pleased to announce a new project to create a Lua/Ravi parser and
code generator that is not a replacement for the default one in
Lua/Ravi but can be used for more specialised code generation, as well
as as a means of understanding how the parser and code generator
works. My hope is that it will enable the study of Lua implementation
in educational institutions, as I intend to use classical methods of
compilation. Moreover, ease of maintenance is a bigger priority than
sheer memory or performance efficiency.

The project was initially part of Ravi because I was reusing the Lua
lexer. However, I have now split this off into a standalone project
independent of Ravi.

The project is currently under development. You can view the code at:

https://github.com/dibyendumajumdar/ravi-compiler

All feedback and contributions welcome!

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Dibyendu Majumdar
On Fri, 21 Feb 2020 at 21:55, Dibyendu Majumdar <[hidden email]> wrote:
> I am pleased to announce a new project to create a Lua/Ravi parser and
> code generator that is not a replacement for the default one in
> Lua/Ravi but can be used for more specialised code generation, as well
> as as a means of understanding how the parser and code generator
> works. My hope is that it will enable the study of Lua implementation
> in educational institutions, as I intend to use classical methods of
> compilation. Moreover, ease of maintenance is a bigger priority than
> sheer memory or performance efficiency.
>

Hi,

So I am looking for ideas / suggestions for what the api for this
library look like.
I need a C api to start with. I have been looking at other libraries
to see what they do, but none stands out as ideal. Perhaps Go is the
only language that has a built-in module that gives you lexer, parser
etc.

Suggestions welcome. If you know of a great example then I would
appreciate if you can give me a reference to the project.

In terms of modules, there will be at least following:

1. Lexer
2. Parser
3. Linearizer (Intermediate Linear Code)
4. Optimizer
5. Code generator

Regards
Dibyendu

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:

> On Fri, 21 Feb 2020 at 21:55, Dibyendu Majumdar <[hidden email]> wrote:
> > I am pleased to announce a new project to create a Lua/Ravi parser and
> > code generator that is not a replacement for the default one in
> > Lua/Ravi but can be used for more specialised code generation, as well
> > as as a means of understanding how the parser and code generator
> > works. My hope is that it will enable the study of Lua implementation
> > in educational institutions, as I intend to use classical methods of
> > compilation. Moreover, ease of maintenance is a bigger priority than
> > sheer memory or performance efficiency.
> >
>
> Hi,
>
> So I am looking for ideas / suggestions for what the api for this
> library look like.
> I need a C api to start with. I have been looking at other libraries
> to see what they do, but none stands out as ideal.

  Without knowing what you looked at, there's LLVM for the code generation
part, and clang for more of the lexing and parsing.

  I have worked with TCC (Tiny C Compiler) which can be embedded into a
larger context as you can call it as a library.  It does all the steps,
lexer, parser, code generation (but very little optimization). The official
repo is at <https://repo.or.cz/tinycc.git>.  

  I have wrapped TCC up as a Lua module.  I'm not sure how well it works
with the latest TCC (I stopped updating my TCC several years ago---the
development model of TCC is a bit of a mess really).  But if you just want
to check it out:

        The low level module:
       
        https://github.com/spc476/lua-conmanorg/blob/master/src/tcc.c

        The slightly higher module that allows one to load Lua modules
        directly from C code:
       
        https://github.com/spc476/lua-conmanorg/blob/master/lua/cc.lua

  I'm not sure if this will help you in any way, but who knows?  It might.

> Perhaps Go is the
> only language that has a built-in module that gives you lexer, parser
> etc.
>
> Suggestions welcome. If you know of a great example then I would
> appreciate if you can give me a reference to the project.

  Hmm ... have you heard of nanopass compilers?

        http://lambda-the-ultimate.org/node/1589

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Dibyendu Majumdar
On Mon, 9 Mar 2020 at 21:37, Sean Conner <[hidden email]> wrote:
> > So I am looking for ideas / suggestions for what the api for this
> > library look like.
> > I need a C api to start with. I have been looking at other libraries
> > to see what they do, but none stands out as ideal.
>
>   Without knowing what you looked at, there's LLVM for the code generation
> part, and clang for more of the lexing and parsing.
>

I am familiar with LLVM api of course. The C api is not bad  but of
course it is for generating LLVM bytecode.
I have only briefly looked at Clang parser api - will look into it in depth.

>   I have worked with TCC (Tiny C Compiler) which can be embedded into a
> larger context as you can call it as a library.  It does all the steps,
> lexer, parser, code generation (but very little optimization). The official
> repo is at <https://repo.or.cz/tinycc.git>.

Does it offer api for each layer?
My intention is that if you just need the lexer, or the parser, then
you should be able to use the library for that without having to
bother with the rest.
The libtcc.h seems more about just using tcc as a black box.

>
>   I have wrapped TCC up as a Lua module.  I'm not sure how well it works
> with the latest TCC (I stopped updating my TCC several years ago---the
> development model of TCC is a bit of a mess really).  But if you just want
> to check it out:
>
>         The low level module:
>
>         https://github.com/spc476/lua-conmanorg/blob/master/src/tcc.c
>
>         The slightly higher module that allows one to load Lua modules
>         directly from C code:
>
>         https://github.com/spc476/lua-conmanorg/blob/master/lua/cc.lua
>
>   I'm not sure if this will help you in any way, but who knows?  It might.
>

Thank you. I had a quick look. As mentioned above my goal is slightly
different. That is, how to expose the lexer? Or the parser?

> > Suggestions welcome. If you know of a great example then I would
> > appreciate if you can give me a reference to the project.
>
>   Hmm ... have you heard of nanopass compilers?
>
>         http://lambda-the-ultimate.org/node/1589
>

No, I hadn't. Thank you for the link.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Sean Conner
It was thus said that the Great Dibyendu Majumdar once stated:
> On Mon, 9 Mar 2020 at 21:37, Sean Conner <[hidden email]> wrote:
> >
> >   I have worked with TCC (Tiny C Compiler) which can be embedded into a
> > larger context as you can call it as a library.  It does all the steps,
> > lexer, parser, code generation (but very little optimization). The official
> > repo is at <https://repo.or.cz/tinycc.git>.
>
> Does it offer api for each layer?

  No, it does not.  It does allow some control though---the location of the
header files, predefined defines, output format, but no hooks into the
individual portions.  In fact, I don't think it has many internal portions
as it's just a one-pass compiler.

> > > Suggestions welcome. If you know of a great example then I would
> > > appreciate if you can give me a reference to the project.
> >
> >   Hmm ... have you heard of nanopass compilers?
> >
> >         http://lambda-the-ultimate.org/node/1589
>
> No, I hadn't. Thank you for the link.

  You're welcome.

  -spc


Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Domingo Alvarez Duarte
In reply to this post by Dibyendu Majumdar
Hello !

Not exactly what you are describing but there is this project
http://terralang.org/

 From their website:

====

Terra is a low-level system programming language that is embedded in and
meta-programmed by the Lua programming language.

====

Cheers !

On 9/3/20 21:28, Dibyendu Majumdar wrote:

> On Fri, 21 Feb 2020 at 21:55, Dibyendu Majumdar <[hidden email]> wrote:
>> I am pleased to announce a new project to create a Lua/Ravi parser and
>> code generator that is not a replacement for the default one in
>> Lua/Ravi but can be used for more specialised code generation, as well
>> as as a means of understanding how the parser and code generator
>> works. My hope is that it will enable the study of Lua implementation
>> in educational institutions, as I intend to use classical methods of
>> compilation. Moreover, ease of maintenance is a bigger priority than
>> sheer memory or performance efficiency.
>>
> Hi,
>
> So I am looking for ideas / suggestions for what the api for this
> library look like.
> I need a C api to start with. I have been looking at other libraries
> to see what they do, but none stands out as ideal. Perhaps Go is the
> only language that has a built-in module that gives you lexer, parser
> etc.
>
> Suggestions welcome. If you know of a great example then I would
> appreciate if you can give me a reference to the project.
>
> In terms of modules, there will be at least following:
>
> 1. Lexer
> 2. Parser
> 3. Linearizer (Intermediate Linear Code)
> 4. Optimizer
> 5. Code generator
>
> Regards
> Dibyendu
>

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Dibyendu Majumdar
On Tue, 10 Mar 2020 at 08:28, Domingo Alvarez Duarte <[hidden email]> wrote:
>
> Not exactly what you are describing but there is this project
> http://terralang.org/
>
 > So I am looking for ideas / suggestions for what the api for this
> > library look like.

Hi, I don't think Terra has a library api exposing the lexer or parser
etc. but thank you for the feedback.
In case it wasn't clear - I have already got a lexer and a parser. I
am trying to expose this to users, example a way for users to traverse
the AST. I have looked at how Clang does it, and Go does it -  I am
leaning more towards a Visitor pattern so that the AST itself doesn't
need to be exposed.

Regards

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Dibyendu Majumdar
On Tue, 10 Mar 2020 at 21:15, Dibyendu Majumdar <[hidden email]> wrote:
> In case it wasn't clear - I have already got a lexer and a parser. I
> am trying to expose this to users, example a way for users to traverse
> the AST. I have looked at how Clang does it, and Go does it -  I am
> leaning more towards a Visitor pattern so that the AST itself doesn't
> need to be exposed.

It seems that it is unclear what I am looking to do, so here is the
current api state.
Right now I have exposed the lexer which is pretty much based on Lua's lexer.
The api allows you submit a Lua source, iterate through the tokens
until the input is exhausted.
The api gives you everything you need hopefully.

You can also see the parser api but it doesn't expose the AST -
abstract syntax tree or provide a way to traverse this - at the
moment.

https://github.com/dibyendumajumdar/ravi-compiler/blob/master/include/ravi_compiler.h

Regards

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Dibyendu Majumdar
In reply to this post by Dibyendu Majumdar
Hi,

I am pleased to say that there is now a public API defined for the
following components:

a) Lexer
b) Parser
c) AST walking

You can view the API definitions at
https://github.com/dibyendumajumdar/ravi-compiler/blob/master/include/ravi_compiler.h
I decided to create a relatively lower level AST walking api - it is a
bit verbose but it is an attempt to make the use of the API intuitive
and safe.

You can see an example of the AST walking api usage here:
https://github.com/dibyendumajumdar/ravi-compiler/blob/master/tests/tastwalk.c

I will have more details regarding the linear IR generator later but
for now as it is work in progress. The IR is more like standard
compiler IR rather then the Lua bytecode IR. If you are curious, you
can view examples of linear IR output here (note the output contains
AST dumps as well).
https://github.com/dibyendumajumdar/ravi-compiler/blob/master/tests/expected/results.expected

I welcome any feedback.

Regards
Dibyendu

On Fri, 21 Feb 2020 at 21:55, Dibyendu Majumdar <[hidden email]> wrote:

>
> Lua's inbuilt parser and code generator is a work of art, very compact
> and low overhead but extremely fast. It uses minimal memory and
> produces bytecodes as it parses the source code (single pass
> compiler). This is great for Lua and Ravi given the use cases of these
> languages, but makes the parser and code generator quite hard to
> understand, play with, or reuse in tools such as IDEs. It also makes
> it harder to perform any advanced type checking or performance
> optimizations.
>
> I am pleased to announce a new project to create a Lua/Ravi parser and
> code generator that is not a replacement for the default one in
> Lua/Ravi but can be used for more specialised code generation, as well
> as as a means of understanding how the parser and code generator
> works. My hope is that it will enable the study of Lua implementation
> in educational institutions, as I intend to use classical methods of
> compilation. Moreover, ease of maintenance is a bigger priority than
> sheer memory or performance efficiency.
>
> The project was initially part of Ravi because I was reusing the Lua
> lexer. However, I have now split this off into a standalone project
> independent of Ravi.
>
> The project is currently under development. You can view the code at:
>
> https://github.com/dibyendumajumdar/ravi-compiler
>
> All feedback and contributions welcome!
>
> Regards
> Dibyendu
_______________________________________________
lua-l mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Development announcement - Ravi/Lua Compiler project

Dibyendu Majumdar
Hi,

In the following location, you can see some of the control flow graph
output from the Ravi/Lua compiler project.
These outputs represent the initial state of the intermediate code.
They are mainly for my benefit to help me debug issues (you will see
some of the generated IR is kind of strange).

https://github.com/dibyendumajumdar/ravi-compiler/tree/master/docs/cfg

I have started to work on analysis passes that will hopefully start to
improve the intermediate IR. Recently I did some work on Dominator
Tree generation. I will be working on Data Flow Analysis next.

I hope this project is of interest to folks that want to use Lua as a
target language in compiler courses. My goal is to create a reusable
library that is helpful for people who want to learn this type of
stuff.

Regards
Dibyendu

On Sat, 2 May 2020 at 13:22, Dibyendu Majumdar <[hidden email]> wrote:

>
> Hi,
>
> I am pleased to say that there is now a public API defined for the
> following components:
>
> a) Lexer
> b) Parser
> c) AST walking
>
> You can view the API definitions at
> https://github.com/dibyendumajumdar/ravi-compiler/blob/master/include/ravi_compiler.h
> I decided to create a relatively lower level AST walking api - it is a
> bit verbose but it is an attempt to make the use of the API intuitive
> and safe.
>
> You can see an example of the AST walking api usage here:
> https://github.com/dibyendumajumdar/ravi-compiler/blob/master/tests/tastwalk.c
>
> I will have more details regarding the linear IR generator later but
> for now as it is work in progress. The IR is more like standard
> compiler IR rather then the Lua bytecode IR. If you are curious, you
> can view examples of linear IR output here (note the output contains
> AST dumps as well).
> https://github.com/dibyendumajumdar/ravi-compiler/blob/master/tests/expected/results.expected
>
> I welcome any feedback.
>
> Regards
> Dibyendu
>
> On Fri, 21 Feb 2020 at 21:55, Dibyendu Majumdar <[hidden email]> wrote:
> >
> > Lua's inbuilt parser and code generator is a work of art, very compact
> > and low overhead but extremely fast. It uses minimal memory and
> > produces bytecodes as it parses the source code (single pass
> > compiler). This is great for Lua and Ravi given the use cases of these
> > languages, but makes the parser and code generator quite hard to
> > understand, play with, or reuse in tools such as IDEs. It also makes
> > it harder to perform any advanced type checking or performance
> > optimizations.
> >
> > I am pleased to announce a new project to create a Lua/Ravi parser and
> > code generator that is not a replacement for the default one in
> > Lua/Ravi but can be used for more specialised code generation, as well
> > as as a means of understanding how the parser and code generator
> > works. My hope is that it will enable the study of Lua implementation
> > in educational institutions, as I intend to use classical methods of
> > compilation. Moreover, ease of maintenance is a bigger priority than
> > sheer memory or performance efficiency.
> >
> > The project was initially part of Ravi because I was reusing the Lua
> > lexer. However, I have now split this off into a standalone project
> > independent of Ravi.
> >
> > The project is currently under development. You can view the code at:
> >
> > https://github.com/dibyendumajumdar/ravi-compiler
> >
> > All feedback and contributions welcome!
> >
> > Regards
> > Dibyendu