[ANN] ltokenp, a token processor for Lua

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] ltokenp, a token processor for Lua

Luiz Henrique de Figueiredo
ltokenp is now available at
        http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/5.3/ltokenp.tar.gz

It should work for Lua 5.1, 5.2, and 5.3.

The README is attached. Enjoy. All feedback welcome.
--lhf

ltokenp is a token processor for Lua: it allows you to process the
stream of tokens coming from the Lua lexer.

Potential uses include:
- Compressing Lua programs by removing comments and whitespace (strip.lua)
- Removing assertions (assert.lua)
- Adding new syntax sugar (self.lua)
- Experimenting with new syntax without hacking the Lua source (reserved.lua)

ltokenp accepts Lua scripts to run and files to process.
The scripts and files are executed and processed as seen.
Each script appears as a separate argument after '-s', one '-s' per script.

Scripts typically output the contents of the files with some modifications.
Unfortunately, all comments and whitespace are eaten by the lexer and never
reach the token stream.

A typical usage is
        ltokenp -s script.lua [file.lua ...]
but you can also do
        ltokenp -s s1.lua f1.lua -s s2.lua f2.lua

A global function named FILTER will be called once for each token seen
in the files. So, scripts typically define FILTER. If no scripts are
given, ltokenp just dumps the token stream with this:

        function FILTER(line,token,text,value)
                print(line,token,text,value)
        end

As can been above, the FILTER function receives 4 arguments from the lexer:
- the line number where the token appears
- the token as a number
- the token as text
- the value of names, numbers, and strings;
  for other tokens, the value is the same as the text.

To try ltokenp, just edit Makefile to reflect your installation of Lua
and then run make. This will build ltokenp and run a simple test. Note
that ltokenp needs inside information from the Lua private headers and
so it needs to be built using a Lua build directory, not just the usual
installation directories.

This code is hereby placed in the public domain.
Please send comments, suggestions, and bug reports to [hidden email] .

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] ltokenp, a token processor for Lua

Dirk Laurie-2
2016-05-04 7:17 GMT+02:00 Luiz Henrique de Figueiredo <[hidden email]>:
> ltokenp is now available at
>         http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/5.3/ltokenp.tar.gz
>
> It should work for Lua 5.1, 5.2, and 5.3.

So it does, though I needed to satisfy myself that the following
warning is garmless:

ltokenp.c: In function ‘main’:
ltokenp.c:41:14: warning: unused parameter ‘argc’ [-Wunused-parameter]

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] ltokenp, a token processor for Lua

Xavier Wang
In reply to this post by Luiz Henrique de Figueiredo
2016-05-04 13:17 GMT+08:00 Luiz Henrique de Figueiredo <[hidden email]>:
> ltokenp is now available at
>         http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/5.3/ltokenp.tar.gz
>
I have made a C module of Lua lexer, based on Lua's source code, here:

https://gist.github.com/starwing/8dc070733393dcfc54c7d52fdabd489c


--
regards,
Xavier Wang.

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] ltokenp, a token processor for Lua

Luiz Henrique de Figueiredo
> I have made a C module of Lua lexer, based on Lua's source code, here:
>
> https://gist.github.com/starwing/8dc070733393dcfc54c7d52fdabd489c

Nice. But my point with ltokenp and its parent tool lstrip is to reuse
the Lua lexer intact to guarantee consistency with the Lua core.