[ANN] LuaSrcDiet-0.12.1 (minor) release

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] LuaSrcDiet-0.12.1 (minor) release

KHMan
Hi all,

LuaSrcDiet reduces the size of Lua 5.1.x source files by
aggressively removing all unnecessary whitespace and comments,
optimizing constant tokens, and renaming local variables to
shorter names.

Location: http://code.google.com/p/luasrcdiet/

Version 0.12.1 changes:

* BUG FIX: Long comment adds an extra 2 characters when using
--keep option. (Thanks to Jeff)
* Faster function call syntax sugar optimization using a one-pass
token deletion loop.

Notes on future plans, Lua 5.2 plans:

'local' keyword removal isn't looking good -- gets very awkward
attempting a general optimizer using the ad hoc data structures
LuaSrcDiet currently has. Better to do it for the Lua 5.2 version.

That said, I'm not in a hurry to do LuaSrcDiet for Lua 5.2. There
was a recent sale at a toy store (Farnell/Newark) so LuaSrcDiet is
currently on the back burner...

If there are serious users who want LuaSrcDiet for Lua 5.2 in
2012, please ping or reply on the list so that the level of
interest can be gauged and plans adjusted. It is after all, an
obscure tool and one cannot expend unlimited manpower on it. This
would be relevant for lstrip and squish too.

Thanks,
--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaSrcDiet-0.12.1 (minor) release

Matthew Wild
On 7 April 2012 07:44, KHMan <[hidden email]> wrote:
> Hi all,
>

> If there are serious users who want LuaSrcDiet for Lua 5.2 in 2012, please
> ping or reply on the list so that the level of interest can be gauged and
> plans adjusted. It is after all, an obscure tool and one cannot expend
> unlimited manpower on it. This would be relevant for lstrip and squish too.
>

I'm obviously interested in the long term, though none of my projects
decisively support 5.2 either yet. When the time comes that I do need
it, I'm sure I'd be able to help with the development.

Regards,
Matthew

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaSrcDiet-0.12.1 (minor) release

Patrick Rapin
In reply to this post by KHMan
> 'local' keyword removal isn't looking good -- gets very awkward attempting a
> general optimizer using the ad hoc data structures LuaSrcDiet currently has.
> Better to do it for the Lua 5.2 version.

By "local" keyword removal, are you thinking of making the following
substitution ?
  local a = 12
  local b = "str"
==>
  local a,b=12,"str"
Because otherwise, I don't think "local" can be considered optional at all.

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaSrcDiet-0.12.1 (minor) release

KHMan
On 4/7/2012 11:53 PM, Patrick Rapin wrote:

>> 'local' keyword removal isn't looking good -- gets very awkward attempting a
>> general optimizer using the ad hoc data structures LuaSrcDiet currently has.
>> Better to do it for the Lua 5.2 version.
>
> By "local" keyword removal, are you thinking of making the following
> substitution ?
>    local a = 12
>    local b = "str"
> ==>
>    local a,b=12,"str"
> Because otherwise, I don't think "local" can be considered optional at all.

Well, it pretty easy to draw up a big list of possible
optimizations. The possibilities are obvious. That is not an issue
at all.

The actual problem: Someone has to spend the time implementing
them. :-) As you can probably guess, no one is rushing to do them.

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaSrcDiet-0.12.1 (minor) release

David Manura
In reply to this post by Matthew Wild
On Sat, Apr 7, 2012 at 10:15 AM, Matthew Wild <[hidden email]> wrote:
> On 7 April 2012 07:44, KHMan <[hidden email]> wrote:
>> This would be relevant for lstrip and squish too.

When using squish, I found the module organization confusing and
wanted a published API to call from a Lua program (not command-line)
like this:

  package.path = 'squish-0.2.0/lua/?.lua;' .. package.path
  local SQ = require 'squish'
  local err; s, err = SQ.minify_string(s)

Instead, I ended up loading it via dofile [1].

My other impression was that squish forked/superseded LuaSrcDiet, so I
used squish.  However, I see some useful documentation in LuaSrcDiet
that is not in squish like TechNotes.html, so I'm unsure how these
relate.

The calls to getfenv() should also be removed in squish for 5.2 compat.

The feature I think these programs can benefit from the most is dead
code elimination (DCE).  Some of that is done in LuaInspect where AST
nodes for certain statements/blocks are marked as dead and AST nodes
of certain local variables (including local functions) are marked as
unused, and the same applies for corresponding tokens in the token
list.  One may want to extend this to inter-module DCE:

  -- foo.lua
  local M = {}
  function M.a() . . . end; function M.b() . . . end; function M.c() . . . end
  return M

You would, however, need to have the full list of modules that might
require 'foo' and then determine which functions in 'foo' are not used
by those other modules, probably skipping DCE in more complex cases
(e.g. metatables on M).  If you merge modules into a single source
file prior to minimization, then this is somewhat simplified.  This
was also mentioned in relation to soar [2,3] for the purpose of
merging modules into a single file, in which case the goal is not
necessarily to minimize/obfuscate--in fact, perhaps quite the
opposite.

>> so that the level of interest can be gauged and plans adjusted

Since minimized Lua source is considered in a way a type of byte code
that's portable, some have used this type of thing in LuaJit -prior-
to its addition of bytecode loading/saving support, and it may still
have applications like that.  The interesting thing about DCE is that
it will also reduce the native byte code size, particularly when
dealing with larger libraries like penlight, of which you might only
use a couple functions (so many people manually do copy/paste).
Perhaps merging+DCE is orthogonal and best done in a tool separate
LuaSrcDist but which could be interfaced to it in a pipeline.

[1] http://lua-users.org/lists/lua-l/2012-03/msg00722.html
[2] http://lua-users.org/lists/lua-l/2012-02/msg00609.html
[3] http://lua-users.org/lists/lua-l/2012-04/msg00151.html

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaSrcDiet-0.12.1 (minor) release

Matthew Wild
On 7 April 2012 18:53, David Manura <[hidden email]> wrote:

> On Sat, Apr 7, 2012 at 10:15 AM, Matthew Wild <[hidden email]> wrote:
>> On 7 April 2012 07:44, KHMan <[hidden email]> wrote:
>>> This would be relevant for lstrip and squish too.
>
> When using squish, I found the module organization confusing and
> wanted a published API to call from a Lua program (not command-line)
> like this:
>
>  package.path = 'squish-0.2.0/lua/?.lua;' .. package.path
>  local SQ = require 'squish'
>  local err; s, err = SQ.minify_string(s)
>
> Instead, I ended up loading it via dofile [1].

Right, Squish is intended as a command-line utility, that just happens
to be written in Lua. Converting it to a module would be entirely
possible of course, I just never saw a use-case for that. I expected
Squish to be used as part of a build process, etc.

> My other impression was that squish forked/superseded LuaSrcDiet, so I
> used squish.  However, I see some useful documentation in LuaSrcDiet
> that is not in squish like TechNotes.html, so I'm unsure how these
> relate.

Not at all the case. Squish at the highest level just combines
modules, scripts and resources into a single valid .lua file. As a
feature it has a variety of filters it can run the produced file
through. Minify is one of these filters, and the majority of that
filter's code comes from LuaSrcDiet. Similarly, the majority of the
gzip filter's code is your deflatelua library.

> The calls to getfenv() should also be removed in squish for 5.2 compat.

Noted, thanks. As I mentioned, none of my projects are 5.2-tested yet.

> The feature I think these programs can benefit from the most is dead
> code elimination (DCE).  Some of that is done in LuaInspect where AST
> nodes for certain statements/blocks are marked as dead and AST nodes
> of certain local variables (including local functions) are marked as
> unused, and the same applies for corresponding tokens in the token
> list.  One may want to extend this to inter-module DCE:
>

I agree, dead-code elimination would be great. Actual effectiveness
would vary greatly depending on the kind of input, but you're right
that when depending on something like Penlight, it could be very
helpful.

>>> so that the level of interest can be gauged and plans adjusted
>
> Since minimized Lua source is considered in a way a type of byte code
> that's portable, some have used this type of thing in LuaJit -prior-
> to its addition of bytecode loading/saving support, and it may still
> have applications like that.

LuaJIT's bytecode loader still isn't compatible with Lua's as far as I
know, so the same problem still applies if you want to target both
with the same code.

> Perhaps merging+DCE is orthogonal and best done in a tool separate
> LuaSrcDist but which could be interfaced to it in a pipeline.

Sounds rather like Squish and its filters :)

Regards,
Matthew

Reply | Threaded
Open this post in threaded view
|

Re: [ANN] LuaSrcDiet-0.12.1 (minor) release

KHMan
In reply to this post by David Manura
On 4/8/2012 1:53 AM, David Manura wrote:

> On Sat, Apr 7, 2012 at 10:15 AM, Matthew Wild wrote:
>> On 7 April 2012 07:44, KHMan wrote:
>>> This would be relevant for lstrip and squish too.
>
> When using squish, I found the module organization confusing and
> wanted a published API to call from a Lua program (not command-line)
> like this:
>
>    package.path = 'squish-0.2.0/lua/?.lua;' .. package.path
>    local SQ = require 'squish'
>    local err; s, err = SQ.minify_string(s)
>
> Instead, I ended up loading it via dofile [1].
>
> My other impression was that squish forked/superseded LuaSrcDiet, so I
> used squish.  However, I see some useful documentation in LuaSrcDiet
> that is not in squish like TechNotes.html, so I'm unsure how these
> relate.

I should point out for the benefit of others that LuaSrcDiet is
really "experimental software", so there should be no expectation
of maintenance or support because of the very limited number of
users of such tools. Hence, the concept of official or good API is
hard to apply rigorously here, compared to Lua libraries that are
stably maintained.

squish and LuaSrcDiet aims for different objectives, so there is
no problem with that. The niche for such utilities is tiny, so
they are more in the realm of academic toys rather than something
to be treated like an ubiquitous tool that others depend and rely
on. Being a practical person, I cannot justify expenditure of too
much time on it.

> The calls to getfenv() should also be removed in squish for 5.2 compat.
>
>[snip]
>>> so that the level of interest can be gauged and plans adjusted
>
> Since minimized Lua source is considered in a way a type of byte code
> that's portable, some have used this type of thing in LuaJit -prior-
> to its addition of bytecode loading/saving support, and it may still
> have applications like that. [snip]

Most of the optimizations still to do would be best done on a
parsed syntax tree, doing it properly like metalua. No one seems
to be in a hurry to go 5.2, so I'm thinking about sticking with
5.1 stuff a little while longer.

But apart from stuff already mentioned on the list, I didn't
really think new specific use cases would be uncovered, given the
circumstances.

So it will be hacked on from time to time, only don't hold your
breath waiting for the next release... :-)

--
Cheers,
Kein-Hong Man (esq.)
Kuala Lumpur, Malaysia