[ANN] Penlight Lua Libraries

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

[ANN] Penlight Lua Libraries

steve donovan
Hi guys,

Penlight is a set of pure Lua libraries designed to make some common
tasks easier and more standard. The dependencies are just lfs and
alien (if you're on Windows and want a path.copyfile that works
properly)

1) Inspired by Nick Trout's Python-like List class, there are Lua
ports of several useful Python libraries:
List,extended string methods, path operations, directory operations
(including the very useful dir.walk)

2) Support for functional programming, including Boost-style
placeholder expressions:
     tablex.map2(t1,t2,_1()+_2())
     f = bind(g,_1,false)

3) Some libraries for handling common file formats, like delimited
text files with headers, AWK-style field iterators and configuration
files.  There is also SIP (Simple Input Patterns) which aims to be
Regular Expressions for Dummies (which is a class I sometimes find
myself in)

Until LuaForge and I have a better day, some temporary hosting:

Find it here at:

http://mysite.mweb.co.za/residents/sdonovan/lua/penlight/pl.0.6b.zip
(includes User Guide and module help)

For the merely curious:

Users Guide at:
http://mysite.mweb.co.za/residents/sdonovan/lua/penlight/penlight.html

LuaDoc help at:
http://mysite.mweb.co.za/residents/sdonovan/lua/penlight/index.html

steve d.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

Andrew Wilson-4
Nice library, tons of useful bits.    Andrew

On Sat, Apr 18, 2009 at 8:31 AM, steve donovan
<[hidden email]> wrote:

> Hi guys,
>
> Penlight is a set of pure Lua libraries designed to make some common
> tasks easier and more standard. The dependencies are just lfs and
> alien (if you're on Windows and want a path.copyfile that works
> properly)
>
> 1) Inspired by Nick Trout's Python-like List class, there are Lua
> ports of several useful Python libraries:
> List,extended string methods, path operations, directory operations
> (including the very useful dir.walk)
>
> 2) Support for functional programming, including Boost-style
> placeholder expressions:
>     tablex.map2(t1,t2,_1()+_2())
>     f = bind(g,_1,false)
>
> 3) Some libraries for handling common file formats, like delimited
> text files with headers, AWK-style field iterators and configuration
> files.  There is also SIP (Simple Input Patterns) which aims to be
> Regular Expressions for Dummies (which is a class I sometimes find
> myself in)
>
> Until LuaForge and I have a better day, some temporary hosting:
>
> Find it here at:
>
> http://mysite.mweb.co.za/residents/sdonovan/lua/penlight/pl.0.6b.zip
> (includes User Guide and module help)
>
> For the merely curious:
>
> Users Guide at:
> http://mysite.mweb.co.za/residents/sdonovan/lua/penlight/penlight.html
>
> LuaDoc help at:
> http://mysite.mweb.co.za/residents/sdonovan/lua/penlight/index.html
>
> steve d.
>
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

steve donovan
In reply to this post by steve donovan
On Sat, Apr 18, 2009 at 2:31 PM, steve donovan
<[hidden email]> wrote:
> Penlight is a set of pure Lua libraries designed to make some common
> tasks easier and more standard.

Actually got a LuaForge release and an Icon!

http://penlight.luaforge.net/penlight.html   For the User Guide

http://penlight.luaforge.net/index.html    For the module Docs

And the download:

http://luaforge.net/frs/download.php/4047/pl.0.6.2.zip

There is a reasonably detailed example (luash.lua, just over 600
lines) showing PL in action; see the examples and the tests
directories.

To anticipate a question, yes there is a lot of overlap with stdlib.
I've tried to like stdlib, but eventually failed, for reasons similar
to Mike Pall's and because of dissatisfaction with the documentation.

The module docs are not totally complete. I suspect I am hitting some
obscure LuaDoc bug.

steve d.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

David Manura
On Thu, Apr 23, 2009 at 10:01 AM, steve donovan wrote:
>> Penlight is a set of pure Lua libraries designed to make some common
>> tasks easier and more standard.

Thank you.  The tutorial is especially useful.  I feel that a lot of
modules don't well advertise themselves in terms of how the module is
intended to be used in context with examples and design comments.
Sometimes the docs are hidden in tarballs.

Incidentally, I've been back working on Perl, were some algorithmic
tasks are so convenient, and then returning to Lua, which though as
good as Lua is, there's a lack of these building blocks such as you've
developed.  In that light, here's a few "common patterns" of my own
and analysis of how well Penlight addresses them.

For example, I wanted to print a table, sorting by keys.  In Perl we just do

  for my $k (sort keys %t) {
    print "$k,$t{$k}\n";
  }

In Lua, using only standard libraries, we have code that is a bit
removed from the problem domain:

  local ks = {}
  for k in pairs(t) do ks[#ks+1] = k end
  table.sort(ks)
  for _,k in ipairs(ks) do
    print(k,t[k])
  end

With some helpful utility functions though, such as in Penlight, this
is clarified to

  local ks = tablex.keys(t)
  table.sort(ks)
  for k in seq.list(ks) do
    print(k,t[k])
  end

If the sort didn't have to be in-place, we could chain the
transformations on a single line:

  for k in seq.list(tablex.sort(tablex.keys(t))) do
    print(k,t[k])
  end

That's not too bad.  A minor point is that the parens start to
accumulate in the chain.  We might use compose if you extended it to
take more than two arguments:

  for k in func.compose(seq.list, tablex.sort, tablex.keys)(t) do
    print(k,t[k])
  end

We can also do this nicely with sequences:

  for k in seq.sort(seq.all_keys(t)) do
    print(k,t[k])
  end

Alternately, this can be done with lists:

  local ks = List(tablex.keys(t))
  ks:sort()
  for k in pl.list.iter(ks) do
    print(k,t[k])
  end

Similarly, it would be nice to have the sort allowed to not be in-place:

  for k in pl.list.iter(List(tablex.keys(t)):sort()) do
    print(k,t[k])
  end

That's a bit ugly.  We're dealing with table arrays, lists, and
sequences.  I feel there's a potential for confusion between these
three.  The distinction between the three is not inherent to the
problem domain (ordered sequences), yet the programmer has to decide
which of these to use and how to fit them together.


Another recent example I had was to select one element from the list
1..10 that matched a predicate f but was not in the list t.
Mathematically, {x in 1..10 : f(x) and x not in t}.

In Perl, it's fairly concise:

  my %s = map {$_ => 1} @t;
  my ($x) = grep { ! $s{$_} && f($_) } (1..10);

Using only the Lua standard library, we have a bit of coding to do:

  local function makeset(t)
    local s = {}
    for _,v in ipairs(t) do s[v] = true end
    return s
  end
  local s = makeset(t)
  local x
  for i=1,10 do if not s[i] and f(i) then
    x = i
  end end

With library or utility functions, we obtain

  local s = tablex.index_map(t)
  local x = tablex.filter(List.range(10):map(_1+1), function(i) return
not s[i] and f(i) end)[1]

The above could be simplified if range accepted bounds, as it does in
sequence form:

  local s = tablex.index_map(t)
  local x = seq.filter(seq.range(1,10), function(i) return not s[i]
and f(i) end)()

which is not too bad.  Perhaps some syntax support for list
comprehensions could make it a bit cleaner.


I'm not certain I like your argument order for functions though.  As
the docs say,

  <quote>
  ...and functions which take a function argument (like tablex.map)
will have the function as the last argument, because of the syntax of
Lua anonymous functions:

  s = tablex.map(t,function(x)
      return x*x
  end)
  </quote>
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

steve donovan
On Sat, Apr 25, 2009 at 9:17 AM, David Manura <[hidden email]> wrote:
> Thank you.  The tutorial is especially useful.  I feel that a lot of
> modules don't well advertise themselves in terms of how the module is
> intended to be used in context with examples and design comments.
> Sometimes the docs are hidden in tarballs.

Documentation is key. It isn't the thing that gives us the greatest
pleasure, however!

> That's a bit ugly.  We're dealing with table arrays, lists, and
> sequences.  I feel there's a potential for confusion between these
> three.  The distinction between the three is not inherent to the
> problem domain (ordered sequences), yet the programmer has to decide
> which of these to use and how to fit them together.

OK, this is the real problem. Python makes the consequence of sequence
more explicit, whereas it isn't really a native Lua concept.  There is
for instance both seq.reduce and tablex.reduce, whereas a smart reduce
could work out from its arguments what operation was required. The
result in either case is a scalar. But the tablex/seq is mostly about
what the returned type is, so seq.map/tablex.map are returning rather
different things. An overloaded version would either return a sequence
or a table. If things were really consistent, it should not matter,
but this requires some very careful thinking.

> Perhaps some syntax support for list
> comprehensions could make it a bit cleaner.

They are a great notation, and I know there have been several proposals.

> I'm not certain I like your argument order for functions though.

Yes, I was being deliberately backwards.  But if we are _mostly_ using
prepackaged functions, then the argument is moot. The concern was that
when using the traditional order, the actual
table/sequence/list/whatever ends up at the end of a potentially long
anonymous function definition.

t1 = map(function(x)
           return x*(x-1)
     end,t1)

We need to think these things out - the current API is not frozen at
this point! The idea remains to capture as many common patterns as
possible, and give them _clear names_. (that's not easy!)  Hopefully,
of the potentially thousands of patterns, we get a subset which
somehow spans the universe of patterns efficiently.

There's another danger, which I know from my C++ days. It is the curse
of the clever one-liner.  At university, we had some APL golfball
terminals, and the game was to print out a whole histogram with one
line of deliciously interesting symbols. Not a good way to write
actual _software_, IMHO.

steve d.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

David Manura
On Sat, Apr 25, 2009 at 4:00 AM, steve donovan wrote:
> On Sat, Apr 25, 2009 at 9:17 AM, David Manura wrote:
>> for k in pl.list.iter(List(tablex.keys(t)):sort()) do
>>   print(k,t[k])
>> end
>> That's a bit ugly.

That would be nicer if the API allowed method call chaining:

  for k in List.keys(t):sort():iter() do
    print(k,t[k])
  end

That would follow a useful pattern of

  (1) wrap a table into a list ... e.g. List(t) or List.keys(t)
  (2) chain any number of list operations ....
       e.g. :sort(f):map(g):filter(h)
  (3) convert to a sequence to iterate over
       (It would be preferable if this could be avoided as in Python,
        defaulting to the type of iteration we usually want.
        An __iter metamethod has been proposed:
        http://lua-users.org/wiki/GeneralizedPairsAndIpairs )

>> If the sort didn't have to be in-place, we could chain the
>> transformations on a single line:

In-place v.s. non-in-place operations...Penlight currently departs
from its Python inspiration in that it provides additional List
methods like "map/filter" that are not in-place like its methods
"sort/reverse".  This feels inconsistent/arbitrary.  Python
not-in-place operations are separated out as standalone functions:
"sorted/map/filter".

However, consider the approach in Chapter 16 of Lua Programming Gems
used to implement a matrix type (p.180).  All operations (even map)
are in-place, like Python list methods.  However, there is an explicit
copy() method, and all operations return the object itself to allow
chaining.

  -- two different uses
  return a:copy():emwul(b):linfold()
  return a:emwul(b):linfold()

This is flexible and avoids excessive temporary copies.

In fact, Penlight already does the copy when constructing a list from a table.

> Python makes the consequence of sequence
> more explicit, whereas it isn't really a native Lua concept.  There is
> for instance both seq.reduce and tablex.reduce

Shouldn't there also be a List:reduce?

> whereas a smart reduce
> could work out from its arguments what operation was required.

Perhaps.

>> I'm not certain I like your argument order for functions though.
>
> Yes, I was being deliberately backwards.  But if we are _mostly_ using
> prepackaged functions, then the argument is moot. The concern was that
> when using the traditional order, the actual
> table/sequence/list/whatever ends up at the end of a potentially long
> anonymous function definition.
>
> t1 = map(function(x)
>           return x*(x-1)
>     end,t1)

The problem I think is when we chain these operations together:

  t1 = sort(map(filter(t1, f), g), h)

Now, "h" modifies the verb "sort", "g" modifies "map", and "f"
modifies "filter".  These are farther apart and read less naturally
than if we had written

  t1 = sort(h, map(g, filter(f, t1)))

We can also compose and curry them:

  t1 = compose(curry(sort,h), curry(map,g), curry(filter,f))(t1)

On the other hand, the form map(t, f) is useful when map a method of
list, list:map(f), rather than a standalone function.
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

Bertrand Mansion
On Sat, Apr 25, 2009 at 9:23 PM, David Manura <[hidden email]> wrote:

> However, consider the approach in Chapter 16 of Lua Programming Gems
> used to implement a matrix type (p.180).  All operations (even map)
> are in-place, like Python list methods.  However, there is an explicit
> copy() method, and all operations return the object itself to allow
> chaining.
>
>  -- two different uses
>  return a:copy():emwul(b):linfold()
>  return a:emwul(b):linfold()
>
> This is flexible and avoids excessive temporary copies.

They actually named this Fluent interface:
http://www.martinfowler.com/bliki/FluentInterface.html

--
Bertrand Mansion
Mamasam
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

David Manura
In reply to this post by David Manura
On Sat, Apr 18, 2009 at 8:31 AM, steve donovan wrote:
> Penlight is a set of pure Lua libraries designed to make some common
> tasks easier and more standard.

"memoize" [1] is a handy function.  The basic form that appears in a
number of my own modules is the one in[2].

  function memoize(func)
    return setmetatable({}, {
      __index = function(self, k) local v = func(k); self[k] = v; return v end,
      __call = function(self, k) return self[k] end
    })
  end

Also note that Metalua implements its own standard library[3-4], which
has some overlap with Penlight.  There could be some collaboration
there.  I know Fabien advocates against the reinvent-the-wheel thing,
and having your work incorporated into Metalua would be a good thing
for Penlight's promotion.

[1] http://lua-users.org/wiki/FuncTables
[2] http://lua-users.org/wiki/CodeGeneration
[3] http://metalua.luaforge.net/manual005.html
[4] http://lua-users.org/wiki/StandardLibraries
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

David Manura
> On Sat, Apr 18, 2009 at 8:31 AM, steve donovan wrote:
>> Penlight is a set of pure Lua libraries designed to make some common
>> tasks easier and more standard.

The pl.operator names perhaps should resemble Lua metamethod names
(e.g. "lt" rather than "less").

Instead of (or in addition to) operator names like ops.mul, one could
have ops['a*b'] or ops 'a*b' containing the representation in Lua
code.  This is generalizable to things like ops 'a+b^2' via code
generation techniques [1] (with memoization above) and is an
alternative to placeholder expressions (e.g. _1 + _2^2).

This could also be used in the pl.test module, allowing asserteq(x,y)
to alternately be written as test(ops'a=b', x, y) and more generally
allow things like test(ops'a<b', x, y), possibly with shorthand
test('<', x, y).

[1] http://lua-users.org/wiki/ShortAnonymousFunctions
Reply | Threaded
Open this post in threaded view
|

Re: [ANN] Penlight Lua Libraries

steve donovan
On Tue, Apr 28, 2009 at 4:11 AM, David Manura <[hidden email]> wrote:
> The pl.operator names perhaps should resemble Lua metamethod names
> (e.g. "lt" rather than "less").

Agreed.

> Instead of (or in addition to) operator names like ops.mul, one could
> have ops['a*b'] or ops 'a*b' containing the representation in Lua
> code.  This is generalizable to things like ops 'a+b^2' via code
> generation techniques [1] (with memoization above) and is an
> alternative to placeholder expressions (e.g. _1 + _2^2).

I had actually converged on something similar

   t = map2('+',t1,t2)

  t = map('*',t,-1)    (i.e result is the original with all elements
multiplied by -1)

This just looks up the corresponding operator.mul function.

(Yes, I'm persuaded, function goes first ;))

>
> This could also be used in the pl.test module, allowing asserteq(x,y)
> to alternately be written as test(ops'a=b', x, y) and more generally
> allow things like test(ops'a<b', x, y), possibly with shorthand
> test('<', x, y).

test('==',x,y) feels nice...

(The only other thing asserteq brings to the party is default
deepcompare of tables, but == could mean that)

But do we need an alternative to placeholder expressions?  Having a
number of ways to do the same thing can be confusing. (It's true that
memoizing PEs is rather more tricky than 'executable' strings)

On a different track, I am having difficulty coming up with sensible
names ;)  Most of these operations are very straightforward to write
(although there are subtleties), but I can't avoid things like map2
because map can take extra arguments to be passed to the function.
seq.copy is perhaps intuitive enough, but seq.copy2 (which takes a
double-valued sequence and makes it into {{x1,y1},{x2,y2}...} feels a
bit forced. Perhaps seq.copy_tuples?

steve d.