Module modularity

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Module modularity

Geoff Leyland
Hi,

I’m the author of lua-csv [1].  It’s a single Lua file and it parses CSV.  No surprises there.

Embedded in lua-csv is a little column mapping tool that allows you to say “I’m looking for a column named ‘distance’ or ‘how far’ and in the table I get, I want it to be named ‘length’ and have the value in the column multiplied by 25.4 since I work in mm, not inches”.  It’s just over 100 lines of code.

I also have some as-yet-unreleased code for reading some other formats - notably dbf and shapefiles, and these used to use the same bit of code, until I pushed lua-csv out and decided one file was better than two.

So, module writers, any opinions on the best way to package these?

 - split lua-csv in two, with a column manager module that’s kind of useless on its own, but which lua-csv depends on, and then release lua-dbf and lua-shapfile also depending on the column manager?

 - just push them out separately all containing an initially similar copy of the same 100 or so lines of column map code?

 - some other bright idea?

Cheers,
Geoff


[1] https://github.com/geoffleyland/lua-csv
Reply | Threaded
Open this post in threaded view
|

Re: Module modularity

Rena
On Wed, Sep 17, 2014 at 9:50 PM, Geoff Leyland
<[hidden email]> wrote:

> Hi,
>
> I’m the author of lua-csv [1].  It’s a single Lua file and it parses CSV.  No surprises there.
>
> Embedded in lua-csv is a little column mapping tool that allows you to say “I’m looking for a column named ‘distance’ or ‘how far’ and in the table I get, I want it to be named ‘length’ and have the value in the column multiplied by 25.4 since I work in mm, not inches”.  It’s just over 100 lines of code.
>
> I also have some as-yet-unreleased code for reading some other formats - notably dbf and shapefiles, and these used to use the same bit of code, until I pushed lua-csv out and decided one file was better than two.
>
> So, module writers, any opinions on the best way to package these?
>
>  - split lua-csv in two, with a column manager module that’s kind of useless on its own, but which lua-csv depends on, and then release lua-dbf and lua-shapfile also depending on the column manager?
>
>  - just push them out separately all containing an initially similar copy of the same 100 or so lines of column map code?
>
>  - some other bright idea?
>
> Cheers,
> Geoff
>
>
> [1] https://github.com/geoffleyland/lua-csv

Generalize the shared code into a standalone module which those modules use?

--
Sent from my Game Boy.

Reply | Threaded
Open this post in threaded view
|

Re: Module modularity

Geoff Leyland
On 18/09/2014, at 1:53 pm, Rena <[hidden email]> wrote:

> Generalize the shared code into a standalone module which those modules use?

Yes, that was the first option above.  I guess I'd just feel a bit silly releasing a 100-line module that has no use on its own.  The license and readme will be longer than the code.


Reply | Threaded
Open this post in threaded view
|

RE: Module modularity

Thijs Schreijer

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Geoff Leyland
> Sent: donderdag 18 september 2014 4:09
> To: Lua mailing list
> Subject: Re: Module modularity
>
> On 18/09/2014, at 1:53 pm, Rena <[hidden email]> wrote:
>
> > Generalize the shared code into a standalone module which those modules
> use?
>
> Yes, that was the first option above.  I guess I'd just feel a bit silly
> releasing a 100-line module that has no use on its own.  The license and
> readme will be longer than the code.
>

Difficult to judge. While things are in motion, I tend to keep the code in a separate module. Once they settle and little further development is expected, I merge them so the main modules become standalone.
Only for small modules (100 lines satisfies that imo)

When you publish them as rocks, it's harder, because the standalone version of this small 'useless to almost everyone' module will linger around forever.

Typical for me is that I quite often 'steal' utility stuff from Penlight. In some cases the project grows and more and more code is imported, to the point that I remove it all and call Penlight as a dependency.

I think its personal, but I'm fond of standalone modules. I hate small modules that pull in everything but the kitchen sink.

My 2cts.
Thijs


Reply | Threaded
Open this post in threaded view
|

Re: Module modularity

steve donovan
On Thu, Sep 18, 2014 at 9:30 AM, Thijs Schreijer
<[hidden email]> wrote:
> Typical for me is that I quite often 'steal' utility stuff from Penlight.

I've done that! ;)

> I think its personal, but I'm fond of standalone modules. I hate small modules that pull in everything but the kitchen sink.

Yes, chasing-the-dependencies can be a pain for actual users; cannot
assume they use LuaRocks...

The submodule in question could get an independent existence when it
does something useful for other modules, basically.

steve d.

Reply | Threaded
Open this post in threaded view
|

Is there an #include equivalent in Lua (I do not mean require)

Tony Papadimitriou
Hi all,

How can one #include a bunch of files so that when compiled, a single
self-contained executable is produced.

'require' does not qualify as equivalent because it needs to find the files
at run-time.

TIA
Tony


Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Udo Schroeter
Hey Tony,

no, Lua does not have a preprocessor, so raw includes are not possible. However, you could just roll your own. For example, if you chose a syntax like

--include: foo.lua

you could write a short Lua script that processes this file and loads these includes, producing one big file which you then execute.

On Thu, Sep 18, 2014 at 3:07 PM, Tony Papadimitriou <[hidden email]> wrote:
Hi all,

How can one #include a bunch of files so that when compiled, a single self-contained executable is produced.

'require' does not qualify as equivalent because it needs to find the files at run-time.

TIA
Tony


Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Jerome Vuarand
In reply to this post by Tony Papadimitriou
2014-09-18 14:07 GMT+01:00 Tony Papadimitriou <[hidden email]>:
> How can one #include a bunch of files so that when compiled, a single
> self-contained executable is produced.

You can concatenate the files in a certain way with a bit of
boilerplate between each. I'm sure others will chime in to offer
suggestions of tools doing that (I can't remember any).

> 'require' does not qualify as equivalent because it needs to find the files
> at run-time.

require can look for modules in plenty of places, not only in separate
files. If your modules and main script already use require, you can
concatenate them as suggested above without modification. For example:

local main = "main.lua"
local modules = {"foo", "foo.fff", "bar"}
local output = "all.lua"
----------
local file = io.open(output, 'wb')
for _,modname in ipairs(modules) do
    local modfile = modname:gsub('%.', '/')..".lua"
    file:write("package.preload['"..modname.."'] = function(...)\n")
    file:write(io.open(modfile, 'rb'):read('*all'))
    file:write("end\n")
end
file:write(io.open(main, 'rb'):read('*all'))
file:close()

Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Roger Ivie
In reply to this post by Tony Papadimitriou
On Thu, 18 Sep 2014, Tony Papadimitriou wrote:
>
> How can one #include a bunch of files so that when compiled, a single
> self-contained executable is produced.

dofile( "whatever.lua" )

--
roger ivie
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Rodrigo Azevedo
In reply to this post by Tony Papadimitriou
I use the make! for example, if mod3.lua requires mod1.lua and mod2.lua the simple makefile

 all: mod1.luah mod3.luah mod2.luah
        luac -o prog.exe main.lua

mod1.luah: mod1.lua
        cat mod1.lua >> main.lua

mod2.luah: mod2.lua
        cat mod2.lua >> main.lua

mod3.luah: mod1.luah mod2.luah
        cat mod3.lua >> main.lua

clean:
        rm main.lua

solves my problems.

2014-09-18 10:07 GMT-03:00 Tony Papadimitriou <[hidden email]>:
Hi all,

How can one #include a bunch of files so that when compiled, a single self-contained executable is produced.

'require' does not qualify as equivalent because it needs to find the files at run-time.

TIA
Tony




--
Rodrigo Azevedo Moreira da Silva
Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

aryajur

On Thu, Sep 18, 2014 at 8:04 AM, Rodrigo Azevedo <[hidden email]> wrote:
I use the make! for example, if mod3.lua requires mod1.lua and mod2.lua the simple makefile

 all: mod1.luah mod3.luah mod2.luah
        luac -o prog.exe main.lua

mod1.luah: mod1.lua
        cat mod1.lua >> main.lua

mod2.luah: mod2.lua
        cat mod2.lua >> main.lua

mod3.luah: mod1.luah mod2.luah
        cat mod3.lua >> main.lua

clean:
        rm main.lua

solves my problems.

2014-09-18 10:07 GMT-03:00 Tony Papadimitriou <[hidden email]>:

Hi all,

How can one #include a bunch of files so that when compiled, a single self-contained executable is produced.

'require' does not qualify as equivalent because it needs to find the files at run-time.

TIA
Tony




--
Rodrigo Azevedo Moreira da Silva

Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Steven Degutis
In reply to this post by Tony Papadimitriou
Quoth Tony Papadimitriou:

> How can one #include a bunch of files so that when compiled, a single
> self-contained executable is produced.

Lua files are not self-contained executables. And there is no way to
produce one without great hackery (that I know of). So, even if you do
concatenate all Lua files into a single source file, if must still be
executed by a Lua interpreter on the target user's machine.

-Steven

Reply | Threaded
Open this post in threaded view
|

RE: Is there an #include equivalent in Lua (I do not mean require)

Thijs Schreijer
In reply to this post by Tony Papadimitriou

> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Tony Papadimitriou
> Sent: donderdag 18 september 2014 15:08
> To: Lua mailing list
> Subject: Is there an #include equivalent in Lua (I do not mean require)
>
> Hi all,
>
> How can one #include a bunch of files so that when compiled, a single
> self-contained executable is produced.
>
> 'require' does not qualify as equivalent because it needs to find the files
> at run-time.
>
> TIA
> Tony
>

Try http://matthewwild.co.uk/projects/squish/home 

Reply | Threaded
Open this post in threaded view
|

RE: Is there an #include equivalent in Lua (I do not mean require)

Thijs Schreijer
In reply to this post by Tony Papadimitriou


> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On
> Behalf Of Tony Papadimitriou
> Sent: donderdag 18 september 2014 15:08
> To: Lua mailing list
> Subject: Is there an #include equivalent in Lua (I do not mean require)
>
> Hi all,
>
> How can one #include a bunch of files so that when compiled, a single
> self-contained executable is produced.
>
> 'require' does not qualify as equivalent because it needs to find the files
> at run-time.
>
> TIA
> Tony
>

And another one; http://lua-users.org/wiki/SimpleLuaPreprocessor 

Iirc there even was one that packaged everything into an executable (unix I think) but couldn't find it just now.

Thijs
Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Luiz Henrique de Figueiredo
In reply to this post by Steven Degutis
> Lua files are not self-contained executables. And there is no way to
> produce one without great hackery (that I know of).

I don't think srlua is great hackery...
        http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#srlua

Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

steve donovan
On Thu, Sep 18, 2014 at 7:02 PM, Luiz Henrique de Figueiredo
<[hidden email]> wrote:
> I don't think srlua is great hackery...
>         http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#srlua

I agree, it simply exploits the fact that you may append readable data
to the end of your executable, and read it later (works fine on both
Windows and POSIX systems, i.e. not requiring any specific 'resource'
compiler).

Of course, that data must be a single Lua script.  luabuild [1]
reduces a Lua program to a standalone executable like so:
   - uses Jay Carlson's soar to pack the program into a single
'archive'. soar can use either static or dynamic analysis, either
looking for requires in bytecode or tracking requires dynamically when
the program is run.  It makes an archive, and finds what C extensions
were used
 - that list is used to compile a custom statically-linked Lua 5.2 stub
- which is glued to the archive with srlua.

luabuild is a 'micro' distribution, it just has the usual suspects,
like lfs, socket, lpeg, sql, posix, winapi, etc.  A general solution
would probably require the backing of a full distribution like LuaDist
and I believe something like this is on that project's roadmap.

steve d.
[1] https://github.com/stevedonovan/luabuild

Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Tony Papadimitriou
In reply to this post by Steven Degutis
Maybe I wasn't clear enough.

What I meant by executable was a Lua executable (which maintains the
advantage of portability to any system Lua can run), not something like an
EXE file.

I need to give someone a compiled Lua program that s/he can run using just
the Lua interpreter but
a. without being able to look at the source, and
b. given as a single file which contains the complete application, rather
than a tree structure of library files.

Thank you all for your suggestions, I will try them ASAP.

-----Original Message-----
From: Steven Degutis
Sent: Thursday, September 18, 2014 7:20 PM
To: Lua mailing list
Subject: Re: Is there an #include equivalent in Lua (I do not mean require)

Quoth Tony Papadimitriou:

> How can one #include a bunch of files so that when compiled, a single
> self-contained executable is produced.

Lua files are not self-contained executables. And there is no way to
produce one without great hackery (that I know of). So, even if you do
concatenate all Lua files into a single source file, if must still be
executed by a Lua interpreter on the target user's machine.

-Steven


Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Sean Conner
It was thus said that the Great [hidden email] once stated:

> Maybe I wasn't clear enough.
>
> What I meant by executable was a Lua executable (which maintains the
> advantage of portability to any system Lua can run), not something like an
> EXE file.
>
> I need to give someone a compiled Lua program that s/he can run using just
> the Lua interpreter but
> a. without being able to look at the source, and
> b. given as a single file which contains the complete application, rather
> than a tree structure of library files.
>
> Thank you all for your suggestions, I will try them ASAP.

  I did start a project a few months ago to address this issue [1], but the code
is still very rough and there were some issues that I could not fully
resolve.  The intent was to package up a Lua application into a ZIP file and
run all the code from there.  It does work (I'm able to run LuaRocks from
such a package) but the issues I ran into were:

        1) Lua modules in C had to be written to disk first before loading;
           I could not run C-based modules directly out of the ZIP file.

        2) I was able to patch Lua's loadfile() function; I have yet to fix
           dofile() or modify io.open() to handle read-only files.

        3) Any other type of file that the Lua application required needed
           to be written out to disk (becuase they could be opened for
           writing).  This is still an unresolved issue.

        4) It requires a few other modules to be already installed for this
           to all work.  It's not entirely self-contained.

  Now, speaking of ZIP files, perhaps the following will work for you.
Compile the Lua application using luac.  Then, write a shell script (or
batch file, depending upon operating system) to run the resulting Lua code,
fixing $LUA_PATH and $LUA_CPATH as required.  Take the resulting compiled
Lua code and shell script/batch file, zip them up and give her the resulting
zip file, which the person can unzip and run.

  This a) prevents them from seeing the source code and
       b) is a single file that contains the complete application.

  Yes, there's the addtional extraction phase, but everything should be in
one central location.

  -spc

[1] https://github.com/spc476/LEM

Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Steven Degutis
In reply to this post by Luiz Henrique de Figueiredo
Quoth Luiz Henrique de Figueiredo:
>
> I don't think srlua is great hackery...
>         http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/#srlua
>

Thanks for sharing this program. It looks incredibly useful.

You're right, it does not look hacky at all. Although, I don't fully
understand how the load() function works in srlua.c, namely how it is
able to locate the end of the executable data and the beginning of the
Lua code, because I am not familiar with the structure of Unix
executables.

If it is indeed robust, I may begin using it in more projects. It's
very convenient to use self-contained executables with no
dependencies.

-Steven

Reply | Threaded
Open this post in threaded view
|

Re: Is there an #include equivalent in Lua (I do not mean require)

Luiz Henrique de Figueiredo
> I don't fully understand how the load() function works in srlua.c,
> namely how it is able to locate the end of the executable data and
> the beginning of the Lua code, because I am not familiar with the
> structure of Unix executables.

glue just appends a Lua script to a Lua interpreter and then adds a
few data bytes at the end containing the size of the Lua script.

srlua then seeks to its end, reads the data bytes, seeks to the beginning
of the script, and loads it from there.

12