Library bindings or os.execute?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Library bindings or os.execute?

Dirk Laurie-2
I find, increasingly often, that on my Linux system I no longer bother
looking whether someone has written a library with Lua bindings
for some sophisticated standard library [1]. Instead, I do it with
os.execute.

This sounds quick-and-dirty, but I wonder whether in the long run,
it might not be more portable and require less maintenance.

-- Dirk

[1] For example, suppose I have ripped a CD into a disk image.
There is a huge binary file sitting there that in principle you could
edit, play etc. The `sox` utility seems intent on supporting the tiniest
obscure feature. Sure, reading the manpage for `sox` is hard work,
but reading the documentation for `libsox` is a damn sight harder.
So I write a Lua frontend that simply assembles the command lines
for the editing/playback calls.

Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

Sean Conner
It was thus said that the Great Dirk Laurie once stated:

> I find, increasingly often, that on my Linux system I no longer bother
> looking whether someone has written a library with Lua bindings
> for some sophisticated standard library [1]. Instead, I do it with
> os.execute.
>
> This sounds quick-and-dirty, but I wonder whether in the long run,
> it might not be more portable and require less maintenance.
>
> -- Dirk
>
> [1] For example, suppose I have ripped a CD into a disk image.
> There is a huge binary file sitting there that in principle you could
> edit, play etc. The `sox` utility seems intent on supporting the tiniest
> obscure feature. Sure, reading the manpage for `sox` is hard work,
> but reading the documentation for `libsox` is a damn sight harder.
> So I write a Lua frontend that simply assembles the command lines
> for the editing/playback calls.

  Several years ago, I starting writing a program to index all the files on
my computer (a Linux system).  Part of this indexing involved classifying
the type of file and for that, Linux came with a program called 'file' that
does that.  It supported the option "-f" to read filenames from stdin and it
would write the file type (I used the "-i" option to print out mime types).

  So my thought was:  I can invoke "file", feeding it filenames in stdin,
and reading the file types on stdout.  Only issue was: popen() (and by
extension, io.popen() but this was before I got into Lua) only supports
reading or writing (but not both).  I was trying to avoid having to generate
a list prior (1,499,996 entries---probably more now) to generating the
types.

  "Not a problem," thought I.  "I wrote a Unix shell as a college project,
I know how to execute a program with redirection."  And so I did a
bi-directional "popen()" function---

---which failed miserably.

  Turns out the standard C library only defaults to line buffering if
stdin/stdout is an actual TTY, not a pipe.  For a pipe (or a file) it does
full buffering.  I solved the issue I had (not the problem---there is a
difference) using some runtime dynanamic linker tricks (LD_PRELOAD mumble
mumble) to get the line buffering and it worked.

  The *real* solution was to use the magic library and avoid the hack
solution.  Thus this:

        local fsys  = require "org.conman.fsys"
        local magic = require "org.conman.fsys.magic"

        magic:flags('mime')

        for entry in fsys.dir() do
          print(magic(entry),entry)
        end

will run in reasonable time for a large directory (and not chew up CPU time
continuously spawning a program or heaven forbid attempt that horrible
hack mentioned above).

  -spc (Who thinks that executables should also be shared libraries ... )

Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

Daurnimator
On 14 June 2017 at 16:35, Sean Conner <[hidden email]> wrote:
>   -spc (Who thinks that executables should also be shared libraries ... )


They can be (on ELF systems)! Just pass `-pie` (and -fPIC, and
probably -Wl,-E unless you use a linker script etc) to gcc.

Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

Adrian Perez de Castro
In reply to this post by Sean Conner
Hello,

On Wed, 14 Jun 2017 02:35:38 -0400, Sean Conner <[hidden email]> wrote:

> [...]
>
>   -spc (Who thinks that executables should also be shared libraries ... )

This is doable at least when dealing with ELF objects. Of course the binary
has to _both_ provide a program entry point (likely as a weak symbol, so it
can be overriden when the ELF object is linked in as a shared library), and
also export dynamic symbols. You can try this today, by going ahead and
executing the glibc main binary:

  % /lib/libc-2.25.so
  ...

Of course, building such a binary can get tricky (symbol visibility, weak
symbols) and most likely it will be a portability nightmare. Still doable,
though!

Cheers,


--
 Adrián 🎩

attachment0 (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

Michal Kottman
In reply to this post by Sean Conner
On Jun 14, 2017 08:36, "Sean Conner" <[hidden email]> wrote:
  -spc (Who thinks that executables should also be shared libraries ... )

This also works on Windows (or at least did, I haven't used it in >10y), where .DLL and .EXE are both of the PE format, and can export symbols - I used this for example to compile lighttpd plugin DLLs and linked them back against lighttpd.exe.

Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

Vaughan McAlley-2
In reply to this post by Dirk Laurie-2
On 14 June 2017 at 15:00, Dirk Laurie <[hidden email]> wrote:

>
> I find, increasingly often, that on my Linux system I no longer bother
> looking whether someone has written a library with Lua bindings
> for some sophisticated standard library [1]. Instead, I do it with
> os.execute.
>
> This sounds quick-and-dirty, but I wonder whether in the long run,
> it might not be more portable and require less maintenance.
>
> -- Dirk
>
> [1] For example, suppose I have ripped a CD into a disk image.
> There is a huge binary file sitting there that in principle you could
> edit, play etc. The `sox` utility seems intent on supporting the tiniest
> obscure feature. Sure, reading the manpage for `sox` is hard work,
> but reading the documentation for `libsox` is a damn sight harder.
> So I write a Lua frontend that simply assembles the command lines
> for the editing/playback calls.
>

I do this a lot, and like Dirk I sometimes wonder if it's best practice. I'm usually only writing for myself though.

Vaughan
 
Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

steve donovan
On Wed, Jun 14, 2017 at 3:59 PM, Vaughan McAlley <[hidden email]> wrote:
> I do this a lot, and like Dirk I sometimes wonder if it's best practice. I'm
> usually only writing for myself though.

Well, exactly. Us casual users should not worry too much - "works on
my machine" is usually good enough! But public libraries? That gets
awkward.

Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

Parke
In reply to this post by Sean Conner
On Tue, Jun 13, 2017 at 11:35 PM, Sean Conner <[hidden email]> wrote:

>   Several years ago, I starting writing a program to index all the files on
> my computer (a Linux system).  Part of this indexing involved classifying
> the type of file and for that, Linux came with a program called 'file' that
> does that.  It supported the option "-f" to read filenames from stdin and it
> would write the file type (I used the "-i" option to print out mime types).
>
>   So my thought was:  I can invoke "file", feeding it filenames in stdin,
> and reading the file types on stdout.  Only issue was: popen() (and by
> extension, io.popen() but this was before I got into Lua) only supports
> reading or writing (but not both).  I was trying to avoid having to generate
> a list prior (1,499,996 entries---probably more now) to generating the
> types.
>
>   "Not a problem," thought I.  "I wrote a Unix shell as a college project,
> I know how to execute a program with redirection."  And so I did a
> bi-directional "popen()" function---
>
> ---which failed miserably.

Why do you need bi-directional popen for this task?

for entry in io.popen 'find / | file -f -' : lines () do
  print ( entry ) end

A similar approach should work in C.

-Parke

Reply | Threaded
Open this post in threaded view
|

Re: Library bindings or os.execute?

Sean Conner
It was thus said that the Great Parke once stated:

> On Tue, Jun 13, 2017 at 11:35 PM, Sean Conner <[hidden email]> wrote:
> >   Several years ago, I starting writing a program to index all the files on
> > my computer (a Linux system).  Part of this indexing involved classifying
> > the type of file and for that, Linux came with a program called 'file' that
> > does that.  It supported the option "-f" to read filenames from stdin and it
> > would write the file type (I used the "-i" option to print out mime types).
> >
> >   So my thought was:  I can invoke "file", feeding it filenames in stdin,
> > and reading the file types on stdout.  Only issue was: popen() (and by
> > extension, io.popen() but this was before I got into Lua) only supports
> > reading or writing (but not both).  I was trying to avoid having to generate
> > a list prior (1,499,996 entries---probably more now) to generating the
> > types.
> >
> >   "Not a problem," thought I.  "I wrote a Unix shell as a college project,
> > I know how to execute a program with redirection."  And so I did a
> > bi-directional "popen()" function---
> >
> > ---which failed miserably.
>
> Why do you need bi-directional popen for this task?
>
> for entry in io.popen 'find / | file -f -' : lines () do
>   print ( entry ) end
>
> A similar approach should work in C.

  The main reason is I didn't think of that.  A secondary reason was that I
was doing a bit more than just calling 'file' on each entry (I was also
calling stat() and recording the metainformation so that could be indexed as
well).

  -spc