Process pipelines in Lua

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Process pipelines in Lua

Reuben Thomas-5
The latest (5.1.27) of luaposix has a somewhat experimental addition,
posix.pipeline, which allows process pipelines to be constructed in
Lua, a mixture of shell commands and forked Lua functions run in
subprocesses.

For example, you can write something like:

posix.pipeline({
  "tail -3 COPYING",
  "wc",
  function () s = posix.read(posix.STDIN_FILENO, 1000) print (s) end
})

Even more experimentally, there's an iterator which adds an extra
process and pipes the output of the end of the pipeline back into the
calling process.

Normally, posix.pipe is used to construct the pipe, but any function
that returns paired file descriptors also works, e.g. posix.openpty.

The reason for mentioning all this on the list is that I think it's
pretty cool (being able to write process pipelines simply in Lua), but
I have no idea if it's of interest to anyone else, and, if so, whether
the API and, secondarily, code, seem sensible.

I'm already using it in another project for coloring the output of
command-line commands:

https://github.com/rrthomas/cw

Comments about the pipeline code and API are most welcome,
particularly on the relevant github issue

https://github.com/luaposix/luaposix/issues/74

--
http://rrt.sc3d.org

Reply | Threaded
Open this post in threaded view
|

Re: Process pipelines in Lua

Pierre-Yves Gérardy
I don't remember why, but I remember looking for a way to achieve this
with luaposix, not grasping how to do it (for now obvious reasons),
and looking elsewhere.

Pretty cool :-)

-- Pierre-Yves


On Sun, Mar 17, 2013 at 11:42 PM, Reuben Thomas <[hidden email]> wrote:

> The latest (5.1.27) of luaposix has a somewhat experimental addition,
> posix.pipeline, which allows process pipelines to be constructed in
> Lua, a mixture of shell commands and forked Lua functions run in
> subprocesses.
>
> For example, you can write something like:
>
> posix.pipeline({
>   "tail -3 COPYING",
>   "wc",
>   function () s = posix.read(posix.STDIN_FILENO, 1000) print (s) end
> })
>
> Even more experimentally, there's an iterator which adds an extra
> process and pipes the output of the end of the pipeline back into the
> calling process.
>
> Normally, posix.pipe is used to construct the pipe, but any function
> that returns paired file descriptors also works, e.g. posix.openpty.
>
> The reason for mentioning all this on the list is that I think it's
> pretty cool (being able to write process pipelines simply in Lua), but
> I have no idea if it's of interest to anyone else, and, if so, whether
> the API and, secondarily, code, seem sensible.
>
> I'm already using it in another project for coloring the output of
> command-line commands:
>
> https://github.com/rrthomas/cw
>
> Comments about the pipeline code and API are most welcome,
> particularly on the relevant github issue
>
> https://github.com/luaposix/luaposix/issues/74
>
> --
> http://rrt.sc3d.org
>

Reply | Threaded
Open this post in threaded view
|

Re: Process pipelines in Lua

Sean Conner
In reply to this post by Reuben Thomas-5
It was thus said that the Great Reuben Thomas once stated:
>
> Comments about the pipeline code and API are most welcome,
> particularly on the relevant github issue

  Years ago, in my own homebrewed language, I too, did something similar
(with piping of STDIN and STDOUT) but I also had the ability to pipe STDERR
to a separate chain of commands [1]---the actual piping could get quite
convoluted, but I'm wondering---can you do the same with luaposix?

  -spc (Or even of arbitrary file descriptors, ala bash?)

[1] Unix commands were first class objects, as were pipe lines of Unix
        commands.  It's just that the syntax of the language was pretty
        nasty for this type of stuff. [2]

[2] It was based on Forth.  


Reply | Threaded
Open this post in threaded view
|

Re: Process pipelines in Lua

Leo Razoumov
In reply to this post by Reuben Thomas-5
On Sun, Mar 17, 2013 at 6:42 PM, Reuben Thomas <[hidden email]> wrote:
> [..snip..]
> The reason for mentioning all this on the list is that I think it's
> pretty cool (being able to write process pipelines simply in Lua), but
> I have no idea if it's of interest to anyone else, and, if so, whether
> the API and, secondarily, code, seem sensible.

Rueben,
this new feature is VERY useful. IMHO, easy pipelining has been the
only feature that is keeping shell programming around. With
'posix.pipeline' one can safely retire shell scripts in favor of Lua
scripts.

--Leo--

Reply | Threaded
Open this post in threaded view
|

Re: Process pipelines in Lua

oliver-2
Very useful for sure. Is there anything like this on Windows? Python's subprocess module allows to feed a process's stdin and read its stdout and stderr independent of OS platform.


On Tue, Mar 26, 2013 at 6:49 AM, Leo Razoumov <[hidden email]> wrote:
On Sun, Mar 17, 2013 at 6:42 PM, Reuben Thomas <[hidden email]> wrote:
> [..snip..]
> The reason for mentioning all this on the list is that I think it's
> pretty cool (being able to write process pipelines simply in Lua), but
> I have no idea if it's of interest to anyone else, and, if so, whether
> the API and, secondarily, code, seem sensible.

Rueben,
this new feature is VERY useful. IMHO, easy pipelining has been the
only feature that is keeping shell programming around. With
'posix.pipeline' one can safely retire shell scripts in favor of Lua
scripts.

--Leo--


Reply | Threaded
Open this post in threaded view
|

Re: Process pipelines in Lua

steve donovan
On Thu, Apr 11, 2013 at 3:52 AM, oliver <[hidden email]> wrote:
Very useful for sure. Is there anything like this on Windows? Python's subprocess module allows to feed a process's stdin and read its stdout and stderr independent of OS platform.

For sure indeed!  winapi allows the capture of interactive processes:

http://stevedonovan.github.io/winapi/topics/readme.md.html

If you use read_async that calls a callback with each received chunk, then you can capture multiple such processes in parallel.

The Unix concept of 'pseudo terminals' (ptys) is most elegant - you arrange the process to run in a pty, and then control the properties of that pty as if it were a 'real' terminal connected e.g. to a serial connection. So you can arrange to buffer by line, etc.  Whereas with the Windows CreateProcess method used by winapi, the received data won't generally be single lines, and stitching them together is up to you.

I've occaisionally thought it would be cool to have a cross-platform module which provided these useful OS features in a high-level way, rather like that Python module. The implementation would use winapi, luaposix or Justin Cormark's ljsyscall for LuaJIT.


Reply | Threaded
Open this post in threaded view
|

Re: Process pipelines in Lua

oliver-2
Excellent, thanks. I wouldn't have known where to find this. 


On Thu, Apr 11, 2013 at 3:13 AM, steve donovan <[hidden email]> wrote:
On Thu, Apr 11, 2013 at 3:52 AM, oliver <[hidden email]> wrote:
Very useful for sure. Is there anything like this on Windows? Python's subprocess module allows to feed a process's stdin and read its stdout and stderr independent of OS platform.

For sure indeed!  winapi allows the capture of interactive processes:

http://stevedonovan.github.io/winapi/topics/readme.md.html

If you use read_async that calls a callback with each received chunk, then you can capture multiple such processes in parallel.

The Unix concept of 'pseudo terminals' (ptys) is most elegant - you arrange the process to run in a pty, and then control the properties of that pty as if it were a 'real' terminal connected e.g. to a serial connection. So you can arrange to buffer by line, etc.  Whereas with the Windows CreateProcess method used by winapi, the received data won't generally be single lines, and stitching them together is up to you.

I've occaisionally thought it would be cool to have a cross-platform module which provided these useful OS features in a high-level way, rather like that Python module. The implementation would use winapi, luaposix or Justin Cormark's ljsyscall for LuaJIT.