(multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

(multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

Soni "They/Them" L.
I'm writing an IRC library as the existing ones aren't really good
enough (or pure Lua enough) for my purposes. I'm trying to decide on how
to handle the messages, but I need some input.

First method: (I prefer this way as I think it gives the coder more
freedom, you can register a single handler and handle any number of
events you want, and you can also use this to implement the 2nd method.
I could also add priorities to this.)

irc.addHandler(coroutine.create(function(...)
     while true do
         -- something here
         coroutine.yield()
     end
end))

irc.addHandler(function(...)
     while true do
         -- something here
         coroutine.yield() -- functions are automagically wrapped in
coroutines
     end
end)

Second method: (this is how LuaIRC does it... not very friendly,
especially when you're doing a modular bot...)

irc.addHandler("eventName", function(...)
end)

So, what do you say? Should I go for the 1st or the 2nd method, or
should I go for something else entirely?

Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

Steven Degutis
Quoth Thiago L.:
> So, what do you say? Should I go for the 1st or the 2nd method, or should I
> go for something else entirely?

The first two examples conflate waiting for events and handling
events. I would go with the third example.

-Steven

Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

Daurnimator
In reply to this post by Soni "They/Them" L.
On 23 September 2014 13:23, Thiago L. <[hidden email]> wrote:
irc.addHandler("eventName", function(...)
end)


It also has a priority as an optional 3rd argument.
Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

William Ahern
In reply to this post by Soni "They/Them" L.
On Tue, Sep 23, 2014 at 02:23:12PM -0300, Thiago L. wrote:

> I'm writing an IRC library as the existing ones aren't really good
> enough (or pure Lua enough) for my purposes. I'm trying to decide on how
> to handle the messages, but I need some input.
>
> First method: (I prefer this way as I think it gives the coder more
> freedom, you can register a single handler and handle any number of
> events you want, and you can also use this to implement the 2nd method.
> I could also add priorities to this.)
>
> irc.addHandler(coroutine.create(function(...)
>     while true do
>         -- something here
>         coroutine.yield()
>     end
> end))
<snip>
> Second method: (this is how LuaIRC does it... not very friendly,
> especially when you're doing a modular bot...)
>
> irc.addHandler("eventName", function(...)
> end)
>
> So, what do you say? Should I go for the 1st or the 2nd method, or
> should I go for something else entirely?

IMO both of these options are too high-level and make too many assumptions
about the caller. I dislike APIs which make too many assumptions. I'd rather
they make fewer assumptions, but make it trivial for me to extend things.

Whether I'm doing this in Lua, C, or some other language, I typically
implement event emitters using a pull model. That is, the API I expose from
the library is something like:

        local event = foo:get()

        if event.type == "bar" then
                ...
        end

Basically like :read or :write, but with objects instead of strings. That
makes the object 100% self-contained, and I don't necessarily have to reason
about how it executes far flung bits of code if I don't want it to.

Of course, many times I'm just going to end up implementing my own function
dispatch table. But many times I won't need to. And other times I really
want to do it myself because I might handle concurrency differently or in a
way that would be incompatible with the way the library dispatches handlers
directly. Fortunately, Lua's coroutines means I can usually invert the
producer-consumer protocol; that is, coroutines often let me convert a push
API into a pull API. But that's needless complexity.

You can also layer a convenience API atop this pull model. The best APIs are
ones that are layered atop each other. First, the user then has a choice of
how to handle things. Secondly, nothing will shake the bugs out of an API
better than using it yourself to build some higher-level API.

Also, to keep things really self-contained I don't let these things do any
I/O or data retrieval internally, either. I manually push the data into the
object (like a :write) and then manually pull it out (like a :read). That
makes it infinitely easier to create abstractions over the object, rather
than relying on a bunch of clunky callback interfaces.

Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

Soni "They/Them" L.

On 23/09/14 03:18 PM, William Ahern wrote:

> On Tue, Sep 23, 2014 at 02:23:12PM -0300, Thiago L. wrote:
>> I'm writing an IRC library as the existing ones aren't really good
>> enough (or pure Lua enough) for my purposes. I'm trying to decide on how
>> to handle the messages, but I need some input.
>>
>> First method: (I prefer this way as I think it gives the coder more
>> freedom, you can register a single handler and handle any number of
>> events you want, and you can also use this to implement the 2nd method.
>> I could also add priorities to this.)
>>
>> irc.addHandler(coroutine.create(function(...)
>>      while true do
>>          -- something here
>>          coroutine.yield()
>>      end
>> end))
> <snip>
>> Second method: (this is how LuaIRC does it... not very friendly,
>> especially when you're doing a modular bot...)
>>
>> irc.addHandler("eventName", function(...)
>> end)
>>
>> So, what do you say? Should I go for the 1st or the 2nd method, or
>> should I go for something else entirely?
> IMO both of these options are too high-level and make too many assumptions
> about the caller. I dislike APIs which make too many assumptions. I'd rather
> they make fewer assumptions, but make it trivial for me to extend things.
What do you mean too many assumptions about the caller?

>
> Whether I'm doing this in Lua, C, or some other language, I typically
> implement event emitters using a pull model. That is, the API I expose from
> the library is something like:
>
> local event = foo:get()
>
> if event.type == "bar" then
> ...
> end
>
> Basically like :read or :write, but with objects instead of strings. That
> makes the object 100% self-contained, and I don't necessarily have to reason
> about how it executes far flung bits of code if I don't want it to.
>
> Of course, many times I'm just going to end up implementing my own function
> dispatch table. But many times I won't need to. And other times I really
> want to do it myself because I might handle concurrency differently or in a
> way that would be incompatible with the way the library dispatches handlers
> directly. Fortunately, Lua's coroutines means I can usually invert the
> producer-consumer protocol; that is, coroutines often let me convert a push
> API into a pull API. But that's needless complexity.
>
> You can also layer a convenience API atop this pull model. The best APIs are
> ones that are layered atop each other. First, the user then has a choice of
> how to handle things. Secondly, nothing will shake the bugs out of an API
> better than using it yourself to build some higher-level API.
>
> Also, to keep things really self-contained I don't let these things do any
> I/O or data retrieval internally, either. I manually push the data into the
> object (like a :write) and then manually pull it out (like a :read). That
> makes it infinitely easier to create abstractions over the object, rather
> than relying on a bunch of clunky callback interfaces.
>


Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

William Ahern
On Tue, Sep 23, 2014 at 03:41:29PM -0300, Thiago L. wrote:

> On 23/09/14 03:18 PM, William Ahern wrote:
> >On Tue, Sep 23, 2014 at 02:23:12PM -0300, Thiago L. wrote:
> >>I'm writing an IRC library as the existing ones aren't really good
> >>enough (or pure Lua enough) for my purposes. I'm trying to decide on how
> >>to handle the messages, but I need some input.
> >>
> >>First method: (I prefer this way as I think it gives the coder more
> >>freedom, you can register a single handler and handle any number of
> >>events you want, and you can also use this to implement the 2nd method.
> >>I could also add priorities to this.)
> >>
> >>irc.addHandler(coroutine.create(function(...)
> >>     while true do
> >>         -- something here
> >>         coroutine.yield()
> >>     end
> >>end))
> ><snip>
> >>Second method: (this is how LuaIRC does it... not very friendly,
> >>especially when you're doing a modular bot...)
> >>
> >>irc.addHandler("eventName", function(...)
> >>end)
> >>
> >>So, what do you say? Should I go for the 1st or the 2nd method, or
> >>should I go for something else entirely?
> >IMO both of these options are too high-level and make too many assumptions
> >about the caller. I dislike APIs which make too many assumptions. I'd
> >rather
> >they make fewer assumptions, but make it trivial for me to extend things.
>
> What do you mean too many assumptions about the caller?

For one thing, that the library users wants to handle events through a
callback (push rather than pull). Depending on how the object acquires
input, that the event dispatcher will loop indepenently rather than being
controlled in a step-by-step fashion by the library user.

Google "push pull parsing". It's generally considered that pull parsers are
much easier to integrate into applications. For example,

        http://docs.oracle.com/cd/E19316-01/819-3669/bnbdy/index.html

Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

Soni "They/Them" L.

On 23/09/14 03:54 PM, William Ahern wrote:

> On Tue, Sep 23, 2014 at 03:41:29PM -0300, Thiago L. wrote:
>> On 23/09/14 03:18 PM, William Ahern wrote:
>>> On Tue, Sep 23, 2014 at 02:23:12PM -0300, Thiago L. wrote:
>>>> I'm writing an IRC library as the existing ones aren't really good
>>>> enough (or pure Lua enough) for my purposes. I'm trying to decide on how
>>>> to handle the messages, but I need some input.
>>>>
>>>> First method: (I prefer this way as I think it gives the coder more
>>>> freedom, you can register a single handler and handle any number of
>>>> events you want, and you can also use this to implement the 2nd method.
>>>> I could also add priorities to this.)
>>>>
>>>> irc.addHandler(coroutine.create(function(...)
>>>>      while true do
>>>>          -- something here
>>>>          coroutine.yield()
>>>>      end
>>>> end))
>>> <snip>
>>>> Second method: (this is how LuaIRC does it... not very friendly,
>>>> especially when you're doing a modular bot...)
>>>>
>>>> irc.addHandler("eventName", function(...)
>>>> end)
>>>>
>>>> So, what do you say? Should I go for the 1st or the 2nd method, or
>>>> should I go for something else entirely?
>>> IMO both of these options are too high-level and make too many assumptions
>>> about the caller. I dislike APIs which make too many assumptions. I'd
>>> rather
>>> they make fewer assumptions, but make it trivial for me to extend things.
>> What do you mean too many assumptions about the caller?
> For one thing, that the library users wants to handle events through a
> callback (push rather than pull). Depending on how the object acquires
> input, that the event dispatcher will loop indepenently rather than being
> controlled in a step-by-step fashion by the library user.
>
> Google "push pull parsing". It's generally considered that pull parsers are
> much easier to integrate into applications. For example,
>
> http://docs.oracle.com/cd/E19316-01/819-3669/bnbdy/index.html
>
So just use raw sockets...?

Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

Soni "They/Them" L.

On 23/09/14 03:58 PM, Thiago L. wrote:

>
> On 23/09/14 03:54 PM, William Ahern wrote:
>> On Tue, Sep 23, 2014 at 03:41:29PM -0300, Thiago L. wrote:
>>> On 23/09/14 03:18 PM, William Ahern wrote:
>>>> On Tue, Sep 23, 2014 at 02:23:12PM -0300, Thiago L. wrote:
>>>>> I'm writing an IRC library as the existing ones aren't really good
>>>>> enough (or pure Lua enough) for my purposes. I'm trying to decide
>>>>> on how
>>>>> to handle the messages, but I need some input.
>>>>>
>>>>> First method: (I prefer this way as I think it gives the coder more
>>>>> freedom, you can register a single handler and handle any number of
>>>>> events you want, and you can also use this to implement the 2nd
>>>>> method.
>>>>> I could also add priorities to this.)
>>>>>
>>>>> irc.addHandler(coroutine.create(function(...)
>>>>>      while true do
>>>>>          -- something here
>>>>>          coroutine.yield()
>>>>>      end
>>>>> end))
>>>> <snip>
>>>>> Second method: (this is how LuaIRC does it... not very friendly,
>>>>> especially when you're doing a modular bot...)
>>>>>
>>>>> irc.addHandler("eventName", function(...)
>>>>> end)
>>>>>
>>>>> So, what do you say? Should I go for the 1st or the 2nd method, or
>>>>> should I go for something else entirely?
>>>> IMO both of these options are too high-level and make too many
>>>> assumptions
>>>> about the caller. I dislike APIs which make too many assumptions. I'd
>>>> rather
>>>> they make fewer assumptions, but make it trivial for me to extend
>>>> things.
>>> What do you mean too many assumptions about the caller?
>> For one thing, that the library users wants to handle events through a
>> callback (push rather than pull). Depending on how the object acquires
>> input, that the event dispatcher will loop indepenently rather than
>> being
>> controlled in a step-by-step fashion by the library user.
>>
>> Google "push pull parsing". It's generally considered that pull
>> parsers are
>> much easier to integrate into applications. For example,
>>
>>     http://docs.oracle.com/cd/E19316-01/819-3669/bnbdy/index.html
>>
> So just use raw sockets...?
Wait what if I combine them?

Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

Michal Kolodziejczyk-3
In reply to this post by Soni "They/Them" L.


On 23.09.2014 19:23, Thiago L. wrote:

> So, what do you say? Should I go for the 1st or the 2nd method, or
> should I go for something else entirely?

You may look at the Observable pattern, introduced here:

http://techblog.netflix.com/2013/02/rxjava-netflix-api.html
https://github.com/ReactiveX/RxJava/wiki

Regards,
        miko

Reply | Threaded
Open this post in threaded view
|

Re: (multiserver/multinetwork) IRC libraries: Events/coroutines vs callbacks vs something else?

William Ahern
In reply to this post by Soni "They/Them" L.
On Tue, Sep 23, 2014 at 03:58:13PM -0300, Thiago L. wrote:
> On 23/09/14 03:54 PM, William Ahern wrote:
<snip>

> >For one thing, that the library users wants to handle events through a
> >callback (push rather than pull). Depending on how the object acquires
> >input, that the event dispatcher will loop indepenently rather than being
> >controlled in a step-by-step fashion by the library user.
> >
> >Google "push pull parsing". It's generally considered that pull parsers are
> >much easier to integrate into applications. For example,
> >
> > http://docs.oracle.com/cd/E19316-01/819-3669/bnbdy/index.html
> >
> So just use raw sockets...?

All my implementations for things like HTTP, SMTP, and even JSON (the
lexer), whether in C, Lua, or Perl, all have these two methods

  parser:parse(data) --> parses chunk of input octet stream
  parser:get() --> returns next buffered event object

If I also support composing a stream, they'll have

  composer:put(event) --> composes event object onto octet stream buffer
  composer:compose() --> returns next chunk of octet stream from buffer

They don't do any I/O interinally, although I'll also usually add a more
convenient, optional, higher-level API which handles socket I/O (e.g. the
get method is overloaded with a method which first does socket read and
parse operations). But with the low-level API I or somebody else can use any
other socket library. It also allows me to easily integrate the library into
an application regardless of whether it's threaded or non-blocking,
callback-based or stateful.

More importantly, it also makes it easier to integrate the library into
another library. So, for example, I can tie my low-level HTTP parser into a
higher-level HTTP/MIME session library, or some other HTTP-based protocol
module. Source/sink callbacks are superficially nice, but when you try
combining two separate pieces of software written by different developers
(which may be past-you and future-you) which both use source/sink callbacks,
it becomes quite ugly combining them at the boundary.[1]

Also, it makes it _much_ easier to do regression testing! Running regression
tests on libraries which do socket I/O is a pain in the butt. Even if they
offer source/sink callbacks, it's still pretty ugly compared to something
like

        parser:parse("GET / HTTP/1.1\r\nContent-Type: text/plain\r\n\r\n")
        assert(parser:get().protocol == 1.1)
        assert(parser:get().subtype == "plain")

Doing things in a way which puts the caller 100% in charge of control flow
is admittedly more tedious than simply implementing source/sink callbacks at
the interface. But it puts all the complexity where it belongs: in a black
box. And because it's easier to write regression tests, even assuming added
complexity in the logic (often you must resort to explicit state
machines--either switch based, or as has been pointed out to me before on
this list (by Dirk Laurie, I think), functional-style, tail-recursive
continuation passing[2]) your production code will often be less buggy and
easier to maintain.


[1] Every developer goes through a phase of creating "glue" libraries which
they try to use in all their own software, but which few others will use.
Take, for example, http://w3.impa.br/~diego/software/luasocket/ltn12.html.
It's an obviously great idea, tried-and-true (the interface protocol is a
superset of the above), but getting everybody to use that library in
particular is like herding cats. Also, such grand solutions can be too
abstract--i.e. they make easy problems easier but hard problems more
difficult. In any event, if you use the method above, you can easily include
an LTN12 wrapper interface; it just doesn't need to be a dependency.

[2] This is similar to callback dispatching, but it's _hidden_ in the
implementation. And it doesn't need to be generic, can be more strongly
typed, and can be refactored without worrying about breaking external
interfaces.