Module for easy multithreading

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Module for easy multithreading

Илья Удалов
Hi,

Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.

Install it from  luarocks: luarocks install effil
Sources hosted on GitHub https://github.com/effil/effil

Ilia and Miha
Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Vadi
Hi!

Looks quite good - thanks for making it!

On Fri, Sep 29, 2017 at 3:13 PM Илья Удалов <[hidden email]> wrote:
Hi,

Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.

Install it from  luarocks: luarocks install effil
Sources hosted on GitHub https://github.com/effil/effil

Ilia and Miha
Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Matthew Wild
In reply to this post by Илья Удалов
On 29 September 2017 at 14:13, Илья Удалов <[hidden email]> wrote:
> Hi,
>
> Take a look at new gorgeous lua module for multithreading - Effil.

Ah thanks! I look forward to trying this out. I cry whenever I need to
use threads in Lua (and was considering writing a new library like
this one). All the existing ones I've used try too hard, and have
weird bugs, deadlocks and crashes.

Regards,
Matthew

Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Илья Удалов
Yep. This what you get when dealing with complicated multithreading models.
We caught a lot of bugs and now it seems to me stable enough.

Meanwhile bug reports and PR are welcome:)

29 сент. 2017 г., в 17:16, Matthew Wild <[hidden email]> написал(а):

All the existing ones I've used try too hard, and have
weird bugs, deadlocks and crashes.

Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Hisham
In reply to this post by Илья Удалов
On 29 September 2017 at 10:13, Илья Удалов <[hidden email]> wrote:

> Hi,
>
> Take a look at new gorgeous lua module for multithreading - Effil.
> Main feature are:
> - thread safe tables.
> - threads can be paused/resumed and canceled.
> - FIFO channels.
>
> Install it from  luarocks: luarocks install effil
> Sources hosted on GitHub https://github.com/effil/effil
> Blog post about Effil:
> https://mihacooper.github.io/effil/lua/2017/09/28/effil/

Looking really good! I really like the documentation! I'll give it a
spin next time I need to work with threads. :)

-- Hisham

Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Russell Haley
In reply to this post by Илья Удалов
Sorry for the top post. 

Nice stuff. I like the documentation. The little comment over each part of the example code is immensely helpful.  Does the system support Windows? There was nothing I saw immediately and a search of the readme page on my phone gave no hits. ‎I did see reference to vs2015 though. 

I don't know about 'pausability', ‎but I've been looking forward to trying the threading model in cqueues (No windows support though). 

Cheers,
Russ

Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
From: Илья Удалов
Sent: Friday, September 29, 2017 6:13 AM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Module for easy multithreading

Hi,

Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.

Install it from  luarocks: luarocks install effil
Sources hosted on GitHub https://github.com/effil/effil

Ilia and Miha

Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Илья Удалов
Yes. System supports windows. It works with all features as for Mac and Linux. 

But
1. There is no continuous integration for windows builds now, only Linux and Mac due to lack of time and experience with appveyor.
2. Effil needs C++ compiler with C++14 standard compliance (introduced since Visual Studio 2015) because of dependency https://github.com/thephd/sol2.

About «pausablility», it basically means you can run thread like this:
local thread = effil.thread(function()
        while true do
            print 'Hello world'
        end
return nil
end)()

And pause/resume it like this:
thread:pause() -- suspended
thread:resume() -- works again  

Or even cancel:
thread:cancel() -- synchronous cancelation
thread:cancel(0) -- asynchronous cancelation 

P.S.
Great thanks to ThePhD for sol2. It is really easy and nice to use wrapper.
Ilia

29 сент. 2017 г., в 19:04, Russell Haley <[hidden email]> написал(а):

Sorry for the top post. 

Nice stuff. I like the documentation. The little comment over each part of the example code is immensely helpful.  Does the system support Windows? There was nothing I saw immediately and a search of the readme page on my phone gave no hits. ‎I did see reference to vs2015 though. 

I don't know about 'pausability', ‎but I've been looking forward to trying the threading model in cqueues (No windows support though). 

Cheers,
Russ

Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
From: Илья Удалов
Sent: Friday, September 29, 2017 6:13 AM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Module for easy multithreading

Hi,

Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.

Install it from  luarocks: luarocks install effil
Sources hosted on GitHub https://github.com/effil/effil

Ilia and Miha


Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Илья Удалов
In reply to this post by Илья Удалов
Yes. Effil sets debug hook. Per 200 instructions by default.
Yes. runner.step = 0 completely disable this hook.
This hook insufficiently effects on performance. About 0.5 % with default value.

You also can add cancelation point effil.yield()

--
Ilia

> On 29 Sep 2017, at 20:53, Hisham <[hidden email]> wrote:
>
> Hi,
>
> A tiny off-list question, sorry for being lazy and not checking in the code:
>
> the docs mention the runner.step function, which is specified in terms
> of number of Lua instructions. Does that mean that every thread runs
> with a debug hook plugged in by default? If so, did you notice any
> performance impact in that? Does setting it to 0 fully disable the
> hook?
>
> Thank you for this library! I have used Lanes in the past but I got an
> even better vibe from Effil's docs!
>
> -- Hisham
>
>
>> On 29 September 2017 at 10:13, Илья Удалов <[hidden email]> wrote:
>> Hi,
>>
>> Take a look at new gorgeous lua module for multithreading - Effil.
>> Main feature are:
>> - thread safe tables.
>> - threads can be paused/resumed and canceled.
>> - FIFO channels.
>>
>> Install it from  luarocks: luarocks install effil
>> Sources hosted on GitHub https://github.com/effil/effil
>> Blog post about Effil:
>> https://mihacooper.github.io/effil/lua/2017/09/28/effil/
>>
>> —
>> Ilia and Miha

Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Russell Haley
In reply to this post by Илья Удалов
So for redistribution on Windows it will require the specific vcredist file associated with vs2015 I'm assuming? 

Russ
Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
From: Илья Удалов
Sent: Friday, September 29, 2017 10:51 AM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Re: Module for easy multithreading

Yes. System supports windows. It works with all features as for Mac and Linux. 

But
1. There is no continuous integration for windows builds now, only Linux and Mac due to lack of time and experience with appveyor.
2. Effil needs C++ compiler with C++14 standard compliance (introduced since Visual Studio 2015) because of dependency https://github.com/thephd/sol2.

About «pausablility», it basically means you can run thread like this:
local thread = effil.thread(function()
        while true do
            print 'Hello world'
        end
return nil
end)()

And pause/resume it like this:
thread:pause() -- suspended
thread:resume() -- works again  

Or even cancel:
thread:cancel() -- synchronous cancelation
thread:cancel(0) -- asynchronous cancelation 

P.S.
Great thanks to ThePhD for sol2. It is really easy and nice to use wrapper.
Ilia

29 сент. 2017 г., в 19:04, Russell Haley <[hidden email]> написал(а):

Sorry for the top post. 

Nice stuff. I like the documentation. The little comment over each part of the example code is immensely helpful.  Does the system support Windows? There was nothing I saw immediately and a search of the readme page on my phone gave no hits. ‎I did see reference to vs2015 though. 

I don't know about 'pausability', ‎but I've been looking forward to trying the threading model in cqueues (No windows support though). 

Cheers,
Russ

Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
From: Илья Удалов
Sent: Friday, September 29, 2017 6:13 AM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Module for easy multithreading

Hi,

Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.

Install it from  luarocks: luarocks install effil
Sources hosted on GitHub https://github.com/effil/effil

Ilia and Miha



Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Илья Удалов
Probably not. 
I think it is up to your tool chain. Effil needs only c++14 compiler compliance and STL. These just language features, not system dependencies. So I hope you will be able to run it on any win host. Just find a suitable compiler. 

Or you can use Linux subsystem for Windows:)
Ilia

On 29 Sep 2017, at 21:17, Russell Haley <[hidden email]> wrote:

So for redistribution on Windows it will require the specific vcredist file associated with vs2015 I'm assuming? 

Russ
Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
From: Илья Удалов
Sent: Friday, September 29, 2017 10:51 AM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Re: Module for easy multithreading

Yes. System supports windows. It works with all features as for Mac and Linux. 

But
1. There is no continuous integration for windows builds now, only Linux and Mac due to lack of time and experience with appveyor.
2. Effil needs C++ compiler with C++14 standard compliance (introduced since Visual Studio 2015) because of dependency https://github.com/thephd/sol2.

About «pausablility», it basically means you can run thread like this:
local thread = effil.thread(function()
        while true do
            print 'Hello world'
        end
return nil
end)()

And pause/resume it like this:
thread:pause() -- suspended
thread:resume() -- works again  

Or even cancel:
thread:cancel() -- synchronous cancelation
thread:cancel(0) -- asynchronous cancelation 

P.S.
Great thanks to ThePhD for sol2. It is really easy and nice to use wrapper.
Ilia

29 сент. 2017 г., в 19:04, Russell Haley <[hidden email]> написал(а):

Sorry for the top post. 

Nice stuff. I like the documentation. The little comment over each part of the example code is immensely helpful.  Does the system support Windows? There was nothing I saw immediately and a search of the readme page on my phone gave no hits. ‎I did see reference to vs2015 though. 

I don't know about 'pausability', ‎but I've been looking forward to trying the threading model in cqueues (No windows support though). 

Cheers,
Russ

Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
From: Илья Удалов
Sent: Friday, September 29, 2017 6:13 AM
To: Lua mailing list
Reply To: Lua mailing list
Subject: Module for easy multithreading

Hi,

Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.

Install it from  luarocks: luarocks install effil
Sources hosted on GitHub https://github.com/effil/effil

Ilia and Miha



Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Hisham
In reply to this post by Илья Удалов
On 29 September 2017 at 15:05, Илья Удалов <[hidden email]> wrote:
> Yes. Effil sets debug hook. Per 200 instructions by default.
> Yes. runner.step = 0 completely disable this hook.
> This hook insufficiently effects on performance. About 0.5 % with default value.

Well, since you replied on-list, I'll move the conversation on-list. :)

Wouldn't it be possible to leave the hook disabled by default and only enable it
when a thread calls thr:pause()? lua_sethook is async-safe so you'd only need
a lock to make sure that the state is not lua_close'd in the meantime (and not
to lock the actual execution of the thread).

(The lua.c interpreter does a roughly similar thing to forward a
cancellation request via the signal handler.)

Just throwing an idea out there!

-- Hisham

> You also can add cancelation point effil.yield()
>
> --
> Ilia
>> On 29 Sep 2017, at 20:53, Hisham <[hidden email]> wrote:
>>
>> Hi,
>>
>> A tiny off-list question, sorry for being lazy and not checking in the code:
>>
>> the docs mention the runner.step function, which is specified in terms
>> of number of Lua instructions. Does that mean that every thread runs
>> with a debug hook plugged in by default? If so, did you notice any
>> performance impact in that? Does setting it to 0 fully disable the
>> hook?
>>
>> Thank you for this library! I have used Lanes in the past but I got an
>> even better vibe from Effil's docs!
>>
>> -- Hisham
>>
>>
>>> On 29 September 2017 at 10:13, Илья Удалов <[hidden email]> wrote:
>>> Hi,
>>>
>>> Take a look at new gorgeous lua module for multithreading - Effil.
>>> Main feature are:
>>> - thread safe tables.
>>> - threads can be paused/resumed and canceled.
>>> - FIFO channels.
>>>
>>> Install it from  luarocks: luarocks install effil
>>> Sources hosted on GitHub https://github.com/effil/effil
>>> Blog post about Effil:
>>> https://mihacooper.github.io/effil/lua/2017/09/28/effil/
>>>
>>> —
>>> Ilia and Miha

Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Joshua Jensen-2
In reply to this post by Илья Удалов
Илья Удалов wrote on 9/29/2017 7:13 AM:
Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.
Hi.

Over the years, I have invested a lot of time and scripting solutions in Lua Lanes. In general, I love it, but I am always looking for better technologies.

I assume you are aware of Lua Lanes. If so, could you cite some of the primary differences between Effil and Lua Lanes?

Thanks for your work!

Josh
Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Miha Cooper
HI, Jushua!

As far as I know Lanes uses Linda objects for data exchange. These Linda objects are key-value storages + FIFO channels with pretty specific interface. You set\send and get\receive data for particular key.

Effil was created to provide more natural or native Lua abstractions for data exchange. We had implemented shared tables which looks like regular Lua tables:
 - Our shared tables can be placed inside each other
 - All our objects are compatible with other: you can send tables through channels and put channels into tables
 - You even don't need to create shared tables directly, you can pass regular Lua table to another thread and it will be converted automatically
 - ...

So, Effil has higher level of abstraction and it's more easier to use. Btw, If it will be interesting we can make detailed comparison of Effil with Lua Lanes.


On 29 September 2017 at 23:03, Joshua Jensen <[hidden email]> wrote:
Илья Удалов wrote on 9/29/2017 7:13 AM:
Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.
Hi.

Over the years, I have invested a lot of time and scripting solutions in Lua Lanes. In general, I love it, but I am always looking for better technologies.

I assume you are aware of Lua Lanes. If so, could you cite some of the primary differences between Effil and Lua Lanes?

Thanks for your work!

Josh



--
With best regards,
Kupriyanov Mikhail.
Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Илья Удалов
In reply to this post by Илья Удалов
Hi Joshua!

Basically Effil brings inter threads shares tables.
You to use nested  tables over threads (even recursively):
local effil = require "effil"

local t = effil.table {1, "I am string"}
local res = effil.thread(
function(t)
    t.self = t
end)(t):get()

print(t.self.self.self.self[2])

Also with metatables:
local greeting = effil.table()
effil.setmetatable(greeting, { __call = function(t, name)
    print("Hi " .. name)
end })

effil.thread(function (greeting)
    greeting("Lua mailing list")
end)(greeting):wait()

Effil have linda like objects with almost same API - channels. It also provides operations with timeouts.
Effil can cancel thread like lanes. But currently it does not have API for change thread priority https://github.com/effil/effil/issues/77
Effil doesn't have locks and times as separate API. Locks seems to be dangerous in multithreading-cancelable environment. Very easy make deadlock. But if you really need them - you can implement locks and timers over effil.channel. 

Ilia
29 сент. 2017 г., в 16:13, Илья Удалов <[hidden email]> написал(а):

Hi,

Take a look at new gorgeous lua module for multithreading - Effil.
Main feature are:
- thread safe tables.
- threads can be paused/resumed and canceled.
- FIFO channels.

Install it from  luarocks: luarocks install effil
Sources hosted on GitHub https://github.com/effil/effil

Ilia and Miha

Reply | Threaded
Open this post in threaded view
|

Re: Module for easy multithreading

Илья Удалов
In reply to this post by Hisham
Ok, Thank you for idea:)

Ilia

29 сент. 2017 г., в 22:44, Hisham <[hidden email]> написал(а):

Well, since you replied on-list, I'll move the conversation on-list. :)

Wouldn't it be possible to leave the hook disabled by default and only enable it
when a thread calls thr:pause()? lua_sethook is async-safe so you'd only need
a lock to make sure that the state is not lua_close'd in the meantime (and not
to lock the actual execution of the thread).

(The lua.c interpreter does a roughly similar thing to forward a
cancellation request via the signal handler.)

Just throwing an idea out there!