Quantcast

p2p parrarel/ cluster computing.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

p2p parrarel/ cluster computing.

Sam Bavi
Hello everybody, 
Could those that know please help me by pointing me In the right direction for learning p2p parallel computing and grid computing from the beginning the right way. Im fairly comfy with programing in Lua, Python, C++ but do not have a computer science degree. Thats important to point out because most resources that I have stumbled upon the last couple Of days do seem to be directed at people with a CS-degree. Normally I would just jump In and learn its and bits here and there, but I feel like when It comes to parrarel computing I would like to learn some good basics first. So any tips on books, topics or other resources would be much appreciated. What Im specifically looking for at the moment Is a server-less p2p parallel processing framework. Apache hadboop Is very nice but Its very much server oriented and Im looking for more something like local client/server distribution of recourses. Apache Spark Is on top on my list right now because It does have a standalone cluster mode, but Its very thin and seems to be more oriented towards testing purposes. The other candidate Is PGU File and CPU Distributor but there docs are way to thin for me to be able to dig in, but to do have some nice pdf:s (semester papers) that are appreciated. I could use something like GPU but then my hands would be very much so tied because of my slim knowledge. Sorry for my bad grammars, english Is not my first language, please let me know If the question is wage and I will do my best to clarify. The thing Is, Its hard for me to say I need info on this specific thing because I don't know yet what exactly It Is that I need to learn. The only thing that I do know Is that I have read some really good papers the last couple of days, and I can follow along fairly good but I feel like Im failing to understand the basic principles and the logic behind grids, clusters and so on. 

Best Regards
Sam Bavi
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Dirk Laurie-2
2017-05-13 15:52 GMT+02:00 Sam Bavi <[hidden email]>:

> Could those that know please help me by pointing me In the right direction
> for learning p2p parallel computing and grid computing from the beginning
> the right way.

What is the computing task that you need parallel computing for?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Martin
In reply to this post by Sam Bavi
On 05/13/2017 06:52 AM, Sam Bavi wrote:

> Could those that know please help me by pointing me In the right
> direction for learning p2p parallel computing and grid computing from
> the beginning the right way. Im fairly comfy with programing in Lua,
> Python, C++ but do not have a computer science degree. Thats important
> to point out because most resources that I have stumbled upon the last
> couple Of days do seem to be directed at people with a CS-degree.
> Normally I would just jump In and learn its and bits here and there, but
> I feel like when It comes to parrarel computing I would like to learn
> some good basics first. So any tips on books, topics or other resources
> would be much appreciated. What Im specifically looking for at the
> moment Is a server-less p2p parallel processing framework. Apache
> hadboop Is very nice but Its very much server oriented and Im looking
> for more something like local client/server distribution of recourses.
> Apache Spark Is on top on my list right now because It does have a
> standalone cluster mode, but Its very thin and seems to be more oriented
> towards testing purposes. The other candidate Is PGU File and CPU
> Distributor but there docs are way to thin for me to be able to dig in,
> but to do have some nice pdf:s (semester papers) that are appreciated. I
> could use something like GPU but then my hands would be very much so
> tied because of my slim knowledge. Sorry for my bad grammars, english Is
> not my first language, please let me know If the question is wage and I
> will do my best to clarify. The thing Is, Its hard for me to say I need
> info on this specific thing because I don't know yet what exactly It Is
> that I need to learn. The only thing that I do know Is that I have read
> some really good papers the last couple of days, and I can follow along
> fairly good but I feel like Im failing to understand the basic
> principles and the logic behind grids, clusters and so on.

I think "right direction" is use line breaks more often. There are no
taxes for empty lines.

-- Martin

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Charles Heywood
Not only that, but it makes anything (code, text) more readable.

On Sat, May 13, 2017, 1:53 PM Martin <[hidden email]> wrote:
On 05/13/2017 06:52 AM, Sam Bavi wrote:
> Could those that know please help me by pointing me In the right
> direction for learning p2p parallel computing and grid computing from
> the beginning the right way. Im fairly comfy with programing in Lua,
> Python, C++ but do not have a computer science degree. Thats important
> to point out because most resources that I have stumbled upon the last
> couple Of days do seem to be directed at people with a CS-degree.
> Normally I would just jump In and learn its and bits here and there, but
> I feel like when It comes to parrarel computing I would like to learn
> some good basics first. So any tips on books, topics or other resources
> would be much appreciated. What Im specifically looking for at the
> moment Is a server-less p2p parallel processing framework. Apache
> hadboop Is very nice but Its very much server oriented and Im looking
> for more something like local client/server distribution of recourses.
> Apache Spark Is on top on my list right now because It does have a
> standalone cluster mode, but Its very thin and seems to be more oriented
> towards testing purposes. The other candidate Is PGU File and CPU
> Distributor but there docs are way to thin for me to be able to dig in,
> but to do have some nice pdf:s (semester papers) that are appreciated. I
> could use something like GPU but then my hands would be very much so
> tied because of my slim knowledge. Sorry for my bad grammars, english Is
> not my first language, please let me know If the question is wage and I
> will do my best to clarify. The thing Is, Its hard for me to say I need
> info on this specific thing because I don't know yet what exactly It Is
> that I need to learn. The only thing that I do know Is that I have read
> some really good papers the last couple of days, and I can follow along
> fairly good but I feel like Im failing to understand the basic
> principles and the logic behind grids, clusters and so on.

I think "right direction" is use line breaks more often. There are no
taxes for empty lines.

-- Martin

--
--

Software Developer / System Administrator
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Sam Bavi
In reply to this post by Dirk Laurie-2
Point taken about line breaks and your right.

Well the ide is to be able to do 3d animation and art network rendering through some sort of p2p, just for an example the way It’s done now in Cinema4D which is my primary design software most people will either need to be on a large lan and have cinema 4D plus every plugin that is being used on your project on the host computer, that means multiple licenses and some of the plugins don’t come cheap.

The other options is render farm where we basically hire the amount of pc power that one may need, but thats not always so easy as the render farm must have the plugins that I have to be able to render the work and thats not always the case.

Now my thought Is to create an online p2p render-farm where anybody could login and make there assets available and at the sometime get help with power when needed. Take me for example I have my 16gig i7 windows PC lying around here doing nothing because I use my work computer, that would do a huge amount of help for anybody that needed It. But this would just extend the current cinema 4D lan render functionality so one would still need to match the clients plugin, licenses and so forth.

The second option would be to just lend resources, something like GPU which Is a Gnutella client that shares CPU-resources. That would eliminate the need to have extra licenses and installed plugins and much of the hassle, but Im not sure If that is possible when It comes to 3D rendering or any render for that matter, right now Its just a thought. I use arnold renderer for maya and c4d, and arnold has Its own special way of doing node calculations so the software Is definitely needed to render a scene, but Is it possible to just get help from another pc to do calculations and not the main render It self? 

I may be wrong but basically the the reason why graphic renders are so heavy duty like arnold, c4d, filter forge to name a few Is lightening calculations, the more and bigger light-setup the more iterations one needs. So Im thinking If It would help (logically It should) to just get help with the calculations form say a bigger p2p farm. But like a said before my knowledge of how cpu’s work and the low level programing is very thin so Im writing what I think should logically work. Something like how OpenMP works and the only Lua framework that I know of that comes to mind here is Torch.

http://lua-users.org/wiki/MultiTasking and ”LUA-ZMQ” looks interesting and theres also lua-parallel which is a framework for Torch and also uses ZeroMq like lua-zmq, anybody had any experience with these two?

But like I said all that last bit seems just like a thought cloud at the moment, the first part thou with 2p2 rendering thats what I would like to start with as I think Its a little bit closer to my knowledge scope.  My thought Is to have something like the way dc++ was built once upon a time, It has to be secure, encrypted, have some sort of que system, one2one file sharing obviously. 

It should be able to ask the client/ server If one has all the necessary plugins and resources for a good render before sending parts of the file, some sort of error check to se If the part that came back is what Its should be, and also I think I would add some sort of busy check so that a service that Is already doing heavy work locally doesn’t get bogged down, sadly that check Is not done at the moment with the network renders from maya and maxon, and Of course last but not least some sort of cluster grid parallel processing. 

thats what I can think of right now, please anybody fill in an blanks If you se them.

On a side note, I recall visiting a page last time I tried Torch that had a full list of downloadable frameworks for specifically Luarocks/Torch. I cant find the page now and was wondering anybody knows which one I mean? It had some Torch - Phyton bindings that I would like to find.

Regards

På 13 maj 2017 kl. 18:29:49, Dirk Laurie ([hidden email]) skrev:

2017-05-13 15:52 GMT+02:00 Sam Bavi <[hidden email]>:

> Could those that know please help me by pointing me In the right direction
> for learning p2p parallel computing and grid computing from the beginning
> the right way.

What is the computing task that you need parallel computing for?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Sean Conner
It was thus said that the Great Sam Bavi once stated:

>
> The second option would be to just lend resources, something like GPU
> which Is a Gnutella client that shares CPU-resources. That would eliminate
> the need to have extra licenses and installed plugins and much of the
> hassle, but Im not sure If that is possible when It comes to 3D rendering
> or any render for that matter, right now Its just a thought. I use arnold
> renderer for maya and c4d, and arnold has Its own special way of doing
> node calculations so the software Is definitely needed to render a scene,
> but Is it possible to just get help from another pc to do calculations and
> not the main render It self? 

  It doesn't work that way.  At the very bottom you have a "unit of
execution" (not a standard term at all, but I find it a very useful
concept).  I'll use the x86 to explain the "unit of execution" as it's the
most popular CPU in use and it'll give a concrete example of what I mean.  A
"unit of execution" is the CPU state required to run a program.  That
includes a unique IP (instruction pointer---the actual name depends upon the
mode, IP is 16 bit, EIP is 32 bit, RIP is 64 bit, but they're the same
register), an execution stack (recording subroutine return addresses) stored
in SP and the rest of the registers.  While two different "unit of
executions" might be running the same code (their IP registers match) they
don't necessarily have the same state (the stack will be different and quite
possibly the registers as well).  It's just that they are distinct "CPU
states".

  A CPU executes instructions.  It reads the next instruction pointed to by
the IP, decodes and executes it.  It really doesn't matter if it's a
physical CPU (like the x86) or a VM (like the Lua VM).  At a high level,
they both pull in the next instruction, decode and execute it.

  A "unit of execution" can then be used to construct coroutines, threads,
processes and other, more well known concepts of "units of execution".  So,
"unit of execution" is a CPU state.

  Now, a brief overview of concurrency and parallelism.  A good answer to
this is:

http://stackoverflow.com/questions/1897993/what-is-the-difference-between-concurrent-programming-and-parallel-programming/3982782#3982782

  And a nice diagram (a good if slightly misleading) is further down on that
page, which can be summed up as:

        Concurrency:  two queues leading to one coffee machine
        Parallelism: two queues leading to two coffee machines

  Before multiprocessing machines became mainstream, everything (at least in
a multitasking operating system) was concurrent.  In fact:

        multitasking: running multiple programs on a single CPU
        multiprocessing: running multiple programs on multiple CPUs

  You have a program that does heavy calculations (rendering, math
simulation, generating fractals, what have you).  It can easily be split to
run on multiple "units of execution" but to really gain speed, each "unit of
execution" has to be a separate CPU so they can run in parallel.  On a
system with multiple CPUs this is easy, as each CPU shares the same memory
(there are exceptions but generally this is true).

  But to move the "unit of execution" to another CPU on another system is
not so straightforward.  You can't just *move* the CPU from system X to your
system; you have to *move* the program from your system to system X.  Or
system X has to have the program already on it.  There's no way around it,
and both come down to the same thing:

        the program has to exist on all systems

> Something like how
> OpenMP works and the only Lua framework that I know of that comes to mind
> here is Torch.

  OpenMP makes it easy to write software for a multiprocessor system; it
doesn't help in writing software to run across distinct (separate)
computers.

> http://lua-users.org/wiki/MultiTasking and ”LUA-ZMQ” looks interesting and
> theres also lua-parallel which is a framework for Torch and also uses
> ZeroMq like lua-zmq, anybody had any experience with these two?

  ZeroMQ is a networking abstraction that makes it easy to write network
code that can do one-to-one, one-to-many, many-to-one, and many-to-many
distribution of messages; it has nothing to do with distribution of work per
se.  You *can* use it to distribute work and programs, but you us ZeroMQ to
distribute the messages that contain the work or program (and you certainly
can do the same without ZeroMQ and use the underlying OS networking layer;
like I said, ZeroMQ is just an abstraction over that).

> and also I think I would add some sort of busy check so that a
> service that Is already doing heavy work locally doesn’t get bogged down,

  The easy way to solve that is to post the "job" at a central location, and
each client then pulls a "job" to work on.  When it finishes with the job,
it posts the results and then grabs the next "job" to work on.  This is a
"pull" architecture.  What you are describing is a "push" architecture,
which is not a good fit for this particular problem (it has uses, but not
for this).

  -spc

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Sam Bavi
Thank you for that thorough answer, really appreciated =)) So basically loosely speaking the pull mvc is much like in ftp where I choose to tell model Im ready now present me with new data. What Is a p2p torrent application considered where the relationship between the view and the model is much more of a blur? Its a interesting choice which I haven’t been thinking about,  on one hand push leaves less room for idle clients and so on but Is more careless with the data, logically pull creates a more sensible flow of data but leaves more room for time wast because Its up to the client to check in, something like that? 

Based on your suggestions I will check out torch-ipc tomorrow and se where It goes, https://github.com/twitter/torch-ipc. From what I’ve read It looks very promising. Again Thanks for taking your time and have a good night =)) Regards

På 14 maj 2017 kl. 00:44:44, Sean Conner ([hidden email]) skrev:

It was thus said that the Great Sam Bavi once stated:

>
> The second option would be to just lend resources, something like GPU
> which Is a Gnutella client that shares CPU-resources. That would eliminate
> the need to have extra licenses and installed plugins and much of the
> hassle, but Im not sure If that is possible when It comes to 3D rendering
> or any render for that matter, right now Its just a thought. I use arnold
> renderer for maya and c4d, and arnold has Its own special way of doing
> node calculations so the software Is definitely needed to render a scene,
> but Is it possible to just get help from another pc to do calculations and
> not the main render It self? 

It doesn't work that way. At the very bottom you have a "unit of
execution" (not a standard term at all, but I find it a very useful
concept). I'll use the x86 to explain the "unit of execution" as it's the
most popular CPU in use and it'll give a concrete example of what I mean. A
"unit of execution" is the CPU state required to run a program. That
includes a unique IP (instruction pointer---the actual name depends upon the
mode, IP is 16 bit, EIP is 32 bit, RIP is 64 bit, but they're the same
register), an execution stack (recording subroutine return addresses) stored
in SP and the rest of the registers. While two different "unit of
executions" might be running the same code (their IP registers match) they
don't necessarily have the same state (the stack will be different and quite
possibly the registers as well). It's just that they are distinct "CPU
states".

A CPU executes instructions. It reads the next instruction pointed to by
the IP, decodes and executes it. It really doesn't matter if it's a
physical CPU (like the x86) or a VM (like the Lua VM). At a high level,
they both pull in the next instruction, decode and execute it.

A "unit of execution" can then be used to construct coroutines, threads,
processes and other, more well known concepts of "units of execution". So,
"unit of execution" is a CPU state.

Now, a brief overview of concurrency and parallelism. A good answer to
this is:

http://stackoverflow.com/questions/1897993/what-is-the-difference-between-concurrent-programming-and-parallel-programming/3982782#3982782

And a nice diagram (a good if slightly misleading) is further down on that
page, which can be summed up as:

Concurrency: two queues leading to one coffee machine
Parallelism: two queues leading to two coffee machines

Before multiprocessing machines became mainstream, everything (at least in
a multitasking operating system) was concurrent. In fact:

multitasking: running multiple programs on a single CPU
multiprocessing: running multiple programs on multiple CPUs

You have a program that does heavy calculations (rendering, math
simulation, generating fractals, what have you). It can easily be split to
run on multiple "units of execution" but to really gain speed, each "unit of
execution" has to be a separate CPU so they can run in parallel. On a
system with multiple CPUs this is easy, as each CPU shares the same memory
(there are exceptions but generally this is true).

But to move the "unit of execution" to another CPU on another system is
not so straightforward. You can't just *move* the CPU from system X to your
system; you have to *move* the program from your system to system X. Or
system X has to have the program already on it. There's no way around it,
and both come down to the same thing:

the program has to exist on all systems

> Something like how
> OpenMP works and the only Lua framework that I know of that comes to mind
> here is Torch.

OpenMP makes it easy to write software for a multiprocessor system; it
doesn't help in writing software to run across distinct (separate)
computers.

> http://lua-users.org/wiki/MultiTasking and ”LUA-ZMQ” looks interesting and
> theres also lua-parallel which is a framework for Torch and also uses
> ZeroMq like lua-zmq, anybody had any experience with these two?

ZeroMQ is a networking abstraction that makes it easy to write network
code that can do one-to-one, one-to-many, many-to-one, and many-to-many
distribution of messages; it has nothing to do with distribution of work per
se. You *can* use it to distribute work and programs, but you us ZeroMQ to
distribute the messages that contain the work or program (and you certainly
can do the same without ZeroMQ and use the underlying OS networking layer;
like I said, ZeroMQ is just an abstraction over that).

> and also I think I would add some sort of busy check so that a
> service that Is already doing heavy work locally doesn’t get bogged down,

The easy way to solve that is to post the "job" at a central location, and
each client then pulls a "job" to work on. When it finishes with the job,
it posts the results and then grabs the next "job" to work on. This is a
"pull" architecture. What you are describing is a "push" architecture,
which is not a good fit for this particular problem (it has uses, but not
for this).

-spc

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Sean Conner
It was thus said that the Great Sam Bavi once stated:
> Thank you for that thorough answer, really appreciated =)) So basically
> loosely speaking the pull mvc is much like in ftp where I choose to tell
> model Im ready now present me with new data. What Is a p2p torrent
> application considered where the relationship between the view and the
> model is much more of a blur?

  I can't answer that as I've never used Bit Torrent, nor have I even looked
at Bit Torrent.  I also found it a bit surprising you mentioned MVC as it
doesn't really fit here (it's was originally intended for user interfaces
and is a staple of the object-oriented crowd, of which I'm not one).

> Its a interesting choice which I haven’t
> been thinking about,  on one hand push leaves less room for idle clients
> and so on but Is more careless with the data, logically pull creates a
> more sensible flow of data but leaves more room for time wast because Its
> up to the client to check in, something like that? 

  With push, you need to inform the server of all clients, and its up to the
server to keep them busy.  I have a job, which client is free for the work?
This one ... oh, it's down.  I need to check another one ... and so on.  For
your render farm, it's easier to implement the pull method, where each
client asks the server for a job (or jobs, depending upon the amount of work
it can do and the client has more information about what its capable of than
trying to keep the server up to date with such information).

  You can think of pull vs push also as polling vs interrupts.  If work (or
events) are spradic, it makes more sense to interrupt (or push); but if the
work (or events) are constant, then polling (or pulling) is better as there
usually is something always available and you can avoid the overhead of
interrupts (or pushing).  

  Which you use really depends upon the frequency of work, as well as the
workload of each job.  In rendering, there's a large frequency of work, each
of which takes time; in that case, having the clients pull for jobs makes
the distribution of jobs easier (you can always add more clients).

> Based on your suggestions I will check out torch-ipc tomorrow and se where
> It goes, https://github.com/twitter/torch-ipc. From what I’ve read It
> looks very promising. Again Thanks for taking your time and have a good
> night =)) Regards

  From a quick look, torch-ipc just seems like yet another IPC mechanism,
like ZeroMQ.  Whatever works I guess.

  -spc


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Pierre Chapuis
In reply to this post by Sam Bavi
May 13, 2017 10:44 PM, "Sam Bavi" <[hidden email]> wrote:
Now my thought Is to create an online p2p render-farm where anybody could login and make there assets available and at the sometime get help with power when needed. Take me for example I have my 16gig i7 windows PC lying around here doing nothing because I use my work computer, that would do a huge amount of help for anybody that needed It. But this would just extend the current cinema 4D lan render functionality so one would still need to match the clients plugin, licenses and so forth.

This is complicated stuff. Don't start from scratch and reinvent the wheel.
In particular, look at BOINC and BURP.

http://burp.renderfarming.net/intro.php
https://en.wikipedia.org/wiki/Big_and_Ugly_Rendering_Project

--
Pierre Chapuis
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Pierre Chapuis
In reply to this post by Sam Bavi
May 13, 2017 10:44 PM, "Sam Bavi" <[hidden email]> wrote:
> Now my thought Is to create an online p2p render-farm where anybody could login and make there
> assets available and at the sometime get help with power when needed. Take me for example I have my
> 16gig i7 windows PC lying around here doing nothing because I use my work computer, that would do a
> huge amount of help for anybody that needed It. But this would just extend the current cinema 4D
> lan render functionality so one would still need to match the clients plugin, licenses and so
> forth.

Sorry about the formatting of my previous message, this one should be better.

This is complicated stuff. Don't start from scratch and reinvent the wheel.
In particular, look at BOINC and BURP.

http://burp.renderfarming.net/intro.php
https://en.wikipedia.org/wiki/Big_and_Ugly_Rendering_Project

--
Pierre Chapuis

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: p2p parrarel/ cluster computing.

Andrew Starks-2
In reply to this post by Pierre Chapuis

On Sun, May 14, 2017 at 04:58 Pierre Chapuis <[hidden email]> wrote:
May 13, 2017 10:44 PM, "Sam Bavi" <[hidden email]> wrote:
Now my thought Is to create an online p2p render-farm where anybody could login and make there assets available and at the sometime get help with power when needed. Take me for example I have my 16gig i7 windows PC lying around here doing nothing because I use my work computer, that would do a huge amount of help for anybody that needed It. But this would just extend the current cinema 4D lan render functionality so one would still need to match the clients plugin, licenses and so forth.

This is complicated stuff. Don't start from scratch and reinvent the wheel.
In particular, look at BOINC and BURP.

http://burp.renderfarming.net/intro.php
https://en.wikipedia.org/wiki/Big_and_Ugly_Rendering_Project


--
Pierre Chapuis
I will add Lua Lanes to your list, as well:



Loading...