Energy Benchmarks

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Energy Benchmarks

Italo Maia

Opinions?

--
"A arrogância é a arma dos fracos."

===========================
Me. Italo Moreira Campelo Maia
Co-fundador do Grupo de Usuários Python do Ceará
Secretário ForHacker (fb.com/ForHackerSpace)
Desenvolvedor Full-Stack, Escritor, Empresário, Visionário
-----------------------------------------------------
Meu Livro, Site, Blog
===========================
Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Dirk Laurie-2
I had a brief look at the Lua source code for the problem on which I
have some expertise, namely the digits-of-pi benchmark.

It uses a frontend to a Lua binding to GMP.

Sample function from that frontend:

local function checkf(obj)
assert(isf(obj), "gmp floating point expected")
end

local function checkfopt(obj)
assert(obj == nil or isf(obj), "gmp floating point expected")
end

function zmeta:add(a, res)
    checkz(self)
    checkzopt(res)
    local t = ntype(a)
    if t == "d" then
        return prv.mpz_add(self, dtoz(a), res)
    elseif t == "u" then
        return prv.mpz_add_ui(self, a, res)
    elseif t == "s" or t == "n" then
        return prv.mpz_sub_ui(self, -a, res)
    elseif t == "z" then
        return prv.mpz_add(self, a, res)
    else
        error("unsupported type")
    end
end

I.e. all debugging aids have been left in, and the rest of the code
shows every sign of having been translated thoughtlessly from C.

Frankly, that tells me more about the level of competence of the team
who conducted the study than about the languages involved.

-- Dirk


Op Di. 2 Apr. 2019 om 09:24 het Italo Maia <[hidden email]> geskryf:

>
> https://thenewstack.io/which-programming-languages-use-the-least-electricity/
>
> Opinions?
>
> --
> "A arrogância é a arma dos fracos."
>
> ===========================
> Me. Italo Moreira Campelo Maia
> Co-fundador do Grupo de Usuários Python do Ceará
> Secretário ForHacker (fb.com/ForHackerSpace)
> Desenvolvedor Full-Stack, Escritor, Empresário, Visionário
> -----------------------------------------------------
> Meu Livro, Site, Blog
> ===========================

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Frank Kastenholz-2


> On Apr 2, 2019, at 4:24 AM, Dirk Laurie <[hidden email]> wrote:
>
>  ... the rest of the code
> shows every sign of having been translated thoughtlessly from C.
>
> Frankly, that tells me more about the level of competence of the team
> who conducted the study than about the languages involved.
>
> -- Dirk

Indeed
Looking at the performance numbers (C is 1.0 vs lua at 83...) makes me agree.

Several years ago a project I was on took a C program and ported it to Lua.  The initial port, by the writers of the C program, ended up with the Lua program running 50-100x as slow as the original C program.  A few simple optimizations (using lots of locals and rewriting some loops that kept evaluating a constant) got us into the 5-10x range ... which was satisfactory for our needs.

My take from that project was that the “problem” was that the app developers were not really proficient in Lua AND had a narrow view of things (‘the compiler will optimize things so I don’t have to worry about it’) - leading to suboptimal performance.

The memory footprint of the program we were doing was not an issue.  The data for the program, as well as the fixed size of the environment and libraries we developed, vastly outweighed the lua source and LVM bytecode size so any inefficiencies on Lua’s part were really irrelevant.  

We did not look at power consumption

Frank Kastenholz




Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Scott Morgan
In reply to this post by Italo Maia
On 02/04/2019 08:23, Italo Maia wrote:
>

One thing I've run into recently was a need to sleep the app regularly.

I don't think there's anything in ANSI C that can handle that, so the
alternatives are: to loop whilst checking os.time, not very energy
efficient, or os.execute an external sleep util (the solution I took),
not very portable.

Of course, there are libraries that provide the functionality. Just
depends how 'pure' you want things.

Scott

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Peter Hickman-3
On Tue, 2 Apr 2019 at 12:00, Scott Morgan <[hidden email]> wrote:
On 02/04/2019 08:23, Italo Maia wrote:
I don't think there's anything in ANSI C that can handle that

Seems to be, but to use it in Lua you need to add it as a library. This is how I handled it https://github.com/PeterHickman/LuaLibs look in the sleep directory. Works on OSX and Linux, I suspect that it will break on Windows (which I do not have access to)

sleep really does seem to be a glaring omission in Lua, along with time at the millisecond level, but nothing that can't be fixed :)

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Hugo Musso Gualandi
In reply to this post by Frank Kastenholz-2



>> Frankly, that tells me more about the level of competence of the team
>> who conducted the study than about the languages involved.
>>
>> -- Dirk
>
>Indeed
>Looking at the performance numbers (C is 1.0 vs lua at 83...) makes me
>agree.

Thr benchmarks in this study were taken from the Computer Language Benchmarks Game set, which has some known peculiarities. For example, many examples measure library speed more than language speed. Nevertheless, this set of benchmarks continues to be very popular for this kind of study, due to its breadth. Nothing comes as close to including as many languages as the CLBG.

As for the unusual C vs Lua difference, most of the Lua benchmarks were written by competent programmers, including Mike Pall. If anything it is the C code that is too optimized here. C benchmark implementations in the CLBG are often an unreadable mix of calls to compiler intrinsics, SIMD instructions, and parallelization primitives. This kind of code can easily be an order of magnitude faster than the usual idiomatic C alternative.

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Egor Skriptunoff-2
In reply to this post by Italo Maia
On Tue, Apr 2, 2019 at 10:24 AM Italo Maia wrote:


The measurements are probably incorrect,
but the central idea is pretty valid:
Lua gives comfort of programming
at the cost of CPU/memory consumption.

So, we have new motto for Lua:

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

joy mondal
These benchmarks in my opinion are pointless.

Trying to measure programming languages based on how much electricity they use is a lot like ranking aircrafts by their weight - completely meaningless metric.

If Erlang was so slow, it wouldn't be running WhatsApp.

If C is so efficient then no programming languages would have been created since 1970s.

Lua was designed to be tiny and easy to embed.

- Easy to write.
- Easy to upgrade and modify.

If Lua was geared towards performance it would look a lot like C++ or Rust - thank God it doesn't .

-----------------------------

Also the biggest consumers of electric power are humans, by that logic Java would be at the bottom due to the sheer biomass of people writing it.

Programmer time is worth a lot more than CPU time, that should be the biggest thing people concentrate on optimizing.

Also I using Lua for Solar Energy, in theory Lua is creating electricity !!

On Thu, Apr 4, 2019 at 9:04 AM Egor Skriptunoff <[hidden email]> wrote:
On Tue, Apr 2, 2019 at 10:24 AM Italo Maia wrote:


The measurements are probably incorrect,
but the central idea is pretty valid:
Lua gives comfort of programming
at the cost of CPU/memory consumption.

So, we have new motto for Lua:

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Enrico Colombini
In reply to this post by Egor Skriptunoff-2
On 04-Apr-19 07:03, Egor Skriptunoff wrote:

> On Tue, Apr 2, 2019 at 10:24 AM Italo Maia wrote:
>
>     https://thenewstack.io/which-programming-languages-use-the-least-electricity/
>
>     Opinions?
>
> The measurements are probably incorrect,
> but the central idea is pretty valid:
> Lua gives comfort of programming
> at the cost of CPU/memory consumption.
>
> So, we have new motto for Lua:
> https://i.imgur.com/JdcFbr0.png

If you don't count the energy used while writing a complex high-level
program in C :-)

Another possible conclusion could be: do not think, the brain requires
energy.

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Frank Kastenholz-2
In reply to this post by joy mondal


> On Apr 4, 2019, at 1:35 AM, joy mondal <[hidden email]> wrote:
>
> Trying to measure programming languages based on how much electricity they use is a lot like ranking aircrafts by their weight - completely meaningless metric.

There are environments where the energy supply is extremely limited, such as battery powered (or pv powered or motion powered) devices and where the device MUST remain operational for a specified period of time, so consumption is important to measure.

Whether _this_ benchmark accurately measures that consumption is a separate question.

Frank



Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Scott Morgan
On 04/04/2019 10:44, Frank Kastenholz wrote:

>
>
>> On Apr 4, 2019, at 1:35 AM, joy mondal <[hidden email]> wrote:
>>
>> Trying to measure programming languages based on how much electricity they use is a lot like ranking aircrafts by their weight - completely meaningless metric.
>
> There are environments where the energy supply is extremely limited, such as battery powered (or pv powered or motion powered) devices and where the device MUST remain operational for a specified period of time, so consumption is important to measure.
>
> Whether _this_ benchmark accurately measures that consumption is a separate question.
>

Also, in big data centres, where thousands of machines are running the
same code, power consumption (and heat dispersal) are important issues
that make these sorts of benchmarks useful.

I'll echo the question about how good this particular paper is at
measuring these things.

Scott

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Italo Maia
I wonder how this benchmark would look like without the worst case algorithm implementation for each language. 

Another thing that worries me is the bad propaganda of such results given tests might have issues, as mentioned.

I have a hard time phantoming what could cause that Lua memory result. 

Em qui, 4 de abr de 2019 12:09, Scott Morgan <[hidden email]> escreveu:
On 04/04/2019 10:44, Frank Kastenholz wrote:
>
>
>> On Apr 4, 2019, at 1:35 AM, joy mondal <[hidden email]> wrote:
>>
>> Trying to measure programming languages based on how much electricity they use is a lot like ranking aircrafts by their weight - completely meaningless metric.
>
> There are environments where the energy supply is extremely limited, such as battery powered (or pv powered or motion powered) devices and where the device MUST remain operational for a specified period of time, so consumption is important to measure.
>
> Whether _this_ benchmark accurately measures that consumption is a separate question.
>

Also, in big data centres, where thousands of machines are running the
same code, power consumption (and heat dispersal) are important issues
that make these sorts of benchmarks useful.

I'll echo the question about how good this particular paper is at
measuring these things.

Scott

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

joy mondal
Anybody who looks at this results and says :

" Jeez Lua is so inefficient, I won't use it "

I am glad they go and use C.

Lua was never designed to minimize energy usage, i do not see energy use mentioned anywhere on lua's wiki page.

I want a ranking of :

- embedding with C, Lua will beat every other language on this.

- compiler complexity, Lua will beat every other language on this.

If somebody creates a language that causes lua to lose on those front, I would agree that that would be the time to start freaking out, but for now Lua maintains a monopoly.

Its like somebody coming to me and asking them to help them do a cardiac bypass surgery - its not my job !

Lua excels in things it set as its goals. Many languages that set their goals to be something and never get to achieve it.

- "Java write once run everywhere" someone ??


On Sat, Apr 6, 2019 at 4:22 PM Italo Maia <[hidden email]> wrote:
I wonder how this benchmark would look like without the worst case algorithm implementation for each language. 

Another thing that worries me is the bad propaganda of such results given tests might have issues, as mentioned.

I have a hard time phantoming what could cause that Lua memory result. 

Em qui, 4 de abr de 2019 12:09, Scott Morgan <[hidden email]> escreveu:
On 04/04/2019 10:44, Frank Kastenholz wrote:
>
>
>> On Apr 4, 2019, at 1:35 AM, joy mondal <[hidden email]> wrote:
>>
>> Trying to measure programming languages based on how much electricity they use is a lot like ranking aircrafts by their weight - completely meaningless metric.
>
> There are environments where the energy supply is extremely limited, such as battery powered (or pv powered or motion powered) devices and where the device MUST remain operational for a specified period of time, so consumption is important to measure.
>
> Whether _this_ benchmark accurately measures that consumption is a separate question.
>

Also, in big data centres, where thousands of machines are running the
same code, power consumption (and heat dispersal) are important issues
that make these sorts of benchmarks useful.

I'll echo the question about how good this particular paper is at
measuring these things.

Scott

Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

Italo Maia
I understand that energt efficiency and memory usage (not sure on this) could not be gols for Lua, as it's bread and butter are being simple (not simplistic) and easely embedable but ... should those be the only bread and butter of the language? 

As Lua outgrows its niche, I think benchmarks like those will have more and more weight when deciding to adopt Lua. Even in the embedable and enterprise world, as Scott put it, energy efficiency is a thing. 

More importantly: do we have leads on why Lua scored so poorly in this benchmark besides some copy/paste algorithms? 


Em dom, 7 de abr de 2019 às 23:03, joy mondal <[hidden email]> escreveu:
Anybody who looks at this results and says :

" Jeez Lua is so inefficient, I won't use it "

I am glad they go and use C.

Lua was never designed to minimize energy usage, i do not see energy use mentioned anywhere on lua's wiki page.

I want a ranking of :

- embedding with C, Lua will beat every other language on this.

- compiler complexity, Lua will beat every other language on this.

If somebody creates a language that causes lua to lose on those front, I would agree that that would be the time to start freaking out, but for now Lua maintains a monopoly.

Its like somebody coming to me and asking them to help them do a cardiac bypass surgery - its not my job !

Lua excels in things it set as its goals. Many languages that set their goals to be something and never get to achieve it.

- "Java write once run everywhere" someone ??


On Sat, Apr 6, 2019 at 4:22 PM Italo Maia <[hidden email]> wrote:
I wonder how this benchmark would look like without the worst case algorithm implementation for each language. 

Another thing that worries me is the bad propaganda of such results given tests might have issues, as mentioned.

I have a hard time phantoming what could cause that Lua memory result. 

Em qui, 4 de abr de 2019 12:09, Scott Morgan <[hidden email]> escreveu:
On 04/04/2019 10:44, Frank Kastenholz wrote:
>
>
>> On Apr 4, 2019, at 1:35 AM, joy mondal <[hidden email]> wrote:
>>
>> Trying to measure programming languages based on how much electricity they use is a lot like ranking aircrafts by their weight - completely meaningless metric.
>
> There are environments where the energy supply is extremely limited, such as battery powered (or pv powered or motion powered) devices and where the device MUST remain operational for a specified period of time, so consumption is important to measure.
>
> Whether _this_ benchmark accurately measures that consumption is a separate question.
>

Also, in big data centres, where thousands of machines are running the
same code, power consumption (and heat dispersal) are important issues
that make these sorts of benchmarks useful.

I'll echo the question about how good this particular paper is at
measuring these things.

Scott



--
"A arrogância é a arma dos fracos."

===========================
Me. Italo Moreira Campelo Maia
Co-fundador do Grupo de Usuários Python do Ceará
Secretário ForHacker (fb.com/ForHackerSpace)
Desenvolvedor Full-Stack, Escritor, Empresário, Visionário
-----------------------------------------------------
Meu Livro, Site, Blog
===========================
Reply | Threaded
Open this post in threaded view
|

Re: Energy Benchmarks

steve donovan
On Mon, Apr 8, 2019 at 11:40 AM Italo Maia <[hidden email]> wrote:
> As Lua outgrows its niche, I think benchmarks like those will have more and more weight when deciding to adopt Lua. Even in the embedable and enterprise world, as Scott put it, energy efficiency is a thing.

This is true, but apart from programmer efficiency considerations, Lua
is often used as 'glue' between highly-efficient components. (Classic
Ousterhout system vs scripting). The glue does higher-level logic, and
would hardly be noticeable.

The games people really care about efficiency, and they love Lua....