why benchmarks need salt

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

why benchmarks need salt

Roberto Ierusalimschy
I was doing some quick-and-dirty benchmarks about 'table.move'. The
file 'temp' is here:

--------------------------------------------------------------
local a = {}
local lim = 10000
local rep = 1000
for i = 1, lim do a[i] = i end

local t = os.clock()
for i = 1, rep do table.move(a, 1, lim, 2) end
print(os.clock() - t)
--------------------------------------------------------------

Now, see how the times change with the name of the executable:

roberto@arraial:~/prj/lua$ lua temp
0.91
roberto@arraial:~/prj/lua$ lua temp
0.91
roberto@arraial:~/prj/lua$ mv lua lua1
roberto@arraial:~/prj/lua$ lua1 temp
0.57
roberto@arraial:~/prj/lua$ lua1 temp
0.57
roberto@arraial:~/prj/lua$ mv lua1 sol
roberto@arraial:~/prj/lua$ sol temp
0.91
roberto@arraial:~/prj/lua$ sol temp
0.91
roberto@arraial:~/prj/lua$ mv sol sol1
roberto@arraial:~/prj/lua$ sol1 temp
0.57
roberto@arraial:~/prj/lua$ sol1 temp
0.58


-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Scott Morgan
On 20/07/15 17:00, Roberto Ierusalimschy wrote:
> I was doing some quick-and-dirty benchmarks about 'table.move'. The
> file 'temp' is here:
...

> Now, see how the times change with the name of the executable:
>
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ mv lua lua1
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ mv lua1 sol
> roberto@arraial:~/prj/lua$ sol temp
> 0.91
> roberto@arraial:~/prj/lua$ sol temp
> 0.91
> roberto@arraial:~/prj/lua$ mv sol sol1
> roberto@arraial:~/prj/lua$ sol1 temp
> 0.57
> roberto@arraial:~/prj/lua$ sol1 temp
> 0.58

What happens if you use full paths to the executable? That is, is it
just the file name itself, or something to do with argv data.

Scott


Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Soni "They/Them" L.
In reply to this post by Roberto Ierusalimschy


On 20/07/15 01:00 PM, Roberto Ierusalimschy wrote:

> I was doing some quick-and-dirty benchmarks about 'table.move'. The
> file 'temp' is here:
>
> --------------------------------------------------------------
> local a = {}
> local lim = 10000
> local rep = 1000
> for i = 1, lim do a[i] = i end
>
> local t = os.clock()
> for i = 1, rep do table.move(a, 1, lim, 2) end
> print(os.clock() - t)
> --------------------------------------------------------------
>
> Now, see how the times change with the name of the executable:
>
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ mv lua lua1
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ mv lua1 sol
> roberto@arraial:~/prj/lua$ sol temp
> 0.91
> roberto@arraial:~/prj/lua$ sol temp
> 0.91
> roberto@arraial:~/prj/lua$ mv sol sol1
> roberto@arraial:~/prj/lua$ sol1 temp
> 0.57
> roberto@arraial:~/prj/lua$ sol1 temp
> 0.58
>
>
> -- Roberto
>
So you can boost the speed of Lua scripts by renaming the Lua binary?

--
Disclaimer: these emails are public and can be accessed from <TODO: get a non-DHCP IP and put it here>. If you do not agree with this, DO NOT REPLY.


Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Roberto Ierusalimschy
In reply to this post by Scott Morgan
> What happens if you use full paths to the executable? That is, is it
> just the file name itself, or something to do with argv data.

Probably argv:

roberto@arraial:~/prj/lua$ lua temp
0.9
roberto@arraial:~/prj/lua$ ./lua temp
0.57
roberto@arraial:~/prj/lua$ lua temp a
0.58

(I already had that kind of behavior before, but it was clearly some
issue with the GC versus differences in the initial memory allocations
caused by the 'arg' table. But this test with 'table.move' does zero
allocations during the measured code.)

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Dirk Laurie-2
In reply to this post by Roberto Ierusalimschy
2015-07-20 18:00 GMT+02:00 Roberto Ierusalimschy <[hidden email]>:

> Now, see how the times change with the name of the executable:
>
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ mv lua lua1
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ mv lua1 sol
> roberto@arraial:~/prj/lua$ sol temp
> 0.91
> roberto@arraial:~/prj/lua$ sol temp
> 0.91
> roberto@arraial:~/prj/lua$ mv sol sol1
> roberto@arraial:~/prj/lua$ sol1 temp
> 0.57
> roberto@arraial:~/prj/lua$ sol1 temp
> 0.58

Is this on your old Pentium? I can't reproduce the phenomenon on
a one-year old desktop.

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Roberto Ierusalimschy
> Is this on your old Pentium? I can't reproduce the phenomenon on
> a one-year old desktop.

Not that old; I bought it in January 2006 :-)

  model name : Intel(R) Pentium(R) 4 CPU 2.93GHz
  cache size : 1024 KB

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Enrico Colombini
In reply to this post by Dirk Laurie-2
On 20-Jul-15 18:46, Dirk Laurie wrote:
> Is this on your old Pentium? I can't reproduce the phenomenon on
> a one-year old desktop.

Neither can I, with a Core i7 under Windows 8.1 (lua 5.3.0, monolithic
executable, not sure if I got it from Lua Binaries or compiled it).

Either with a Lua executable in the same directory as 'temp', or with a
long pathname, with or without arguments, I consistently get the same
results (0.301..0.309).

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Florian Weimer
In reply to this post by Roberto Ierusalimschy
* Roberto Ierusalimschy:

> I was doing some quick-and-dirty benchmarks about 'table.move'. The
> file 'temp' is here:
>
> --------------------------------------------------------------
> local a = {}
> local lim = 10000
> local rep = 1000
> for i = 1, lim do a[i] = i end
>
> local t = os.clock()
> for i = 1, rep do table.move(a, 1, lim, 2) end
> print(os.clock() - t)
> --------------------------------------------------------------
>
> Now, see how the times change with the name of the executable:

Something extremely similar has been observed before:

Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, Peter F. Sweeney,
“Producing Wrong Data Without Doing Anything Obviously Wrong!”

<http://www-plan.cs.colorado.edu/klipto/mytkowicz-asplos09.pdf>

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Enrico Colombini
In reply to this post by Roberto Ierusalimschy
On 20-Jul-15 19:12, Roberto Ierusalimschy wrote:
> Not that old; I bought it in January 2006 :-)
>
>    model name : Intel(R) Pentium(R) 4 CPU 2.93GHz
>    cache size : 1024 KB

Does not seem to happen for me on an equivalent machine (under Windows XP).

Probably unrelated: on that machine I once got 10-15% longer times after
entering and exiting the Lua interpreter, and the first time I added
many arguments (but not on subsequent repetitions).

--
   Enrico

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Jay Carlson
In reply to this post by Roberto Ierusalimschy
On 2015-07-20, at 12:00 PM, Roberto Ierusalimschy <[hidden email]> wrote:

>
> Now, see how the times change with the name of the executable:
>
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ sol temp
> 0.91
> roberto@arraial:~/prj/lua$ sol1 temp
> 0.57
>

Three-letter vs four-letter smells like alignment issues. Stack alignment?

Jay
Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Roberto Ierusalimschy
In reply to this post by Florian Weimer
> Something extremely similar has been observed before:
>
> Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, Peter F. Sweeney,
> “Producing Wrong Data Without Doing Anything Obviously Wrong!”
>
> <http://www-plan.cs.colorado.edu/klipto/mytkowicz-asplos09.pdf>

Many thanks for the pointer!

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Tim Hill
In reply to this post by Roberto Ierusalimschy

> On Jul 20, 2015, at 9:00 AM, Roberto Ierusalimschy <[hidden email]> wrote:
>
> Now, see how the times change with the name of the executable:
>
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ lua temp
> 0.91
> roberto@arraial:~/prj/lua$ mv lua lua1
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
> roberto@arraial:~/prj/lua$ lua1 temp
> 0.57
>
>
> -- Roberto
>

Wow .. a 40% improvement .. imagine the gain if you changed the name to lua9 !

Clearly, this technique needs to be patented.

—Tim


Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Elias Barrionovo
Soon:

"Abstract: In this paper we present a new optimization technique,
where we use an artificial neural network together with other
heuristics to find a suitable name for (...)"

On Mon, Jul 20, 2015 at 4:05 PM, Tim Hill <[hidden email]> wrote:

>
>> On Jul 20, 2015, at 9:00 AM, Roberto Ierusalimschy <[hidden email]> wrote:
>>
>> Now, see how the times change with the name of the executable:
>>
>> roberto@arraial:~/prj/lua$ lua temp
>> 0.91
>> roberto@arraial:~/prj/lua$ lua temp
>> 0.91
>> roberto@arraial:~/prj/lua$ mv lua lua1
>> roberto@arraial:~/prj/lua$ lua1 temp
>> 0.57
>> roberto@arraial:~/prj/lua$ lua1 temp
>> 0.57
>>
>>
>> -- Roberto
>>
>
> Wow .. a 40% improvement .. imagine the gain if you changed the name to lua9 !
>
> Clearly, this technique needs to be patented.
>
> —Tim
>
>



--
NI!

() - www.asciiribbon.org
/\ - ascii ribbon campaign against html e-mail and proprietary attachments

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Pierpaolo Bernardi
In reply to this post by Roberto Ierusalimschy
Does the same percentual difference hold for longer running
benchmarks?  Is it a fixed startup delay, or a real performance
difference?


On Mon, Jul 20, 2015 at 7:12 PM, Roberto Ierusalimschy
<[hidden email]> wrote:

>> Is this on your old Pentium? I can't reproduce the phenomenon on
>> a one-year old desktop.
>
> Not that old; I bought it in January 2006 :-)
>
>   model name : Intel(R) Pentium(R) 4 CPU 2.93GHz
>   cache size : 1024 KB
>
> -- Roberto
>

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Roberto Ierusalimschy
> Does the same percentual difference hold for longer running
> benchmarks?  Is it a fixed startup delay, or a real performance
> difference?

The difference is too big for a startup delay. This is Lua!! :-) Morever,
we are measuring only an internal loop, not the entire process.

-- Roberto

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Coda Highland
On Mon, Jul 20, 2015 at 1:05 PM, Roberto Ierusalimschy
<[hidden email]> wrote:
>> Does the same percentual difference hold for longer running
>> benchmarks?  Is it a fixed startup delay, or a real performance
>> difference?
>
> The difference is too big for a startup delay. This is Lua!! :-) Morever,
> we are measuring only an internal loop, not the entire process.
>
> -- Roberto
>

I assume the difference is that it's catching some pathological
behavior in the hashtable -- introducing a collision where otherwise
there would be none, or similar.

/s/ Adam

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Elias Barrionovo
On Mon, Jul 20, 2015 at 5:06 PM, Coda Highland <[hidden email]> wrote:
> I assume the difference is that it's catching some pathological
> behavior in the hashtable -- introducing a collision where otherwise
> there would be none, or similar.


Talking about names and hash collisions reminded me of PHP:
http://news.php.net/php.internals/70691


--
NI!

() - www.asciiribbon.org
/\ - ascii ribbon campaign against html e-mail and proprietary attachments

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Pierre-Yves Gérardy
In reply to this post by Coda Highland
On Mon, Jul 20, 2015 at 10:06 PM, Coda Highland <[hidden email]> wrote:
I assume the difference is that it's catching some pathological
behavior in the hashtable -- introducing a collision where otherwise
there would be none, or similar.

Indeed. Roberto, does it still happen if you cache table.move in a local?

—Pierre-Yves

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Luke Mewburn
In reply to this post by Roberto Ierusalimschy
On Mon, Jul 20, 2015 at 01:00:47PM -0300, Roberto Ierusalimschy wrote:
  | I was doing some quick-and-dirty benchmarks about 'table.move'. The
  | file 'temp' is here:
  |
  | --------------------------------------------------------------
  | local a = {}
  | local lim = 10000
  | local rep = 1000
  | for i = 1, lim do a[i] = i end
  |
  | local t = os.clock()
  | for i = 1, rep do table.move(a, 1, lim, 2) end
  | print(os.clock() - t)
  | --------------------------------------------------------------
  |
  | Now, see how the times change with the name of the executable:
  |
  | roberto@arraial:~/prj/lua$ lua temp
  | 0.91
  | roberto@arraial:~/prj/lua$ lua temp
  | 0.91
  | roberto@arraial:~/prj/lua$ mv lua lua1
  | roberto@arraial:~/prj/lua$ lua1 temp
  | 0.57
  | roberto@arraial:~/prj/lua$ lua1 temp
  | 0.57
  | roberto@arraial:~/prj/lua$ mv lua1 sol
  | roberto@arraial:~/prj/lua$ sol temp
  | 0.91
  | roberto@arraial:~/prj/lua$ sol temp
  | 0.91
  | roberto@arraial:~/prj/lua$ mv sol sol1
  | roberto@arraial:~/prj/lua$ sol1 temp
  | 0.57
  | roberto@arraial:~/prj/lua$ sol1 temp
  | 0.58

Do you have different programs called "lua" or "sol" in your PATH
that are getting executed instead of the copy in the current
directory?

Or is this definitely a reproducible problem on your platform when the
argv[0] is any 3 character string?

(I don't have "." in my PATH, even at the end, to avoid issues running
the "wrong" version of a program when cd-ed to arbitrary directories.)

regards,
Luke.

attachment0 (205 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Roberto Ierusalimschy
In reply to this post by Pierre-Yves Gérardy
> On Mon, Jul 20, 2015 at 10:06 PM, Coda Highland <[hidden email]> wrote:
>
> > I assume the difference is that it's catching some pathological
> > behavior in the hashtable -- introducing a collision where otherwise
> > there would be none, or similar.
> >
> > Indeed. Roberto, does it still happen if you cache table.move in a local?

Sure. (You should all see the paper Florian mentioned.)

-- Roberto

12