why benchmarks need salt

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Gunnar Zötl

> Do you have different programs called "lua" or "sol" in your PATH
> that are getting executed instead of the copy in the current
> directory?
>
> Or is this definitely a reproducible problem on your platform when the
> argv[0] is any 3 character string?
>
> (I don't have "." in my PATH, even at the end, to avoid issues running
> the "wrong" version of a program when cd-ed to arbitrary directories.)

I can see the same effect on my linux bos (core 2 6400 cpu).
Not as drastic, it's 0.45 vs. 0.48 secs, but it is reproducible.
Weird...

Gunnar

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Enrico Colombini
In reply to this post by Elias Barrionovo
On 20-Jul-15 21:12, Elias Barrionovo wrote:
> Soon:
>
> "Abstract: In this paper we present a new optimization technique,
> where we use an artificial neural network together with other
> heuristics to find a suitable name for (...)"

While you are busy writing that paper ;-) I ran a few tests under Linux
with a freshly compiled 5.3.1, randomizing the executable name length
from 1 to 10 characters.

I raised 'rep' to 10000 in Roberto's code, to minimize os.clock()
granularity influence.

On 64-bit xubuntu (i7), len=1..10, 100 repetitions:
   min=1.074592, max=1.223682, avg=1.08894445

On 32-bit xubuntu (Pentium 4 HT), len=1..10, 10 repetitions:
   min=5.215278, max=9.036871 (!?!), avg=5.8223072

Strangely enough, the above pathological result (that happened in two
instances with length=6, towards the end of the 10 runs) did not happen
again with 100 repetitions. Now on the 32-bit machine I got:
   min=5.214088, max=5.517392, avg=5.2637858

(quick & dirty code attached - of course it could add its own problems)

Two notes:
- could swapping and/or other processes be involved?
   (e.g. something making heavy use of the system heap)
- my test program runs from the same intepreter that it calls
   to run 'temp'.

Now I *have* to turn off the old Pentium: 120 W in the middle of a heat
wave...

--
   Enrico

test.lua (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Josh Simmons
On Tue, Jul 21, 2015 at 11:15 PM, Enrico Colombini <[hidden email]> wrote:
>
> While you are busy writing that paper ;-) I ran a few tests under Linux with
> a freshly compiled 5.3.1, randomizing the executable name length from 1 to
> 10 characters.
>

STABILISER is also very interesting for the wider case of dealing with
benchmark variance.

http://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf

Reply | Threaded
Open this post in threaded view
|

Re: why benchmarks need salt

Enrico Colombini
On 22-Jul-15 05:05, Josh Simmons wrote:
> STABILISER is also very interesting for the wider case of dealing with
> benchmark variance.
>
> http://people.cs.umass.edu/~emery/pubs/stabilizer-asplos13.pdf

Interesting, thanks.

--
   Enrico

12