[Omaha.pm] Benchmark flakiness

[Jay Hannah <jhannah@omnihotels.com>]

Title: Benchmark flakiness

Weird...

-----------
use Benchmark qw( :hireswallclock );
timethis(100000, '
      for ($x=0; $x<=200; $x++) { sin($x/($x+2)); }
');
-----------

On my Mac this takes 10 seconds and outputs this:

timethis 100000: 9.88306 wallclock secs ( 9.80 usr +  0.02 sys =  9.82 CPU) @ 10183.30/s (n=100000)

On my SuSE box it takes between 10 and 20 seconds and outputs this regardless of how long it really took:

timethis 100000: 10.2632 wallclock secs ( 9.13 usr +  0.27 sys =  9.40 CPU) @ 10638.30/s (n=100000)

So the reported wallclock is fine on my Mac (idle or under load) but lies on SuSE under load.

It gets worse:

-----------
use Benchmark qw( timethese cmpthese :hireswallclock ) ;
 $x = 3;
 $r = timethese( 10000000, {
     a => sub{$x*$x},
     b => sub{$x**2},
 } );
 cmpthese $r;
------------

On my Mac this takes 22 seconds and outputs this:

Benchmark: timing 10000000 iterations of a, b...
         a: 1.02988 wallclock secs ( 1.02 usr + -0.01 sys =  1.01 CPU) @ 9900990.10/s (n=10000000)
         b: 1.10844 wallclock secs ( 1.12 usr + -0.01 sys =  1.11 CPU) @ 9009009.01/s (n=10000000)
       Rate   b   a
b 9009009/s  -- -9%
a 9900990/s 10%  --

On SuSE it takes 30 seconds and outputs this:

Benchmark: timing 10000000 iterations of a, b...
         a: 1.94711 wallclock secs ( 1.56 usr +  0.13 sys =  1.69 CPU) @ 5917159.76/s (n=10000000)
         b: 1.65905 wallclock secs ( 1.52 usr +  0.11 sys =  1.63 CPU) @ 6134969.33/s (n=10000000)
       Rate   a   b
a 5917160/s  -- -4%
b 6134969/s  4%  --

So in timethese() wallclock is entirely hosed...

Bummer.

Roll our own, I guess...   -sigh-

j