[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Omaha.pm] Benchmark.pm (cont'd from August)

Benchmark.pm:
  http://search.cpan.org/~rgarcia/perl-5.10.0/lib/Benchmark.pm
Back in August I complained. Yesterday I got interesting & useful info on irc.perl.org #poe: 



12:44 <@Leolo> Benchmark: timing 1000000 iterations of amore, less, more...
12:44 <@Leolo> amore: 4 wallclock secs ( 2.57 usr + 0.00 sys = 2.57 CPU) @ 389105.06/s (n=1000000) 12:44 <@Leolo> less: 2 wallclock secs ( 2.52 usr + 0.00 sys = 2.52 CPU) @ 396825.40/s (n=1000000) 12:44 <@Leolo> more: 2 wallclock secs ( 2.58 usr + 0.00 sys = 2.58 CPU) @ 387596.90/s (n=1000000) 12:47 <@jhannah> Am I the only one that gave up on Benchmark? http://mail.pm.org/pipermail/omaha-pm/2008-August/001855.html 12:54 <@dngor> jhannah: Yes, wall clock is useless for benchmarking. That's why I use a CPU-time cutoff and compare the iterations/cpu time. 
12:55 <@jhannah> wouldn't an accurate wallclock be insanely trivial to list?
12:55 <@Leolo> wallclock is useless
12:55 <@dngor> Wall time isn't accurate in a pre-emptive OS.
12:55 <@dngor> You're at the whim of other processes.
12:55 <@Leolo> other processes will hog the CPU, throwing off the .... what dngor said 12:55 <@dngor> Now, if you nice -19 the benchmark, that would make it better. :) 12:57 <@dngor> It may also require a reboot. Be sure your benchmark will eventually exit. :) 12:57 <@jhannah> I understand the point you're making. What I don't understand why Benchmark would label something "wallclock" when it has no bearing, whatsoever, on how much clock-on-the-wall-time elapsed. It should be removed or accurate...? 
12:57  * jhannah adds an "is" in there somewhere
12:58 <@dngor> Oh, I see. Well, there's Benchmark.pm overhead which I presume it tries to discount. 13:00 <@dngor> I admit I'm just guessing at this point. The CPU/wall ratio implies that the program wasn't pre-empted much. 13:01 <@dngor> Program run time, from start to exit, also involves loading Perl and the source, compiling it, and tearing down the process. Those may be significant, but Benchmark.pm won't report them. 
13:02 <@dngor> Whether these apply, I don't know.
13:03  * jhannah hugs purl, shoots Benchmark
13:03 <@jhannah> call it anything EXCEPT "wallclock" and I'm happy.  :)
13:07 <@dngor> Example of a cpu-time limit: http://poe.dyndns.org/~troc/benchmarks/bench-calls.perl 13:08 <@dngor> Example of "rolling your own" for jhannah: http://poe.dyndns.org/~troc/benchmarks/arguments.perl 13:11 <@jhannah> dngor: so you just ignore the "wallclock secs"? Or you trust them as accurate relative to each other but unrelated to a clock on the wall? 
13:11 <@dngor> I ignore them and look at n per X cpu seconds.
13:11 <@dngor> CPU time is the only objective measurement.
13:12 <@dngor> The CPU is the universe.
13:13 <@dngor> There is no coupling at all between the CPU and the wall clock. Consider that many machines dynamically clock now to conserve power.