<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi Noel<br>
<br>
Thanks for that.<br>
<br>
It comes out in the numbers, but, for the record, I am doing
something very similar to that.... only the structures are slightly
(very) different.<br>
<br>
I do a bunch of GC's, and I do one in a different thread with the
current thread sleeping, then I repeat the GC's until the size
becomes 'stable' at a change of less than 128 bytes.<br>
<a class="moz-txt-link-freetext" href="https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R33">https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R33</a><br>
<br>
I do a complete once-through of the test suite to warm things up.<br>
Each once-through runs the code through 6 times (hmmm... I thought
it was 12, but that was something else I did yesterday). Each of the
actual tests 'exercises' the code repeatedly because it's all sort
of loop-based code (parsing, scanning, etc.).<br>
<br>
Anyway, the output of the 'warmup' run is always much slower than
the remaining 5 'real' runs, and I do the 'real' runs multiple times
to ensure there is some stability.<br>
<br>
What you see in the web-page is the result of what I believe to be
fully JIT-compiled and 'clean' and 'reliable enough' for the
purposes I want.<br>
<br>
I know that the Java VM testing is 'tricky' when it comes to
performance, and as such I understand that it's easy to get things
wrong, and I'll spend more time looking at it to ensure I'm doing
the reasonable thing, but, are you suggesting that the code I am
running is not actually getting reliable results?<br>
<br>
The code is structured differently to what you have suggested below,
but, the entire 'main' loop is warmed up:
<a class="moz-txt-link-freetext" href="https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R124">https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R124</a><br>
<br>
Then, the main loop is run 5 times, and I visually inspect the
numbers to ensure that they are consistent:
<a class="moz-txt-link-freetext" href="https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R135">https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R135</a><br>
<br>
Between each 'test' I do a full GC with 'bells and whistles'<br>
<a class="moz-txt-link-freetext" href="https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R160">https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb#L1R160</a><br>
<br>
It is quite obvious that the runs that come out of the 'real' loops
are optimized, cached, etc.<br>
<br>
What is not clear is whether the optimizer has completely compiled
out some of the code. I have tried to ensure that it does not by
doing some sort of test on each element so that it is not completely
ignored.<br>
now that I think about it though, maybe the 'devnull' Writer is too
'light' and the optimizer may have completely skipped it
entirely..... and the whole XMLOUtputter code with it.... I will
check.<br>
<br>
So, I appreciate the insight, and I will play around with things to
see if increasing the number of warmup and actual 'real' runs
changes the numbers.<br>
<br>
I'll look in to making sure that some of the code is not being
optimized out completely.<br>
<br>
But, my code already is doing pretty much exactly what you are
suggesting... (it does not calculate the deviation, but it does
ignore the fastest and slowest run.....).<br>
<br>
In fact, it does more because it then repeats the exact same loops
multiple times to ensure the averages remain consistent over runs
(as it happens, it essentially does 20 'runs' of the code to get the
results - 5 loops of 6 runs but the 6 only counts as 4 because the
best and worst are eliminated).<br>
<br>
Have you got specific concerns about the code? Did you run it? Do
you think the results are 'wrong'?<br>
<br>
Thanks for the insight in to the commons-math code. I'm always
'discovering' more and more 'stuff' in commons code. I have some
'stuff' I've done at work I am trying to convince my boss (actually
legal&compliance) to let me use in JDOM, but it's the sort of
thing that belongs in a 'commons' type location, not JDOM....<br>
<br>
Rolf<br>
<br>
On 14/10/2011 4:08 AM, Noel Grandin wrote:
<blockquote cite="mid:4E97EE04.3070307@peralex.com" type="cite">
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
<font face="Tahoma">Hi<br>
<br>
Performance testing on the Java VM is tricky. <br>
To avoid getting caught out by cache-hot/cache-cold and JIT vs.
not-JIT things, it's preferrable to do something like this in
PerfTest#timeRun(Runnnable)<br>
<br>
// warm up the caches and get the JIT going<br>
for (int i=0; i<10; i++) {<br>
runnable.run();<br>
}<br>
<br>
// give the JIT time to run, and get GC to run - GC can be
stubborn sometimes<br>
</font><font face="Tahoma">for (int i=0; i<3; i++) {<br>
</font><font face="Tahoma">Thread.sleep(100);<br>
</font><font face="Tahoma">System.gc();<br>
}<br>
<br>
// need 20 runs to get a decent average and standard deviation<br>
ArithmeticMean mean = new ArithmeticMean(); // these two classes
are in jakarata-commons-math<br>
</font><font face="Tahoma">Variance deviation = new Variance();<br>
</font><font face="Tahoma">for (int i=0; i<20; i++) {<br>
long time1 = System.currentTimeNanos();<br>
runnable.run();<br>
</font><font face="Tahoma"> long time2 =
System.currentTimeNanos();<br>
mean.increment(time2 - time1);<br>
deviation</font><font face="Tahoma">.increment(time2 - time1);</font><br>
<font face="Tahoma">}<br>
<br>
System.out.println("result = " + mean.getMean() + " +- " +
deviation.getVariance());<br>
</font><br>
Regards, Noel Grandin<br>
<br>
Rolf wrote:
<blockquote cite="mid:4E9781E5.1080609@tuis.net" type="cite">Hi
all. <br>
<br>
I have put together a 'simple' system for measuring the relative
performance of JDOM2. The idea is that I need to know whether I
am improving or breaking JDOM performance as the code evolves. <br>
<br>
Currently the metric code is only useful of you compare apples
to apples, and, in this case, it means processing a single
(medium size) XML document on my laptop, yada-yada-yada. But, it
should be useful as a tool to get a feel for what a code-change
does. <br>
<br>
Already I can see that I probably have an issue in the
SAXHandler (possibly an issue in JDOM-1.1.2 actually) because
1.1.2 is 5-times faster in that area than JDOM2. <br>
<br>
I have put together a results page here: <br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://hunterhacker.github.com/jdom/jdom2/performance.html">http://hunterhacker.github.com/jdom/jdom2/performance.html</a>
<br>
<br>
It also describes what each test does. If you are interested in
seeing the code and what it does have a look here (it is not
well documented and it is still perhaps evolving): <br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb">https://github.com/hunterhacker/jdom/commit/8b719c86913398ace8e197b6de145b33d9d300bb</a>
<br>
<br>
<br>
Rolf <br>
_______________________________________________ <br>
To control your jdom-interest membership: <br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.jdom.org/mailman/options/jdom-interest/youraddr@yourhost.com">http://www.jdom.org/mailman/options/jdom-interest/youraddr@yourhost.com</a>
<br>
<br>
</blockquote>
<br>
<br>
<br>
<hr><font color="808080" size="-2">Disclaimer: <a
moz-do-not-send="true"
href="http://www.peralex.com/disclaimer.html">http://www.peralex.com/disclaimer.html</a><br>
<br>
</font></blockquote>
<br>
</body>
</html>