|
A.
Machine Tested:
Sun Microsystems - Enterprise 420R
All Results quoted or displayed in this section have been
reviewed and accepted
by the OSG Java Subcommittee of the
Standard Performance Evaluation Corporation (SPEC®).
A hyper-link is provided on each result to the
"SPEC full disclosure report"
which is located on the
OSG JBB2000 Web-site.
|
Back to Machine Tested
a. Results with the "Generally Available (GA)" JVM Code:
Table 1. Effect of heap size and tuning options for
Java 1.3.0 and Java 1.2.2.
All runs were done on the same Enterprise 420R system with no change in
physical configuration.
"Generally Available" code is that which has the full faith and support
of the vendor company. This code is production ready and recommended
by the vendor.
Table 1 above, gives us insight into the questions posed above about
why it is interesting to look at a range of results!
- Tuning: definitions.
The first data column of Table 1 describes the type of tuning used for
each row of the benchmark.
gcocco software describes "high opt" tuning (ga0, ga1, ga2 and ga3) as
highly optimized
for a production environment, but perhaps "normal" in a benchmarker's eye.
In "high opt", one always
turns on as many "helpful", reasonable-risk parameters as possible.
Parameters which are described as "experimental" or "subject to change"
are not used.
The exact set of parameters that were used in all the runs discussed
on this page can be found by viewing the "SPEC full disclosure report"
link in the last column of each result table.
There is a full disclosure for each benchmark discussed.
Row (ga4) of Table 1 is called "aggressive", and makes use
of some "tricks of the trade" specific for benchmarking on Solaris.
The "aggressive" parameters for tuning memory are not for the faint of heart.
The ones used for these runs are describe in the Sun online manual:
"autoup,
tune_t_fsflush
and
rechoose_interval
.
For some light reading, go ahead and research this set of
"tunables"(sic) at Sun's website.
Be sure to look up what Sun describes in their manual as
"Interface Stability".
(You'll have to page down a few pages on the preceding link.)
OK, so now that you know everything there is to know about
these parameters, why should you be concerned?
You are concerned because all three of the
parameters listed above, which are used in all of Sun's
published runs to date, are described
in the Sun online manual as "Unstable"!
OK, so "Unstable" in this sense means "subject to change", but
none-the-less, you can be sure these are not parameters
one should use casually (if at all)!
Another important point to note, is that line (ga4) in Table 1 above
used the three "aggressive" parameters described above.
That run yielded
[(9351-9069)/9351] = 0.03 or a 3% performance gain.
So, in this case, is the potential gain of 3%, worth the risk of
dealing with a changing, "unstable" interface?
We would recommend that you use only parameters in production that
are described as "stable" unless
you have a very good reason to do so.
- Heap: The effect of heap size for the SPECjbb2000 workload
A heap is a chunk of virtual memory allocated to a program
for its' local storage needs.
How efficiently the program (in this case JVM) manipulates the storage
has a huge effect on the programs efficiency and scalability.
The first three runs (ga1, ga2, ga3)
are identical except for the variation in the heap size,
from a modest 512 MB to almost 2 GB (the maximum allowed
for this 32 bit Java Virtual Machine).
Although most would expect the performance to be dramatically better
with much more
heap,
it is not the case.
[(9069-8853)/9069] = 0.02 or 2%
So, is it worth it to dedicate 4 times as much memory
to one application for the heap
and only gain about 2% in performance?
We don't think so, therefore in this case we
would recommend running with a 512MB heap and save some money on real memory!
- JVM: Java Virtual Machine Level
Java Virtual Machine(JVM) level 1.3.0_02 was the "then current",
downloadable version of the Sun's recommended "Generally Available" code.
It was used for runs ga1-ga4 in Table 1.
- Accepted: SPECjbb2000 Committee acceptance date
This is the date of the SPECjbb2000 review meeting for this
benchmark run.
All Results quoted or displayed in this section have been
reviewed and accepted
by the OSG Java Subcommittee of the
Standard Performance Evaluation Corporation (SPEC®).
- Result: operations/second (ops/s)
The only approved metric used to compare SPECjbb2000 results
from vendor to vendor is operations/second!
Back to Machine Tested
b. Results generated with the "Beta" JVM
Table 2. Effect of tuning options for Java 1.3.1 Beta and 1.4.0 Beta.
Now let's look at the heart of getting
really
good benchmark numbers. In this case, it was done using "Beta" code.
Beta code is sometimes experimental and
may
eventually make it into a product, but it
may
not! In this case, as ß1a shows, the actual performance achieved
was slightly better when the product finally came out.
- Tuning: definitions.
In recommending the use of the Beta JVM, Sun also recommended
parameter changes for us to try.
We were asked to use the
new experimental heap parameter for these "aggressively tuned" runs.
This parameter is appropriately named the "-XX:+AggressiveHeap"
parameter.
This and other
standard as well as non-standard parameters
are described on java.sun.com site.
The "-XX" in the parameter name means the parameter is "Non-Standard" and
"is subject to change in future releases".
The "-XX:+AggressiveHeap" parameter allocates "almost" all
of the available memory for the heap.
The JVM then does "lazy" heap management to defer all work
to a time outside of the benchmark measurement window.
It is certainly much faster than expected,
[(15,990-8,893)]/8,893 = 0.80 or 80%.
If we use the newer version 1.4.0 beta, we see an even better speedup
which is
[(18,489-8,893)]/8,893 = 1.08 or 108% faster.
If we were the hardware vendor for these runs, we would be quoting
this
as our "single number" indicating the performance of this machine!
Now let's learn about the down-side. Read about the use of the
Aggressive Heap Option
on the Java site.
Sun strongly cautions one about using this parameter except where
the situation warrants and you have the skill to do so.
It's explicitly not for the "casual user".
- Heap: actual heap size for the run
This is always an important parameter.
Compare the first "high opt" run (ß1) in Table 2 with run
(ga2) from Table 1 which have the same heap size (1024 MB).
The median size heap was chosen for comparison of of runs with "GA" code
to the runs with "Beta" code.
The comparison was also done on runs with "high opt" tuning parameters,
similar to what one would be running in a highly tuned "production"
environment.
The performance gain is a very nice improvement over the "stable" and
supported release of about
[(11,157-8,893)]/8,893 = 0.25 or 25%.
So far we like it!
Now lets go back to the on-line manual, and look at this
Intimate Shared Memory (ISM) option.
Those of us who are OS people, usually just
call this "pinned" or "locked down" virtual pages.
That means, no one else can
use this critical memory resource.
Since no one else can "move" them, the operating system can
skip lots of checks and gain on the order of a 5%-10%
performance boost on the workload.
This is worthwhile, and is sometimes done for servers which host performance
critical workloads.
However, if you can't or won't use ISM, remember to apply some
performance degradation to any SPECjbb number you are looking at
that Sun has produced.
Table 3. All of the published benchmarks by Sun use the parameters
that have been discussed above.
Anyway, go back to the ISM on-line reference manual and study the part on
"Problems and Dangers".
Be sure to read the references for ISM which is described
by Sun, as "not for the casual user"!
Our experience with ISM on the E420R wasn't all that good.
There was an embarrassing problem (for Sun) of an immediate
core dump
when trying to use this option.
Basically, it appears that the combination of "Aggressive Heap" and
"useISM" parameters was targeted for Sun's larger memory machines.
As the truss data shows, it could never run on "our" 4GB system and
Sun couldn't tell us how to make it work.
After ignoring the "useISM" parameter problem for almost a year(
Sun Java "Review
(ID:120953) -XX:+UseISM causes core dump in 1.3.1 RC1")
Sun assures us that this problem will not occur in the "released" version
of Java 1.4.0.
For the general case,
we would never recommend using this "ISM" code in production based on
Sun's own cautions.
The question is if the dangers are worth the 7-10% improvement that "ISM"
provides.
We certainly don't think it is worth it.
- JVM: Java Virtual Machine Level
Java Virtual Machine(JVM) Beta level "1.3.1 RC1" was the "then current (4/2001)",
downloadable version of the lastest and greatest "beta".
We were personally asked by a Sun Performance Manager to switch to
this level of code and make our runs on this level of JVM.
It was used for all runs in Table 2.
And should be quite similar to what Sun has used for its runs
in Table 3.
- Accepted: Date SPECjbb2000 Committee accepted run.
This is the date of the SPECjbb2000 review meeting for this
benchmark run.
All Results quoted or displayed in this section have been
reviewed and accepted
by the OSG Java Subcommittee of the
Standard Performance Evaluation Corporation (SPEC®).
- Result: operations/second (ops/s)
The only approved metric used to compare SPECjbb2000 results
from vendor to vendor is operations/second!
Back to Machine Tested
c. Performance Range Variability from 6,000 to 18,000 ops/sec!
So, if the basic question is how fast is a certain piece of hardware,
the data below presents a dilemma.
If the same exact machine performs SPECjbb2000 from 6K to 18K ops/sec,
which value should you use?
The following table summarizes the 3 runs in question:
Table 4. Summary of performance range of runs on identical hardware.
Table 4 above shows the gains that can be made through software improvements
only on the same hardware.
Tripling your performance on the same workload on the identical platform is
impressive.
Basically, ga3 uses the old version 1.2.2 JVM which ships as the
standard JVM on Solaris. If you don't do anything, this is what you get.
If you instead download the now (1/2002) current JVM, you will get
results similar to ß1a.
If you are willing to live on the bleeding edge, you can pull down
the version 1.4 beta (ß4).
You will see performance like the 18K ops/sec result.
This is of course not ready for prime time, but it has been shown in
the past to be a good indicator of the performance you can get on the
next release of the JVM code.
Back to Machine Tested
d. Performance Boost using Intimate Shared Memory (ISM) is at least 8%.
Intimate Shared Memory (ISM) is one of the options that Sun uses to
squeeze out the last few percent of the performance on its' machine.
While it is a "real" option and can be used, it is described as
"dangerous" by Sun themselves (see discussion above)
.
So we set out to give you some data points on why they are using
it in spite of their own best advice.
The answer is that with the addition of this parameter, Sun can
boost its' results about 8-10%.
This could mean the difference in claiming leadership performance
and being second or third.
So it is clear that Sun will use this for its' benchmarks, but
you have to factor this in to your interpretation of Sun's
results.
|
Tuning |
Heap |
CPUs |
JVM |
Run Date |
Result |
Report |
| 256 |
high |
256 MB |
4 |
1.3.1_01 |
05-Dec-2001 |
10,846 |
full disclosure |
| i256 |
high,ism |
256 MB |
4 |
1.3.1_01 |
04-Dec-2001 |
11,711 |
full disclosure |
| 512 |
high |
512 MB |
4 |
1.3.1_01 |
04-Dec-2001 |
11,412 |
full disclosure |
| i512 |
high,ism |
512 MB |
4 |
1.3.1_01 |
04-Dec-2001 |
12,392 |
full disclosure |
| 1024 |
high |
1024 MB |
4 |
1.3.1_01 |
04-Dec-2001 |
11,674 |
full disclosure |
| i1024 |
high,ism |
1024 MB |
4 |
1.3.1_01 |
03-Dec-2001 |
12,432 |
full disclosure |
| 2010 |
high |
2010 MB |
4 |
1.3.1_01 |
01-Dec-2001 |
11,846 |
full disclosure |
| i2010 |
high,ism |
2010 MB |
4 |
1.3.1_01 |
30-Nov-2001 |
12,816 |
full disclosure |
Note: Results in the above table have been tested in accordance
with SPEC run rules but have not been submitted for formal review.
Back to Machine Tested
e. Run to Run Reproducability
By their very nature, benchmarks push systems into a very extreme
runtime condition.
How well a system performs under such a stress-load is a testament
to its' design.
We decided to regression test a larger sample of runs to see
what sort of distribution of results we would achieve.
With time constraints a factor, we ran 23 identical runs and
only 30% would be considered valid benchmark runs.
This system was obviously running at the edge or beyond its'
design envelope.
This is usually how most if not all "one number benchmarking"
takes place! Keep running until you get a really good one
and submit it for review!
Back to Machine Tested
f. Tricks/Tips/Cautions:
- No I/O during Benchmark
This is important if you "actually" do I/O! For more
background on the design of this benchmark read the
SPECjbb2000 Whitepaper.
- SMP "scaling" test
SPECjbb2000 is useful as an indicator for scaling (of this workload).
If you have to decide on an upgrade (for instance),
all the data you need can be found in the full disclosures on the
SPEC website.
Symmetric Multi-Processor (SMP) issues generally come
down to "scaling",
so, if your workload is like SPECjbb2000, you have your indicator!
- Potentially large variability in runs
Software platforms for this class of machines are complex.
The combinations of Hardware, Operating System, Java Version as well as
the fixes applied to these components is mind numbing.
Care must be exersized by the system administrator
or system evaluator since a large range
of performance can be had on the SAME hardware (even if each
combination of components is tuned to its' best possible performance).
Back to Machine Tested
g. Net/Net - The Performance you should expect!
For production use, with the "Generally Available" code, you should
expect the Sun Enterprise 420R to be about 6150 ops/sec
if you must use a JVM that is 1.2.2 compatible.
Otherwise, you can now get about 8900 ops/sec with version
1.3.0, 11,500 ops/sec with version 1.3.1 or 18,500 ops/sec on the latest
version 1.4.0 code.
Back to Machine Tested |
Back to top
|
