[<< wikibooks] ROSE Compiler Framework/SPEC CPU 2006 benchmark
== spec cpu 2006 ==
Assuming the source package is under /usr/casc/overture/ROSE/svn/spec_cpu2006_v1.1


=== installation ===
./install.sh -d /home/yourAccount/opt/spec_cpu2006


=== files and directories ===
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench
Subdirectories

build  data  Docs  exe  run  Spec  src


=== benchmark summary ===
12+17= 29 benchmarks
12 Integer benchmarks:  http://www.spec.org/cpu2006/CINT2006/

400.perlbench  	C  	 Programming Language
Derived from Perl V5.8.7. The workload includes SpamAssassin, MHonArc (an email indexer), and specdiff (SPEC's tool that checks benchmark outputs).

401.bzip2 	C 	Compression
Julian Seward's bzip2 version 1.0.3, modified to do most work in memory, rather than doing I/O.

403.gcc 	C 	C Compiler
Based on gcc Version 3.2, generates code for Opteron.

429.mcf 	C 	Combinatorial Optimization
Vehicle scheduling. Uses a network simplex algorithm (which is also used in commercial products) to schedule public transport.

445.gobmk 	C 	Artificial Intelligence: Go
Plays the game of Go, a simply described but deeply complex game.

456.hmmer 	C 	Search Gene Sequence
Protein sequence analysis using profile hidden Markov models (profile HMMs)

458.sjeng 	C 	Artificial Intelligence: chess
A highly-ranked chess program that also plays several chess variants.

462.libquantum 	C 	Physics / Quantum Computing
Simulates a quantum computer, running Shor's polynomial-time factorization algorithm.

464.h264ref 	C 	Video Compression
A reference implementation of H.264/AVC, encodes a videostream using 2 parameter sets. The H.264/AVC standard is expected to replace MPEG2

471.omnetpp 	C++ 	Discrete Event Simulation
Uses the OMNet++ discrete event simulator to model a large Ethernet campus network.

473.astar 	C++ 	Path-finding Algorithms
Pathfinding library for 2D maps, including the well known A* algorithm.

483.xalancbmk 	C++ 	XML Processing
A modified version of Xalan-C++, which transforms XML documents to other document types.

17 Floating point benchmarks: http://www.spec.org/cpu2006/CFP2006/

410.bwaves  	Fortran  	Fluid Dynamics  	 Computes 3D transonic transient laminar viscous flow.
416.gamess 	Fortran 	Quantum Chemistry. 	 Gamess implements a wide range of quantum chemical computations. For the SPEC workload, self-consistent field calculations are performed using the Restricted Hartree Fock method, Restricted open-shell Hartree-Fock, and Multi-Configuration Self-Consistent Field
433.milc 	C 	Physics / Quantum Chromodynamics 	 A gauge field generating program for lattice gauge theory programs with dynamical quarks.
434.zeusmp 	Fortran 	Physics / CFD 	ZEUS-MP is a computational fluid dynamics code developed at the Laboratory for Computational Astrophysics (NCSA, University of Illinois at Urbana-Champaign) for the simulation of astrophysical phenomena.
435.gromacs 	C,Fortran 	Biochemistry / Molecular Dynamics 	Molecular dynamics, i.e. simulate Newtonian equations of motion for hundreds to millions of particles. The test case simulates protein Lysozyme in a solution.

436.cactusADM 	C,Fortran 	Physics / General Relativity 	Solves the Einstein evolution equations using a staggered-leapfrog numerical method

437.leslie3d 	Fortran 	Fluid Dynamics 	Computational Fluid Dynamics (CFD) using Large-Eddy Simulations with Linear-Eddy Model in 3D. Uses the MacCormack Predictor-Corrector time integration scheme.
444.namd 	C++ 	Biology / Molecular Dynamics 	Simulates large biomolecular systems. The test case has 92,224 atoms of apolipoprotein A-I.
447.dealII 	C++ 	Finite Element Analysis 	deal.II is a C++ program library targeted at adaptive finite elements and error estimation. The testcase solves a Helmholtz-type equation with non-constant coefficients.
450.soplex 	C++ 	Linear Programming, Optimization 	Solves a linear program using a simplex algorithm and sparse linear algebra. Test cases include railroad planning and military airlift models.
453.povray 	C++ 	Image Ray-tracing 	Image rendering. The testcase is a 1280x1024 anti-aliased image of a landscape with some abstract objects with textures using a Perlin noise function.
454.calculix 	C,Fortran 	Structural Mechanics 	Finite element code for linear and nonlinear 3D structural applications. Uses the SPOOLES solver library.

459.GemsFDTD 	Fortran 	Computational Electromagnetics 	Solves the Maxwell equations in 3D using the finite-difference time-domain (FDTD) method.
465.tonto 	Fortran 	Quantum Chemistry 	An open source quantum chemistry package, using an object-oriented design in Fortran 95. The test case places a constraint on a molecular Hartree-Fock wavefunction calculation to better match experimental X-ray diffraction data.
470.lbm 	C 	Fluid Dynamics 	Implements the "Lattice-Boltzmann Method" to simulate incompressible fluids in 3D
481.wrf 	C,Fortran 	Weather 	Weather modeling from scales of meters to thousands of kilometers. The test case is from a 30km area over 2 days.

482.sphinx3 	C 	Speech recognition 	A widely-known speech recognition system from Carnegie Mellon University


=== language summary ===
C : 14 benchmarks/ 12 real ones excluding specrand

   400.perlbench     	 Programming Language
   401.bzip2            Compression
   403.gcc              C Compiler
   429.mcf              Combinatorial Optimization
   433.milc             Physics / Quantum Chromodynamics
   445.gobmk            Artificial Intelligence: Go
   456.hmmer            Search Gene Sequence
   458.sjeng            Artificial Intelligence: chess
   462.libquantum       Physics / Quantum Computing
   464.h264ref          Video Compression
   470.lbm      	 Fluid Dynamics
   482.sphinx3          Speech recognition
   998.specrand
   999.specrand

C++: 7 benchmarks  : namd dealII soplex povray omnetpp astar xalancbmk

   444.namd
   447.dealII
   450.soplex
   453.povray
   471.omnetpp
   473.astar
   483.xalancbmk

Fortran: 6 benchmarks: bwaves gamess zeusmp leslie3d GemsFDTD tonto

   410.bwaves
   416.gamess
   434.zeusmp
   437.leslie3d
   459.GemsFDTD
   465.tonto

Mixed C and Fortran: 4:  gromacs cactusADM  calculix wrf

   435.gromacs 
   436.cactusADM 
   454.calculix
   481.wrf


=== configuration ===
enviornment Source the shrc or cshrc in /home/youAccount/opt/spec_cpu2006 to set up your environment for the benchmark. . ./shrc

configuration  prepare config/tux268-gcc42.cfg  set compilers and flags  based on existing examples thereTable summary

output_format = asc, pdf, Screen, html


=== use ===
http://www.spec.org/cpu2006/Docs/runspec.html#section3.1actions

build only:bash-3.00$ which runspec
/home/youAccount/opt/spec_cpu2006/bin/runspec
bash-3.00$ runspec --config=tux268-gcc42.cfg --action=build --tune=base bzip2

clean up first

runspec --config=rose-linux64-gcc41 --tune=base --size=test --iterations=1 --noreportable --action=clobber allRun with test data set default is --size=ref : run with reference data setrunspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2

runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2

run all integer or fp benchmarks      12 total integerrunspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable int   // total 12

400.perlbench      --     3.78           -- S
401.bzip2          --     7.52           -- S
403.gcc            --     1.42           -- S
429.mcf            --     5.37           -- S
445.gobmk          --    20.0            -- S

456.hmmer          --     5.35           -- S
458.sjeng          --     5.00           -- S
462.libquantum     --     0.0951         -- S
464.h264ref        --    21.7            -- S
471.omnetpp        --     0.545          -- S

473.astar          --    10.7            -- S
483.xalancbmk      --     0.115          -- S

 Est. SPECint_base2006                   --
 Est. SPECint2006                                                   Not Run

runspec --config=tux268-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable fp   // total 17

410.bwaves         --    12.4            -- S
416.gamess         --     0.510          -- S
433.milc           --     8.71           -- S
434.zeusmp         --    25.2            -- S
435.gromacs        --     1.74           -- S

436.cactusADM      --     4.66           -- S
437.leslie3d       --    32.7            -- S
444.namd           --    15.6            -- S
447.dealII         --    22.6            -- S
450.soplex         --     0.0263         -- S

453.povray         --     0.874          -- S
454.calculix       --     0.0546         -- S
459.GemsFDTD       --     4.52           -- S
465.tonto          --     1.35           -- S
470.lbm            --    13.6            -- S

481.wrf            --     5.66           -- S
482.sphinx3        --     2.26           -- S

 Est. SPECfp_base2006                    --
 Est. SPECfp2006                                                    Not Run

test gcc/g++/gfortran first
then build only


=== cleanup/clobber ===
clean up: --action=clean  or clobber
then identityTranslator
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable gobmk


=== data set ===
Reportable runs ensure that your binaries can produce correct results with the test and train workloads (this run rule requirement is new as of CPU2006 V1.0) and then run the ref workload 3 times for the actual measurements.
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=ref --iterations=1 --noreportable perlbench
data sizes

test: smallest data set
train: test the ability to run more than one size in a single invocation
ref (default): run 3 times for reportable results


=== selective run ===
runspec --action=clobber int fp
runspec --config=tux268-intel.cfg --tune=base --size=test --iterations=1 --noreportable bzip2
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable bzip2

bash-3.00$ pwd
/home/yourAccount/opt/spec_cpu2006
bash-3.00$ runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable fp


=== modified source run ===
Changed 
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench/src/spec_config.h
to have #include 
Have to add a line in the config file (eg. config/tux268-rose-gcc42.cfg) to avoid corrupted source warning

strict_rundir_verify = 0


=== debug ===
pwd
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/400.perlbench/src

/home/yourAccount/opt/roseLatest/bin/identityTranslator -c -o sv.o -DSPEC_CPU -DNDEBUG -DPERL_CORE   -O2 -DSPEC_CPU_LINUX_IA32        sv.c

Adding argv[3] = sv.o to p_objectFileNameList
Using strcmp(): This was not a valid string (buffer = define returned)
Using strcmp(): This was not a valid string (buffer = define returned)
Error: Unknown cppIndentifier =
identityTranslator: ../../../../rose/src/frontend/SageIII/rose_attributes_list.C:1676: bool ROSEAttributesList::isCppDirective(const std::string&, PreprocessingInfo::DirectiveType&, std::string&): Assertion `false' failed.
Aborted


=== results verification ===
/home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/401.bzip2/run/run_base_test_i386-m32-gcc42-nn.0001

.misbash-3.00$ cat dryer.jpg.out.mis
0008:  Compressed data 1434910 bytes in length
       Compressed data 1434894 bytes in length
                           ^
0013:  Compressed data 1127895 bytes in length
       Compressed data 1127995 bytes in length
                           ^
0018:  Compressed data 1067335 bytes in length
       Compressed data 1067427 bytes in length
                           ^


== run manually ==
each benchmark has a command line file: 

 run_base_test_amd64-m64-gcc41-nn.0000/speccmds.cmd

sphinx_livepretend_base.amd64-m64-gcc41-nn ctlfile . args.an4
403.gcc  ~/opt/spec_cpu2006/benchspec/CPU2006/403.gcc/run/run_base_test_amd64-m64-gcc41-nn.0000]

[yourAccount@tux322]cat speccmds.cmd
-C /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/403.gcc/run/run_base_test_amd64-m64-gcc41-nn.0000
-o cccp.out -e cccp.err ../run_base_test_amd64-m64-gcc41-nn.0000/gcc_base.amd64-m64-gcc41-nn cccp.i -o cccp.s

# ignore -C etc. focus on the final part! 

gdb -args ../run_base_test_amd64-m64-gcc41-nn.0000/gcc_base.amd64-m64-gcc41-nn cccp.i -o cccp.s


== ROSE identityTranslator ==


=== formal text ===
We use ROSE's identityTranslator. Like any other ROSE-based translators, it is designed to automatically detect input source file types and invoke the right EDG capability.
We do test ROSE on a subset of SPEC CPU2006 as part of our daily regression test.
ROSE can fully support 10 out 12 spec C benchmarks (compiling them and having verified execution results).
The 10 C benchmars are perlbench, bzip2, mcf, milc, gobmk, hmmer, sjeng, libquantum, lbm, and sphinx3.
gcc and h264ref have some minor issues (mostly related to ROSE's handling of macro calls) but we are working on this.
The way we compile SPEC using ROSE is:
1) install (via make install) ROSE to a directory, like ROSE_INS (specified via configure --prefix=ROSE_INS).
2) set up the environment for using the installed ROSE, identityTranslator is located under ROSE_INS/bin
PATH=$ROSE_INS/bin:$PATH
LD_LIBRARY_PATH=$ROSE_INS/lib:$LD_LIBRARY_PATH
export PATH LD_LIBRARY_PATH
3) preparing an SPEC configuration file (e.g. spec_installation_path/config/rose.cfg) to use ROSE
A set of relevant options in rose.cfg are:

#We want to the test to abort on errors and report immediately
ignore_errors = no
# we want have ascii and table-based output (Screen) for results
output_format = asc, Screen
#The result is not intended for official reports to the SPEC organization
reportable = 0
# compilers to compile benchmarks
CC = identityTranslator
CXX = identityTranslator
FC = identityTranslator
# compilation options: turn off ROSE ’s EDG frontend warnings
# since we are not interesting in fixing the benchmarks
COPTIMIZE = -O2 --edg:no_warnings
CXXOPTIMIZE = -O2 --edg:no_warnings
FOPTIMIZE = -O2 --edg:no_warnings

4) finally you can test ROSE on spec

cd $SPEC_CPU2006_INS
# set up environment to run spec
. ./shrc
# clean up previous object/executable files for all floating point and integer benchmarks
runspec --config=rose.cfg --tune=base --size=test --iterations=1 --noreportable fp int --action=clobber

runspec --config=$SPEC_CPU2006_CONFIG --tune=base --size=test --iterations=1 --noreportable \
perlbench bzip2 mcf milc gobmk hmmer sjeng libquantum lbm sphinx3


=== 12 C benchmarks ===
10/12 passed now
Passing set: runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable  perlbench bzip2 mcf milc gobmk hmmer sjeng libquantum lbm sphinx3

perlbench: pass !!! compilation  -D_GNUG__ problem bool problem
bzip2:     pass !!!  line continuation for two empty line problem: compilation but wrong results
gcc :                noncompilable rose_x  pending on a bug 347?, Unparse_ExprStmt::unparseEnumDeclStmt() SgInitializedName should be moved to located node
           test case test2009_05.c
mcf :      pass!!!
milc:      pass!!!

gobmk :    pass!!!   file path issue include/liberty.h vs. engine/liberty.h   rose_out.c engine/rose_out.c
hmmer :    pass!!!   EDG frontend failure: implicit C function decl
sjeng :    pass!!!
libquantum:pass!!!
h264ref:             noncompilable rose_x  recursive macro replace after macro expansion: 

lbm :       pass!!!
sphinx3:    pass!!! __FILE__ macro expansion problem. file name vs. full path+filename

runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable sphinx3

/home/yourAccount/opt/spec_cpu2006/bin/specinvoke -d /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482.sp hinx3/run/run_base_test_i386-m32-gcc42-nn.0000 -e speccmds.err -o speccmds.stdout -f speccmds.cmd -C
/home/yourAccount/opt/spec_cpu2006/bin/specinvoke -E -d /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482 .sphinx3/run/run_base_test_i386-m32-gcc42-nn.0000 -c 1 -e compare.err -o compare.stdout -f compare.c md

*** Miscompare of an4.log; for details see
    /home/yourAccount/opt/spec_cpu2006/benchspec/CPU2006/482.sphinx3/run/run_base_test_i386-m32-gcc42-nn.0 000/an4.log.mis
Error: 1x482.sphinx3


=== 7 C++ benchmarks ===
runspec --config=tux268-rose-gcc42.cfg --tune=base --size=test --iterations=1 --noreportable namd dealII soplex povray omnetpp astar xalancbmk


== References ==
Henning, J.L., SPEC CPU2000 Measuring CPU Performance in the New Millennium, 2000
http://www.spec.org/cpu2006/