Using SPEChpc™ 2021: The 'runhpc' Command

1. Basics

2. Commonly used options

--action --check_version --config --flagsurl --help --ignore_errors --iterations --loose --output_format --pmodel New --ranks --rawformat --rebuild --reportable --threads --tune

3. Less common options

--baseonly --basepeak --nobuild --comment --define --delay --deletework --expid --fake --fakereport --fakereportable --[no]graph_auto --graph_max --graph_min --http_proxy --http_timeout --info_wrap_column --keeptmp --labelNew --log_timestamp --make_no_clobber --notes_wrap_column --output_root --preenv --reportonly --review --[no]setprocgroup --showtimer --size --[no]table --test --undef --update --use_submit_for_compare --use_submit_for_speed --username --verbose --version

4 Removed/unsupported options

5 Quick reference

1. Basics

What is runhpc?   runhpc is the primary tool for SPEChpc 2021. You use it from a Linux shell command line to build and run benchmarks, with commands such as these:

runhpc --config=eniac.cfg    --action=build 605.lbm_s
runhpc --config=colossus.cfg --threads=16   528.pot3d_t
runhpc --config=z3.cfg       --ranks=64    tiny 

The first command compiles the benchmark named 605.lbm_s. The second runs the benchmark 528.pot3d_s using 16 threads. The third runs 64 ranks of all the SPEChpc Tiny benchmarks.

New with SPEChpc: The former runspec utility is renamed runhpc in SPEChpc.

Before reading this document: If you have not already done so, please install and test your SPEChpc 2021 distribution. This document assumes that you have already:

• installed SPEChpc 2021
• found a config file
• done basic edits (e.g. path)
• used one of the several shrc files to set your path.

If you have not done the above, please see the brief instructions in the Quick Start guide, or the more detailed section "Testing Your Installation" Linux.

1.1 Defaults

The SPEChpc default settings described in this document may be adjusted by config files.

The order of precedence for settings is:

Therefore, when this document tells you that something is the default, bear in mind that your config file may have changed that setting. With luck, the author of the config file will tell you so.

1.2 Syntax

The syntax for the runhpc command is:

runhpc [options] [list of benchmarks to run]

Options are described in the following sections. There, you will notice that many options have both long and short names. The long names are invoked using two dashes, and the short names use only a single dash. For long names that take a parameter, you can optionally use an equals sign. Long names can also be abbreviated, provided that you still enter enough letters for uniqueness. For example, the following commands all do the same thing:

runhpc --config=dianne_july25a --debug=99 small
runhpc --config dianne_july25a --debug 99 small
runhpc --conf dianne_july25a   --deb 99   small
runhpc -c dianne_july25a       -v 99      small

1.3 Benchmarks and Suites

In the list of benchmarks to run, you can use one or more individual benchmarks, such as 605.lbm_s, or you can run entire suites, using one of the Short Tags below.

Benchmark names: Individual benchmarks can be named, numbered, or both.
Separate them with a space.
Names can be abbreviated, as long as you enter enough characters for uniqueness.
Each of the following commands does the same thing:

runhpc -c jason_july09d --noreportable 605.lbm_s 618.tealeaf_s 635.weather_s
runhpc -c jason_july09d --noreportable 605 618 635

To exclude a benchmark: Use a hat (^, also known as carat, typically found as shift-6). Note that if hat has significance to your shell, you may need to protect it from interpretation by the shell, for example by putting it in single quotes.

bash-n.n.n$ runhpc --noreportable -c cathy_sep14c small ^619
pickyShell% runhpc --noreportable -c cathy_sep14c small '^605' 

Turning off reportable: If your config file sets reportable=yes then you cannot run a subset unless you turn that option off.

[/usr/cathy/spechpc2021]$ runhpc --config cathy_apr21b --noreportable small ^lbm 

1.4 Run order

A reportable run does these steps:

1. Ref: Run the tiny, small, medium, or large workload

   • Set up the directories for all benchmarks. If both base and peak tuning are requested, set up both.
   • Run one benchmark at a time, in order by benchmark number within the requested set, verifying that each gets the correct answer, and recording times.
   • Repeat. (*)
   • If --iterations=3, repeat again. 2 is the minimum for a reportable run. (*)
   • If both base and peak --tuning are requested, all of base is completed before starting peak.

2. Report:

   • During the reporting phase, if three runs were used, the median run time is selected from each benchmark. If two runs are used, then the slower of the two runs is selected.
   • Use the selected runs to compute ratios and overall metrics.
   • Write reports as requested in the --output_format switch.

Summarizing reportable run order: The order can be summarized as:

  setup for ref
          ref1, ref2 [, ref3] (*)

 (*) One benchmark at a time.  Third run only if --iterations=3.

Reportable order when more than one tuning is present: If you run both base and peak tuning, base is always run first.

 setup for ref
          base ref1, base ref2 [, base ref3] (*)
          peak ref1, peak ref2 [, peak ref3] (*)

 (*) One benchmark at a time.  Third run only if --iterations=3.

Reportable order when more than one suite is present: If you start a reportable using more than one suite, all the work is done for one suite before proceeding to the next.

1.5 Disk Usage

1.5.1 Directory tree

The structure of the SPEChpc 2021 directory tree is:

$SPEC - the root directory
   benchspec    - Some suite-wide files
   HPC          - The benchmarks
   bin          - Tools to run and report on the suite
   config       - Config files
   Docs         - HTML documentation
   result       - Log files and reports
   tmp          - Temporary files
   tools        - Sources for the SPEChpc tools

Within each of the individual benchmarks, the structure is:

nnn.benchmark - root for this benchmark
   build      - Benchmark binaries are built here
   data        
      all     - Data used by all runs (if needed by the benchmark)
      ref     - The timed data set
      test    - Data for a simple test that an executable is functional
      train   - Data for feedback-directed optimization (not used in SPEChpc. Contains same workload as test)
   Docs       - Documentation for this benchmark
   exe        - Compiled versions of the benchmark
   run        - Benchmarks are run here
   Spec       - SPEC metadata about the benchmark
   src        - The sources for the benchmark

Tiny, Medium and Larger benchmarks (5nn.benchmark_t, 7nn.benchmark_m,8nn.benchmark_l) share content that is located under a corresponding Small benchmark (6nn.benchmark_s).

Look for the output of your runhpc command in the directory $SPEC/result. There, you will find log files and result files. More information about log files can be found in the Config Files document.

The format of the result files depends on what was selected in your config file, but will typically include at least .txt for ASCII text, and will always include .rsf, for raw (unformatted) run data. More information about result formats can be found below, under --output_format. Note that you can always re-generate the output, using the --rawformat option, also documented below.

1.5.2 Hey! Where did all my disk space go?

When you find yourself wondering "Where did all my disk space go?", the answer is usually "The run directories." Most activity takes place in automatically created subdirectories of $SPEC/benchspec/HPC/*/run/. Other consumers of disk space underneath individual nnn.benchmark directories include the build/ and exe/ directories.

At the top of the directory tree, space is used by your config/ and result/ directories, and for temporary directories

$SPEC/tmp
output_root/tmp

Usually, the largest amount of space is in the run directories.

If you use the config file label feature, then directories are named to try to make it easy for you to hunt them down, For example, suppose Holger has a config file that he is using to test some new memory optimizations using the Small workload with OpenMP. He has set

in his config file. In that case, the tools would create directories such as these:

$ pwd
/Users/holger/hpc2021/benchspec/HPC/605.lbm_s:
$ ls -d */*Holger*
build/build_base_HolgerMemoryOpt.0000
exe/mcf_r_base.HolgerMemoryOpt
run/run_base_ref_HolgerMemoryOpt.0000
run/run_base_ref_HolgerMemoryOpt.0001
run/run_base_ref_HolgerMemoryOpt.0002
run/run_base_ref_HolgerMemoryOpt.0003
run/run_base_ref_HolgerMemoryOpt.0004
run/run_base_ref_HolgerMemoryOpt.0005
run/run_base_ref_HolgerMemoryOpt.0006
run/run_base_ref_HolgerMemoryOpt.0007
run/run_base_ref_HolgerMemoryOpt.0008
run/run_base_ref_HolgerMemoryOpt.0009
run/run_base_ref_HolgerMemoryOpt.0010
run/run_base_ref_HolgerMemoryOpt.0011
run/run_base_ref_HolgerMemoryOpt.0012
run/run_base_test_HolgerMemoryOpt.0000
$  

To get your disk space back, see the documentation of the various cleaning options, below.

1.6 Multi-user support

SPEChpc 2021 supports multiple users sharing an installation; however you must choose carefully regarding file protections. This section describes the multi-user features and protection options.

Features that are always enabled:

• SPEC's original source and data directories are not changed during testing.
• Work directories are created under nnn.benchmark/build and nnn.benchmark/run.
• Usage is tracked in nnn.benchmark/{run,build}/list.
• Directories created for one user are not re-used for another.
• Multiple users can run tests at the same time. Of course, if the jobs compete with each other for resources, it is likely that they will run more slowly.

Limitations: The default methods impose two key limitations, which will not be safe in some environments:

1. The directory tree must be writable by each of the users, which means that they have to trust each other not to modify or delete each others' files.
2. Directories such as result/ and nnn.benchmark/exe/ and nnn.benchmark/run/ are not segregated by user. Therefore you can have only one version of (for example) 605.lbm_s/exe/lbm_base.Ofast and different users will have their result logs intermixed with each others in the result/ directory.

Partial solution(?) expid+conventions:
You can deal with limitation #2 if users adopt certain habits. For example, Swen could name all his config files swen-something.cfg. He could use runhpc --expid=swen or the corresponding config file expid=swen to cause his results to be placed under $SPEC/result/swen and binaries under nnn.benchmark/exe/swen. Unfortunately, this alleged solution still requires that the tree be writeable by all users, and will not help Swen at all when Chris comes along and blithely does one of the alternate cleaning methods.

Solution(?) Give up:
You could just choose to spend the disk space to give each person their own tree. For SPEChpc 2021, this may increase disk space requirement by about 8 GB per user.

Recommended Solution: output_root. The recommended method uses 4 steps:

More detail:

1. Most of the SPEChpc 2021 tree is shared, and can be protected read-only. For example, on a Unix system, you might set protections with:

   chmod -R ugo-w $SPEC
   

2. The one exception is the config directory, $SPEC/config/ which needs to be a read/write directory shared by all the users, and config files must be writeable. On most Linux system, chmod 1777 is very useful: it lets anyone create files, which they own, control, and protect. (1777 is commonly used for /tmp for this very reason.)

   chmod 1777 $SPEC/config
   chmod u+w $SPEC/config/*cfg
   

New in SPEChpc 2021: The SPEC installation tree may now be fully read-only including the config directory. However in this case, you must to provode the full path to the config file to runhpc since it can no longer be in the default location.
Example: runhpc --config /path/to/config.cfg.

3. Config files usually would not be shared between users. For example, students might create their own copies of a config file:

   Kevin enters: cd /cs403/hpc2021 . ./shrc cd config cp assignment1.cfg kevin1.cfg chmod u+w kevin1.cfg runhpc --config=kevin1 --action=build 605.lbm_s

   Bert enters: cd /cs403/hpc2021 . ./shrc cd config cp assignment1.cfg bert1.cfg chmod u+w bert1.cfg runhpc --config=bert1 --action=build 605.lbm_s

4. Set output_root in the config files to change the destinations of the outputs. For example, if config files include (near the top):

   output_root=/home/${username}/hpc2021
   label=feb27a
   

   then these directories will be used for the above runhpc command:

   Kevins's directories	build: /home/kevin/hpc2021/benchspec/HPC/605.lbm_s/build/build_base_feb27a.0001 Logs: /home/kevin/hpc2021/result
   Bert's	build: /home/bert/hpc2021/benchspec/HPC/605.lbm_s/build/build_base_feb27a.0001 Logs: /home/bert/hpc2021/result

Navigation: Unix users can easily navigate an output_root tree using ogo

1.7 Actions

Most runhpc commands perform an action on a set of benchmarks.

The default action is validate.
The actions are described in two tables below: first, actions that relate to building and running; and then actions regarding cleanup.

Cleaning actions are listed in order from least thorough to most:

runhpc --action clean --config may12a small

Clean by hand:
If you prefer, you can clean disk space by entering commands such as the following:

rm -Rf $SPEC/benchspec/HPC/*/run
rm -Rf $SPEC/benchspec/HPC/*/build
rm -Rf $SPEC/benchspec/HPC/*/exe 

The above commands not only empty the contents of the run and exe directories; they also delete the directories themselves. That's fine; the tools will re-create the run and exe directories if they are needed again later on.

result directories can be cleaned or renamed. Don't worry about creating a new directory; runhpc will do so automatically. You should be careful to ensure no surprises for any currently-running users. If you move result directories, it is a good idea to also clean temporary directories at the same time.
Example:

cd $SPEC
mv result old-result
rm -Rf tmp/
cd output_root     # (If you use an output_root)
rm -Rf tmp/

I have so much disk space, I'll never use all of it:

Run directories are automatically re-used for subsequent runs. If you prefer, you can ask the tools to never touch a used run directory. Do this by setting the environment variable:

     SPEC_NO_RUNDIR_DEL

In this case, you should be prepared to do frequent cleaning, perhaps after reviewing the results of each run.

2 Commonly used options

Most users of runhpc will want to become familiar with the following options.

This section is organized alphabetically without regard to upper/lower case and without regard to presence or absence of no at the start of the switch.

--action action

• Synonym: -a action
• Default: validate
• Meaning: One of the following actions that runhpc can do for you:   build, buildsetup, clean, clobber, onlyrun, realclean, report, run, runsetup, scrub, setup, trash, or validate. For more information, see Actions, above

--check_version

• Default: check only when doing reportable runs
• Meaning: From time to time, SPEC may update the SPEChpc 2021 suite. Typically, SPEC will not publish results at www.spec.org/hpg/hpc2021 unless they have been run using the most recent update. Prior to investing substantial time running your tests, it is recommended that you check whether your version is up to date, either by saying runhpc --check_version, or by adding the switch to whatever else you were doing on your runhpc command. Note that if you are running within a firewall, you may need to add a proxy server to the command line. For example:

     runhpc --check_version --http_proxy http://webcache.tom.spokewrenchdad.com:8080 
  

  or, equivalently, for those who prefer to abbreviate to the shortest possible amount of typing:

     runhpc --ch --http_p http://webcache.tom.spokewrenchdad.com:8080 
  

  The command downloads a small file (~15 bytes) from www.spec.org which contains information about the most recent release, and compares that to your release. If your version is out of date, a warning will be printed.

--config name

• Synonym: -c name
• Default: default.cfg
• Meaning: Use config file $SPEC/config/name.cfg (Unix) or %SPEC%\config\name.cfg (Windows). If not specified, the tools will try default.cfg

--copies number

• Synonyms: -C number
• Default: 1

• Meaning: Use number copies for a SPECrate run. Not used with SPEChpc.

--flagsurl URL[,URL...]

• Synonym: -F URL
• Default: none

• Meaning: A "flags file" tells runhpc -- and the reader -- how to interpret tuning options, for example -O3 or -Ofast.
  If you want more than one, separate them with commas, or repeat the --flagsurl switch.
  These are equivalent:

  runhpc --flagsurl=$SPEC/compiler.xml,$SPEC/platform.xml      
  runhpc --flagsurl=$SPEC/compiler.xml --flagsurl=$SPEC/platform.xml

  You can use either a file path or an http:// address. If needed, add an --http_proxy (or use the corresponding config file option).

  The special value noflags may be used to cause rawformat to remove a stored flags file when re-formatting a previously run result.

  Help, I got an error message about INVALID RUN:

  ##############################################################################
  #   INVALID RUN -- INVALID RUN -- INVALID RUN -- INVALID RUN -- INVALID RUN  #
  #                                                                            #
  # Your run was marked invalid because it has one or more flags in the        #
  # "unknown" category. You might be able to resolve this problem without      #
  # re-running your test; see                                                  #
  #      https://www.spec.org/hpg/hpc2021/Docs/runhpc.html#flagsurl            #
  # for more information.                                                      #
  #                                                                            #
  #   INVALID RUN -- INVALID RUN -- INVALID RUN -- INVALID RUN -- INVALID RUN  #
  ##############################################################################

  Flags files are required by rule 4.6. If you don't have one, or if your flags file is obsolete, you will see the above error.
  To fix it:

  1. Find the sections of the report marked "Unknown".

     • If there are many unknown flags, you probably forgot to use a flags file, or you used the wrong one.
     • If there are only a few, perhaps your flags file just needs a few updates.

  2. You can ask your compiler vendor for help, or you can adapt a flags file from other results, or you can fix it yourself.

  3. Once your new flags files are available, make a copy of your rawfile.
     Then, insert the flags files, using either of these two equivalent commands:

     rawformat --flagsurl=...
     runhpc --rawformat --flagsurl=...

  4. Linux example:

     cp results/hpc2021_sml.005.small.ref.rsf results/retry.rsf
     rawformat --flagsurl $SPEC/new.compiler.xml,$SPEC/new.platform.xml retry.rsf 

--help

• Synonyms: -h | -?
• Default: Help text is not displayed.
• Meaning: Print helpful text

--ignore_errors

• Synonyms: -I | --ignoreerror
• Default: Stop on first error.
• Meaning: Continue running even if errors occur.

--iterations number

• Synonym: -n number
• Default: 2
• Meaning: How many times to run each benchmark. For a reportable run, must be 2 or 3.

--loose

• Synonyms: -l | --nostrict | --noreportable
• Default: loose
• Meaning: Unset reportable. For example, suppose that you have a golden config file that is used for reportable runs, which sets the config file option reportable. However, you aren't ready to do a reportable run, because you first want to ensure that the system under test is adequately configured to run the benchmark 635.weather_s. Since the config file sets the option reportable, the following command will result in an error message:

  [/usr/mayara/spechpc2021]$ runhpc --config golden --iterations 1 635.weather_s
  

  as the SPEC tools will inform you that you cannot change the number of iterations on a reportable run. But either of the following commands will override the config file and just run 635.weather_r once:

  [/usr/mayara/spechpc2021]$ runhpc --config golden --iterations 1 --loose 635.weather_s
  [/usr/mayara/spechpc2021]$ runhpc --config golden --iterations 1 --noreportable 635.weather_s
  

--output_format format

• Synonym: -o format
• Default: HTML and text
• Meaning: Desired report format, one or more of the following. An output format may be deselected by prepending any of the names with the word "no". If more than one option is used, separate them by commas or colons. The values are not case-sensitive.

config
   cfg|cfgfile
   configfile
   conffile

check
   subcheck
   reportcheck
   reportable
   reportablecheck
   chk|sub|subtest|test

csv
   spreadsheet

default

flag|flags

html
   xhtml|www|web

mail
   mailto|email

pdf
   adobe

postscript
   ps|printer|print

raw
   rsf

screen|scr|disp
   display|terminal|term

text
   txt|ASCII|asc

--pmodel model (New in SPEChpc)

• Synonym: none
• Default: MPI

• Meaning: When the benchmarks are run, set the config file $pmodel variable used to select the node level parallel model when compiling the benchmark. Valid options are:

  MPI	Use MPI only without a node level parallel model.
  ACC	Use MPI+OpenACC.
  OMP	Use MPI+OpenMP using task/thread based directives.
  TGT	Use MPI+OpenMP using 'target' based directives.

  Notes

  1. This option sets the appropriate define flags to unguard the model's directives in the source.
  2. The appropriate compiler flag to enable compilation of the model must also be set as part of the compiler's optimization flags!.
  3. Not all compilers support all models. Please consult you compiler's documentation on which are supported and appropriate flags to use.
  4. You may prefer to set the config file option pmodel because setting it there allows you to vary the node level parallel model for different benchmarks when running peak.
  5. If you set it on the command line with runhpc --pmodel=[MPI|ACC|OMP|TGT] then the command line setting will override your config file settings.

--ranks N

• Synonym: none
• Default: 1

• Meaning: When the benchmarks are run, set the config file $ranks variable used with a submit command to control the number of MPI ranks to use.

  Notes

  1. You may prefer to set the config file option ranks= because setting it there allows you to vary the count for different benchmarks when running peak.
  2. If you set it on the command line with runhpc --ranks=N then the command line setting will override your config file settings.

--rawformat rawfiles

• Synonym: -R rawfile
• Default: Do not use rawformat mode.

• Meaning: Do not attempt to do a run; instead, just generate reports from an existing rawfile.

  Output will always include the results of format check unless you add nocheck to your list of output_formats.

  Using this option will cause any specified --actions to be ignored. The runhpc program is actually exited and rawformat is executed instead. These commands do the same thing:

  runhpc --rawformat something
  rawformat something

  The rawformat utility or the --rawformat switch can be useful if (for example) you are just doing ASCII output during most of your runs, but now you would like to create additional reports for one or more especially interesting runs. To create the html and PDF files for experiment number 77, you could say either of these:

  runhpc --rawformat --output_format html,ps $SPEC/result/hpc2021_sml.005.small.ref.rsf 
  rawformat          --output_format html,ps $SPEC/result/hpc2021_sml.005.small.ref.rsf 

  For more information about rawformat, please see utility.html.

--rebuild

• Synonym: -D
• Default: Do not rebuild unless binary is missing or does not match the config file.
• Meaning: Delete existing binaries and recompile. Use this if, for example, you have installed a new version of your compiler and want to force a rebuild of the benchmarks.

--reportable

• Synonyms: -s | --strict | --noloose
• Default: loose
• Meaning: Wherever it is practical to do so in an automated fashion, enforce the SPEChpc 2021 run rules, so as to produce a result which is suitable for public reporting and/or submission to SPEC. This option forces various other options, for example iterations must be 2 or 3. Note that when you do a reportable run, the list of benchmarks to run must be an entire suite.

--threads N

• Synonym: none
• Default: 1

• Meaning: When the benchmarks are run, set the environment variables OMP_NUM_THREADS=N and ACC_NUM_CORES=N

  Notes

  1. If you have set OMP_NUM_THREADS or ACC_NUM_CORES in your environment prior to using runhpc, that setting will be ignored. To use multiple benchmark threads, you must use either runhpc --threads=N or the corresponding config file option threads=N.
  2. You may prefer to set the config file option threads= because setting it there allows you to vary the count for different benchmarks when running peak.
  3. If you set it on the command line with runhpc --threads=N then the command line setting will override your config file settings.
  4. The number of threads will be set for every MPI rank. Example: runhpc --ranks=8 --threads=16 will use 8 MPI ranks, each spawing 16 CPU threads, for a total of 128 CPU threads.

--tune tuning

• Synonyms: --tuning tuning | -T tuning
• Default: base
• Meaning: Selects tuning to use: base, peak, or all. For a reportable run, must be either base or all. Reportable runs do base first, then (optionally) peak.

3. Less common options

This section is organized alphabetically without regard to upper/lower case and without regard to presence or absence of no at the start of the switch.

--baseonly

• Synonym: None
• Default: Discard peak results and report only base.
• Meaning: This switch only has meaning in --rawformat mode, when reformatting previously-run results.

--basepeak [bench,bench,...]

• Synonym: None
• Default: Do not do basepeak when reformatting results.
• Meaning: This switch only has meaning in --rawformat mode, when reformatting previously-run results. When no arguments are specified, this switch will perform base to peak substitution for all benchmarks in the result file. If arguments are given, they must be the name of benchmarks or benchmark suites, and the base results for only those benchmarks will be copied over the peak results for only those benchmarks.
  Note: Use of this option will cause rawformat to output only a raw file. This is to help ensure that there is always a raw file present to match any generated results.

--nobuild

• Synonym: -N
• Default: normally, the tools will make their own decision of whether to rebuild, based on whether the binaries exist, or whether they match the expected checksums.

• Meaning: Do not build binaries, even if they don't exist or checksums don't match.

  The --nobuild feature can be very handy if, for example, you have script with multiple invocations of runhpc, and you would like to ensure that the build is only attempted once. (Perhaps your thought process might be, "If it fails the first time, fine, just forget about it until I come in Monday and look things over.") By adding --nobuild --ignore_errors to all runs after the first one, no attempt will be made to build the failed benchmarks after the first attempt.

  The --nobuild feature also comes in handy when testing whether proposed config file options would potentially force an automatic rebuild.

--comment "comment text"

• Synonym: None
• Default: Add no comment.
• Meaning: Adds a comment to the log file and the stored configuration file.

--define SYMBOL[=VALUE]
--define SYMBOL:VALUE

• Synonym: -S SYMBOL[=VALUE] or -S SYMBOL:VALUE
• Default: Do not define config file preprocessor macros.

• Meaning: Define a preprocessor macro named SYMBOL and optionally give it the value VALUE.
  If no value is specified, the macro is defined with no value.
  SYMBOL may not contain equals signs ("=") or colons (":").
  This option may be used multiple times.

  Many of the Example config files in your config/ directory have sections similar to this:

  %ifndef %{build_ncpus} 
  %   define  build_ncpus 8      
  %endif
     .
     .
     .
  makeflags            = --jobs=%{build_ncpus}

  If you have a large server and want compiles to complete more quickly, you could say runhpc --define build_ncpus=99 and specmake will create up to 99 compile jobs at a time.

--delay secs

• Synonym: None
• Default: Add no delay.
• Meaning: Add a delay of the specified number of seconds before and after each benchmark (specinvoke) invocation. The delay is not counted toward the benchmark runtime. This option is silently ignored during reportable runs.

--deletework

• Synonym: -d
• Default: Do not delete working directories.
• Meaning: Delete run directories and re-populate. Use this if you suspect that the run directories may have been corrupted, for example due to a system crash during setup of an earlier run. You might also use this if you just wish to ensure clean run directories, but note that the tools will automatically provide what they deem to be a sufficient level of cleaning.

--expid subdir

• Synonym: None
• Default: None
• Meaning: If set to a non-empty value, will cause executables, run directories, results, and log files to be put in a subdirectory (with the same name as the value set) in their normal directories. For example, using --expid=CDS will cause benchmark binaries to end up in exe/CDS, run directories to end up in run/CDS, and results and logs in $SPEC/result/CDS.

--fake

• Synonyms: --dryrun | --dry-run
• Default: Really do the actions requested.
• Meaning: List, but do not execute, the commands needed to build or run the benchmarks.
  • Use --fake --action build to get a listing of the commands needed to build a benchmark.
  • Use --fake --action run to get a listing of the commands needed to run a benchmark. (If the benchmark executable does not exist, you will also get a listing of the needed build commands.)
  • Warning: Fake mode is not implemented for the cleaning actions.
    For example, if you say runhpc --fake --action=clean the cleaning really happens.
  Runcpu uses specmake -n and specinvoke -n to generate these listings.

--fakereport

• Synonyms: --action=report | --reportonly
• Default: Not set
• Meaning: Process the config file and generate output as if a run had been done. The results will all be zero, but otherwise the report will be correctly formatted, as if it had come from an actual run. Note that if your config file or command line includes options that would cause a real run to be marked invalid (for example, setting iterations=1 or --size=test), then the tools will also mark the fake report as invalid. If your goal is to get a preview of what the final, golden run will look like, you might prefer --fakereportable.

--fakereportable

• Synonyms: --mockup
• Default: Not set
• Meaning: Process the config file and generate output as if a "reportable" run had been done. The results will all be zero, but otherwise the report will be correctly formatted, as if it had come from a reportable run.

--[no]graph_auto

• Synonyms: --[no]graphauto
• Default: --nograph_auto
• Meaning: frees the tools to pick a value that they think appropriate for the top and bottom of the Y axis. The maximum and minimum chosen will attempt to let the viewer observe differences among the benchmarks. By default, the minimum will instead be anchored to zero.

--graph_max N

• Synonyms: --graphmax
• Default: big enough (plus a little)
• Meaning: Allows you to specify a new value for the maximum point on the graph scale.

--graph_min N

• Synonyms: --graphmin N
• Default: 0
• Meaning: Allows you to specify a new value for the minimum point on the graph scale.

--http_proxy proxy[:port]

• Synonyms: none
• Default: none

• Meaning: In some cases, such as when doing version checks and loading flag description files, runhpc will attempt to fetch a file, using http. If your web browser needs a proxy server in order to access the outside world, then runhpc will probably want to use the same proxy server. The proxy server can be set by:

  • The runhpc --http_proxy switch, or
  • The environment variable http_proxy, or
  • Using the config file http_proxy option

  For example, a failure of this form:

  $ runhpc --rawformat --output_format txt \
     --flagsurl http://portlandcyclers.net/evan.xml SPEChpc 2021.007.fprate.rsf
  ...
  Retrieving flags file (http://portlandcyclers.net/evan.xml)...
  
  ERROR: Specified flags URL (http://portlandcyclers.net/evan.xml) could not be retrieved.
         The error returned was: 500 Can't connect to portlandcyclers.net:80 
         (Bad hostname 'portlandcyclers.net')
  

  improves when a proxy is provided:

  $ runhpc --rawformat --output_format txt \
     --flagsurl http://portlandcyclers.net/evan.xml  \
     --http_proxy=http://webcache.tom.spokewrenchdad.com:8080 SPEChpc 2021.007.fprate.rsf
  

  Note that this setting will override the value of the http_proxy environment variable, as well as any setting in the config file.

  By default, no proxy is used. The special value none may be used to unset any proxies set in the environment or via config file.

--http_timeout N

• Synonyms: none
• Default: 30 seconds
• Meaning: This is the amount of time (in seconds) to wait while attempting to fetch a file via HTTP. If the connection cannot be established in this amount of time, the attempt will be aborted.

--info_wrap_columns N

• Synonym: --infowrap N
• Default: 50
• Meaning: When set to a value greater than 0, attempts to split non-notes informational lines such that they are no longer than N columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line.
  If a line contains an item that is longer than N, a warning is logged and the original line is left unchanged.

--[no]keeptmp

• Synonym: None
• Default: --nokeeptmp
• Meaning: When set, temporary files such as compiler option output and temporary log files will not be removed. For a list of temporary file locations, see the discussion in config.html.

--label name

• Synonyms: -L name
• Default: none
• Meaning: Label to build or run. Normally used only if the config file has been written to handle more than one label. The config file author should tell you what labels to enter under which circumstances.
  Note: The label used may only consist of alphanumeric characters, underscores, hyphens, and periods.

--[no]log_timestamp

• Synonym: --[no]log-timestamp
• Default: --nolog_timestamp
• Meaning: When set, lines output to the log file will have a time stamp prepended. The timestamp is always present in the debug log.

--make_no_clobber

• Synonym: -M
• Default: clobber

• Meaning: Do not delete existing object files before attempting to build. This option should only be used for troubleshooting a problematic compile. It cannot be used for a reportable run.

  Rather than using this option, it would probably be easier to just go to the build directory and use specmake

--notes_wrap_columns N

• Synonym: --noteswrap
• Default: 0 (disabled)
• Meaning: When set to a value greater than 0, attempts to split notes lines such that they are no longer than N columns wide. Lines are split on whitespace, and newly created lines are guaranteed to have at least the same indentation as the original line.
  If a line contains an item that is longer than N, a warning is logged and the original line is left unchanged.

--output_root directory

• Synonym: --output-root directory
• Default: None

• Meaning: If set to a non-empty value, all output files will be rooted under the named directory, instead of under $SPEC (or %SPEC%).
  If directory is not an absolute path (one that begins with "/" on Unix, or a device name on Windows), the path will be created under $SPEC.

  This option can be useful for sharing an installation.
  It can also be useful if you want to optimize your I/O, as discussed in the corresponding SPEChpc 2021 Config Files section on output_root.

--preenv, --nopreenv

• Synonym: none
• Default: True (preENV_ lines are used)
• Meaning: When set, causes preENV_ lines in the config file to be used. Use of --nopreenv will disable this behavior.

--review, --noreview

• Synonym: none
• Default: --noreview
• Meaning: Format results for review, meaning that additional detail will be printed that normally would not be present. In lists of Portability Flags, benchmarks that do not use any flags are nevertheless listed, along with the notation "No flags used". Flags marked to be hidden (typically, splitters) will be displayed anyway, along with a notation that they are usually hidden.

--setprocgroup, --nosetprocgroup

• Synonyms: none
• Default: --setprocgroup
• Meaning: Attempt to create all processes in a single group. Improves the chances that ^C will get the whole run, not just one of the children.

--showtimer