\chapter{Running the benchmarks }
\hypertarget{benchmarking_ginkgo}{}\label{benchmarking_ginkgo}\index{Running the benchmarks@{Running the benchmarks}}
\label{benchmarking_ginkgo_md__2build_2ginkgo-hpc_2src_2ginkgo-1_811_80_2BENCHMARKING}%
\Hypertarget{benchmarking_ginkgo_md__2build_2ginkgo-hpc_2src_2ginkgo-1_811_80_2BENCHMARKING}%


In addition to the unit tests designed to verify correctness, Ginkgo also includes an extensive benchmark suite for checking its performance on all Ginkgo supported systems. The purpose of Ginkgo\textquotesingle{}s benchmarking suite is to allow easy and complete reproduction of Ginkgo\textquotesingle{}s performance, and to facilitate performance debugging as well. Most results published in Ginkgo papers are generated thanks to this benchmarking suite and are accessible online under the \href{https://github.com/ginkgo-project/ginkgo-data/}{\texttt{ginkgo-\/data repository}}. These results can also be used for performance comparison in order to ensure that you get similar performance as what is published on this repository.

To compile the benchmarks, the flag {\ttfamily -\/DGINKGO\+\_\+\+BUILD\+\_\+\+BENCHMARKS=ON} has to be set during the {\ttfamily cmake} step. In addition, the \href{https://github.com/ginkgo-project/ssget}{\texttt{{\ttfamily ssget} command-\/line utility}} can be installed on the system to load matrices. The purpose of this file is to explain in detail the capacities of this benchmarking suite as well as how to properly setup everything.

There are two ways to benchmark Ginkgo. When compiling the benchmark suite, executables are generated for collecting matrix statistics, running sparse-\/matrix vector product, solvers (possibly distributed) benchmarks. Thus, one can run the needed executable for benchmarking \texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}manually\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}. Another way to run (almost) \texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*}all benchmarks\texorpdfstring{$\ast$}{*}\texorpdfstring{$\ast$}{*} is through the convenience script {\ttfamily run\+\_\+all\+\_\+benchmarks.\+sh}, but not all features are exposed through this tool!

Here is a short description of the content of this file:\+
\begin{DoxyEnumerate}
\item Ginkgo setup and best practice guidelines
\item Using ssget to fetch the matrices
\item Running benchmarks manually
\item Benchmarking with \`{}make benchmark\`{} or \`{}run\+\_\+all\+\_\+benchmarks.sh\`{}
\item Publishing the results on Github and analyze the results with the GPE (optional)
\item Detailed performance analysis and debugging
\item Available benchmark options
\end{DoxyEnumerate}\hypertarget{benchmarking_ginkgo_autotoc_md11}{}\doxysubsection{\texorpdfstring{1:\+ Ginkgo setup and best practice guidelines}{1:\+ Ginkgo setup and best practice guidelines}}\label{benchmarking_ginkgo_autotoc_md11}
Before benchmarking Ginkgo, make sure that you follow the general guidelines in order to ensure best performance.


\begin{DoxyEnumerate}
\item The code should be compiled in {\ttfamily Release} mode.
\item Make sure the machine has no competing jobs. On a Linux machine multiple commands can be used, {\ttfamily last} shows the currently opened sessions, {\ttfamily top} or {\ttfamily htop} allows to show the current machine load, and if considering using specific GPUs, {\ttfamily nvidia-\/smi} or {\ttfamily rocm-\/smi} can be used to check their load.
\item By default, Ginkgo\textquotesingle{}s benchmarks will always do at least one warm-\/up run. For better accuracy, every benchmark is also averaged over 10 runs, except for the solver benchmark which are usually fairly long. These parameters can be tuned at the command line to either shorten benchmarking time or improve benchmarking accuracy.
\end{DoxyEnumerate}

In addition, the following specific options can be considered:\+
\begin{DoxyEnumerate}
\item When specifically using the adaptive block jacobi preconditioner, enable the {\ttfamily GINKGO\+\_\+\+JACOBI\+\_\+\+FULL\+\_\+\+OPTIMIZATIONS} CMake flag. Be careful that this will use much more memory and time for the compilation due to compiler performance issues with register optimizations, in particular.
\item The current benchmarking setup also allows to benchmark only the overhead by using as either (or for all) preconditioner/\+spmv/\+solver, the special {\ttfamily overhead} Lin\+Op. If your purpose is to check Ginkgo\textquotesingle{}s overhead, make sure to try this mode.
\end{DoxyEnumerate}\hypertarget{benchmarking_ginkgo_autotoc_md12}{}\doxysubsection{\texorpdfstring{2:\+ Using ssget to fetch the matrices}{2:\+ Using ssget to fetch the matrices}}\label{benchmarking_ginkgo_autotoc_md12}
To benchmark {\ttfamily ginkgo}, matrices need to be provided as input in the {\ttfamily Matrix Market} format. A convenient way is to run benchmark with the \href{https://sparse.tamu.edu/}{\texttt{Suite\+Sparse matrix collection}}. A helper tool, the \href{https://github.com/ginkgo-project/ssget}{\texttt{{\ttfamily ssget} command-\/line utility}} can be used to facilitate downloading and extracting matrices from the suitesparse collection. When running the benchmarks with the helper script {\ttfamily run\+\_\+all\+\_\+benchmarks.\+sh} (or calling {\ttfamily make benchmark}), the {\ttfamily ssget} tool is required.

To install {\ttfamily ssget}, access the repository and copy the file {\ttfamily ssget} into a directory present in your {\ttfamily PATH} variable as per the tool\textquotesingle{}s {\ttfamily README.\+md} instructions. The tool can be installed either in a global system path or a local directory such as {\ttfamily \$HOME/\+.local/\+bin}. After installing the tool, it is important to review the {\ttfamily ssget} script and configure as needed the variable {\ttfamily ARCHIVE\+\_\+\+LOCATION} on line 39. This is where the matrices will be stored into. As of \href{https://github.com/ginkgo-project/ssget/pull/7}{\texttt{PR\#7}}, one can also configure the archive location using the {\ttfamily -\/a} option without modifying the script itself. You can also use this script without installing it.

The Ginkgo benchmark can be set to run on only a portion of the Suite\+Sparse matrix collection as we will see in the following section. Please note that the entire collection requires roughly 100GB of disk storage in its compressed format, and roughly 25GB of additional disk space for intermediate data (such us uncompressing the archive). Additionally, the benchmark runs usually take a long time (Sp\+MV benchmarks on the complete collection take roughly 24h using the K20 GPU), and will stress the system.

Before proceeding, it can be useful in order to save time to download the matrices as preparation. This can be done by using the {\ttfamily ssget -\/f -\/i i} command where {\ttfamily i} is the ID of the matrix to be downloaded. The following loop allows to download the full Suite\+Sparse matrix collection:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{for\ i\ in\ \$(seq\ 0\ \$(ssget\ -\/n));\ do}
\DoxyCodeLine{\ \ \ \ ssget\ -\/f\ -\/i\ \$\{i\}}
\DoxyCodeLine{done}

\end{DoxyCode}


Note that {\ttfamily ssget} can also be used to query properties of the matrix and filter the matrices which are downloaded. For example, the following will download only positive definite matrices with less than 500M non zero elements and 10M columns. Please refer to the \href{https://github.com/ginkgo-project/ssget/blob/master/README.md}{\texttt{{\ttfamily ssget} documentation}} for more information.


\begin{DoxyCode}{0}
\DoxyCodeLine{for\ i\ in\ \$(seq\ 0\ \$(ssget\ -\/n));\ do}
\DoxyCodeLine{\ \ \ \ posdef=\$(ssget\ -\/p\ posdef\ -\/i\ \$\{i\})}
\DoxyCodeLine{\ \ \ \ cols=\$(ssget\ -\/p\ cols\ -\/i\ \$\{i\})}
\DoxyCodeLine{\ \ \ \ nnz=\$(ssget\ -\/p\ nonzeros\ -\/i\ \$\{i\})}
\DoxyCodeLine{\ \ \ \ if\ [\ "{}\$posdef"{}\ -\/eq\ 1\ -\/a\ "{}\$cols"{}\ -\/lt\ 10000000\ -\/a\ "{}\$nnz"{}\ -\/lt\ 500000000\ ];\ then}
\DoxyCodeLine{\ \ \ \ \ \ \ \ ssget\ -\/f\ -\/i\ \$\{i\}}
\DoxyCodeLine{\ \ \ \ fi}
\DoxyCodeLine{done}

\end{DoxyCode}


For extracting the matrices, {\ttfamily ssget -\/f -\/i \$\{i\}} can be used.\hypertarget{benchmarking_ginkgo_autotoc_md13}{}\doxysubsection{\texorpdfstring{3:\+ Running benchmarks manually}{3:\+ Running benchmarks manually}}\label{benchmarking_ginkgo_autotoc_md13}
As mentioned above, one way to run benchmarks is through the executable of a certain needed type of benchmarking.

When compiling Ginkgo with the flag {\ttfamily -\/DGINKGO\+\_\+\+BUILD\+\_\+\+BENCHMARKS=ON}, a suite of executables will be generated depending on the CMake configuration. These executables are the backbone of the benchmarking suite. Note that all of these executables describe the available options and the required input format when running them with the {\ttfamily -\/-\/help} option. All executables have multiple variants depending on the precision, by default {\ttfamily double} precision is used for the type of values, but variants with {\ttfamily single} and {\ttfamily complex} (single and double) value types are also available. Here is a non exhaustive list of the available benchmarks:\+


\begin{DoxyItemize}
\item {\ttfamily blas/\+blas}:\+ supports benchmarking many of Ginkgo\textquotesingle{}s BLAS operations:\+ dot products, axpy, copy, etc.
\item {\ttfamily conversion/\+conversion}:\+ conversion between matrix formats.
\item {\ttfamily matrix\+\_\+generator/\+matrix\+\_\+generator}:\+ mostly allows generating block diagonal matrices (to benchmark the block-\/jacobi preconditioner).
\item {\ttfamily matrix\+\_\+statistics/\+matrix\+\_\+statistics}:\+ computes size and other matrix statistics (such as variance, load imbalance, ...).
\item {\ttfamily preconditioner/\+preconditioner}:\+ benchmarks most Ginkgo preconditioner.
\item {\ttfamily solver/\+solver}:\+ benchmark most of Ginkgo\textquotesingle{}s solvers in a non distributed setting.
\item {\ttfamily sparse\+\_\+blas/\+sparse\+\_\+blas}:\+ benchmarks Sparse BLAS operations, such as Sp\+GEMM, Sp\+GEAM, transpose.
\item {\ttfamily spmv/\+spmv}:\+ benchmarks Ginkgo\textquotesingle{}s matrix formats (Sparse-\/\+Matrix Vector product).
\end{DoxyItemize}

Optionally when compiling with MPI support:\+
\begin{DoxyItemize}
\item {\ttfamily blas/\+distributed/\+multi\+\_\+vector}:\+ measures BLAS performance on (distributed) multi-\/vectors.
\item {\ttfamily solver/\+distributed/\+solver}:\+ distributed solver benchmarks.
\item {\ttfamily spmv/\+distributed/\+spmv}:\+ distributed matrix Sparse-\/\+Matrix Vector (Sp\+MV) product benchmark.
\end{DoxyItemize}

All benchmarks require input data as in a {\ttfamily JSON} format. The json file has to consist of exactly one array, and within that array the test cases are defined. The exact syntax can change between executables, the {\ttfamily -\/-\/help} option will explain the necessary {\ttfamily JSON} input format. For example for the {\ttfamily spmv} benchmark case, and many other benchmarks the following minimal input should be provided:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{[}
\DoxyCodeLine{\ \ \{}
\DoxyCodeLine{\ \ \ \ "{}filename"{}:\ "{}path/to/your/matrix"{},}
\DoxyCodeLine{\ \ \ \ "{}rhs"{}:\ "{}path/to/your/rhs"{}}
\DoxyCodeLine{\ \ \},}
\DoxyCodeLine{\ \ \{\ ...\ \}}
\DoxyCodeLine{]}

\end{DoxyCode}
 The files have to be in matrix market format.

Some benchmarks require some extra fields. For example, the {\bfseries{solver benchmarks}} requires the field {\ttfamily "{}optimal"{}:\+ \{"{}spmv"{}:\+ "{}matrix format (such as csr)"{}\}}. This is automatically populated when running the {\ttfamily spmv} benchmark which finds the optimal (fastest) format among all requested formats.

The {\ttfamily "{}rhs"{}} field (right-\/hand side) is generally optional for the {\bfseries{solver benchmarks}} if you generate the RHS with the {\ttfamily -\/rhs\+\_\+generation} flag (use {\ttfamily -\/-\/help} with benchmark executables if this is unclear).

After writing the necessary data in a JSON file, the benchmark can be called by passing in the input via stdin, i.\+e.


\begin{DoxyCode}{0}
\DoxyCodeLine{./solver\ <\ input.json}

\end{DoxyCode}


The output of our benchmarks is again JSON, and it is printed to stdout, while our status messages are printed to stderr. So, the output can be stored with


\begin{DoxyCode}{0}
\DoxyCodeLine{./solver\ <\ input.json\ >\ output.json}

\end{DoxyCode}


Note that in most cases, the JSON output by our benchmarks is compatible with other benchmarks, therefore it is possible to first call the {\ttfamily spmv} benchmark, use the resulting output JSON as input to the {\ttfamily solver} benchmark, and finally use the resulting solver JSON output as input to the {\ttfamily preconditioner} benchmark.\hypertarget{benchmarking_ginkgo_autotoc_md14}{}\doxysubsection{\texorpdfstring{4:\+ Benchmarking with {\ttfamily make benchmark} or {\ttfamily run\+\_\+all\+\_\+benchmarks.\+sh}}{4:\+ Benchmarking with {\ttfamily make benchmark} or {\ttfamily run\+\_\+all\+\_\+benchmarks.\+sh}}}\label{benchmarking_ginkgo_autotoc_md14}
The benchmark suite is invoked using the {\ttfamily make benchmark} command in the build directory. Under the hood, this command simply calls the script {\ttfamily benchmark/\+run\+\_\+all\+\_\+benchmarks.\+sh} so it is possible to manually launch this script as well. The behavior of the suite can be modified using environment variables. Assuming the {\ttfamily bash} shell is used, these can either be specified via the {\ttfamily export} command to persist between multiple runs:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{export\ VARIABLE="{}value"{}}
\DoxyCodeLine{...}
\DoxyCodeLine{make\ benchmark}

\end{DoxyCode}


or specified on the fly, on the same line as the {\ttfamily make benchmark} command:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{VARIABLE="{}value"{}\ ...\ make\ benchmark}

\end{DoxyCode}


Since {\ttfamily make} sets any variables passed to it as temporary environment variables, the following shorthand can also be used:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{make\ benchmark\ VARIABLE="{}value"{}\ ...}

\end{DoxyCode}


A combination of the above approaches is also possible (e.\+g. it may be useful to {\ttfamily export} the {\ttfamily SYSTEM\+\_\+\+NAME} variable, and specify the others at every benchmark run).

The benchmark suite can take a number of configuration parameters. Benchmarks can be run only for {\ttfamily sparse matrix vector products (spmv)}, for full solvers (with or without preconditioners), or for preconditioners only when supported. The benchmark suite also allows to target a sub-\/part of the Suite\+Sparse matrix collection. For details, see the available benchmark options. Here are the most important options:\+
\begin{DoxyItemize}
\item {\ttfamily BENCHMARK=\{spmv, solver, preconditioner\}} -\/ allows to select the type of benchmark to be run.
\item {\ttfamily EXECUTOR=\{reference,cuda,hip,omp,dpcpp\}} -\/ select the executor and platform the benchmarks should be run on.
\item {\ttfamily SYSTEM\+\_\+\+NAME=\texorpdfstring{$<$}{<}name\texorpdfstring{$>$}{>}} -\/ a name which will be used to designate this platform (e.\+g. V100, Radeon\+VII, ...).
\item {\ttfamily SEGMENTS=\texorpdfstring{$<$}{<}N\texorpdfstring{$>$}{>}} -\/ Split the benchmarked matrix space into {\ttfamily \texorpdfstring{$<$}{<}N\texorpdfstring{$>$}{>}} segments. If specified, {\ttfamily SEGMENT\+\_\+\+ID} also has to be set.
\item {\ttfamily SEGMENT\+\_\+\+ID=\texorpdfstring{$<$}{<}I\texorpdfstring{$>$}{>}} -\/ used in combination with the {\ttfamily SEGMENTS} variable. {\ttfamily \texorpdfstring{$<$}{<}I\texorpdfstring{$>$}{>}} should be an integer between 1 and {\ttfamily \texorpdfstring{$<$}{<}N\texorpdfstring{$>$}{>}}, the number of {\ttfamily SEGMENTS}. If specified, only the {\ttfamily \texorpdfstring{$<$}{<}I\texorpdfstring{$>$}{>}}-\/th segment of the benchmark suite will be run.
\item {\ttfamily BENCHMARK\+\_\+\+PRECISION} -\/ defines the precision the benchmarks are run in. Supported values are:\+ "{}double"{} (default), "{}single"{}, "{}dcomplex"{} and "{}scomplex"{}
\item {\ttfamily MATRIX\+\_\+\+LIST\+\_\+\+FILE=/\+path/\+to/\+matrix\+\_\+list.file} -\/ allows to list Suite\+Sparse matrix id or name to benchmark. As an example, a matrix list file containing the following will ensure that benchmarks are ran for only those three matrices:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{1903}
\DoxyCodeLine{Freescale/circuit5M}
\DoxyCodeLine{thermal2}

\end{DoxyCode}

\end{DoxyItemize}\hypertarget{benchmarking_ginkgo_autotoc_md15}{}\doxysubsection{\texorpdfstring{5:\+ Publishing the results on Github and analyze the results with the GPE (optional)}{5:\+ Publishing the results on Github and analyze the results with the GPE (optional)}}\label{benchmarking_ginkgo_autotoc_md15}
The previous experiments generated json files for each matrices, each containing timing, iteration count, achieved precision, ... depending on the type of benchmark run. These files are available in the directory {\ttfamily \$\{ginkgo\+\_\+build\+\_\+dir\}/\+benchmark/\+results/\+}. These files can be analyzed and processed through any tool (e.\+g. python). In this section, we describe how to generate the plots by using Ginkgo\textquotesingle{}s \href{https://ginkgo-project.github.io/gpe/}{\texttt{GPE}} tool. First, we need to publish the experiments into a Github repository which will be then linked as source input to the GPE. For this, we can simply fork the ginkgo-\/data repository. To do so, we can go to the github repository and use the forking interface:\+ \href{https://github.com/ginkgo-project/ginkgo-data/}{\texttt{https:\+/\+/\+github.\+com/\+ginkgo-\/project/\+ginkgo-\/data/\+}}

Once it\textquotesingle{}s done, we want to clone the repository locally, put all results online and access the GPE for plotting the results. Here are the detailed steps:\+


\begin{DoxyCode}{0}
\DoxyCodeLine{git\ clone\ https://github.com/<username>/ginkgo-\/data.git\ \$HOME/ginkgo\_benchmark/ginkgo-\/data}
\DoxyCodeLine{\#\ send\ the\ benchmarked\ data\ to\ the\ ginkgo-\/data\ repository}
\DoxyCodeLine{\#\ If\ needed,\ remove\ the\ old\ data\ so\ that\ no\ previous\ data\ is\ left.}
\DoxyCodeLine{\#\ rm\ -\/r\ \$\{HOME\}/ginkgo\_benchmark/ginkgo-\/data/data/\$\{SYSTEM\_NAME\}}
\DoxyCodeLine{rsync\ -\/rtv\ \$\{ginkgo\_build\_dir\}/benchmark/results/\ \$HOME/ginkgo\_benchmark/ginkgo-\/data/data/}
\DoxyCodeLine{cd\ \$\{HOME\}/ginkgo\_benchmark/ginkgo-\/data/data/}
\DoxyCodeLine{\#\ The\ following\ updates\ the\ main\ \`{}.json`\ files\ with\ the\ list\ of\ data.}
\DoxyCodeLine{\#\ Ensure\ a\ python\ 3\ installation\ is\ available.}
\DoxyCodeLine{./build-\/list\ .\ >\ list.json}
\DoxyCodeLine{./agregate\ <\ list.json\ >\ agregate.json}
\DoxyCodeLine{./represent\ .\ >\ represent.json}
\DoxyCodeLine{git\ config\ -\/-\/local\ user.name\ "{}<Name>"{}}
\DoxyCodeLine{git\ config\ -\/-\/local\ user.email\ "{}<email>"{}}
\DoxyCodeLine{git\ commit\ -\/am\ "{}Ginkgo\ benchmark\ \$\{BENCHMARK\}\ of\ \$\{SYSTEM\_NAME\}..."{}}
\DoxyCodeLine{git\ push}

\end{DoxyCode}


Note that depending on what data is of interest, you may need to update the scripts {\ttfamily build-\/list} or {\ttfamily agregate} to change which files you want to agglomerate and summarize (depending on the system name), or which data you want to select (solver results, spmv results, ...).

For the generating the plots in the GPE, here are the steps to go through:\+
\begin{DoxyEnumerate}
\item Access the GPE:\+ \href{https://ginkgo-project.github.io/gpe/}{\texttt{https:\+/\+/\+ginkgo-\/project.\+github.\+io/\+gpe/\+}}
\item Update data root URL, from {\ttfamily \href{https://raw.githubusercontent.com/ginkgo-project/ginkgo-data/master/data}{\texttt{https:\+/\+/\+raw.\+githubusercontent.\+com/\+ginkgo-\/project/\+ginkgo-\/data/\+master/\+data}}} to {\ttfamily \href{https://raw.githubusercontent.com/}{\texttt{https:\+/\+/\+raw.\+githubusercontent.\+com/\+}}\texorpdfstring{$<$}{<}username\texorpdfstring{$>$}{>}/\+ginkgo-\/data/\+\texorpdfstring{$<$}{<}branch\texorpdfstring{$>$}{>}/\+data}
\item Click on the arrow to load the data, select the {\ttfamily Result Summary} entry above.
\item Click on {\ttfamily select an example} to choose a plotting script. Multiple scripts are available by default in different branches. You can use the {\ttfamily jsonata} and {\ttfamily chartjs} languages to develop your own as well.
\item The results should be available in the tab "{}plot"{} on the right side. Other tabs allow to access the result of the processed data after invoking the processing script.
\end{DoxyEnumerate}\hypertarget{benchmarking_ginkgo_autotoc_md16}{}\doxysubsection{\texorpdfstring{6:\+ Detailed performance analysis and debugging}{6:\+ Detailed performance analysis and debugging}}\label{benchmarking_ginkgo_autotoc_md16}
Detailed performance analysis can be run by passing the environment variable {\ttfamily DETAILED=1} to the benchmarking script. This detailed run is available for solvers and allows to log the internal residual after every iteration as well as log the time taken by all operations. These features are also available in the {\ttfamily performance-\/debugging} example which can be used instead and modified as needed to analyze Ginkgo\textquotesingle{}s performance.

These features are implemented thanks to the loggers located in the file {\ttfamily \$\{ginkgo\+\_\+src\+\_\+dir\}/\+benchmark/\+utils/\+loggers.hpp}. Ginkgo possesses hooks at all important code location points which can be inspected thanks to the logger. In this fashion, it is easy to use these loggers also for tracking memory allocation sizes and other important library aspects.\hypertarget{benchmarking_ginkgo_autotoc_md17}{}\doxysubsection{\texorpdfstring{7:\+ Available benchmark options}{7:\+ Available benchmark options}}\label{benchmarking_ginkgo_autotoc_md17}
There are a set amount of options available for benchmarking. Most important options can be configured through the benchmarking script itself thanks to environment variables. Otherwise, some specific options are not available through the benchmarking scripts but can be directly configured when running the benchmarking program itself. For a list of all options, run for example {\ttfamily \$\{ginkgo\+\_\+build\+\_\+dir\}/\+benchmark/\+solver/\+solver -\/-\/help}.

The supported environment variables are described in the following list:\+
\begin{DoxyItemize}
\item {\ttfamily BENCHMARK=\{spmv, solver, preconditioner\}} -\/ allows to select the type of benchmark to be run. Default is {\ttfamily spmv}.
\begin{DoxyItemize}
\item {\ttfamily spmv} -\/ Runs the sparse matrix-\/vector product benchmarks on the Suite\+Sparse collection.
\item {\ttfamily solver} -\/ Runs the solver benchmarks on the Suite\+Sparse collection. The matrix format is determined by running the {\ttfamily spmv} benchmarks first, and using the fastest format determined by that benchmark.
\item {\ttfamily preconditioner} -\/ Runs the preconditioner benchmarks on artificially generated block-\/diagonal matrices.
\end{DoxyItemize}
\item {\ttfamily EXECUTOR=\{reference,cuda,hip,omp,dpcpp\}} -\/ select the executor and platform the benchmarks should be run on. Default is {\ttfamily cuda}.
\item {\ttfamily SYSTEM\+\_\+\+NAME=\texorpdfstring{$<$}{<}name\texorpdfstring{$>$}{>}} -\/ a name which will be used to designate this platform (e.\+g. V100, Radeon\+VII, ...) and not overwrite previous results. Default is {\ttfamily unknown}.
\item {\ttfamily SEGMENTS=\texorpdfstring{$<$}{<}N\texorpdfstring{$>$}{>}} -\/ Split the benchmarked matrix space into {\ttfamily \texorpdfstring{$<$}{<}N\texorpdfstring{$>$}{>}} segments. If specified, {\ttfamily SEGMENT\+\_\+\+ID} also has to be set. Default is {\ttfamily 1}.
\item {\ttfamily SEGMENT\+\_\+\+ID=\texorpdfstring{$<$}{<}I\texorpdfstring{$>$}{>}} -\/ used in combination with the {\ttfamily SEGMENTS} variable. {\ttfamily \texorpdfstring{$<$}{<}I\texorpdfstring{$>$}{>}} should be an integer between 1 and {\ttfamily \texorpdfstring{$<$}{<}N\texorpdfstring{$>$}{>}}, the number of {\ttfamily SEGMENTS}. If specified, only the {\ttfamily \texorpdfstring{$<$}{<}I\texorpdfstring{$>$}{>}}-\/th segment of the benchmark suite will be run. Default is {\ttfamily 1}.
\item {\ttfamily MATRIX\+\_\+\+LIST\+\_\+\+FILE=/\+path/\+to/\+matrix\+\_\+list.file} -\/ allows to list Suite\+Sparse matrix id or name to benchmark. As an example, a matrix list file containing the following will ensure that benchmarks are run for only those three matrices:\+ 
\begin{DoxyCode}{0}
\DoxyCodeLine{1903}
\DoxyCodeLine{Freescale/circuit5M}
\DoxyCodeLine{thermal2}

\end{DoxyCode}

\item {\ttfamily DEVICE\+\_\+\+ID} -\/ the accelerator device ID to target for the benchmark. The default is {\ttfamily 0}.
\item {\ttfamily DRY\+\_\+\+RUN=\{true, false\}} -\/ If set to {\ttfamily true}, prepares the system for the benchmark runs (downloads the collections, creates the result structure, etc.) and outputs the list of commands that would normally be run, but does not run the benchmarks themselves. Default is {\ttfamily false}.
\item {\ttfamily PRECONDS=\{\doxylink{group__jacobi}{Jacobi Preconditioner},ic,ilu,paric,parict,parilu,parilut,ic-\/isai,ilu-\/isai,paric-\/isai,parict-\/isai,parilu-\/isai,parilut-\/isai,none\}} the preconditioners to use for either {\ttfamily solver} or {\ttfamily preconditioner} benchmarks. Multiple options can be passed to this variable. Default is {\ttfamily none}.
\item {\ttfamily FORMATS=\{csr,coo,ell,hybrid,sellp,hybridxx,cusparse\+\_\+xx,hipsparse\+\_\+xx\}} the matrix formats to benchmark for the {\ttfamily spmv} phase of the benchmark. Run {\ttfamily \$\{ginkgo\+\_\+build\+\_\+dir\}/\+benchmark/\+spmv/\+spmv -\/-\/help} for a full list. If needed, multiple options for hybrid with different optimization parameters are available. Depending on the libraries available at build time, vendor library formats (cu\+SPARSE with {\ttfamily cusparse\+\_\+} prefix or hip\+SPARSE with {\ttfamily hipsparse\+\_\+} prefix) can be used as well. Multiple options can be passed. The default is {\ttfamily csr,coo,ell,hybrid,sellp}.
\item {\ttfamily SOLVERS=\{bicgstab,bicg,cg,cgs,fcg,pipe\+\_\+cg,gmres,cb\+\_\+gmres\+\_\+\{keep,reduce1,reduce2,integer,ireduce1,ireduce2\},lower\+\_\+trs,upper\+\_\+trs\}}
\begin{DoxyItemize}
\item the solvers which should be benchmarked. Multiple options can be passed. The default is {\ttfamily bicgstab,cg,cgs,fcg,gmres,idr}. Note that {\ttfamily lower/\+upper\+\_\+trs} by default don\textquotesingle{}t use a preconditioner, as they are by default exact direct solvers.
\end{DoxyItemize}
\item {\ttfamily SOLVERS\+\_\+\+PRECISION=\texorpdfstring{$<$}{<}precision\texorpdfstring{$>$}{>}} -\/ the minimal residual reduction before which the solver should stop. The default is {\ttfamily 1e-\/6}.
\item {\ttfamily SOLVERS\+\_\+\+MAX\+\_\+\+ITERATIONS=\texorpdfstring{$<$}{<}number\texorpdfstring{$>$}{>}} -\/ the maximum number of iterations with which a solver should be run. The default is {\ttfamily 10000}.
\item {\ttfamily SOLVERS\+\_\+\+RHS=\{1, random, sinus\}} -\/ whether to use a vector of all ones, random values or b = A \texorpdfstring{$\ast$}{*} (s /\+ \texorpdfstring{$\vert$}{|}s\texorpdfstring{$\vert$}{|})\$ with s(idx) = sin(idx) (for complex numbers, s(idx) = sin(2\texorpdfstring{$\ast$}{*}idx) + i \texorpdfstring{$\ast$}{*} sin(2\texorpdfstring{$\ast$}{*}idx+1)) as the right-\/hand side in solver benchmarks. Default is {\ttfamily 1}.
\item {\ttfamily SOLVERS\+\_\+\+INITIAL\+\_\+\+GUESS=\{rhs,0,random\}} -\/ the initial guess generation of the solvers. {\ttfamily rhs} uses the right-\/hand side, {\ttfamily 0} uses a zero vector and {\ttfamily random} generates a random vector as the initial guess.
\item {\ttfamily DETAILED=\{0,1\}} -\/ selects whether detailed benchmarks should be run. This generally provides extra, verbose information at the cost of one or more extra benchmark runs. It can be either {\ttfamily 0} (off) or {\ttfamily 1} (on).
\item {\ttfamily GPU\+\_\+\+TIMER=\{true, false\}} -\/ If set to {\ttfamily true}, use the gpu timer, which is valid for cuda/\+hip executor, to measure the timing. Default is {\ttfamily false}.
\item {\ttfamily SOLVERS\+\_\+\+JACOBI\+\_\+\+MAX\+\_\+\+BS} -\/ sets the maximum block size for the Jacobi preconditioner (if used, otherwise, it does nothing) in the solvers benchmark. The default is \textquotesingle{}32\textquotesingle{}.
\item {\ttfamily SOLVERS\+\_\+\+GMRES\+\_\+\+RESTART} -\/ the maximum dimension of the Krylov space to use in GMRES. The default is {\ttfamily 100}. 
\end{DoxyItemize}