Benchmarking

Use this guidance for performance-sensitive changes and benchmark reviews. Performance claims need a benchmark that ships in the same merge request; correctness tests still ship separately and run before timing.

Projects and Builds

The supported and unsupported benchmark trees are separate, standalone CMake projects. They are not part of Eigen's main test build and both require Google Benchmark:

cmake -G Ninja -S benchmarks -B build-bench -DCMAKE_BUILD_TYPE=Release
cmake --build build-bench --target <benchmark-target>

cmake -G Ninja -S unsupported/benchmarks -B build-unsupported-bench -DCMAKE_BUILD_TYPE=Release
cmake --build build-unsupported-bench --target <benchmark-target>

The unsupported parent project automatically adds its GPU subtree when it detects CUDAToolkit. That configuration also requires a working CUDA compiler and architecture selection. On a host with only a partial toolkit installation, configure CPU-only unsupported benchmarks with -DCMAKE_DISABLE_FIND_PACKAGE_CUDAToolkit=TRUE or report the GPU configuration as unavailable.

Consult benchmarks/CMakeLists.txt and unsupported/benchmarks/CMakeLists.txt for current targets and compile settings. CUDA benchmarks also have a standalone project and instructions in unsupported/benchmarks/GPU/CMakeLists.txt. The CI scripts build.benchmark.sh and run.benchmark.sh describe the supported-tree scheduled build and result format; do not assume they validate unsupported/benchmarks changes.

Benchmark Design

  • Benchmark the user-visible operation affected by the change, with representative scalar types, sizes, shapes, storage layouts, sparsity, and thread counts. Include transition sizes where a kernel or blocking strategy changes.
  • Confirm the registered arguments actually reach the changed code — a size that falls off the fast path, or no case at all for the affected configuration, measures something else while looking green. Check hand-written bytes_per_second/items multipliers against the operation; a miscount silently rescales every reported rate.
  • Keep allocation, input generation, validation, and unrelated setup outside the timed region. Prevent dead-code elimination with Google Benchmark's DoNotOptimize and ClobberMemory where appropriate.
  • Validate results outside the measured loop. A faster incorrect kernel is not a useful result.
  • Use enough work per iteration to dominate timer noise without hiding important small-problem behavior. Report meaningful rates or byte/operation counters when they improve interpretation.
  • Compare the change against the relevant baseline with identical compiler, optimization, ISA, dependency, and benchmark arguments. Record the commit, hardware, compiler, flags, and command needed to reproduce the result.

Argument Grids

Express static grids declaratively on the registration:

  • Args({a, b}) for individual points.
  • Range, DenseRange, or Ranges for swept dimensions.
  • ArgsProduct({{...}, {...}}) for Cartesian products.

Do not use Apply(). Its callback is typed on benchmark::internal::Benchmark*, a library-internal name that benchmark sources must not reference. A grid that appears to need it is expressible by enumerating the points in ArgsProduct or Args, or by registering several benchmarks.

Running Measurements

  1. Check uptime and stop or finish competing builds and compute-heavy work. Run only one benchmark process at a time; concurrent benchmarks invalidate both measurements.
  2. Keep the machine, CPU affinity, power/governor policy, thermal state, compiler, flags, ISA, and dependencies as constant as practical. Disclose anything that could not be controlled.
  3. Use multiple repetitions, for example --benchmark_repetitions=10, and retain raw results. Compare medians plus a dispersion measure such as MAD, IQR, or standard deviation; do not select the best run.
  4. For before/after binaries, alternate separate invocations (A, B, A, B) to expose thermal or background-load drift. Use the same benchmark filter and arguments for each pair.
  5. Re-run suspicious or noisy cases. Treat changes smaller than the observed run-to-run variation as inconclusive, not as wins or regressions.

When the machine cannot be made quiet enough for the effect size, deterministic counters are the honest measurement: callgrind instruction counts, allocation counts (e.g. -Wl,--wrap=malloc), with identical result checksums across both variants. Report them as counter measurements naming the tool, not as timings; that plus a statement that wall clock was inconclusive is a complete performance claim, where an unqualified ratio from a loaded host is not.

Never infer a general speedup from one convenient size or one warm run. State the tested domain, include regressions as well as improvements, and keep numerical accuracy results separate from performance measurements.