Benchmarking

Use this guidance for performance-sensitive changes and benchmark reviews. Performance claims need a benchmark that ships in the same merge request; correctness tests still ship separately and run before timing.

Projects and Builds

The supported and contrib benchmark trees are separate, standalone CMake projects. They are not part of Eigen's main test build and both require Google Benchmark:

cmake -G Ninja -S benchmarks -B build-bench -DCMAKE_BUILD_TYPE=Release
cmake --build build-bench --target <benchmark-target>

cmake -G Ninja -S contrib/benchmarks -B build-contrib-bench -DCMAKE_BUILD_TYPE=Release
cmake --build build-contrib-bench --target <benchmark-target>

The contrib parent project automatically adds its GPU subtree when it detects CUDAToolkit. That configuration also requires a working CUDA compiler and architecture selection. On a host with only a partial toolkit installation, configure CPU-only contrib benchmarks with -DCMAKE_DISABLE_FIND_PACKAGE_CUDAToolkit=TRUE or report the GPU configuration as unavailable.

Consult benchmarks/CMakeLists.txt and contrib/benchmarks/CMakeLists.txt for current targets and compile settings. CUDA benchmarks also have a standalone project and instructions in contrib/benchmarks/GPU/CMakeLists.txt. No CI job builds or runs benchmarks, so the pipeline validates neither a benchmark's own compilation nor a performance claim: build and run both locally, and report the measurement conditions this guide requires.

Adding A Benchmark

One family per translation unit, bench_<topic>.cpp under the module directory (benchmarks/LU/bench_lu.cpp), registered beside it with eigen_add_benchmark(<target> <source> [LIBRARIES ...] [DEFINITIONS ...]), which links benchmark_main, compiles at -O3 with NDEBUG, and takes the include path from the tree. Do not merge families into one file: code-layout shifts between combined and separate binaries have shown up as deltas of tens of percent in kernels that did not change. Multi-threaded benchmarks call UseRealTime() on the registration to measure elapsed time. The default CPU timer measures the main thread and omits internally spawned workers; MeasureProcessCPUTime() includes those workers when total CPU consumption is also needed. See Google Benchmark's CPU timers.

Benchmark Design

  • Benchmark the user-visible operation affected by the change, with representative scalar types, sizes, shapes, storage layouts, sparsity, and thread counts. Include transition sizes where a kernel or blocking strategy changes.
  • Confirm the registered arguments actually reach the changed code — a size that falls off the fast path, or no case at all for the affected configuration, measures something else while looking green. Check hand-written bytes_per_second/items multipliers against the operation; a miscount silently rescales every reported rate.
  • Keep allocation, input generation, validation, and unrelated setup outside the timed region. Prevent dead-code elimination with Google Benchmark's DoNotOptimize and ClobberMemory where appropriate.
  • Validate results outside the measured loop. A faster incorrect kernel is not a useful result.
  • Use enough work per iteration to dominate timer noise without hiding important small-problem behavior. Report meaningful rates or byte/operation counters when they improve interpretation.
  • Compare the change against the relevant baseline with identical compiler, optimization, ISA, dependency, and benchmark arguments. Record the commit, hardware, compiler, flags, and command needed to reproduce the result.

Argument Grids

Express static grids declaratively on the registration:

  • Args({a, b}) for individual points.
  • Range, DenseRange, or Ranges for swept dimensions.
  • ArgsProduct({{...}, {...}}) for Cartesian products.

Do not use Apply(). Its callback is typed on benchmark::internal::Benchmark*, a library-internal name that benchmark sources must not reference. A grid that appears to need it is expressible by enumerating the points in ArgsProduct or Args, or by registering several benchmarks.

Running Measurements

  1. Check uptime and stop or finish competing builds and compute-heavy work. Run only one benchmark process at a time; concurrent benchmarks invalidate both measurements.
  2. Keep the machine, CPU affinity, power/governor policy, thermal state, compiler, flags, ISA, and dependencies as constant as practical. Disclose anything that could not be controlled.
  3. Use multiple repetitions, for example --benchmark_repetitions=10, and retain raw results. Compare medians plus a dispersion measure such as MAD, IQR, or standard deviation; do not select the best run.
  4. For before/after binaries, alternate separate invocations (A, B, A, B) to expose thermal or background-load drift. Use the same benchmark filter and arguments for each pair.
  5. Re-run suspicious or noisy cases. Treat changes smaller than the observed run-to-run variation as inconclusive, not as wins or regressions.

When the machine cannot be made quiet enough for the effect size, deterministic counters are the honest measurement: callgrind instruction counts, allocation counts (e.g. -Wl,--wrap=malloc), with identical result checksums across both variants. Report them as counter measurements naming the tool, not as timings; that plus a statement that wall clock was inconclusive is a complete performance claim, where an unqualified ratio from a loaded host is not.

Never infer a general speedup from one convenient size or one warm run. State the tested domain, include regressions as well as improvements, and keep numerical accuracy results separate from performance measurements.