blob: 0148bcb1f68be1b584e72029ae58255facaaa780 [file] [view]
# CI Internals
Use this guide when changing [`.gitlab-ci.yml`](../.gitlab-ci.yml), the job definitions under [`ci/`](../ci), the
test selector [`scripts/affected_tests.py`](../scripts/affected_tests.py), the pass cache
[`ci/scripts/test_cache.py`](../ci/scripts/test_cache.py), or the clang-tidy driver. [`ci.md`](ci.md) is the
consumer's view of the same machinery; the checked-out files are authoritative where the two disagree.
Both selector scripts, `affected_tests.py` and `test_cache.py`, fail closed: an error or an unknown case either fails
the job or widens the selection, never silently narrows it. A wrong answer, however, goes unnoticed: a job that skips
too many tests still reports success. Their unit tests are therefore blocking and run on every merge request in
`checkformat:lint`:
```bash
python3 scripts/test_affected_tests.py
python3 ci/scripts/test_test_cache.py
```
## Artifacts And Test Reports
Build jobs publish the configured build directory as an artifact. Their paired test jobs consume that artifact and
run CTest without rebuilding. When changing either side, keep the test job's `needs`, CTest label or filter, and the
corresponding build target consistent; otherwise CTest can discover tests whose executables are absent.
Publishing is opt-in per job rather than inherited: the bases `.common:linux:cross` and `.common:windows` have no
`artifacts:` key. To make a job publish, add `.artifacts:linux:builddir`, `.artifacts:windows:builddir` or
`.artifacts:test:results` as a second `extends:` parent. Give a test job the results template, not a build-directory
template. A test job links nothing, and the build directory it downloaded is already the build job's artifact. The
results template also registers `JUnitTestResults_*.xml` through `artifacts:reports:junit:`. That registration puts
failures in the job's Tests tab and in the merge request widget, not only in the log. A job that needs neither, such as
`test:linux:buildsystem`, extends the base alone and publishes nothing.
A job that failed and then passed on retry still reports the failed first attempt. The report is converted from the
dashboard file `Test.xml`, and the retry runs `--rerun-failed` without `-T test`, so it never rewrites that file. The
job exits 42 to mark the soft failure. The merge request widget can compare results only against a base-branch report
from a job with the same name. Default-branch pushes run only a small subset of jobs, so most jobs show a summary
without a comparison.
## The Pass Cache
In merge request pipelines, the Linux test jobs keep a content-addressed pass cache in `.testcache/`, with one GitLab
cache per job name. [`test.linux.script.sh`](../ci/scripts/test.linux.script.sh) skips a test when an earlier merge
request pipeline recorded a pass on the first attempt for the same executable, emulator, CTest definition, and
environment fingerprint. The fingerprint covers the image, the `lib*` package state, `ci/scripts/` and `ci/docker/`,
and variables that change behavior, such as `EIGEN_REPEAT` and `QEMU_CPU`. After the run, the script calls
[`test_cache.py`](../ci/scripts/test_cache.py) to record the tests that passed on the first attempt. It reads their
statuses from the dashboard run's `Test.xml`.
Scheduled and web pipelines always run every test their job selects. Each of their runs draws fresh random seeds from
the clock, and those seeds are part of what they test. Sharded jobs never skip, and `EIGEN_CI_TEST_CACHE: "off"` opts a
job out. Skipped tests are absent from that run's JUnit report.
The fingerprint hashes `ci/scripts/` and `ci/docker/` but not the `ci/*.gitlab-ci.yml` files. Every YAML setting that
can change a test's outcome already enters the key by value:
- job variables through `KEYED_ENV_PREFIXES`;
- the image through `CI_JOB_IMAGE`;
- compiler flags and the cross emulator through the digests of the files in the test's command;
- CTest timeouts through the properties hash.
If the key included the YAML files, any edit to them would discard all recorded passes. Leaving them out has two
consequences:
- **When you add a job variable that can change a test's outcome, add it to `KEYED_ENV_PREFIXES`.** Setting it in the
YAML alone does not put it in the key.
- When you move a job to a runner pool whose CPU differs, you should also clear the cache, because the fingerprint does
not see a job's `tags:`. The key cannot tell two hosts in the same tag pool apart either.
## Tier Rules
The `affected-tests` and `all-tests` labels each turn off the smoke jobs (`.rules:libeigen:smoketest`). Both tiers test
more deeply than the fixed smoke list, on the same native runners, so the smoke jobs would add cost but no coverage. The
smoke jobs are turned off only in the `libeigen` namespace, because the affected and full tiers have no jobs in a fork.
In merge request pipelines, `.rules:libeigen:nvhpc` runs the NVHPC build and test pair only when the `nvhpc-tests`
label is set. When the pair was part of the `all-tests` matrix, its compiler frontend used roughly a quarter of the
project's hosted-runner minutes.
Under `affected-tests`, either of two independent triggers adds a platform beyond the four unconditional jobs: a change
under the platform's backend directory, detected through `rules:changes:`, or a label, read from
`$CI_MERGE_REQUEST_LABELS`. GitLab ANDs `if:` with `changes:` within one rule entry, so each rule set has two entries,
one per trigger. Each rule set tests the whole label string on its own, so several labels select the union of their
platforms. The `*-tests` labels are **unscoped** so that a merge request can carry several of them. GitLab makes scoped
labels (`backend::NEON`) mutually exclusive, so it would allow only one.
The rule sets for `arch/SVE` and `arch/SME` also list `Eigen/src/Core/util/ConfigureVectorization.h` in `changes:`,
because that header decides whether either backend is compiled at all. SVE runs one build per vector length. Eigen takes
`EIGEN_ARM64_SVE_VL` from `__ARM_FEATURE_SVE_BITS`, and only `-msve-vector-bits` sets that macro.
`test/sve_vector_length` reads `RDVL`, so a binary run at another width fails instead of computing wrong answers.
`all-platforms` leaves out three rows of the platform table in `ci.md` because their jobs ignore the selection (the list
of affected tests that `select:tests` computes). The AVX512-FP16 pair and the SME build are compile-only, with no paired
test job, and the GPU jobs build `buildtests_gpu`. SME gets compile coverage rather than the selection. Its test jobs,
one per streaming vector length (SVL), already run a curated subset of targets through `EIGEN_CI_CTEST_REGEX`, and the
selection would conflict with that subset. The GPU row adds jobs from outside the tier through the `affected-tests`
entry in `.rules:libeigen:gpu`. `gpu-tests` already triggers those jobs on its own, so it combines with `affected-tests`
without a second rule entry. When adding a runner for a backend that has none (ZVector, MSA, HVX, HIP, SYCL), add its
trigger to these rule sets too.
## The Selector
`select:tests` writes `affected/targets.txt` and `affected/ctest_regex.txt`. The paired build and test jobs on Linux and
Windows read them through `EIGEN_CI_BUILD_TARGET_FILE` and `EIGEN_CI_CTEST_REGEX_FILE`. On Windows, the readers are
[`build.windows.script.ps1`](../ci/scripts/build.windows.script.ps1) and
[`test.windows.script.ps1`](../ci/scripts/test.windows.script.ps1).
The selector follows the textual `#include` graph and ignores preprocessor guards, so it selects a strict superset of
the tests that actually compile a changed file. Changes to CMake, `ci/scripts/`, `ci/docker/`, or the BLAS/LAPACK shims
force the full suite, because they invalidate the mapping itself. The `ci/*.gitlab-ci.yml` files only orchestrate jobs
and cannot change which test includes which header, so they select nothing. Neither do the clang-tidy and lint images
under `ci/tidy/` and `ci/lint/`. Git rename detection is disabled for the input diff, so the selector sees both the old
and the new path of a moved file. An old path absent from the current graph safely forces the full suite.
The selector maps sources to targets by reading the test registrations in the CMake files. The registrations include
executables built from several translation units, and the GPU tests. GPU test sources are `.cu`, because `ei_add_test`
takes the extension from `EIGEN_ADD_TEST_FILENAME_EXTENSION`. The selector treats a changed test source without a
registration as an error; it does not drop the source as an unconfigured target. The selector skips
`test/buildsystem/`. The consumer projects there are separate CMake projects, and only `test:linux:buildsystem`
configures them. An `add_executable` in them is not a test registration, and no test in the main project depends on
their sources.
Some targets need an optional dependency such as CHOLMOD, CUDA or SYCL and are absent from configurations without it.
The build script filters the selection against `ninja -t targets` after the CMake configure, because ninja aborts on an
unknown target. A selection of only such targets builds nothing and is not a failure. A missing selection artifact,
however, must fail the job rather than fall back to the default target, which would silently build everything. A job
reads a `NONE` selection before the toolchain setup and the configure step, so a merge request that affects no test
costs a checkout rather than a full configure. `rules:` cannot keep such a job out of the pipeline, because GitLab
evaluates them when it creates the pipeline, before `select:tests` runs. Only a child pipeline generated from the
selection could skip the job entirely.
The build script shuffles the remaining targets and builds them in batches. Before that, it expands each target through
ninja's phony edges into the targets it aggregates. Most selected names are aggregates: `buildtests`, and the parent
target of every split test, which builds all of that test's parts. Batching limits memory pressure by spreading the
targets it is given across batches. An unexpanded parent target would put a whole test family into one batch and
defeat that limit.
Two kinds of registration need special handling. First, `buildtests` aggregates only the `ei_add_test` targets, so in
full-suite mode the selector names a bare `add_executable`, such as the `bug1213` link regression, explicitly next to
`buildtests`. Second, the compile-failure suite under `failtest/` is `EXCLUDE_FROM_ALL` and is compiled at test time, so
the selector picks those tests by their CTest names, `<name>_ok` and `<name>_ko`, and never hands them to the build job.
Both matter because a `-R` filter silently drops whatever it does not name. The unfiltered runs in the other tiers
include them automatically.
The `buildfailtests` fixture compiles the failtests. Every `_ok` and `_ko` test requires that fixture, so CTest adds it
to any run that selects one of them, even if `-E` excludes it. The fixture builds every failtest target in one
`cmake --build` that keeps going after errors. Separate builds, one per test, would have to run serially: concurrent
builds in one binary directory collide whenever a CMake regeneration is pending. On the hosted runners, that serial
suite took 80-95% of an affected test job's wall time.
The tests themselves only check which executables exist. `_ok` passes when its own executable exists. `_ko` passes when
its own executable is missing and its `_ok` twin's exists. A missing compiler therefore fails both tests instead of
making `_ko` pass. When a job excludes the failtests by name from an "ALL" selection, it must exclude
`^buildfailtests$` as well, because an "ALL" selection applies no `-R`.
The RISC-V affected tier runs the `failtest` label on an amd64 job with the original cross compiler. Its native
runtime job excludes those compile tests and the nested `buildsystem` scenarios: the runtime image has neither Ninja
nor a compiler, and the cached compiler paths name amd64 executables. The separate `test:linux:buildsystem` job covers
the nested consumers when build-system files change.
## Clang-Tidy Compilation Database
For a source in the compilation database, the driver first narrows the database with
[`tidy_compile_db.py`](../scripts/tidy_compile_db.py). A split test has one database entry per `EIGEN_TEST_PART`, or per
range of parts that `cmake/EigenTestPartGroups.cmake` compiles together. clang-tidy parses the file once for each entry
that names it: 14 times for `test/array_cwise.cpp`, which alone uses up the job's timeout. The narrowed database keeps
one entry per distinct compiler configuration. Within a configuration that is split into parts, it keeps the entries
whose parts actually compile the added lines. A line inside a `CALL_SUBTEST_<n>(...)` call or an
`#if defined(EIGEN_TEST_PART_<n>)` guard needs part `<n>`; any other line can be checked under any part. The driver
prints what it left out next to the file name, so a run that hits the cap names the parts it did not check instead of
reporting the file clean.