doc/develop/test/ci_test_plan.rst

.. _ci_test_plan:

CI Test Plan Selector (test_plan_v2)
#####################################

The CI test plan selector is a modular Python script located at
``scripts/ci/test_plan_v2.py``.  Its purpose is to analyse the set of files
changed in a Pull Request and emit a targeted twister test plan — exercising
only the tests that are plausibly affected by the change while avoiding a
full tree-wide run on every commit.

The script writes two artefacts:

* ``testplan.json`` — passed to twister via ``twister --load-tests``.
* ``.testplan`` — a plain ``KEY=value`` environment file consumed by CI
  orchestration scripts, containing ``TWISTER_TESTS``, ``TWISTER_NODES``, and
  ``TWISTER_FULL``.

Architecture
************

The selector is built around a *strategy pipeline*.  Each strategy is an
independent analyser that:

1. Receives the list of changed files (or the subset not yet consumed by
   earlier strategies).
2. Inspects those files according to its own logic.
3. Returns a list of :class:`TwisterCall` descriptors and the set of files it
   has *handled*.

The :class:`Orchestrator` drives the pipeline, merges all results into a
:class:`PlanAccumulator`, deduplicates test suites, and writes the output
files.

Strategy ordering
*****************

Strategies run in a fixed order that reflects two principles:

**Specificity over generality.**
The most precise strategies run first.  A file that lives inside
``tests/kernel/sched/`` is handled definitively by :class:`DirectTestStrategy`
(run exactly those tests) before the catch-all :class:`MaintainerAreaStrategy`
can add an entire Kernel area sweep.

**Consume-before-additive.**
Consuming strategies run before additive ones.  Once a consuming strategy
claims a file, downstream strategies never see it.  This prevents a change
to a board file from also triggering a driver compat scan and a Kconfig sweep
for the same path.

The current order is:

.. list-table::
   :header-rows: 1
   :widths: 5 20 10 65

   * - #
     - Strategy
     - Consumes
     - Rationale
   * - 0
     - :class:`ComplexityStrategy`
     - No
     - Scores the patchset using pydriller and lizard.  Must run first so the
       score is available to :class:`RiskClassifierStrategy`.  Emits no twister
       calls.
   * - 1
     - :class:`IgnoreStrategy`
     - Yes
     - Silently drops files that can never affect tests (documentation, CI
       workflows, tooling).  Runs early so later strategies never waste effort
       on ignored paths.
   * - 1b
     - :class:`BoilerplateFilter`
     - Yes
     - Consumes files whose entire diff consists of whitespace adjustments,
       blank-line changes, SPDX licence identifiers, copyright notices, or
       bare comment delimiters.  Such changes cannot affect runtime behaviour,
       so removing them early keeps the pool clean for substantive strategies.
       Requires a git commit range; no-op otherwise.
   * - 2
     - :class:`DirectTestStrategy`
     - Yes
     - A changed test or sample file must trigger *only* that test, not an
       entire area-wide sweep.  Consuming this file prevents
       :class:`MaintainerAreaStrategy` from also adding every test in the
       Kernel area when ``tests/kernel/sched/main.c`` is modified.
   * - 3
     - :class:`SnippetStrategy`
     - Yes
     - Snippet changes are self-contained: only tests that declare the snippet
       in ``required_snippets`` need to run.  Consuming prevents downstream
       strategies from treating a snippet YAML as an unknown configuration
       file.
   * - 4
     - :class:`BoardStrategy`
     - Yes
     - Board-specific changes require a targeted integration test on every
       board variant.  Consuming prevents a board ``.yaml`` file from also
       matching Kconfig and Header strategies.
   * - 5
     - :class:`SoCStrategy`
     - Yes
     - SoC-level changes affect every board built on that SoC family.
       Consuming prevents the same ``soc/`` path from reaching the Kconfig
       sweep.
   * - 6
     - :class:`ManifestStrategy`
     - Yes
     - A changed ``west.yml`` requires module-tagged integration tests, not a
       generic maintainer-area run.  Consuming the manifest file prevents the
       catch-all from adding unrelated tests.
   * - 7
     - :class:`DriverCompatStrategy`
     - No
     - Additive: driver files may also be covered by area patterns, so the
       file is not consumed and :class:`KconfigImpactStrategy` and
       :class:`MaintainerAreaStrategy` can add further coverage.
   * - 8
     - :class:`DtsBindingStrategy`
     - No
     - Additive: binding changes combine overlay-based test selection with
       board-targeted area calls.
   * - 9
     - :class:`KconfigImpactStrategy`
     - No
     - Additive: Kconfig changes may affect many unrelated files; only consumes
       when a non-widespread symbol is found, leaving widespread-symbol files
       for the catch-all.
   * - 10
     - :class:`HeaderImpactStrategy`
     - No
     - Additive: header changes trace include users back to maintainer areas.
       Widespread headers (included in more than the configured threshold) are
       skipped.
   * - 11
     - :class:`MaintainerAreaStrategy`
     - No
     - Catch-all: matches any remaining file against ``MAINTAINERS.yml`` area
       patterns and emits ``--test-pattern`` calls for each matching area that
       has a non-empty ``tests:`` list.

Boilerplate filter
******************

:class:`BoilerplateFilter` runs immediately after :class:`IgnoreStrategy` and
before any test-selection strategy.  It inspects the actual diff of each
changed file (via ``git diff``) and consumes those whose entire content change
is non-substantive:

* **Whitespace and blank-line changes** — detected by ``git diff -w
  --ignore-blank-lines``: if that command produces no output for a file,
  every changed line is whitespace or blank.

* **SPDX and copyright header edits** — lines that begin with
  ``SPDX-License-Identifier:``, ``SPDX-FileCopyrightText:``, or the word
  ``Copyright``.

* **Comment-delimiter-only lines** — lines whose non-whitespace content
  consists entirely of ``/*``, ``*/``, ``//``, ``#``, or ``*`` characters
  (e.g. reformatted block-comment borders).

A file is consumed only when **every** added or removed line in its diff falls
into one of those categories.  A single substantive line (a changed statement,
macro, or declaration) causes the file to pass through to downstream
strategies unchanged.

The filter is a no-op when ``--commits`` is not supplied (e.g. when using
``--modified-files``), because the diff cannot be computed without a commit
range.  In that case all files remain in the pool.

Consume vs. additive behaviour
*******************************

Every strategy subclass carries a class attribute ``consumes: bool``.

When ``consumes = True``:
  Files returned in the *handled* set are removed from the ``remaining`` pool
  before the next strategy runs.  Use this when the strategy is *authoritative*
  for its file type — i.e. when seeing the file in a later strategy would
  produce redundant or incorrect results.

When ``consumes = False`` (default):
  Files remain in the pool regardless of what the strategy returns.  All
  downstream strategies receive the same file list.  Use this for additive
  strategies that contribute *additional* test coverage without claiming
  exclusive ownership.

.. note::

   The :class:`Orchestrator` skips a strategy entirely when the ``remaining``
   pool is empty.  Because non-consuming strategies see all files, "remaining"
   only shrinks through consuming strategies.  As soon as the pool is empty
   the orchestrator stops calling strategies.

Full-run and fallback conditions
**********************************

The orchestrator signals ``TWISTER_FULL=True`` in ``.testplan`` when either of
the following is true:

1. **Unresolved files.** After all strategies have run, at least one changed
   file was not *handled* by any strategy.  Rather than silently skipping
   coverage for an unknown path, the script falls back to requesting a full
   run so no regression slips through.

2. **Explicit full-run signal.** A strategy returns a :class:`TwisterCall`
   with ``full_run=True``.  When the orchestrator encounters such a call, it
   immediately sets ``TWISTER_FULL=True``, empties the remaining pool, and
   stops executing further calls.  This mechanism allows a strategy to opt out
   of targeted selection (e.g. when a core subsystem header used by the entire
   tree is modified).

When ``TWISTER_FULL=True`` the CI script is expected to discard
``testplan.json`` and run twister without ``--load-tests``.

Node count calculation
**********************

``TWISTER_NODES`` in ``.testplan`` is computed as follows:

* ``0`` — no tests were selected.
* ``1`` — fewer than ``--tests-per-builder`` tests were selected (fits in one
  builder).
* ``ceil(total / tests_per_builder)`` — otherwise.  The ceiling ensures that
  no builder is over-capacity when the division is not exact.

The default value of ``--tests-per-builder`` is ``900``.

Adding a new strategy
*********************

1. **Subclass** :class:`SelectionStrategy` and implement the two abstract
   members:

   .. code-block:: python

      class MyStrategy(SelectionStrategy):

          consumes: bool = False  # or True if authoritative

          @property
          def name(self):
              return "MyStrategy"

          def analyze(self, changed_files):
              # inspect changed_files
              calls = [...]       # list of TwisterCall
              handled = {...}     # subset of changed_files this strategy owns
              return calls, handled

2. **Decide consume vs. additive.**  Ask: "If a downstream strategy also sees
   this file, will it produce useful additional coverage, or redundant/wrong
   results?"  If redundant/wrong: ``consumes = True``.

3. **Insert in the correct position** in :func:`build_strategies`.  As a rule
   of thumb:

   * Consuming strategies belong before all additive strategies.
   * More-specific strategies belong before less-specific ones.
   * Strategies with no side-effects on the file pool (additive) can be
     ordered by cost — cheapest first.

4. **Guard optional dependencies** with a late import inside ``analyze`` and
   return ``[], set()`` gracefully when the dependency is missing.  Use the
   ``# noqa: PLC0415`` comment to silence the late-import warning.

5. **Write unit tests** in ``scripts/tests/ci/test_test_plan_v2.py`` covering:

   * The happy path (files correctly identified and routed).
   * The no-op path (irrelevant files produce no calls and no handled set).
   * Edge cases (missing files on disk, malformed YAML, empty inputs).

Shared pipeline context
***********************

:class:`PipelineContext` is a dataclass threaded through all strategies via
:func:`build_strategies`.  It currently carries:

* ``complexity_score`` — aggregate patchset score written by
  :class:`ComplexityStrategy` and read by :class:`RiskClassifierStrategy`.
* ``file_metrics`` — per-file :class:`ComplexityMetrics` mappings for
  detailed logging and downstream decision-making.

New cross-strategy state should be added as typed fields on
:class:`PipelineContext` rather than as strategy-level instance variables.

Test strategy for validating selection behaviour
*************************************************

The test suite lives in ``scripts/tests/ci/test_test_plan_v2.py`` and is run
with pytest:

.. code-block:: bash

   pytest scripts/tests/ci/test_test_plan_v2.py -v

Tests are organised into one class per strategy or shared component.  The
following categories of tests are expected for each new strategy:

**Unit tests (no Zephyr tree required)**
  Use a ``tmp_path`` fixture to create the minimum filesystem structures
  (``board.yml``, ``snippet.yml``, ``testcase.yaml``, etc.) needed to exercise
  each code path.  These tests should be fast and hermetic.

**Helper method tests**
  Test internal methods (path-walking, YAML parsing, regex extraction) in
  isolation.  Pass synthetic content rather than relying on real files in the
  repository.

**Orchestrator integration tests**
  Use mock strategies and a mock :class:`TwisterExecutor` to test the
  orchestrator's consume-vs-additive logic, full-run signalling, and
  ``.testplan`` output without invoking twister.

**Real-tree integration tests** (optional, ``@pytest.mark.integration``)
  May be added and marked with ``@pytest.mark.integration`` to be excluded
  from the default run.  These are only meaningful on a complete Zephyr
  checkout.

Minimal test checklist for a new strategy
==========================================

* The strategy returns no calls and no handled files when no relevant files
  are in the input list.
* The strategy correctly identifies and returns the expected :class:`TwisterCall`
  for a known-good input.
* The strategy does not crash when required files are absent from the filesystem.
* For consuming strategies: the handled set equals the matched files and
  nothing else.
* For additive strategies: the handled set is empty or equals only the files
  the strategy is authoritative for.

CLI reference
*************

.. code-block:: text

   usage: test_plan_v2.py [-c A..B] [-m FILE] [-f PATH]
                          [-o FILE] [-p PLATFORM] [--maintainers-file FILE]
                          [-T DIR] [--quarantine-list FILE]
                          [--tests-per-builder N] [--disable-strategy NAME]
                          [--detailed-test-id]

   -c A..B              Git commit range (e.g. ``main..HEAD``).  Changed files
                        are derived from ``git diff --name-only A..B``.
   -m FILE              JSON file containing a list of changed file paths.
   -f PATH              Treat PATH as a changed file (repeatable).
   -o FILE              Output JSON file (default: ``testplan.json``).
   -p PLATFORM          Restrict all selections to this platform (repeatable).
   --maintainers-file   Path to ``MAINTAINERS.yml``.
   -T DIR               Extra testsuite root forwarded to every twister call.
   --quarantine-list    Quarantine YAML forwarded to twister.
   --tests-per-builder  Tests per CI builder node (default: 900).
   --disable-strategy   Skip a strategy by name (repeatable).
   --detailed-test-id   Pass ``--detailed-test-id`` to twister.