doc/develop/twister/index.rst

.. _twister_script:

Test Runner (Twister)
#####################

Twister scans for the set of test applications in the git repository
and attempts to execute them. By default, it tries to build each test
application on boards marked as default in the board definition file.

The default options will build the majority of the test applications on a
defined set of boards and will run in an emulated environment if available for
the architecture or configuration being tested.

Because of the limited test execution coverage, twister
cannot guarantee local changes will succeed in the full build
environment, but it does sufficient testing by building samples and
tests for different boards and different configurations to help keep the
complete code tree buildable.

When using (at least) one ``-v`` option, twister's console output
shows for every test application how the test is run (qemu, native_sim, etc.) or
whether the binary was just built. The resultant
:ref:`status <twister_statuses>`
of a test is likewise reported in the ``twister.json`` and other report files.
There are a few reasons why twister only builds a test and doesn't run it:

- The test is marked as ``build_only: true`` in its ``.yaml``
  configuration file.
- The test configuration has defined a ``harness`` but you don't have
  it or haven't set it up.
- The target device is not connected and not available for flashing
- You or some higher level automation invoked twister with
  ``--build-only``.

To run Twister in the local tree, follow the steps below:

.. code-block:: console

   $ west twister

.. note::

   The examples in this document invoke Twister as ``west twister``, the
   :ref:`west <west>` extension command, which works the same on all host
   operating systems. The following invocations are equivalent:

   * ``west twister ...`` (recommended).
   * ``python .\scripts\twister ...`` (Windows): invoking the script
     directly. This requires the Zephyr environment to be set up first (``source
     zephyr-env.sh`` or ``zephyr-env.cmd``).

   All forms accept the same command line options.

If you want to run tests on one or more specific platforms, you can use
the ``--platform`` option, it is a platform filter for testing, with this
option, test suites will only be built/run on the platforms specified.
This option also supports different revisions of one same board,
you can use ``--platform board@revision`` to test on a specific revision.

The list of command line options supported by twister can be viewed with
``west twister --help``. See :ref:`twister_commandline_options` for the full
set of options.

The following pages cover additional Twister topics:

.. toctree::
   :maxdepth: 1

   commandline
   pytest
   twister_statuses
   twister_blackbox


Board Configuration
*******************

To build tests for a specific board and to execute some of the tests on real
hardware or in an emulation environment such as QEMU a board configuration file
is required which is generic enough to be used for other tasks that require a
board inventory with details about the board and its configuration that is only
available during build time otherwise.

The board metadata file is located in the board directory and is structured
using the YAML markup language. The example below shows a board with a data
required for best test coverage for this specific board:

.. code-block:: yaml

  identifier: frdm_k64f
  name: NXP FRDM-K64F
  type: mcu
  arch: arm
  toolchain:
    - zephyr
    - gnuarmemb
  supported:
    - arduino_gpio
    - arduino_i2c
    - netif:eth
    - adc
    - i2c
    - nvs
    - spi
    - gpio
    - usb_device
    - watchdog
    - can
    - pwm
  testing:
    default: true


identifier:
  A string that matches how the board is defined in the build system. This same
  string is used when building, for example when calling ``west build`` or
  ``cmake``:

  .. code-block:: console

     # with west
     west build -b reel_board
     # with cmake
     cmake -DBOARD=reel_board ..

name:
  The actual name of the board as it appears in marketing material.
vendor:
  The board vendor. Used by the ``vendor_allow`` and ``vendor_exclude`` test
  scenario filters.
tier:
  An optional integer indicating the board support tier. Used for reporting and
  to group platforms by their level of support.
type:
  Type of the board or configuration. One of ``mcu``, ``qemu``, ``sim``,
  ``unit`` or ``native``.
simulation:
  Simulator(s) used to simulate the platform, e.g. qemu.

  .. code-block:: yaml

      simulation:
        - name: qemu
        - name: armfvp
          exec: FVP_Some_Platform
        - name: custom
          exec: AnotherBinary

  By default, tests will be executed using the first entry in the simulation array. Another
  simulation can be selected with ``--simulation <simulation_name>``.
  The ``exec`` attribute is optional. If it is set but the required simulator is not available, the
  tests will be built only.
  If it is not set and the required simulator is not available the tests will fail to run.
  The simulation name must match one of the elements of ``SUPPORTED_EMU_PLATFORMS``.
arch:
  Architecture of the board
toolchain:
  The list of supported toolchains that can build this board. This should match
  one of the values used for :envvar:`ZEPHYR_TOOLCHAIN_VARIANT` when building on the command line.
  Twister filters out any test instance whose toolchain is not in this list, unless
  ``--force-toolchain`` is given. This list says which toolchains *may* build the
  board, it does not select one; see :ref:`twister_toolchain_selection`.
preferred_toolchain:
  The toolchain Twister should use for this platform when nothing else selects one.
  This is useful for boards that are nominally buildable with several toolchains but
  should be tested with a specific one. See :ref:`twister_toolchain_selection`.
build_toolchains:
  An optional list of toolchains that every test assigned to this platform should
  be built with. Twister creates one test instance per toolchain in the list, each
  in its own build directory, instead of picking a single toolchain for the
  platform. For example, to build everything on ``native_sim`` with both GCC and
  Clang:

  .. code-block:: yaml

      build_toolchains:
        - host/gnu
        - host/llvm

  Because this multiplies build time, it is usually better to leave it out of the
  board definition and enable it only for CI, using the ``build_toolchains``
  option of the :ref:`Twister configuration file <twister_test_config>`.
  See :ref:`twister_toolchain_selection`.
ram:
  Available RAM on the board (specified in KB). This is used to match test scenario
  requirements.  If not specified we default to 128KB.
flash:
  Available FLASH on the board (specified in KB). This is used to match test scenario
  requirements.  If not specified we default to 512KB.
sysbuild: [True|False] (default False)
  If true, applications for this platform are built using
  :ref:`sysbuild <sysbuild>` by default.
twister: [True|False] (default True)
  If false, Twister ignores this platform entirely and never builds or runs
  tests on it.
supported:
  A list of features this board supports. This can be specified as a single word
  feature or as a variant of a feature class. For example:

  .. code-block:: yaml

        supported:
          - pci

  This indicates the board does support PCI. You can make a test scenario build or
  run only on such boards, or:

  .. code-block:: yaml

        supported:
          - netif:eth
          - sensor:bmi16

  A test scenario can depend on 'eth' to only test ethernet or on 'netif' to run
  on any board with a networking interface.

testing:
  testing relating keywords to provide best coverage for the features of this
  board.

  .. _twister_default_testing_board:

  binaries:
    A list of custom binaries to be kept for device testing.
  default: [True|False]:
    This is a default board, it will tested with the highest priority and is
    covered when invoking the simplified twister without any additional
    arguments.
  ignore_tags:
    Do not attempt to build (and therefore run) tests marked with this list of
    tags.
  only_tags:
    Only execute tests with this list of tags on a specific platform.
  timeout_multiplier: <float> (default 1)
    .. _twister_board_timeout_multiplier:

    Multiply each test scenario timeout by specified ratio. This option allows to tune timeouts only
    for required platform. It can be useful in case naturally slow platform I.e.: HW board with
    power-efficient but slow CPU or simulation platform which can perform instruction accurate
    simulation but does it slowly.

  flash_before: [True|False] (default False)
    For pytest/shell harness hardware testing, flash the device before opening the serial port.
    This prevents serial port disconnection issues during flashing on some boards (e.g., those
    with USB CDC that reset during flash operations).

  renode:
    Configuration for the Renode simulator. Supports two keys: ``uart``, the
    UART peripheral the harness connects to (e.g. ``sysbus.uart0``), and
    ``resc``, the Renode script (``.resc``) used to set up the simulated machine.

env:
  A list of environment variables. Twister will check if all these environment variables are set,
  and otherwise skip this platform. This allows the user to define a platform which should be
  used, for example, only if some required software or hardware is present, and to signal that
  presence to twister using these environment variables.

variants:
  A mapping of board variant (qualifier) names to per-variant overrides. Each
  entry is itself a platform definition and may override any of the keys above
  for that specific variant, while inheriting the remaining values from the
  top-level definition.

.. _twister_toolchain_selection:

Toolchain Selection
*******************

Several options influence which toolchain a test is built with. They fall into
three groups: options that *select* a toolchain, options that *filter* out test
instances whose toolchain is not usable, and options that *multiply* a test into
several builds.

Twister first determines a default toolchain for the whole run by invoking
``cmake/verify-toolchain.cmake``, which honors the :envvar:`ZEPHYR_TOOLCHAIN_VARIANT`
environment variable. This value is reported at the start of the run as
``Using '<toolchain>' toolchain variant.``

For every test scenario and platform pair, the toolchain is then selected using
the first of the following that applies:

#. The test scenario's ``integration_toolchains``, if set. The test is built once
   per listed toolchain.
#. The platform's ``build_toolchains``, if set, either in the board configuration
   or in the :ref:`Twister configuration file <twister_test_config>`. The test is
   built once per listed toolchain.
#. For ``posix`` and ``unit`` platforms, ``host/llvm`` if the run default is
   ``host/llvm``, otherwise ``host/gnu``.
#. The platform's ``preferred_toolchain``, if set.
#. The run default described above, or ``zephyr`` if it could not be determined.

Note that :envvar:`ZEPHYR_TOOLCHAIN_VARIANT` only changes the run default, which
is the last entry in this list. It does not override a platform's
``preferred_toolchain`` or ``build_toolchains``, nor a scenario's
``integration_toolchains``.

Once a toolchain is selected, the resulting test instance can still be filtered
out:

* If the toolchain is not in the platform's ``toolchain`` list of supported
  toolchains, the instance is filtered. ``--force-toolchain`` disables this check
  and uses the selected toolchain unconditionally. The comparison also succeeds
  on the part before the ``/``, so ``host/gnu`` matches a platform listing ``host``.
* The scenario's ``toolchain_allow`` and ``toolchain_exclude`` options filter
  instances by the selected toolchain.

Because ``integration_toolchains`` and ``build_toolchains`` produce one test
instance per toolchain, they multiply build time. ``build_toolchains`` is
therefore normally left out of the board configuration and enabled only for the
configuration file used by CI.

.. _twister_tests_long_version:

Tests
*****

Tests are detected by the presence of a ``tests.yaml`` (``sample.yaml`` and
``testcase.yaml`` support is deprecated) files in the application's project
directory.  This test application configuration file may contain one or more
entries in the ``tests:`` section each identifying a Test Scenario.

.. _twister_test_project_diagram:

.. figure:: figures/twister_test_project.svg
   :alt: Twister and a Test application project.
   :figclass: align-center

   Twister and a Test application project.


Test application configurations are written using the YAML syntax and share the
same structure as samples.

A Test Scenario is a set of conditions and variables defined in a Test Scenario
entry, under which a set of Test Suites will be built and executed.

A Test Suite is a collection of Test Cases which are intended to be used to test
a software program to ensure it meets certain requirements. The Test Cases in a
Test Suite are either related or meant to be executed together.

Test Scenario, Test Suite, and Test Case names must follow to these basic rules:

#. The format of the Test Scenario identifier shall be a string without any spaces or
   special characters (allowed characters: alphanumeric and [\_=]) consisting
   of multiple sections delimited with a dot (``.``).

#. Each Test Scenario identifier shall start with a section name followed by a
   subsection names delimited with a dot (``.``). For example, a test scenario
   that covers semaphores in the kernel shall start with ``kernel.semaphore``.

#. All Test Scenario names must be unique for the Twister execution scope.

#. The full canonical name of a Test Suite is:
   ``<Test Application Project path>/<Test Scenario identifier>``

#. Depending on the Test Suite implementation, its Test Case identifiers consist
   of **at least three sections** delimited with a dot (``.``):

   * **Ztest tests**:
     a Test Scenario identifier from the corresponding ``testcase.yaml`` file,
     a Ztest suite name, and a Ztest test name:
     ``<Test Scenario identifier>.<Ztest suite name>.<Ztest test name>``

   * **Standalone tests and samples**:
     a Test Scenario identifier from the corresponding ``tests.yaml`` file where
     the last section signifies the standalone
     Test Case name, for example: ``debug.coredump.logging_backend``.


The following is an example test configuration with a few options that are
explained in this document.


  .. code-block:: yaml

        tests:
          bluetooth.gatt:
            build_only: true
            platform_allow:
              - qemu_cortex_m3
              - qemu_x86
            tags:
              - bluetooth
          bluetooth.gatt.br:
            build_only: true
            extra_args:
              -CONF_FILE="prj_br.conf"
            filter: not CONFIG_DEBUG
            platform_exclude:
              -up_squared
            platform_allow:
              - qemu_cortex_m3 qemu_x86
            tags:
              bluetooth


A sample with tests will have the same structure with additional information
related to the sample and what is being demonstrated:

  .. code-block:: yaml

        sample:
          name: hello world
          description: Hello World sample, the simplest Zephyr application
        tests:
          sample.basic.hello_world:
            build_only: true
            tags:
              - tests
            min_ram: 16
          sample.basic.hello_world.singlethread:
            build_only: true
            extra_args: CONF_FILE=prj_single.conf
            filter: not CONFIG_BT
            tags:
              - tests
            min_ram: 16

A Test Scenario entry in the ``tests:`` YAML dictionary has its Test Scenario
identifier as a key.

Each Test Scenario entry in the Test Application configuration can define the
following key/value pairs:

..  _test_config_args:

tags: <list of tags> (required)
    A set of string tags for the test scenario. Usually pertains to
    functional domains but can be anything. Command line invocations
    of this script can filter the set of tests to run based on tag.

skip: <True|False> (default False)
    skip test scenario unconditionally. This can be used for broken tests for
    example.

slow: <True|False> (default False)
    Don't run this test scenario unless ``--enable-slow`` or ``--enable-slow-only`` was
    passed in on the command line. Intended for time-consuming test scenarios that
    are only run under certain circumstances, like daily builds. These test
    scenarios are still compiled.

extra_args: <list of extra arguments>
    Extra arguments to pass to build tool when building or running the
    test scenario.

    Using namespacing, it is possible to apply extra_args only to some
    hardware. Currently architectures/platforms/simulation are supported:

    .. code-block:: yaml

        common:
          tags: drivers adc
        tests:
          test:
            depends_on: adc
          test_async:
            extra_args:
              - arch:x86:CONFIG_ADC_ASYNC=y
              - platform:qemu_x86:CONFIG_DEBUG=y
              - platform:mimxrt1060_evk:SHIELD=rk043fn66hs_ctg
              - simulation:qemu:CONFIG_MPU=y

extra_configs: <list of extra configurations>
    Extra configuration options to be merged with a main prj.conf
    when building or running the test scenario. For example:

    .. code-block:: yaml

        common:
          tags: drivers adc
        tests:
          test:
            depends_on: adc
          test_async:
            extra_configs:
              - CONFIG_ADC_ASYNC=y

    Using namespacing, it is possible to apply a configuration only to some
    hardware. Currently both architectures and platforms are supported:

    .. code-block:: yaml

        common:
          tags: drivers adc
        tests:
          test:
            depends_on: adc
          test_async:
            extra_configs:
              - arch:x86:CONFIG_ADC_ASYNC=y
              - platform:qemu_x86:CONFIG_DEBUG=y


extra_conf_files: <list of configuration files>
    Extra Kconfig fragment files to merge into the build, as an alternative to
    passing ``CONF_FILE=`` through ``extra_args``. Entries from ``common`` and
    the test scenario are concatenated. Prefer this field over ``extra_args``
    for configuration files.

extra_overlay_confs: <list of overlay configuration files>
    Extra Kconfig overlay fragments to merge into the build, as an alternative
    to passing ``OVERLAY_CONFIG=`` through ``extra_args``. Entries from
    ``common`` and the test scenario are concatenated.

extra_dtc_overlay_files: <list of devicetree overlay files>
    Extra devicetree overlay files to apply to the build, as an alternative to
    passing ``DTC_OVERLAY_FILE=`` through ``extra_args``. Entries from
    ``common`` and the test scenario are concatenated.

build_only: <True|False> (default False)
    If true, twister will not try to run the test even if the test is runnable
    on the platform.

    This keyword is reserved for tests that are used to test if some code
    actually builds. A ``build_only`` test is not designed to be run in any
    environment and should not be testing any functionality, it only verifies
    that the code builds.

    This option is often used to test drivers and the fact that they are correctly
    enabled in Zephyr and that the code builds, for example sensor drivers. Such
    test shall not be used to verify the functionality of the driver.

build_on_all: <True|False> (default False)
    If true, attempt to build test scenario on all available platforms. This is mostly
    used in CI for increased coverage. Do not use this flag in new tests.

depends_on: <list of features>
    A board or platform can announce what features it supports, this option
    will enable the test only those platforms that provide this feature.

levels: <list of levels>
    Test levels this test should be part of. If a level is present, this
    test will be selectable using the command line option ``--level <level name>``

min_ram: <integer>
    estimated minimum amount of RAM in KB needed for this test to build and run. This is
    compared with information provided by the board metadata.

min_flash: <integer>
    estimated minimum amount of ROM in KB needed for this test to build and run. This is
    compared with information provided by the board metadata.

.. _twister_test_case_timeout:

timeout: <number of seconds>
    Length of time to run test before automatically killing it.
    Default to 60 seconds.

arch_allow: <list of arches, such as x86, arm, arc>
    Set of architectures that this test scenario should only be run for.

arch_exclude: <list of arches, such as x86, arm, arc>
    Set of architectures that this test scenario should not run on.

toolchain_allow: <list of toolchain variants>
    Set of toolchain variants that this test scenario should only be run for.
    The toolchain is the one configured for the run (see
    :envvar:`ZEPHYR_TOOLCHAIN_VARIANT`). Platforms built with any other
    toolchain are filtered out.

toolchain_exclude: <list of toolchain variants>
    Set of toolchain variants that this test scenario should not be run for.

vendor_allow: <list of vendors>
    Set of platform vendors that this test scenario should only be run for.  The
    vendor is defined as part of the board definition. Boards associated with
    this vendors will be included. Other boards, including those without a
    vendor will be excluded.

vendor_exclude: <list of vendors>
    Set of platform vendors that this test scenario should not run on.
    The vendor is defined as part of the board. Boards associated with this
    vendors will be excluded.

platform_allow: <list of platforms>
    Set of platforms that this test scenario should only be run for. Do not use
    this option to limit testing or building in CI due to time or resource
    constraints, this option should only be used if the test or sample can
    only be run on the allowed platform and nothing else.

integration_platforms: <YML list of platforms/boards>
    This option limits the scope to the listed platforms when twister is
    invoked with the ``--integration`` option. Use this instead of
    platform_allow if the goal is to limit scope due to timing or
    resource constraints.

integration_toolchains: <YML list of toolchain variants>
    This option expands the scope to all the listed toolchains variants and
    adds another vector of testing where desired. By default, test
    configurations are generated based on the toolchain configured in the environment:

    test scenario -> platforms1 -> toolchain1
    test scenario -> platforms2 -> toolchain1


    When a platform supports multiple toolchains that are available during the
    twister run, it is possible to expand the test configurations to include
    additional tests for each toolchain. For example, if a platform supports
    toolchains ``toolchain1`` and ``toolchain2``, and the test scenario
    includes:

    .. code-block:: yaml

      integration_toolchains:
        - toolchain1
        - toolchain2

    the following configurations are generated:

    test scenario -> platforms1 -> toolchain1
    test scenario -> platforms1 -> toolchain2
    test scenario -> platforms2 -> toolchain1
    test scenario -> platforms2 -> toolchain2


    .. note::

      This functionality is evaluated always and is not limited to the
      ``--integration`` option.

    This option takes precedence over a platform's ``build_toolchains``. To expand
    the toolchain scope for every test on a platform instead of per test scenario,
    use ``build_toolchains``. See :ref:`twister_toolchain_selection`.

platform_exclude: <list of platforms>
    Set of platforms that this test scenario should not run on.

platform_type: <list of platform types>
    Restrict this test scenario to platforms of the given type(s). A platform's
    type is declared via the ``type:`` key in its board metadata. Supported
    values are ``mcu``, ``qemu``, ``sim``, ``unit`` and ``native``. Platforms
    whose type is not in this list are filtered out.

simulation_exclude: <list of simulators>
    Set of simulators that this test scenario should not run on. Supported
    values are ``qemu``, ``simics``, ``xt-sim``, ``renode``, ``nsim``,
    ``mdb-nsim``, ``tsim``, ``armfvp``, ``native`` and ``custom``.

extra_sections: <list of extra binary sections>
    When computing sizes, twister will report errors if it finds
    extra, unexpected sections in the Zephyr binary unless they are named
    here. They will not be included in the size calculation.

sysbuild: <True|False> (default False)
    Build the project using sysbuild infrastructure. Only the main project's
    generated devicetree and Kconfig will be used for filtering tests.
    on device testing must use the hardware map, or west flash to load
    the images onto the target. The ``--erase`` option of west flash is
    not supported with this option. Usage of unsupported options will result
    in tests requiring sysbuild support being skipped.

harness: <string>
    A harness keyword in the ``testcase.yaml`` file identifies a Twister
    harness needed to run a test successfully. A harness is a feature of
    Twister and implemented by Twister, some harnesses are defined as
    placeholders and have no implementation yet.

    A harness can be seen as the handler that needs to be implemented in
    Twister to be able to evaluate if a test passes criteria. For example, a
    keyboard harness is set on tests that require keyboard interaction to reach
    verdict on whether a test has passed or failed, however, Twister lack this
    harness implementation at the moment.

    Supported harnesses:

    - ztest
    - test
    - console
    - pytest
    - gtest
    - robot
    - ctest
    - shell
    - power
    - display_capture
    - script
    - bsim

    See :ref:`twister_harnesses` for more information.

platform_key: <list of platform attributes>
    Often a test needs to only be built and run once to qualify as passing.
    Imagine a library of code that depends on the platform architecture where
    passing the test on a single platform for each arch is enough to qualify the
    tests and code as passing. The platform_key attribute enables doing just
    that.

    For example to key on (arch, simulation) to ensure a test is run once
    per arch and simulation (as would be most common):

    .. code-block:: yaml

      platform_key:
        - arch
        - simulation

    Adding platform (board) attributes to include things such as soc name,
    soc family, and perhaps sets of IP blocks implementing each peripheral
    interface would enable other interesting uses. For example, this could enable
    building and running SPI tests once for each unique IP block.

harness_config: <harness configuration options>
    Extra harness configuration options to be used to select a board and/or
    for handling generic Console with regex matching. Config can announce
    what features it supports. This option will enable the test to run on
    only those platforms that fulfill this external dependency.


    fixture: <string or list>
        Specify a test scenario dependency on an external device(e.g., sensor),
        and identify setups that fulfill this dependency. It depends on
        specific test setup and board selection logic to pick the particular
        board(s) out of multiple boards that fulfill the dependency in an
        automation setup based on ``fixture`` keyword. Some sample fixture names
        are i2c_hts221, i2c_bme280, i2c_FRAM, ble_fw and gpio_loop.

    ztest_suite_repeat: <int> (default 1)
        This parameter specifies the number of times the entire test suite should be repeated.

    ztest_test_repeat: <int> (default 1)
        This parameter specifies the number of times each individual test within the test suite
        should be repeated.

    ztest_test_shuffle: <True|False> (default False)
        This parameter indicates whether the order of the tests within the test suite should
        be shuffled. When set to ``true``, the tests will be executed in a random order.



    The following is an example yaml file with robot harness_config options.

    .. code-block:: yaml

        tests:
          robot.example:
            harness: robot
            harness_config:
              robot_testsuite: [robot file path]

    It can be more than one test suite using a list.

    .. code-block:: yaml

        tests:
          robot.example:
            harness: robot
            harness_config:
              robot_testsuite:
                - [robot file path 1]
                - [robot file path 2]
                - [robot file path n]

    One or more options can be passed to robotframework.

    .. code-block:: yaml

        tests:
          robot.example:
            harness: robot
            harness_config:
              robot_testsuite: [robot file path]
              robot_option:
                - --exclude tag
                - --stop-on-error

filter: <expression>
    Filter whether the test scenario should be run by evaluating an expression
    against an environment containing the following values:

    .. code-block:: none

            { ARCH : <architecture>,
              PLATFORM : <platform>,
              <all CONFIG_* key/value pairs in the test's generated defconfig>,
              *<env>: any environment variable available
            }

    Twister will first evaluate the expression to find if a "limited" cmake call, i.e. using package_helper cmake script,
    can be done.

    Existence of "dt_*" entries indicates devicetree is needed. Refer to :ref:`twister_dt_filter_expressions`
    for detailed description of the different DT expressions available.

    Existence of "CONFIG*" entries indicates kconfig is needed.
    If there are no other types of entries in the expression a filtration can be done without creating a complete build system.
    If there are entries of other types a full cmake is required.

    The grammar for the expression language is as follows:

    .. code-block:: antlr

        expression : expression 'and' expression
                   | expression 'or' expression
                   | 'not' expression
                   | '(' expression ')'
                   | symbol '==' constant
                   | symbol '!=' constant
                   | symbol '<' NUMBER
                   | symbol '>' NUMBER
                   | symbol '>=' NUMBER
                   | symbol '<=' NUMBER
                   | symbol 'in' list
                   | symbol ':' STRING
                   | symbol
                   ;

        list : '[' list_contents ']';

        list_contents : constant (',' constant)*;

        constant : NUMBER | STRING;

    For the case where ``expression ::= symbol``, it evaluates to ``true``
    if the symbol is defined to a non-empty string.

    Operator precedence, starting from lowest to highest:

       * or (left associative)
       * and (left associative)
       * not (right associative)
       * all comparison operators (non-associative)

    ``arch_allow``, ``arch_exclude``, ``platform_allow``, ``platform_exclude``
    are all syntactic sugar for these expressions. For instance:

    .. code-block:: none

        arch_exclude = x86 arc

    Is the same as:

    .. code-block:: none

        filter = not ARCH in ["x86", "arc"]

    The ``:`` operator compiles the string argument as a regular expression,
    and then returns a true value only if the symbol's value in the environment
    matches. For example, if ``CONFIG_SOC="stm32f107xc"`` then

    .. code-block:: none

        filter = CONFIG_SOC : "stm.*"

    Would match it.

required_snippets: <list of needed snippets>
    :ref:`Snippets <snippets>` are supported in twister for test scenarios that
    require them. As with normal applications, twister supports using the base
    zephyr snippet directory and test application directory for finding
    snippets. Listed snippets will filter supported tests for boards (snippets
    must be compatible with a board for the test to run on them, they are not
    optional).

    The following is an example yaml file with 2 required snippets.

    .. code-block:: yaml

        tests:
          snippet.example:
            required_snippets:
              - cdc-acm-console
              - user-snippet-example

.. _required_applications:

required_applications: <list of required applications> (default empty)
    Specify a list of test applications that must be built before current test can run.
    It enables sharing of built applications between test scenarios, allowing tests
    to access build artifacts from other applications.

    Each required application entry supports:

    - ``application``: Test scenario identifier (required)
    - ``name``: Deprecated alias for ``application`` (still accepted for backward
      compatibility, but ``application`` should be used in new configurations)
    - ``platform``: Target platform (optional, defaults to current test's platform)
    - ``path``: Directory path where Twister should search for the application
      (optional). Can be an absolute path or a path relative to the directory
      containing the test's YAML file. Environment variables and Zephyr module
      directory variables are expanded (see :ref:`twister_module_dir_vars`).
      If not specified, Twister searches in the same directory as the referring
      test's YAML file.

    Required applications are automatically discovered and built by Twister.
    If a required application is not already loaded, Twister searches for it
    in the directory specified by ``path`` or, if ``path`` is not set, in the
    same directory as the referring test's YAML file.
    When reusing build directories (e.g., with ``--no-clean``),
    Twister can find required applications in the current build directory.

    How it works:

    - Twister builds the required applications first
    - The main test application waits for required applications to complete
    - Build directories of required applications are made available to the test harness
    - For pytest harness, build directories are passed via ``--required-build`` arguments
      and accessible through the ``required_build_dirs`` fixture

    When combined with ``build: false``, the current test scenario
    skips its own build step entirely and uses the first required
    application's build artifacts as its image. This is useful for
    scenarios that serve purely as test harnesses for an image built
    elsewhere.

    Example configuration:

    .. code-block:: yaml

        tests:
          # Requires two applications, second one from a different path and with a fixed platform
          sample.required_app_demo:
            harness: pytest
            required_applications:
              - application: sample.shared_app
              - application: other.app
                path: ../other_app
                platform: native_sim
          # No self build, use the first required application as the test image
          sample.no_self_build:
            build: false
            harness: pytest
            required_applications:
              - application: sample.basic.helloworld
                path: $ZEPHYR_BASE/samples/hello_world
          sample.shared_app:
            build_only: true

    Limitations:

    - Not supported with ``--runtime-artifact-cleanup``, as build artifacts of
      required applications must be retained for use by the main test application.
    - Not supported with ``--subset``: a required application and the test
      depending on it may be assigned to different subsets, making build
      artifacts unavailable at test execution time.

build: <True|False> (default True)
    If false, the test scenario skips its own build step and uses the build
    artifacts from the first entry in ``required_applications`` as its image.
    This is useful for scenarios that serve purely as a test harness for an
    image built by another scenario.

    Constraints:

    - ``required_applications`` must be non-empty.
    - Supported harnesses: pytest-based (e.g. ``pytest``, ``shell``) and ``bsim``.
    - QEMU platforms are not supported.

expect_reboot: <True|False> (default False)
    Notify twister that the test scenario is expected to reboot while executing.
    When enabled, twister will suppress warnings about unexpected multiple runs
    of a testsuite or testcase.

modules: <list of module names>
    Build and run this test scenario only when all of the listed
    :ref:`modules <modules>` are present in the workspace. Scenarios that
    require a module which is not available are filtered out.

type: <string> (default integration)
    Test type of the scenario. Set to ``unit`` for unit tests that are built
    for the :ref:`unit_testing board <unit_testing_board>` and run on the host
    without the full Zephyr build system.

testcases: <list of test case names>
    Explicitly declare the list of test case names that make up this scenario.
    This is normally detected automatically (for example from the ztest source)
    and only needs to be set for harnesses that cannot be introspected.

ignore_faults: <True|False> (default False)
    Do not mark the test scenario as failed if a fault is detected in the
    output while the test is running.

ignore_qemu_crash: <True|False> (default False)
    Do not mark the test scenario as failed if QEMU crashes while the test is
    running.

The set of test scenarios that actually run depends on directives in the test scenario
file and options passed in on the command line. If there is any confusion,
running with ``-v`` or examining the :ref:`test plan <twister_output>`
(:file:`testplan.json`) can help show why particular test scenarios were
filtered out.

To load arguments from a file, add ``+`` before the file name, e.g.,
``+file_name``. File content must be one or more valid arguments separated by
line break instead of white spaces.

Most everyday users will run with no arguments.

.. _twister_module_dir_vars:

Expanding paths with module directory variables
===============================================

Path options in the test scenario file (e.g. ``required_applications``,
``harness_config: pytest_root``) are expanded before use. In addition to
environment variables, Twister expands Zephyr module directory
variables, which mirror the CMake variables defined for every module:

* ``ZEPHYR_<MODULE>_MODULE_DIR`` - absolute path to the module's root.
* ``ZEPHYR_<MODULE>_MODULE_NAME`` - the module's name.

``<MODULE>`` is upper-cased with non-alphanumeric characters replaced by ``_``,
exactly as CMake does (for example ``$ZEPHYR_HAL_NORDIC_MODULE_DIR`` for the
``hal_nordic`` module). Unknown references are left unchanged.

Managing tests timeouts
=======================

There are several parameters which control tests timeouts on various levels:

* ``timeout`` option in each test scenario. See :ref:`here <twister_test_case_timeout>` for more
  details.
* ``timeout_multiplier`` option in board configuration. See
  :ref:`here <twister_board_timeout_multiplier>` for more details.
* ``--timeout-multiplier`` twister option which can be used to adjust timeouts in exact twister run.
  It can be useful in case of simulation platform as simulation time may depend on the host
  speed & load or we may select different simulation method (i.e. cycle accurate but slower
  one), etc...

Overall test scenario timeout is a multiplication of these three parameters.

.. _twister_dt_filter_expressions:

Devicetree Filtering Expressions
================================

Expressions starting with "dt_*" are used to filter boards based on specific
devicetree properties, such as compatibles, aliases, node labels, node
properties, chosen nodes, etc. when selecting test scenarios.

.. note::

   The source code for these expressions can be found at
   :zephyr_file:`scripts/pylib/twister/expr_parser.py`.

Expressions
-----------

``dt_compat_enabled(compat)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if any DT node with the specified compatible string (``compat``) is enabled.

**Parameters:**
   - ``compat``: The compatible string to match.

``dt_alias_exists(alias)``
~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if any DT node with the specified alias exists and is enabled.

**Parameters:**
   - ``alias``: The alias (defined in ``aliases`` node) to match.

``dt_enabled_alias_with_parent_compat(alias, compat)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if the DT has an enabled alias node whose parent has the specified compatible string.
   Useful for nodes like ``gpio-leds`` child nodes, which may not have their own compatible.

**Parameters:**
   - ``alias``: The alias (defined in ``aliases`` node) to match.
   - ``compat``: The parent node’s compatible string to match.

``dt_label_with_parent_compat_enabled(label, compat)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if a DT node with the specified label exists, is enabled, and its parent has the
   specified compatible string.

**Parameters:**
   - ``label``: The node label to match.
   - ``compat``: The parent node’s compatible string to match.

``dt_label_compat_enabled(label, compat)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if a DT node with the specified label exists, is enabled, and has the
   specified compatible string.

**Parameters:**
   - ``label``: The node label to match.
   - ``compat``: The node compatible string to match.

``dt_chosen_enabled(chosen)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if a DT chosen property with the specified name exists and the node assigned to it
   is enabled.

**Parameters:**
   - ``chosen``: The name of the chosen property.

``dt_nodelabel_enabled(label)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if a DT node with the specified label exists and is enabled.

**Parameters:**
   - ``label``: The node label to match.

``dt_nodelabel_prop_enabled(label, prop)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if a DT node with the specified label exists, is enabled, and has the specified property
   with a non-empty value.

**Parameters:**
   - ``label``: The node label to match.
   - ``prop``: The node's property to check.

``dt_node_has_prop(node_id, prop)``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

**Purpose:**
   Checks if a DT node (specified by alias or path) has the specified property, regardless of its
   status. Useful for nodes that do not have a status, like ``zephyr,user`` node.

**Parameters:**
   - ``node_id``: The node alias (defined in ``aliases`` node) or node path to match.
   - ``prop``: The node's property to check.

Usage
-----

These expressions are used in Twister’s test scenarios filtering logic to select boards that match
specific DT conditions. For example:

.. code-block:: yaml

   tests:
     - test: my_test
       filter: dt_compat_enabled("my-compat-string")

The test scenario ``my_test`` will only build for boards where a DT node with ``my-compat-string``
is enabled.

.. _twister_harnesses:

Harnesses
*********

A *harness* is the mechanism Twister uses to run a test and decide whether it
passed or failed. After a test image is built and started on its target (real
hardware, an emulator, or the host), the harness drives the interaction with
the running image -- providing input, capturing output, or handing execution
over to an external test runner -- and interprets the result to assign a
:ref:`status <twister_statuses>` to each test case.

A test scenario selects a harness with the ``harness:`` entry in its
``tests.yaml`` and tunes its behavior through ``harness_config``. When no
harness is specified, the default ``test`` harness is used. Different harnesses
serve different needs: some parse the device's console output against expected
patterns, while others delegate execution to an external framework such as
pytest, Robot Framework, or ctest. The pages linked below describe each
supported harness and its ``harness_config`` options.

Harnesses ``ztest``, ``gtest`` and ``console`` are based on parsing of the
output and matching certain phrases. ``ztest`` and ``gtest`` harnesses look
for pass/fail/etc. frames defined in those frameworks.

Some widely used harnesses that are not supported yet:

- keyboard
- net
- bluetooth

The following is an example yaml file with a few harness_config options.

.. code-block:: yaml

      sample:
        name: HTS221 Temperature and Humidity Monitor
      common:
        tags: sensor
        harness: console
        harness_config:
          type: multi_line
          ordered: false
          regex:
            - "Temperature:(.*)C"
            - "Relative Humidity:(.*)%"
          fixture: i2c_hts221
      tests:
        test:
          tags: sensors
          depends_on: i2c

.. toctree::
   :maxdepth: 1

   harness/ctest
   harness/gtest
   harness/pytest
   harness/console
   harness/robot
   harness/power
   harness/display_capture
   harness/script
   harness/bsim
   harness/shell


.. _twister_sidecars:

Sidecars
********

Some tests need a host-side resource to exist for the duration of a run: a
daemon the emulated guest talks to, a shared memory region the host reads back
afterwards, or a network interface the guest attaches to. A *sidecar* models
this. It is selected with the ``sidecar:`` entry in a test scenario's
:file:`tests.yaml` and is orthogonal to the harness: the
harness interprets the guest's output while the sidecar provisions the host side
around the run. Any harness (``console`` for a sample, ``ztest`` for a test,
...) can therefore be paired with any sidecar.

.. code-block:: yaml

   tests:
     some.test:
       harness: ztest
       sidecar: <name>

A sidecar has a small lifecycle, driven by Twister for each test instance:

#. **configure** -- the sidecar reads what it needs from the instance and its
   ``sidecar_config`` block before anything is provisioned.
#. **setup** -- called just before the handler runs the test image; it brings
   the host resource up (starts a daemon, creates an interface, ...). If the
   host side is unavailable -- a required tool is not installed, or bringing the
   resource up needs privileges that are not present -- setup reports this and
   Twister *skips* execution instead of failing the test.
#. **teardown** -- called after the handler returns, in a ``finally`` block, so
   it always runs even if the test failed or timed out. It releases the resource
   and may also collect data the guest left behind (for example reading a shared
   memory region back into the build directory).

Because provisioning is decoupled from output processing, Twister can also
attach a sidecar to an instance itself, without the test opting in -- for
example to route coverage data off a guest that has no other host transport.

Each sidecar defines its own configuration keys under a block of
``sidecar_config`` named after the sidecar. Namespacing by sidecar name keeps
each sidecar's keys separate, so only the block matching the scenario's
``sidecar:`` value is consumed. For example, the ``virtiofs`` sidecar shares a
host directory seeded from a template with:

.. code-block:: yaml

   tests:
     some.test:
       harness: console
       sidecar: virtiofs
       sidecar_config:
         virtiofs:
           shared: shared


Selecting platform scope
************************

One of the key features of Twister is its ability to decide on which platforms a given
test scenario should run. This behavior has its roots in Twister being developed as
a test runner for Zephyr's CI. With hundreds of available platforms and thousands of
tests, the testing tools should be able to adapt the scope and select/filter out what
is relevant and what is not.

Twister always prepares an initial list of platforms in scope for a given test,
based on command line arguments and the :ref:`test's configuration <test_config_args>`. Then,
platforms that don't fulfill the conditions required in the configuration yaml
(e.g. minimum ram) are filtered out from the scope.
Using ``--force-platform`` allows to override filtering caused by ``platform_allow``,
``platform_exclude``, ``arch_allow`` and ``arch_exclude`` keys in test configuration
files.

Command line arguments define the initial scope in the following way:

* ``-p/--platform <platform_name>`` (can be used multiple times): only platforms
  passed with this argument;
* ``-l/--all``: all available platforms;
* ``-G/--integration``: all platforms from an ``integration_platforms`` list in
  a given test configuration file. If a test has no ``integration_platforms``
  *"scope presumption"* will happen;
* No scope argument: *"scope presumption"* will happen.

*"Scope presumption"*: A list of Twister's :ref:`default platforms <twister_default_testing_board>`
is used as the initial list. If nothing is left after the filtration, the ``platform_allow`` list
is used as the initial scope.

Running in Integration Mode
***************************

This mode is used in continuous integration (CI) and other automated
environments used to give developers fast feedback on changes. The mode can
be activated using the ``--integration`` option of twister and narrows down
the scope of builds and tests if applicable to platforms defined under the
integration keyword in the test configuration file (``tests.yaml``).


Running tests on custom emulator
********************************

Apart from the already supported QEMU and other simulated environments, Twister
supports running any out-of-tree custom emulator defined in the board's :file:`board.cmake`.
To use this type of simulation, add the following properties to
:file:`custom_board/custom_board.yaml`:

.. code-block:: yaml

   simulation:
     - name: custom
       exec: <name_of_emu_binary>

This tells Twister that the board is using a custom emulator called ``<name_of_emu_binary>``,
make sure this binary exists in the PATH.

Then, in :file:`custom_board/board.cmake`, set the supported emulation platforms to ``custom``:

.. code-block:: cmake

   set(SUPPORTED_EMU_PLATFORMS custom)

Finally, implement the ``run_custom`` target in :file:`custom_board/board.cmake`.
It should look something like this:

.. code-block:: cmake

   add_custom_target(run_custom
     COMMAND
     <name_of_emu_binary to invoke during 'run'>
     <any args to be passed to the command, i.e. ${BOARD}, ${APPLICATION_BINARY_DIR}/zephyr/zephyr.elf>
     WORKING_DIRECTORY ${APPLICATION_BINARY_DIR}
     DEPENDS ${logical_target_for_zephyr_elf}
     USES_TERMINAL
     )


Running Tests in Random Order
*****************************
Enable ZTEST framework's :kconfig:option:`CONFIG_ZTEST_SHUFFLE` config option to
run your tests in random order.  This can be beneficial for identifying
dependencies between test cases. For native_sim platforms, you can provide
the seed to the random number generator by providing ``--seed=value`` as an
argument to twister. See :ref:`Shuffling Test Sequence <ztest_shuffle>` for more
details.


Running Tests on Hardware
*************************

Beside being able to run tests in QEMU and other simulated environments,
twister supports running most of the tests on real devices and produces
reports for each run with detailed FAIL/PASS results.


Executing tests on a single device
==================================

To use this feature on a single connected device, run twister with
the following new options:

.. tabs::

   .. group-tab:: Linux

      .. code-block:: bash

	      west twister --device-testing --device-serial /dev/ttyACM0 \
	      --device-serial-baud 115200 -p frdm_k64f  -T tests/kernel

   .. group-tab:: Windows

      .. code-block:: bat

	      west twister --device-testing --device-serial COM1 \
	      --device-serial-baud 115200 -p frdm_k64f  -T tests/kernel

The ``--device-serial`` option denotes the serial device the board is connected to.
This needs to be accessible by the user running twister. You can run this on
only one board at a time, specified using the ``--platform`` option.
If the platform supports multiple serial ports, you can provide ``--device-serial``
multiple times, and it will be passed to the pytest harness. Alternatively you can use
the hardware map, see :ref:`multi-core testing <twister_multi_core_testing>` for more details

The ``--device-serial-baud`` option is only needed if your device does not run at
115200 baud.

To support devices without a physical serial port, use the ``--device-serial-pty``
option. In this cases, log messages are captured for example using a script.
In this case you can run twister with the following options:

.. tabs::

   .. group-tab:: Linux

      .. code-block:: bash

         west twister --device-testing --device-serial-pty "script.py" \
         -p intel_adsp/cavs25 -T tests/kernel

   .. group-tab:: Windows

      .. note::

         Not supported on Windows OS

The script is user-defined and handles delivering the messages which can be
used by twister to determine the test execution status.

The ``--device-flash-timeout`` option allows to set explicit timeout on the
device flash operation, for example when device flashing takes significantly
large time.

The ``--device-flash-with-test`` option indicates that on the platform
the flash operation also executes a test scenario, so the flash timeout is
increased by a test scenario timeout.

Executing tests on multiple devices
===================================

To build and execute tests on multiple devices connected to the host PC, a
hardware map needs to be created with all connected devices and their
details such as the serial device, baud and their IDs if available.
Run the following command to produce the hardware map:

.. code-block:: console

   $ west twister --generate-hardware-map map.yml

The generated hardware map file (map.yml) will have the list of connected
devices, for example:

.. tabs::

   .. group-tab:: Linux

      .. code-block:: yaml

         - connected: true
           id: OSHW000032254e4500128002ab98002784d1000097969900
           platform: unknown
           product: DAPLink CMSIS-DAP
           runner: pyocd
           serial: /dev/cu.usbmodem146114202
         - connected: true
           id: 000683759358
           platform: unknown
           product: J-Link
           runner: unknown
           serial: /dev/cu.usbmodem0006837593581

   .. group-tab:: Windows

      .. code-block:: yaml

         - connected: true
           id: OSHW000032254e4500128002ab98002784d1000097969900
           platform: unknown
           product: unknown
           runner: unknown
           serial: COM1
         - connected: true
           id: 000683759358
           platform: unknown
           product: unknown
           runner: unknown
           serial: COM2


Any options marked as ``unknown`` need to be changed and set with the correct
values, in the above example the platform names, the products and the runners need
to be replaced with the correct values corresponding to the connected hardware.
In this example we are using a reel_board and an nrf52840dk/nrf52840:

.. tabs::

   .. group-tab:: Linux

      .. code-block:: yaml

         - connected: true
           id: OSHW000032254e4500128002ab98002784d1000097969900
           platform: reel_board
           product: DAPLink CMSIS-DAP
           runner: pyocd
           serial: /dev/cu.usbmodem146114202
           baud: 9600
         - connected: true
           id: 000683759358
           platform: nrf52840dk/nrf52840
           product: J-Link
           runner: nrfjprog
           serial: /dev/cu.usbmodem0006837593581
           baud: 9600

   .. group-tab:: Windows

      .. code-block:: yaml

         - connected: true
           id: OSHW000032254e4500128002ab98002784d1000097969900
           platform: reel_board
           product: DAPLink CMSIS-DAP
           runner: pyocd
           serial: COM1
           baud: 9600
         - connected: true
           id: 000683759358
           platform: nrf52840dk/nrf52840
           product: J-Link
           runner: nrfjprog
           serial: COM2
           baud: 9600

The baud entry is only needed if not running at 115200.

If the map file already exists, then new entries are added and existing entries
will be updated. This way you can use one single master hardware map and update
it for every run to get the correct serial devices and status of the devices.

With the hardware map ready, you can run any tests by pointing to the map

.. tabs::

   .. group-tab:: Linux

      .. code-block:: bash

         west twister --device-testing --hardware-map map.yml -T samples/hello_world/

   .. group-tab:: Windows

      .. code-block:: bat

         west twister --device-testing --hardware-map map.yml -T samples\hello_world

The above command will result in twister building tests for the platforms
defined in the hardware map and subsequently flashing and running the tests
on those platforms.

.. note::

  Currently only boards with support for pyocd, nrfjprog, jlink, openocd, or dediprog
  are supported with the hardware map features. Boards that require other runners to flash the
  Zephyr binary are still work in progress.

Hardware map allows to set ``--device-flash-timeout`` and ``--device-flash-with-test``
command line options as ``flash-timeout`` and ``flash-with-test`` fields respectively.
These hardware map values override command line options for the particular platform.

Serial PTY support using ``--device-serial-pty``  can also be used in the
hardware map:

.. code-block:: yaml

   - connected: true
     id: None
     platform: intel_adsp/cavs25
     product: None
     runner: intel_adsp
     serial_pty: path/to/script.py
     runner_params:
       - --remote-host=remote_host_ip_addr
       - --key=/path/to/key.pem


The runner_params field indicates the parameters you want to pass to the
west runner. For some boards the west runner needs some extra parameters to
work. It is equivalent to following west and twister commands.

.. tabs::

   .. group-tab:: Linux

      .. code-block:: bash

         west flash --remote-host remote_host_ip_addr --key /path/to/key.pem

         west twister -p intel_adsp/cavs25 --device-testing --device-serial-pty script.py
         --west-flash="--remote-host=remote_host_ip_addr,--key=/path/to/key.pem"

   .. group-tab:: Windows

      .. note::

         Not supported on Windows OS

.. note::

  For serial PTY, the "--generate-hardware-map" option cannot scan it out
  and generate a correct hardware map automatically. You have to edit it
  manually according to above example. This is because the serial port
  of the PTY is not fixed and being allocated in the system at runtime.

If west is not available or does not know how to flash your system, a custom
flash command can be specified using the ``flash-command`` flag. The script is
called with a ``--build-dir`` with the path of the current build, as well as a
``--board-id`` flag to identify the specific device when multiple are available
in a hardware map.

.. tabs::

   .. group-tab:: Linux

      .. code-block:: bash

         west twister -p npcx9m6f_evb --device-testing --device-serial /dev/ttyACM0
         --flash-command './custom_flash_script.py,--flag,"complex, argument"'

   .. group-tab:: Windows

      .. note::

         west twister -p npcx9m6f_evb --device-testing
         --device-serial COM1
         --flash-command 'custom_flash_script.py,--flag,"complex, argument"'

Would result in calling ``./custom_flash_script.py
--build-dir <build directory> --board-id <board identification>
--flag "complex, argument"``.

.. _twister_fixtures:

Fixtures
--------

Some tests require additional setup or special wiring specific to the test.
Running the tests without this setup or test fixture may fail. A test scenario can
specify the fixture it needs which can then be matched with hardware capability
of a board and the fixtures it supports via the command line or using the hardware
map file.

Fixtures are defined in the hardware map file as a list:

.. code-block:: yaml

      - connected: true
        fixtures:
          - gpio_loopback
        id: 0240000026334e450015400f5e0e000b4eb1000097969900
        platform: frdm_k64f
        product: DAPLink CMSIS-DAP
        runner: pyocd
        serial: /dev/ttyACM9

When running ``twister`` with ``--device-testing``, the configured fixture
in the hardware map file will be matched to test scenarios requesting the same fixtures
and these tests will be executed on the boards that provide this fixture.

.. figure:: figures/fixtures.svg
   :figclass: align-center

Fixtures can also be provided via twister command option ``--fixture``, this option
can be used multiple times and all given fixtures will be appended as a list. And the
given fixtures will be assigned to all boards, this means that all boards set by
current twister command can run those test scenarios which request the same fixtures.

Some fixtures allow for configuration strings to be appended, separated from the
fixture name by a ``:``. Only the fixture name is matched against the fixtures
requested by test scenarios.

Notes
-----

It may be useful to annotate board descriptions in the hardware map file
with additional information.  Use the ``notes`` keyword to do this.  For
example:

.. code-block:: yaml

    - connected: false
      fixtures:
        - gpio_loopback
      id: 000683290670
      notes: An nrf5340dk/nrf5340 is detected as an nrf52840dk/nrf52840 with no serial
        port, and three serial ports with an unknown platform.  The board id of the serial
        ports is not the same as the board id of the development kit.  If you regenerate
        this file you will need to update serial to reference the third port, and platform
        to nrf5340dk/nrf5340/cpuapp or another supported board target.
      platform: nrf52840dk/nrf52840
      product: J-Link
      runner: jlink
      serial: null

Overriding Board Identifier
---------------------------

When (re-)generated the hardware map file will contain an ``id`` keyword
that serves as the argument to ``--board-id`` when flashing.  In some
cases the detected ID is not the correct one to use, for example when
using an external J-Link probe.  The ``probe_id`` keyword overrides the
``id`` keyword for this purpose.   For example:

.. code-block:: yaml

    - connected: false
      id: 0229000005d9ebc600000000000000000000000097969905
      platform: mimxrt1060_evk
      probe_id: 000609301751
      product: DAPLink CMSIS-DAP
      runner: jlink
      serial: null

Using Single Board For Multiple Variants
----------------------------------------

  The ``platform`` attribute can be a list of names or a string
  with names separated by spaces. This allows to run tests for
  different platform variants on the same physical board, without
  re-configuring the hardware map file for each variant. For example:

.. code-block:: yaml

    - connected: true
      id: '001234567890'
      platform:
      - nrf5340dk/nrf5340/cpuapp
      - nrf5340dk/nrf5340/cpuapp/ns
      product: J-Link
      runner: nrfjprog
      serial: /dev/ttyACM1

.. _twister_multi_core_testing:

Multi-Core testing support
--------------------------

Twister supports testing multi-core applications where different cores use
separate UART interfaces. This feature works only with the pytest harness
(``harness: pytest``). Generated hardware map should contain multiple entries
for the same physical device, each representing a different core connection.
For example:

.. code-block:: yaml

    - connected: true
      id: 001234567890
      serial: /dev/ttyACM0
    - connected: true
      id: 001234567890
      platform:
      - nrf54l15dk/nrf54l15/cpuapp
      product: J-Link
      runner: nrfutil
      serial: /dev/ttyACM1

Both instances share the same device ID but have different serial ports, allowing
tests to interact with multiple cores simultaneously. Each connection
is handled independently with separate log files.

.. _twister_multi_duts_testing:

Multi-DUTs testing support
--------------------------

Twister supports test scenarios that require more than one device.
This feature works only with the pytest harness (``harness: pytest``).
Hardware and ``native_sim`` execution environments are supported.

To declare that a test needs an additional device, add
``required_devices`` under ``harness_config`` in the test's YAML file.
Each entry in the list describes one extra DUT. An empty entry ``{}``
reserves a second device with the same platform and application as the
main DUT. See :ref:`required_devices <required_devices>` for all available options.

Example test configuration:

.. code-block:: yaml

    tests:
      multidut.basic:
        harness: pytest
        harness_config:
          required_devices:
            - {}

The hardware map must contain at least one entry per required device.
Each entry needs a matching platform and a serial connection:

.. code-block:: yaml

    - connected: true
      id: 01
      platform: nrf52840dk/nrf52840
      serial: /dev/ttyACM0
    - connected: true
      id: 02
      platform: nrf52840dk/nrf52840
      serial: /dev/ttyACM1

Run the test on hardware with:

.. code-block:: console

    $ west twister -vv -ll debug -T tests/subsys/testsuite/multidut \
      --device-testing --hardware-map map.yaml

Run the test on ``native_sim`` (no hardware map required):

.. code-block:: console

    $ west twister -vv -ll debug -T tests/subsys/testsuite/multidut -p native_sim

Twister reserves all required devices (or creates placeholder entries if ``native_sim`` is used),
then passes them to pytest with all necessary information
(platform, serial connection, build artifacts to flash, etc.) so the
test can interact with all devices.

An example multi-DUT test can be found at
:zephyr_file:`tests/subsys/testsuite/multidut`.

Quarantine
----------

Twister allows user to provide configuration files defining a list of tests or
platforms to be put under quarantine. Such tests will be skipped and marked
accordingly in the output reports. This feature is especially useful when
running larger test suits, where a failure of one test can affect the execution
of other tests (e.g. putting the physical board in a corrupted state).

To use the quarantine feature one has to add the argument
``--quarantine-list <PATH_TO_QUARANTINE_YAML>`` to a twister call.
Multiple quarantine files can be used.
The current status of tests on the quarantine list can also be verified by adding
``--quarantine-verify`` to the above argument. This will make twister skip all tests
which are not on the given list.

A quarantine yaml is a sequence of dictionaries. Each dictionary must have
at least one of the following keys: ``scenarios``, ``platforms``, ``architectures``
or ``simulations``. A combination of these entries is allowed.
An optional ``comment`` entry can be used to provide more details
(e.g., a link to a reported issue). These comments will also
be added to the output reports.

When quarantining a class of tests or many scenarios in a single testsuite or
when dealing with multiple issues within a subsystem, it is possible to use
regular expressions, for example, **kernel.*** would quarantine
all kernel tests.

An example of entries in a quarantine yaml:

.. code-block:: yaml

    - scenarios:
        - sample.basic.helloworld
      comment: "Link to the issue: https://github.com/zephyrproject-rtos/zephyr/pull/33287"

    - scenarios:
        - kernel.common
        - kernel.common.(misra|tls)
        - kernel.common.nano64
      platforms:
        - .*_cortex_.*
        - native_sim

    - platforms:
        - qemu_x86
      comment: "filter out qemu_x86"

    - architectures:
        - riscv

    - simulations:
        - armfvp

.. _twister_output:

Test Output and Reports
***********************

By default, Twister writes all of its output to a :file:`twister-out` directory
created in the current working directory. Use ``-O``/``--outdir`` to choose a
different location. On each run this directory is cleaned, unless ``--no-clean``
is given; ``--clobber-output`` controls what cleaning does.

Top-level reports
=================

The following files are written at the root of the output directory:

:file:`twister.json`
    The primary machine-readable report. It contains an entry for every selected
    test suite and test case with its :ref:`status <twister_statuses>`, target
    platform, execution time, memory footprint, any
    :ref:`recorded data <twister_console_harness>`, and the environment and
    options used for the run.

:file:`testplan.json`
    The resolved test plan: every test instance (test scenario times platform)
    that Twister considered, including those that were filtered out together
    with the reason. This is the file to inspect to understand why a given
    scenario did or did not run. It can be reused with ``--load-tests`` to
    replay the same selection.

:file:`twister.xml`
    JUnit XML summary suitable for CI systems.

:file:`twister_report.xml`
    JUnit XML report including all test cases (not just the summary).

:file:`twister_suite_report.xml`
    JUnit XML report grouped by test suite.

:file:`twister.log`
    Human-readable log of the whole run.

:file:`twister_footprint.json`
    ROM/RAM footprint report. Only generated when ``--footprint-report`` is used.

The report base name (``twister``) can be changed with ``--report-name``, and
``--report-suffix`` appends a suffix (for example a version or commit ID) to all
generated file names. Use ``-o``/``--report-dir`` to write the reports to a
directory other than the output directory, and ``--platform-reports`` to
additionally emit a per-platform :file:`<platform>.json` and
:file:`<platform>.xml`. ``--report-summary`` prints a summary of the failures
from the latest run without rebuilding.

Per-test artifacts
==================

Each test instance has its own build directory under the output directory,
named after the platform and the test:
:file:`twister-out/<platform>/<test path>/<scenario>/`. In addition to the
normal Zephyr build artifacts (for example :file:`zephyr/zephyr.elf`), it may
contain:

:file:`build.log`
    Output of the build for this instance.

:file:`handler.log`
    Console output captured from the device or emulator while running the test.

:file:`twister_harness.log`
    Log produced by pytest-based harnesses (for example ``pytest`` and
    ``shell``).

:file:`recording.csv`
    Data fields captured by the ``record`` option of the
    :ref:`console harness <twister_console_harness>`, when configured.

.. _twister_test_config:

Twister Configuration File
**************************

The Twister configuration file (``test_config.yaml``, passed with
``--test-config``) can be used to customize various aspects of twister
and the default enabled options and features. This allows tweaking the filtering
capabilities depending on the environment and makes it possible to adapt and
improve coverage when targeting different sets of platforms.

.. note::

   This file (selected with ``--test-config``) configures a whole Twister run.
   It is distinct from the per-application test configuration in ``tests.yaml``,
   which describes individual :ref:`test scenarios <twister_tests_long_version>`.

The Twister configuration file also adds support for test levels and the ability
to assign a specific test to one or more levels. Using command line options of
twister it is then possible to select a level and just execute the tests
included in this level.

Additionally, the configuration file allows defining level
dependencies and additional inclusion of tests into a specific level if
the test itself does not have this information already.

In the configuration file you can include complete components using
regular expressions and you can specify which test level to import from
the same file, making management of levels easier.

To help with testing outside of upstream CI infrastructure, additional
options are available in the configuration file, which can be hosted
locally. As of now, those options are available:

- Ability to ignore default platforms as defined in board definitions
  (Those are mostly emulation platforms used to run tests in upstream
  CI)
- Option to specify your own list of default platforms overriding what
  upstream defines.
- Ability to override ``build_on_all`` options used in some test scenarios.
  This will treat tests or sample as any other just build for default
  platforms you specify in the configuration file or on the command line.
- Ignore some logic in twister to expand platform coverage in cases where
  default platforms are not in scope.


Platform Configuration
======================

The following options control platform filtering in twister:

- ``override_default_platforms``: override default key a platform sets in board
  configuration and instead use the list of platforms provided in the
  configuration file as the list of default platforms. This option is set to
  False by default.
- ``increased_platform_scope``: This option is set to True by default, when
  disabled, twister will not increase platform coverage automatically and will
  only build and run tests on the specified platforms.
- ``default_platforms``: A list of additional default platforms to add. This list
  can either be used to replace the existing default platforms or can extend it
  depending on the value of ``override_default_platforms``.
- ``build_toolchains``: A mapping of platform names to the list of toolchains
  every test assigned to that platform should be built with. Twister creates one
  test instance per toolchain, each in its own build directory. This sets, or
  overrides, the ``build_toolchains`` option of the board definition; an empty
  list disables multi-toolchain builds for a platform that requests them. Since
  this multiplies build time, it is typically enabled only in the configuration
  file used by CI (``tests/test_config_ci.yaml``) so that local runs keep
  building each test once. See :ref:`twister_toolchain_selection`.

And example platforms configuration:

.. code-block:: yaml

	platforms:
	  override_default_platforms: true
	  increased_platform_scope: false
	  default_platforms:
	    - qemu_x86
	  build_toolchains:
	    native_sim:
	      - host/gnu
	      - host/llvm


Test Level Configuration
========================

The test configuration allows defining test levels, level dependencies and
additional inclusion of tests into a specific test level if the test itself
does not have this information already.

In the configuration file you can include complete components using
regular expressions and you can specify which test level to import from
the same file, making management of levels simple.

And example test level configuration:

.. code-block:: yaml

	levels:
	  - name: my-test-level
	    description: >
	      my custom test level
	    adds:
	      - kernel.threads.*
	      - kernel.timer.behavior
	      - arch.interrupt
	      - boards.*


Combined configuration
======================

To mix the Platform and level configuration, you can take an example as below:

An example platforms plus level configuration:

.. code-block:: yaml

	platforms:
	  override_default_platforms: true
	  default_platforms:
	    - frdm_k64f
	levels:
	  - name: smoke
	    description: >
	        A plan to be used verifying basic zephyr features.
	  - name: unit
	    description: >
	        A plan to be used verifying unit test.
	  - name: integration
	    description: >
	        A plan to be used verifying integration.
	  - name: acceptance
	    description: >
	        A plan to be used verifying acceptance.
	  - name: system
	    description: >
	        A plan to be used verifying system.
	  - name: regression
	    description: >
	        A plan to be used verifying regression.


To run with above test_config.yaml file, only default_platforms with given test level
test scenarios will run.

.. code-block:: console

   $ west twister --test-config=<path to>/test_config.yaml -T tests --level="smoke"