Design: Hermetic Zephyr ZTest Execution in Bazel

Goal

Establish a standardized, native-feeling Starlark macro (ztest) that enables Zephyr's ZTest-based integration and unit test applications to be executed hermetically and cacheably using standard Bazel test workflows: bazel test //path/to:test.

Background & Problem Statement

Zephyr RTOS unit and integration tests rely extensively on the ZTest framework. These tests are fundamentally structured as standalone Zephyr applications (zephyr_app targets) that boot the Zephyr kernel, execute a suite of tests inside the simulated or physical hardware, and print test results to the serial console.

In our Bazel configuration, target firmware binaries are built as executable rules (using the zephyr_app transition rule wrapper). When building for the host simulator platform (@zephyr//boards/native/native_sim:native), the resulting binary can be executed directly in the host terminal (using bazel run), booting the simulator and running the tests.

However, this creates severe limitations for automated workflows:

  1. bazel test Failure: Executing bazel test on a target binary fails with ERROR: No test targets were found, yet testing was requested. Standard test rules (cc_test) cannot link against the Zephyr kernel or target BSP files in our overlay.
  2. Non-Hermetic/Non-Cacheable: Executing test applications manually via bazel run bypasses Bazel's sandboxing and caching features. If a test succeeds, its results cannot be cached by Bazel, forcing complete re-execution.
  3. No Automated Presubmits: Automated tools (like Pigweed's presubmit runner) have no way to reliably execute simulator tests, requiring us to manually decline automated execution ("no_test": true inside workflows.json) and only verify that they compile.

Proposed Solution: Automated ztest Starlark Macro

We propose implementing a custom Bazel test macro called ztest that seamlessly bridges the gap between Zephyr target applications and host-side Bazel test runners.

Instead of demanding developers to write custom shell runners or maintain complex wrapper configurations, a developer can simply declare a ztest target in their BUILD.bazel.

load("@zephyr//:cc_test.bzl", "ztest")

ztest(
    name = "my_integration_test",
    deps = [
        ":test_sources",
    ],
)

Under the Hood Architecture

The ztest macro automatically sets up the following build graph:

graph TD
    UserTest[ztest: my_test] -->|Generates| Bin[zephyr_app: my_test_bin]
    UserTest -->|Generates| Copy[copy_file: my_test_runner_copy]
    UserTest -->|Generates| Test[py_test: my_test]
    Copy -->|Copies| Script["Script:<br>my_test_runner.py"]
    Test -->|Runs| Script
    Test -->|Data Dependency| Bin
    Script -->|1. Runs| Bin
    Script -->|2. Parses Console| Match{"Matches Success Signature?"}
    Match -->|Yes| Success[Exit 0: Test Passed]
    Match -->|No| Fail[Exit 1: Test Failed]
  1. Application Binary (zephyr_app): The macro compiles a transition-ruled zephyr_app firmware target (name + "_bin") containing the ZTest binary.
  2. Runner Script Copying: It instantiates a copy_file rule (name + "_runner_copy") from bazel_skylib to copy the shared static Python runner script (ztest_runner.py) to the local package, working around Bazel's package constraints on py_test's main attribute.
  3. Hermetic Test Wrapper (py_test): The macro wraps the copied runner script in a standard py_test target. This automatically binds the execution to the hermetic Python toolchain managed by rules_python, eliminating host-environment dependencies.
  4. Automatic Simulator Boot: The runner executes the target binary inside the host native_sim simulator (passing -uart_stdinout and guest-side/host-side timeouts to prevent hangs).
  5. Result Assertion: The runner captures the output stream, parses it for Zephyr ZTest's explicit completion signature (PROJECT EXECUTION SUCCESSFUL), integrates exit code validation, and propagates the exit code.
  6. Strict Platform Constraints: The test target automatically configures its target_compatible_with to only run on simulator platforms (@zephyr//boards/native:sim), avoiding execution on physical platforms.

Stream & Console Output Management

Zephyr RTOS targets fundamentally communicate through a physical or simulated UART console device. This maps to a single, shared output channel:

  • Single Console Pipeline: Both standard logs and fatal assert/panic stack traces exit the simulator via the single simulated UART channel, merging into the subprocess's stdout.
  • Subprocess Capture: To ensure no messages are lost (including host-side simulator core dumps or target compiler runtime warnings that could exit via stderr), the Python runner redirects and captures both pipelines: subprocess.run(..., stdout=subprocess.PIPE, stderr=subprocess.STDOUT).
  • Crash Safety & Robust Assertions:
    • The runner prioritizing exit-code evaluation: if the simulator process exits with a non-zero status, the runner aborts immediately and propagates the error back to Bazel without scanning for the success signature. This guarantees that early panics, boot loops, or dynamic link failures are marked as test failures.
    • If the exit code is 0, the runner validates the presence of the success_signature.
  • Custom Success Signatures: To support custom test setups or alternative test runners, the macro exposes a success_signature attribute (defaulting to "PROJECT EXECUTION SUCCESSFUL").
  • JUnit XML Generation: When Bazel executes tests, it injects the XML_OUTPUT_FILE environment variable. The Python runner captures this path and automatically generates a standard JUnit XML report capturing test suites, test execution, and raw logs inside this file, allowing deep integration with CI/CD test dashboards.
  • Hermetic Python Environment Integration: To satisfy strict hermeticity goals, the static runner script avoids depending on the local host environment (#!/usr/bin/env python3). Instead, it is packaged using the hermetic py_test toolchain defined by rules_python to guarantee identical script execution across all environments.
  • Bazel Output Splitting:
    • Test Logs (stdout): The merged stream (Zephyr kernel logs + compiler warnings + ZTest results) is streamed to stdout so Bazel preserves it in the persistent test.log output.
    • Diagnostic Failures (stderr): Runner-level failures (e.g., timeout limits reached, failure to locate the "PROJECT EXECUTION SUCCESSFUL" signature, process crashes) are written directly to the runner‘s stderr. This ensures diagnostic summaries print cleanly onto the developer’s terminal when a test fails.

Detailed Design

1. Starlark Rule and Macro Implementation

We define the public ztest macro and the helper _ztest_runner_gen rule inside bazel_overlay/cc_test.bzl.

The implementation details of the ztest macro, platform selections, and the underlying py_test setup are in bazel_overlay/cc_test.bzl.

2. Host-Executability Requirement & Simulator Restriction

The ztest macro strictly requires a host-executable binary because it executes the compiled Zephyr application directly on the host machine using the Zephyr native_sim simulator.

[!WARNING] If this test target were allowed to build on a non-host architecture (e.g., ARM Cortex-M) without emulator support, executing the test would fail at runtime with an Exec format error on the host.

Consequently, the macro unconditionally restricts compatibility to simulator platforms (like @zephyr//boards/native:sim). If hardware-in-the- loop (HIL) testing or QEMU emulator execution is required in the future, a separate runner and macro implementation must be introduced.

Verification & Unit Testing Plan

To ensure the correctness and long-term maintainability of the ztest macro, rule, and generated runner script, we will implement a multi-tiered testing strategy.

1. End-to-End Integration Tests (Macro & Integration Verification)

Since analysis-phase unit tests require heavy mocking of Zephyr's underlying configuration system (Kconfig, DTS, build transitions), we rely on actual compiling end-to-end targets inside our codebase to verify the build graph and runner execution.

We will add mock applications in examples/ to verify the system:

  • Success Integration Test (//examples/ztest_mock_pass): A minimal ZTest app that succeeds. Running bazel test //examples/ztest_mock_pass must pass and be cacheable by Bazel.
  • Failure Integration Test (//examples/ztest_mock_fail): A minimal ZTest app that deliberately fails (e.g. zassert_true(false)). Running bazel test //examples/ztest_mock_fail must fail cleanly and return a non-zero exit code to the terminal.

2. Python Runner Unit Tests

Since the Python runner script is generated inline inside Starlark, testing it directly can be difficult. To address this, we will extract the runner's core execution and parsing logic into a shared, unit-testable Python script inside scripts/build/ztest_runner_core.py, while the Starlark rule writes a small “bootstrap” file that imports this logic.

  • File: scripts/build/ztest_runner_core_test.py (New)
  • Framework: Standard Python unittest library, executed via a py_test target.
  • Test Cases:
    • Path Resolution Test: Mock the filesystem and environment variables (RUNFILES_DIR, TEST_WORKSPACE) to verify that the runner correctly resolves the absolute path of the simulator binary under different sandboxing conditions.
    • Success Scenario Assertion: Mock subprocess.run to return returncode = 0 and stdout containing PROJECT EXECUTION SUCCESSFUL. Assert that the runner exits with code 0.
    • Recommended Exit Code Enforcement: Mock subprocess.run to return a non-zero exit code (e.g., 1) but stdout containing PROJECT EXECUTION SUCCESSFUL. Assert that the runner correctly exits with code 1.
    • Missing Signature Failure: Mock subprocess.run to return returncode = 0 but empty stdout. Assert that the runner exits with code 1 and prints a descriptive error.
    • Simulator Hang Timeout: Mock subprocess.run to throw subprocess.TimeoutExpired. Assert that the runner terminates safely, prints the accumulated logs, and exits with code 1.

Alternatives Considered

Alternative A: Maintain External Script Files For Each Test

Maintain concrete .sh scripts in individual test directories and execute them using standard sh_test rules.

  • Pros: Standard Bazel rules only.
  • Cons: Creates massive boilerplate. Every single ZTest in the codebase would require duplicate runner files, increasing maintenance overhead and cluttering the repository.

Future Extensions: Twister Integration & Multi-Platform Execution

To address the limitations of the current custom simulator runner, we plan to explore a deeper integration with Zephyr's Twister framework and support for broader execution environments:

1. Twister-Driven Test Analysis

While the current runner relies on simple string matching (PROJECT EXECUTION SUCCESSFUL), future versions could leverage Twister's parsing libraries to:

  • Parse detailed ZTest XML/JSON reports for granular test suite results.
  • Handle complex failure conditions, boot panics, and crash logs with higher fidelity.
  • Support custom success signatures and test configurations without custom Starlark wrappers.

2. Emulator and Hardware-in-the-Loop (HIL) Execution

To support running tests on architectures other than the host simulator (e.g., ARM Cortex-M targets):

  • Emulator Runner: Extend the ztest macro to support launching emulators like QEMU or FVP using Bazel-runnable toolchain wrappers.
  • Twister Integration for HIL: Explore a hermetic wrapper around Twister that allows executing cross-compiled zephyr_app test binaries on physical hardware boards connected to the host or runner environments.