target/ast10x0: simulate an orchestrator that never blocks on a service A QEMU simulation of the real orchestrator-sm core talking to several services at once without ever blocking on one. Three apps in one image: the orchestrator (the state machine plus an executor that turns each effect into a command on a util_requester lane and each response back into an event, on one thread and one WaitGroup), a verifier (a one-channel ServiceLoop that defers each reply and completes it from the loop's deadline), and devices (a two-channel ServiceLoop where each device is a DeviceModel with a boot script, and a long-poll channel reports boots). The services decide how they behave. A device's boot script and each component's verify latency are fixed when the image is built; the orchestrator can only drive reset, name the component to verify and listen, so a bug in it cannot quietly change the script it is judged against. One image runs one scenario, chosen by a build flag, so a failure names its scenario. Scenarios: flaky_boot (a device that does not boot first time, beside a slow verify: the watchdog fires during the verify and the late verdict is dropped), verify_hangs (a verify that outlives its deadline is cancelled and recovered), and never_boots (a device that never boots makes the machine give up after its retry limit and lock). never_boots is the scenario that found the retry bug fixed in the state machine commit before this one. The scenarios race tens of milliseconds against QEMU's wall clock. Each gap that has to hold is documented next to the numbers, the service threads run above the orchestrator's priority, and the tests are exclusive. The review checklist was applied: no indexing, casts that can truncate, or bare arithmetic in the simulation code. A README documents the apps, scenarios, how to run and extend it, and the timing rules. Clippy, the lockfile check and the presubmit checks pass, and the format check passes after fixing apps.bzl. Depends on the four commits before it (the state machine fix, the device model, the requester lane and the util/service fix). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
diff --git a/target/ast10x0/tests/orchestrator/async_sim/BUILD.bazel b/target/ast10x0/tests/orchestrator/async_sim/BUILD.bazel new file mode 100644 index 0000000..cbca940 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/BUILD.bazel
@@ -0,0 +1,22 @@ +# Licensed under the Apache-2.0 license +# SPDX-License-Identifier: Apache-2.0 + +# The shared sources and system config for the async orchestration scenarios. +# Each scenario is its own package (`flaky_boot/`, `verify_hangs/`) that builds +# these through labels, so one image is one scenario. See scenario.bzl. +# +# Thesis under test: the real orchestrator core can drive several services +# without blocking, using ServiceLoop servers and an async client. + +exports_files([ + "devices_main.rs", + "orchestrator_main.rs", + "target.rs", + "verifier_main.rs", +]) + +filegroup( + name = "system_config", + srcs = ["system.json5"], + visibility = ["//visibility:public"], +)
diff --git a/target/ast10x0/tests/orchestrator/async_sim/README.md b/target/ast10x0/tests/orchestrator/async_sim/README.md new file mode 100644 index 0000000..0d83276 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/README.md
@@ -0,0 +1,77 @@ +<!-- Licensed under the Apache-2.0 license --> +<!-- SPDX-License-Identifier: Apache-2.0 --> + +# async_sim: an orchestrator that never blocks on a service + +A QEMU simulation of the real `orchestrator-sm` core talking to several services +at once, without ever blocking on one. It checks that the orchestrator keeps +reacting to boot watchdogs and other services while a slow call is in flight, +that a call that outlives its deadline is abandoned and recovered, and that a +device that never boots makes the machine give up. + +## Apps + +Three apps in one image: + +- **orchestrator**: the system under test. The real state machine, plus an + executor that turns each effect into a command on a `util_requester` lane and + each response back into an event. One thread, one WaitGroup, no blocking call. +- **verifier**: a `util_service::ServiceLoop` server with one channel. It defers + each reply for a latency of its own and completes it from the loop's deadline. +- **devices**: a `ServiceLoop` server with two channels, `reset` and `events`. Each + device is an `openprot_device_model::DeviceModel` with a boot script. `reset` + releases or holds a device; `events` is a long-poll that is answered when a + device boots. + +The services decide how they behave. A device's boot script and each +component's verify latency are fixed when the image is built, and the +orchestrator never says how a service should act. It can only drive reset, name +the component to verify, and listen. So a bug in the orchestrator (a double +release, say) cannot quietly change the script it is judged against. + +## Scenarios + +One image runs one scenario, chosen by a build flag, so each script is fixed at +build time and a failure names its scenario. + +| Scenario | The world | Passes when | +|---|---|---| +| `flaky_boot` | C0 does not boot on its first release; C1's first verify is slow | C0's boot watchdog fires while C1 is still verifying, the core recovers C0, C1's late verdict is dropped as stale, and the re-walk reaches `Ready` | +| `verify_hangs` | C2's first verify outlives the orchestrator's deadline | the executor cancels it and reports a failed verification, the verifier's late reply is dropped, and the re-walk reaches `Ready` | +| `never_boots` | C0 never boots, on any release | the machine gives up after its retry limit (three recoveries), reports recovery failed and locks | + +## Running + +```sh +bazel test --config=virt_ast10x0 //target/ast10x0/tests/orchestrator/async_sim/... +bazel test --config=virt_ast10x0 --test_output=all \ + //target/ast10x0/tests/orchestrator/async_sim/flaky_boot:flaky_boot_qemu_test +``` + +Repeat a run with `--runs_per_test`, not a loop inside the image: the scripts +count from the start of the image. Each image also has a `no_panics_test`. + +## Adding a scenario + +1. Add a `--cfg` name to `SCENARIO_CFGS` in `scenario.bzl`. +2. Add a package (`my_scenario/BUILD.bazel`) that calls `qemu_scenario` with that + flag. The system config names each app by its target name, so a scenario is a + package of its own. +3. Give the services their script for it: `C0_SCRIPT` in `devices_main.rs`, and + `LATENCY_MS` in `verifier_main.rs`. +4. Add the scenario function and its checks to `orchestrator_main.rs`. + +## Timing + +The scenarios race tens of milliseconds against QEMU's wall clock, so the margins +matter. Each gap that has to hold is written next to the numbers: + +- `BOOT_WINDOW` and `VERIFY_DEADLINE` in `orchestrator_main.rs`, and the boot + and verify times in `devices_main.rs` and `verifier_main.rs`, form a chain + (boot < watchdog < slow verify < deadline). Move one and the others move with it. +- `verify_hangs` needs the hang to sit between the deadline and twice the + deadline. Widening only one side breaks the other. +- The service threads run at a higher priority than the orchestrator, so a + service reacts as soon as a request or its timer is ready. +- The tests are `exclusive`: beside other QEMU runs the guest slows down and the + margins go.
diff --git a/target/ast10x0/tests/orchestrator/async_sim/apps.bzl b/target/ast10x0/tests/orchestrator/async_sim/apps.bzl new file mode 100644 index 0000000..e79f1e3 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/apps.bzl
@@ -0,0 +1,56 @@ +# Licensed under the Apache-2.0 license +# SPDX-License-Identifier: Apache-2.0 + +"""The apps every async_sim scenario builds, shared by label. + +Labels rather than file names, because each scenario is a package of its own +and builds the same sources from there. +""" + +HERE = "//target/ast10x0/tests/orchestrator/async_sim:" + +APPS = [ + # The system under test: the orchestrator core and its async executor. + { + "deps": [ + "//services/orchestrator/server:orchestrator_server", + "//services/orchestrator/sm:orchestrator_sm", + "//util/ipc", + "//util/requester", + "//util/transport", + "@pigweed//pw_kernel/userspace", + "@pigweed//pw_log/rust:pw_log", + "@pigweed//pw_status/rust:pw_status", + "@rust_crates//:heapless", + ], + "name": "orchestrator", + "src": HERE + "orchestrator_main.rs", + }, + # Verifies images after a latency of its own choosing (deferred replies). + { + "deps": [ + "//util/service", + "@pigweed//pw_kernel/userspace", + "@pigweed//pw_log/rust:pw_log", + "@pigweed//pw_status/rust:pw_status", + ], + "name": "verifier", + "src": HERE + "verifier_main.rs", + }, + # The managed devices: each follows a boot script through a device model. + { + "deps": [ + "//services/orchestrator/test/device_model:device_model", + "//util/service", + "@pigweed//pw_kernel/userspace", + "@pigweed//pw_log/rust:pw_log", + "@pigweed//pw_status/rust:pw_status", + "@rust_crates//:heapless", + ], + "name": "devices", + "src": HERE + "devices_main.rs", + }, +] + +SYSTEM_CONFIG = HERE + "system_config" +TARGET_SRC = HERE + "target.rs"
diff --git a/target/ast10x0/tests/orchestrator/async_sim/devices_main.rs b/target/ast10x0/tests/orchestrator/async_sim/devices_main.rs new file mode 100644 index 0000000..13d5483 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/devices_main.rs
@@ -0,0 +1,177 @@ +// Licensed under the Apache-2.0 license +// SPDX-License-Identifier: Apache-2.0 + +//! Simulated managed devices: a reset controller and a boot-event source. +//! +//! Each device is an `openprot_device_model::DeviceModel` with a boot script +//! fixed when the image is built. The orchestrator can only drive the reset +//! line and listen; it never says how a device should behave. +//! +//! `reset` channel, answered at once with `[status, id]`: +//! `[OP_RELEASE, id]` release `id`'s reset; the model starts its next boot +//! `[OP_ASSERT, id]` hold `id` in reset; a held device has no news, so +//! any boot report not yet sent is dropped +//! `[OP_RESTORE, id]` restore `id` from its recovery source +//! +//! `events` channel, a long-poll: `[OP_WAIT]` is answered `[0, id]` with the +//! next device that booted. With none ready the reply is deferred, so one +//! request sits parked until a boot happens. That is how an event reaches a +//! client that never blocks. +//! +//! The service sleeps until the earliest moment any model says it will change +//! (`next_event_at`), so a boot is reported when it is due, not on a polling +//! interval. + +#![no_main] +#![no_std] + +use app_devices::handle; +use openprot_device_model::{Behaviour, DeviceModel}; +use pw_status::Result; +use userspace::entry; +use userspace::time::{Clock, Instant, SystemClock}; +use util_service::{Channel, Event, ServiceLoop}; + +const OP_RELEASE: u8 = 2; +const OP_ASSERT: u8 = 3; +const OP_RESTORE: u8 = 4; +const KEY_EVENTS: u32 = 1; +const RESET_INDEX: usize = 0; +const DEVICES: usize = 3; + +/// How long a healthy device takes to come up after its release. It has to +/// beat the orchestrator's boot watchdog by a wide margin; see `BOOT_WINDOW` +/// in orchestrator_main.rs. +const BOOT_MS: u64 = 20; + +const BOOTS: &[Behaviour] = &[Behaviour::Boots { + after_millis: BOOT_MS, +}]; + +/// C0 does not boot the first time it is released, and boots the second. +#[cfg(flaky_boot)] +const C0_SCRIPT: &[Behaviour] = &[ + Behaviour::Hang, + Behaviour::Boots { + after_millis: BOOT_MS, + }, +]; +/// C0 never boots, on any release. +#[cfg(never_boots)] +const C0_SCRIPT: &[Behaviour] = &[Behaviour::Hang]; +#[cfg(not(any(flaky_boot, never_boots)))] +const C0_SCRIPT: &[Behaviour] = BOOTS; + +fn now_ms() -> u64 { + SystemClock::now().ticks().saturating_mul(1000) / SystemClock::TICKS_PER_SEC +} + +/// The kernel instant at `ms` on the same clock as [`now_ms`]. +fn instant_at(ms: u64) -> Instant { + Instant::from_ticks(ms.saturating_mul(SystemClock::TICKS_PER_SEC) / 1000) +} + +/// Drop every queued boot for `id`. +fn forget(ready: &mut heapless::Deque<u8, 8>, id: u8) { + for _ in 0..ready.len() { + if let Some(x) = ready.pop_front() + && x != id + { + let _ = ready.push_back(x); + } + } +} + +fn serve() -> Result<()> { + let mut channels = [Channel::new(handle::RESET), Channel::new(handle::EVENTS)]; + let mut service = ServiceLoop::new(handle::WG, &mut channels)?; + let mut buf = [0u8; 8]; + let mut devices: [DeviceModel<'static>; DEVICES] = [ + DeviceModel::new(C0_SCRIPT), + DeviceModel::new(BOOTS), + DeviceModel::new(BOOTS), + ]; + // Devices that booted and have not been reported yet. + let mut ready: heapless::Deque<u8, 8> = heapless::Deque::new(); + let mut parked = false; + + loop { + // Wake when the earliest model is next due to change. One + // millisecond late on purpose: the conversion rounds down, and a + // wake a tick early would find nothing due and wait again. + let deadline = devices + .iter() + .filter_map(|d| d.next_event_at()) + .min() + .map_or(Instant::MAX, |ms| instant_at(ms.saturating_add(1))); + + if let Event::Request { channel, len } = service.next(deadline, &mut buf)? { + let req = buf.get(..len).unwrap_or(&[]); + if channel.index() == RESET_INDEX { + let now = now_ms(); + let reply = match *req { + [OP_RELEASE, id] => match devices.get_mut(usize::from(id)) { + Some(device) => { + device.set_reset(false, now); + [0, id] + } + None => [1, id], + }, + [OP_ASSERT, id] => match devices.get_mut(usize::from(id)) { + Some(device) => { + device.set_reset(true, now); + // The line is held, so there is no news: drop + // what the device had not yet reported. + while device.take_transition().is_some() {} + forget(&mut ready, id); + [0, id] + } + None => [1, id], + }, + [OP_RESTORE, id] => [0, id], + _ => [1, 0], + }; + let _ = service.reply(channel, &reply); + } else if let Some(id) = ready.pop_front() { + let _ = service.reply(channel, &[0, id]); + } else if service.defer(channel, KEY_EVENTS).is_ok() { + parked = true; + } else { + pw_log::error!("devices: defer failed"); + let _ = service.reply(channel, &[1, 0]); + } + } + + // Whatever the wake was, let time pass for every device and report + // each one that came up. + let now = now_ms(); + for (id, device) in devices.iter_mut().enumerate() { + device.poll(now); + while let Some(change) = device.take_transition() { + if change.ready { + if let Ok(id) = u8::try_from(id) { + let _ = ready.push_back(id); + } + } + } + } + if parked && let Some(id) = ready.pop_front() { + parked = false; + if service.complete(KEY_EVENTS, &[0, id]).is_err() { + pw_log::warn!("devices: boot event dropped (client gone)"); + } + } + } +} + +#[entry] +fn entry() { + if let Err(e) = serve() { + pw_log::error!("devices: loop exited, status {}", e as u32); + } +} + +#[panic_handler] +fn panic(_info: &core::panic::PanicInfo) -> ! { + loop {} +}
diff --git a/target/ast10x0/tests/orchestrator/async_sim/flaky_boot/BUILD.bazel b/target/ast10x0/tests/orchestrator/async_sim/flaky_boot/BUILD.bazel new file mode 100644 index 0000000..07e24b5 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/flaky_boot/BUILD.bazel
@@ -0,0 +1,19 @@ +# Licensed under the Apache-2.0 license +# SPDX-License-Identifier: Apache-2.0 + +# C0 does not boot on its first release and C1's verify is slow, so C0's boot +# watchdog fires while C1 is still verifying. Passes when the core recovers C0, +# drops C1's late verdict, and the re-walk ends in Ready. +# +# bazel test --config=virt_ast10x0 //target/ast10x0/tests/orchestrator/async_sim/flaky_boot:flaky_boot_qemu_test + +load("//target/ast10x0/tests/orchestrator/async_sim:apps.bzl", "APPS", "SYSTEM_CONFIG", "TARGET_SRC") +load("//target/ast10x0/tests/orchestrator/async_sim:scenario.bzl", "qemu_scenario") + +qemu_scenario( + name = "flaky_boot_qemu", + apps = APPS, + cfgs = ["flaky_boot"], + system_config = SYSTEM_CONFIG, + target_src = TARGET_SRC, +)
diff --git a/target/ast10x0/tests/orchestrator/async_sim/never_boots/BUILD.bazel b/target/ast10x0/tests/orchestrator/async_sim/never_boots/BUILD.bazel new file mode 100644 index 0000000..9e3af02 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/never_boots/BUILD.bazel
@@ -0,0 +1,24 @@ +# Licensed under the Apache-2.0 license +# SPDX-License-Identifier: Apache-2.0 + +# C0 never boots, on any release. Every attempt verifies, releases and times +# out. Passes when the machine gives up after its retry limit, reports that +# recovery is exhausted and locks, instead of recovering forever. +# +# This scenario found that the state machine cleared a component's retry count +# on a passing verification and on entering Ready, so a device that verifies and +# never boots recovered forever. The count now ends only on a confirmed boot +# report, and the machine locks after three recoveries. +# +# bazel test --config=virt_ast10x0 //target/ast10x0/tests/orchestrator/async_sim/never_boots:never_boots_qemu_test + +load("//target/ast10x0/tests/orchestrator/async_sim:apps.bzl", "APPS", "SYSTEM_CONFIG", "TARGET_SRC") +load("//target/ast10x0/tests/orchestrator/async_sim:scenario.bzl", "qemu_scenario") + +qemu_scenario( + name = "never_boots_qemu", + apps = APPS, + cfgs = ["never_boots"], + system_config = SYSTEM_CONFIG, + target_src = TARGET_SRC, +)
diff --git a/target/ast10x0/tests/orchestrator/async_sim/orchestrator_main.rs b/target/ast10x0/tests/orchestrator/async_sim/orchestrator_main.rs new file mode 100644 index 0000000..3588089 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/orchestrator_main.rs
@@ -0,0 +1,654 @@ +// Licensed under the Apache-2.0 license +// SPDX-License-Identifier: Apache-2.0 + +//! The orchestrator under test: the real `orchestrator-sm` core driven by an +//! asynchronous effect executor. +//! +//! Nothing here blocks on a service. An effect becomes a command queued for +//! one of three channels (verifier, reset, events); the loop starts it when +//! the channel is idle, then sleeps in one WaitGroup wait whose deadline is +//! the earliest of the boot watchdogs and the in-flight verify deadlines. +//! Whatever wakes it is decoded into an `Event` and fed back to the core. +//! +//! A channel carries one transaction at a time, so commands for the same +//! service queue behind each other. The events channel always has one +//! `WaitBoot` long-poll outstanding. +//! +//! The services decide how they behave (a device's boot script, how long a +//! verify takes), so nothing in this app tells them. One image runs one +//! scenario, chosen by a build flag, so each service's script is fixed when +//! the image is built and a failure names its scenario. +//! +//! | Flag | Shows | +//! |----------------|------------------------------------------------------------| +//! | `flaky_boot` | C0 does not boot on its first release and C1's verify is | +//! | | slow. C0's boot watchdog fires *while* C1 is still | +//! | | verifying, the core recovers C0, C1's late verdict is | +//! | | dropped as stale, and the re-walk ends in `Ready`. | +//! | `verify_hangs` | C2's verify outlives the executor's deadline. The executor | +//! | | cancels it and reports `VerificationFailed`; the | +//! | | verifier's late reply is dropped; the re-walk ends in | +//! | | `Ready`. | +//! | `never_boots` | C0 never boots on any release. The platform has to give up | +//! | | after its retry limit, report the failure and lock, not | +//! | | recover forever. | + +#![no_main] +#![no_std] + +#[cfg(not(any(flaky_boot, verify_hangs, never_boots)))] +compile_error!("build one scenario: pass --cfg flaky_boot, verify_hangs or never_boots"); +#[cfg(any( + all(flaky_boot, verify_hangs), + all(flaky_boot, never_boots), + all(verify_hangs, never_boots) +))] +compile_error!("build one scenario, not several"); + +use app_orchestrator::handle; +use openprot_orchestrator_server::BootWatchdogs; +use openprot_orchestrator_sm::{ + Chain, ComponentAttrs, ComponentId, Effect, EffectError, Event, Orchestrator, PowerOnResult, + State, +}; +use pw_status::{Error, Result}; +use userspace::entry; +use userspace::syscall::{self, Signals}; +use userspace::time::{Clock, Duration, Instant, SystemClock}; +use util_ipc::{AsyncChannelTransport, IpcHandle}; +use util_requester::Command; + +const N: usize = 3; +const E: usize = 2 * N + 2; +const MAX_RETRY: u8 = 3; + +type Core = Orchestrator<N, E>; +type Transport = AsyncChannelTransport<IpcHandle>; + +// Wire opcodes, matching verifier_main.rs and devices_main.rs. +const OP_VERIFY: u8 = 1; +const OP_RELEASE: u8 = 2; +const OP_ASSERT: u8 = 3; +const OP_RESTORE: u8 = 4; +const OP_WAIT: u8 = 5; + +/// A component that has not reported in this long after release is failed. +/// +/// The scenarios race four durations, and each gap has to stay wide on QEMU's +/// wall clock. In order: a healthy device's boot `B` (devices_main.rs, 20 ms), +/// this window `W`, the slow verify `V` that `flaky_boot` arranges +/// (verifier_main.rs, 210 ms), and the verify deadline `D` below (300 ms). +/// +/// - `W - B` = 90 ms: a healthy device must report in before its watchdog. +/// - `V - W` = 100 ms: the flaky device's watchdog must fire *while* the slow +/// verify is still running, so the timeout is seen before that verdict. +/// - `D - V` = 90 ms: the slow verify must still finish inside its deadline. +/// +/// Move one and the others move with it. `verify_hangs` has its own pair of +/// margins against `D`, described in verifier_main.rs. +const BOOT_WINDOW: Duration = Duration::from_millis(110); +/// A verify not answered this long after it started is abandoned. +/// +/// Tied to the `verify_hangs` latency in verifier_main.rs, which must sit +/// between this and twice this (see the margins there). Both are scaled +/// together, not just widened. +const VERIFY_DEADLINE: Duration = Duration::from_millis(300); +/// A whole scenario that takes longer than this has hung. +const SCENARIO_LIMIT: Duration = Duration::from_millis(4000); + +const VERIFIER: usize = 0; +const RESET: usize = 1; +const EVENTS: usize = 2; + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Cmd { + Verify { id: u8 }, + Release { id: u8 }, + Assert { id: u8 }, + Restore { id: u8 }, + WaitBoot, +} + +impl Cmd { + /// A verify that has not answered this long after it started is + /// abandoned. Nothing else has a deadline. + fn deadline(&self) -> Option<Instant> { + match self { + Cmd::Verify { .. } => Some( + SystemClock::now() + .checked_add_duration(VERIFY_DEADLINE) + .unwrap_or(Instant::MAX), + ), + _ => None, + } + } +} + +impl Command for Cmd { + fn encode(&self, out: &mut [u8]) -> Option<usize> { + let (bytes, len) = match *self { + Cmd::Verify { id } => ([OP_VERIFY, id, 0], 2), + Cmd::Release { id } => ([OP_RELEASE, id, 0], 2), + Cmd::Assert { id } => ([OP_ASSERT, id, 0], 2), + Cmd::Restore { id } => ([OP_RESTORE, id, 0], 2), + Cmd::WaitBoot => ([OP_WAIT, 0, 0], 1), + }; + out.get_mut(..len)?.copy_from_slice(bytes.get(..len)?); + Some(len) + } +} + +/// A `util_requester` lane plus the kernel half of it: the transport's handle +/// is in the WaitGroup exactly while a command is in flight. +struct Lane { + index: usize, + inner: util_requester::Lane<Transport, Cmd, Instant, 8>, +} + +impl Lane { + fn new(index: usize, tr: Transport) -> Self { + Self { + index, + inner: util_requester::Lane::new(tr), + } + } + + fn unwatch(&self) -> Result<()> { + syscall::wait_group_remove(handle::WG, self.inner.transport().as_raw()) + } + + /// Start the next queued command if the channel is free. + fn start_next(&mut self) -> Result<()> { + match self.inner.start_next(|cmd| cmd.deadline()) { + Ok(Some(_)) => syscall::wait_group_add( + handle::WG, + self.inner.transport().as_raw(), + Signals::READABLE, + self.index, + ), + Ok(None) => Ok(()), + Err(_) => Err(Error::Internal), + } + } + + /// Collect the response if it has landed. + fn finish(&mut self) -> Result<Option<(Cmd, [u8; 2])>> { + let mut resp = [0u8; 4]; + match self.inner.poll(&mut resp) { + Ok(None) => Ok(None), + Ok(Some(done)) => { + self.unwatch()?; + match (done.len >= 2, resp.first(), resp.get(1)) { + (true, Some(status), Some(id)) => Ok(Some((done.cmd, [*status, *id]))), + _ => Err(Error::Internal), + } + } + Err(_) => { + let _ = self.unwatch(); + Err(Error::Internal) + } + } + } + + /// Abandon the command in flight and return it. + fn cancel(&mut self) -> Result<Option<Cmd>> { + let cancelled = self.inner.cancel(); + let unwatched = self.unwatch(); + let cmd = cancelled.map_err(|_| Error::Internal)?; + if cmd.is_some() { + unwatched?; + } + Ok(cmd) + } +} + +/// What the core was told, in the order the executor delivered it. +#[derive(Clone, Copy, PartialEq, Eq)] +enum Seen { + Timeout(u8), + Passed(u8), + Failed(u8), + Booted(u8), + Restored(u8), +} + +/// What a scenario run is waiting for. +#[derive(Clone, Copy)] +enum Goal { + /// Every component verified, released and booted, the core in `Ready`. + #[cfg_attr(never_boots, allow(dead_code))] + Ready, + /// The core latched `Locked`. Gives up, as a failure, if the core has + /// recovered more than `max_recoveries` times and is still running. + #[cfg_attr(not(never_boots), allow(dead_code))] + Locked { max_recoveries: u32 }, +} + +struct Exec { + lanes: [Lane; 3], + wd: BootWatchdogs<N>, + booted: [bool; N], + seen: heapless::Vec<Seen, 48>, + verify_cancels: u32, + /// Components restored so far: one per `Restored` delivered to the core. + recoveries: u32, + /// Whether the core reported that recovery was exhausted. Only + /// `never_boots` reads it. + #[cfg_attr(not(never_boots), allow(dead_code))] + recovery_failed: bool, +} + +impl Exec { + fn begin(&mut self) { + self.wd = BootWatchdogs::new(); + self.booted = [false; N]; + self.seen.clear(); + self.verify_cancels = 0; + self.recoveries = 0; + self.recovery_failed = false; + } + + fn push(&mut self, lane: usize, cmd: Cmd) -> core::result::Result<(), EffectError> { + self.lanes + .get_mut(lane) + .ok_or(EffectError)? + .inner + .push(cmd) + .map_err(|_| EffectError) + } + + fn mark(&mut self, id: u8, booted: bool) { + if let Some(b) = self.booted.get_mut(usize::from(id)) { + *b = booted; + } + } + + /// Turn one effect into a command. Never blocks and never calls a + /// service: the answer comes back later as an event. + fn effect(&mut self, effect: Effect) -> core::result::Result<Option<Event>, EffectError> { + match effect { + Effect::VerifyFirmware(id) => { + self.push(VERIFIER, Cmd::Verify { id: id.get() })?; + } + Effect::ReleaseReset(id) => { + let id = id.get(); + self.mark(id, false); + self.push(RESET, Cmd::Release { id })?; + self.wd + .arm_boot(ComponentId::new(id), BOOT_WINDOW) + .map_err(|_| EffectError)?; + } + Effect::AssertReset(id) => { + let id = id.get(); + self.mark(id, false); + self.wd.cancel_boot(ComponentId::new(id)); + self.push(RESET, Cmd::Assert { id })?; + } + Effect::RecoverComponent { id, .. } => { + let id = id.get(); + self.mark(id, false); + self.wd.cancel_boot(ComponentId::new(id)); + // The machine expects the component held in reset while it is + // restored. A device that is still running (or hung) with its + // line released ignores a second release, so hold it first. + self.push(RESET, Cmd::Assert { id })?; + self.push(RESET, Cmd::Restore { id })?; + } + // Local, nothing to ask a service for. + Effect::ReadFirmware(_) | Effect::CommitSvnFloor(_) => {} + Effect::ReportRecoveryFailed(_) => self.recovery_failed = true, + Effect::ReportIsolated(_) + | Effect::ReportBootFailed { .. } + | Effect::ReportUpdateDeferred + | Effect::ReportUpdateAborted => {} + // Not exercised by this simulation. + Effect::SignAttestation + | Effect::AuthenticateUpdate + | Effect::StageUpdate + | Effect::ActivateUpdate + | Effect::DiscardStaged + | Effect::LatchLockdown => {} + Effect::Emit(_) => return Err(EffectError), + } + Ok(None) + } + + fn record(&mut self, event: Event) { + let seen = match event { + Event::Timeout(id) => { + pw_log::info!("EV timeout {}", id.get() as u32); + Seen::Timeout(id.get()) + } + Event::VerificationPassed(id) => { + pw_log::info!("EV passed {}", id.get() as u32); + Seen::Passed(id.get()) + } + Event::VerificationFailed(id) => { + pw_log::info!("EV failed {}", id.get() as u32); + Seen::Failed(id.get()) + } + Event::Booted(id) => { + pw_log::info!("EV booted {}", id.get() as u32); + Seen::Booted(id.get()) + } + Event::Restored(id) => { + self.recoveries = self.recoveries.saturating_add(1); + pw_log::info!("EV restored {}", id.get() as u32); + Seen::Restored(id.get()) + } + _ => return, + }; + let _ = self.seen.push(seen); + } + + /// Start whatever can start. The events lane always keeps a long-poll + /// queued. + fn pump(&mut self) -> Result<()> { + if let Some(events) = self.lanes.get_mut(EVENTS) + && events.inner.is_idle() + { + let _ = events.inner.push(Cmd::WaitBoot); + } + for lane in self.lanes.iter_mut() { + lane.start_next()?; + } + Ok(()) + } + + /// Decode a lane's response into an event for the core, if it has one. + fn service(&mut self, lane: usize) -> Result<Option<Event>> { + let Some(l) = self.lanes.get_mut(lane) else { + return Ok(None); + }; + let Some((cmd, resp)) = l.finish()? else { + return Ok(None); + }; + let [status, reported] = resp; + let ok = status == 0; + Ok(match cmd { + Cmd::Verify { id, .. } if ok => Some(Event::VerificationPassed(ComponentId::new(id))), + Cmd::Verify { id, .. } => Some(Event::VerificationFailed(ComponentId::new(id))), + Cmd::Restore { id } if ok => Some(Event::Restored(ComponentId::new(id))), + Cmd::Release { .. } | Cmd::Assert { .. } | Cmd::Restore { .. } if !ok => { + return Err(Error::Internal); + } + Cmd::Release { .. } | Cmd::Assert { .. } => None, + Cmd::WaitBoot if ok => { + let id = reported; + self.mark(id, true); + self.wd.cancel_boot(ComponentId::new(id)); + Some(Event::Booted(ComponentId::new(id))) + } + Cmd::WaitBoot => None, + Cmd::Restore { .. } => None, + }) + } + + /// Give up on a verify that has run past its deadline. The unknown + /// outcome counts as a failed verification. + fn expire_verify(&mut self) -> Result<Option<Event>> { + let now = SystemClock::now(); + let Some(lane) = self.lanes.get_mut(VERIFIER) else { + return Ok(None); + }; + let Some(&cmd) = lane.inner.in_flight() else { + return Ok(None); + }; + if !lane.inner.expired(&now) { + return Ok(None); + } + let _ = lane.cancel()?; + self.verify_cancels = self.verify_cancels.saturating_add(1); + Ok(match cmd { + Cmd::Verify { id, .. } => Some(Event::VerificationFailed(ComponentId::new(id))), + _ => None, + }) + } + + /// The earliest moment the loop must wake without a response. + fn next_deadline(&self) -> Instant { + let mut at = self.wd.wait_deadline(); + if let Some(due) = self.lanes.get(VERIFIER).and_then(|l| l.inner.deadline()) + && due < at + { + at = due; + } + at + } + + /// Nothing but the long-poll is outstanding. + fn quiescent(&self) -> bool { + self.lanes + .iter() + .take(EVENTS) + .all(|lane| lane.inner.is_idle()) + } + + fn all_booted(&self) -> bool { + self.booted.iter().all(|b| *b) + } + + // Only the scenarios that check event order use it. + #[cfg_attr(never_boots, allow(dead_code))] + fn first(&self, what: Seen) -> Option<usize> { + self.seen.iter().position(|s| *s == what) + } +} + +fn deliver(core: &mut Core, exec: &mut Exec, event: Event) { + exec.record(event); + core.dispatch_with(event, |effect| exec.effect(effect)); +} + +fn new_core() -> Result<Core> { + let mut entries = heapless::Vec::<(ComponentId, ComponentAttrs), N>::new(); + for id in 0..N as u8 { + entries + .push((ComponentId::new(id), ComponentAttrs::passive_required())) + .map_err(|_| Error::ResourceExhausted)?; + } + let chain: Chain<N> = entries.try_into().map_err(|_| Error::Internal)?; + Ok(Orchestrator::new(chain, MAX_RETRY)) +} + +/// Run one scenario to `Ready` with every device booted. +fn run_scenario(exec: &mut Exec, goal: Goal) -> Result<Core> { + exec.begin(); + let mut core = new_core()?; + let start = SystemClock::now(); + let limit = start + .checked_add_duration(SCENARIO_LIMIT) + .unwrap_or(Instant::MAX); + + deliver( + &mut core, + exec, + Event::PowerGood(PowerOnResult::Provisioned), + ); + + loop { + exec.pump()?; + + match goal { + Goal::Ready => { + if core.state() == State::Locked { + pw_log::error!("scenario: core latched Locked"); + return Err(Error::Internal); + } + if core.state() == State::Ready && exec.quiescent() && exec.all_booted() { + return Ok(core); + } + } + Goal::Locked { max_recoveries } => { + if core.state() == State::Locked { + return Ok(core); + } + if exec.recoveries > max_recoveries { + pw_log::error!( + "scenario: still not locked after {} recoveries", + exec.recoveries as u32 + ); + return Err(Error::Internal); + } + } + } + if SystemClock::now() >= limit { + pw_log::error!("scenario: timed out"); + return Err(Error::DeadlineExceeded); + } + + let deadline = if exec.next_deadline() < limit { + exec.next_deadline() + } else { + limit + }; + match syscall::object_wait(handle::WG, Signals::READABLE, deadline) { + Ok(wake) => { + if let Some(event) = exec.service(wake.user_data)? { + deliver(&mut core, exec, event); + } + } + Err(Error::DeadlineExceeded) => {} + Err(e) => return Err(e), + } + while let Some(event) = exec.wd.poll_expired() { + deliver(&mut core, exec, event); + } + if let Some(event) = exec.expire_verify()? { + deliver(&mut core, exec, event); + } + } +} + +/// C0 never boots the first time and C1 verifies slowly. C0's watchdog has +/// to fire during C1's verify, so the core is told about the timeout before +/// C1's verdict, which then arrives for a component the re-walk has moved +/// past. +#[cfg(flaky_boot)] +fn scenario(exec: &mut Exec) -> Result<()> { + pw_log::info!("scenario flaky_boot: flaky boot beside a slow verify"); + run_scenario(exec, Goal::Ready)?; + + let timeout = exec.first(Seen::Timeout(0)); + let late_verdict = exec.first(Seen::Passed(1)); + match (timeout, late_verdict) { + (Some(t), Some(v)) if t < v => {} + _ => { + pw_log::error!("flaky_boot: C0 timeout did not precede C1's verdict"); + return Err(Error::Internal); + } + } + if exec.first(Seen::Restored(0)).is_none() { + pw_log::error!("flaky_boot: C0 was never restored"); + return Err(Error::Internal); + } + if exec.verify_cancels != 0 { + pw_log::error!("flaky_boot: a verify was abandoned unexpectedly"); + return Err(Error::Internal); + } + Ok(()) +} + +/// A verify outlives its deadline: abandon it, fail the component, recover, +/// and finish. +#[cfg(verify_hangs)] +fn scenario(exec: &mut Exec) -> Result<()> { + pw_log::info!("scenario verify_hangs: verify outlives its deadline"); + run_scenario(exec, Goal::Ready)?; + + if exec.verify_cancels != 1 { + pw_log::error!("verify_hangs: expected exactly one abandoned verify"); + return Err(Error::Internal); + } + if exec.first(Seen::Failed(2)).is_none() || exec.first(Seen::Restored(2)).is_none() { + pw_log::error!("verify_hangs: C2 was not failed and restored"); + return Err(Error::Internal); + } + Ok(()) +} + +/// C0 never boots, on any release. Every attempt verifies, releases and times +/// out, so the machine recovers and re-walks again and again. It has to stop: +/// give up after its retry limit, report that recovery is exhausted, and lock. +/// The run allows a few attempts past the limit, so a machine that never stops +/// fails here instead of running out the clock. +#[cfg(never_boots)] +fn scenario(exec: &mut Exec) -> Result<()> { + pw_log::info!("scenario never_boots: a device that never boots"); + let limit = u32::from(MAX_RETRY); + run_scenario( + exec, + Goal::Locked { + max_recoveries: limit.saturating_add(3), + }, + )?; + + if exec.recoveries != limit { + pw_log::error!( + "never_boots: {} recoveries before locking, expected {}", + exec.recoveries as u32, + limit as u32 + ); + return Err(Error::Internal); + } + if !exec.recovery_failed { + pw_log::error!("never_boots: locked without reporting recovery failed"); + return Err(Error::Internal); + } + Ok(()) +} + +/// A send and a receive buffer for each of the three lanes. +static mut BUFS: [[[u8; 8]; 2]; 3] = [[[0; 8]; 2]; 3]; + +/// # Safety +/// Called once. Hands out each buffer exactly once. +unsafe fn lanes() -> [Lane; 3] { + // Safety: see function doc; `BUFS` is not touched anywhere else. + let bufs: &'static mut [[[u8; 8]; 2]; 3] = unsafe { &mut *core::ptr::addr_of_mut!(BUFS) }; + let [verifier, reset, events] = bufs; + let make = |index: usize, handle: u32, [send, recv]: &'static mut [[u8; 8]; 2]| { + Lane::new( + index, + AsyncChannelTransport::new(IpcHandle::new(handle), send, recv), + ) + }; + [ + make(VERIFIER, handle::VERIFIER, verifier), + make(RESET, handle::RESET, reset), + make(EVENTS, handle::EVENTS, events), + ] +} + +fn run_all() -> Result<()> { + // Safety: called once. + let mut exec = Exec { + lanes: unsafe { lanes() }, + wd: BootWatchdogs::new(), + booted: [false; N], + seen: heapless::Vec::new(), + verify_cancels: 0, + recoveries: 0, + recovery_failed: false, + }; + // One run per image: the services' scripts count from the start of the + // image. Repeat the run with `--runs_per_test`, not a loop. + scenario(&mut exec) +} + +#[entry] +fn entry() { + let ret = run_all(); + match &ret { + Ok(()) => pw_log::info!("scenario PASSED"), + Err(e) => pw_log::error!("FAILED: status code {}", *e as u32), + } + let _ = syscall::debug_shutdown(ret); + loop {} +} + +#[panic_handler] +fn panic(_info: &core::panic::PanicInfo) -> ! { + let _ = userspace::syscall::debug_shutdown(Err(pw_status::Error::Internal)); + loop {} +}
diff --git a/target/ast10x0/tests/orchestrator/async_sim/scenario.bzl b/target/ast10x0/tests/orchestrator/async_sim/scenario.bzl new file mode 100644 index 0000000..a8ed6f2 --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/scenario.bzl
@@ -0,0 +1,128 @@ +# Licensed under the Apache-2.0 license +# SPDX-License-Identifier: Apache-2.0 + +"""One async_sim scenario: a kernel, its apps, the image and the QEMU test. + +The services decide how they behave (a device's boot script, how long a verify +takes), and they read that from a build flag. So one image is one scenario, +and a failure names itself. Every scenario is the same sources and the same +system config with one `--cfg` set. + +A scenario is a package of its own because the system config names an app by +its target name, and two targets in one package cannot share a name. +""" + +load("@pigweed//pw_kernel/tooling:rust_app.bzl", "rust_app") +load("@pigweed//pw_kernel/tooling:system_image.bzl", "system_image", "system_image_test") +load("@pigweed//pw_kernel/tooling:target_codegen.bzl", "target_codegen") +load("@pigweed//pw_kernel/tooling:target_linker_script.bzl", "target_linker_script") +load("@pigweed//pw_kernel/tooling/panic_detector:rust_binary_no_panics_test.bzl", "rust_binary_no_panics_test") +load("@rules_rust//rust:defs.bzl", "rust_binary") +load("//target/ast10x0:defs.bzl", "TARGET_COMPATIBLE_WITH") + +# Every `--cfg` name a scenario uses. Declared on every build, not just the one +# that sets it, because an undeclared cfg name is a warning and warnings are +# errors here. +SCENARIO_CFGS = [ + "flaky_boot", + "verify_hangs", + "never_boots", +] + +KERNEL_DEPS = [ + "//target/ast10x0:entry", + "@pigweed//pw_kernel/arch/arm_cortex_m:arch_arm_cortex_m", + "@pigweed//pw_kernel/kernel", + "@pigweed//pw_kernel/subsys/console:console_backend", + "@pigweed//pw_kernel/target:target_common", + "@pigweed//pw_kernel/userspace", + "@pigweed//pw_log/rust:pw_log", +] + +def qemu_scenario(name, system_config, target_src, apps, cfgs, test_tags = None): + """Builds one scenario image and the QEMU test that runs it. + + Args: + name: Image name. The test is `<name>_test`. + system_config: Label of the system.json5 filegroup. + target_src: Label of the kernel `target.rs`. + apps: One dict per app, with `name` (which must match the app in the + system config), `src`, and `deps`. + cfgs: `--cfg` names set on every app, from `SCENARIO_CFGS`. + test_tags: Extra tags for the QEMU test. `manual` keeps a scenario that + fails today out of `bazel test //...` while it documents a bug. + """ + for cfg in cfgs: + if cfg not in SCENARIO_CFGS: + fail("unknown scenario cfg {}; add it to SCENARIO_CFGS".format(cfg)) + + rustc_flags = ["--check-cfg=cfg({})".format(c) for c in SCENARIO_CFGS] + rustc_flags += ["--cfg={}".format(c) for c in cfgs] + + target_codegen( + name = "codegen", + arch = "@pigweed//pw_kernel/arch/arm_cortex_m:arch_arm_cortex_m", + system_config = system_config, + target_compatible_with = TARGET_COMPATIBLE_WITH, + ) + + target_linker_script( + name = "linker_script", + system_config = system_config, + tags = ["kernel"], + target_compatible_with = TARGET_COMPATIBLE_WITH, + template = "//target/ast10x0:linker_script_template", + ) + + rust_binary( + name = "target", + srcs = [target_src], + edition = "2024", + tags = ["kernel"], + target_compatible_with = TARGET_COMPATIBLE_WITH, + deps = [ + ":codegen", + ":linker_script", + ] + KERNEL_DEPS, + ) + + app_labels = [] + for app in apps: + rust_app( + name = app["name"], + srcs = [app["src"]], + codegen_crate_name = "app_" + app["name"], + edition = "2024", + rustc_flags = rustc_flags, + tags = ["kernel"], + target_compatible_with = TARGET_COMPATIBLE_WITH, + deps = app["deps"], + ) + app_labels.append(":" + app["name"]) + + system_image( + name = name, + apps = app_labels, + kernel = ":target", + platform = "//target/ast10x0", + system_config = system_config, + tags = ["kernel"], + target_compatible_with = TARGET_COMPATIBLE_WITH, + visibility = ["//visibility:public"], + ) + + # Exclusive: the scenarios race a 50 ms boot watchdog against 20 ms boots + # and similar, on QEMU's wall-clock time. Beside other QEMU runs the guest + # slows down and the margins go. + system_image_test( + name = name + "_test", + image = ":" + name, + tags = ["qemu_only", "exclusive"] + (test_tags or []), + target_compatible_with = TARGET_COMPATIBLE_WITH, + ) + + rust_binary_no_panics_test( + name = name + "_no_panics_test", + binary = ":" + name, + tags = ["kernel"], + )
diff --git a/target/ast10x0/tests/orchestrator/async_sim/system.json5 b/target/ast10x0/tests/orchestrator/async_sim/system.json5 new file mode 100644 index 0000000..2edf3ad --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/system.json5
@@ -0,0 +1,115 @@ +// Licensed under the Apache-2.0 license +// SPDX-License-Identifier: Apache-2.0 + +// AST10x0 async orchestration simulation (QEMU). +// +// orchestrator — the real orchestrator-sm core driven by an async effect +// executor. One thread, one WaitGroup, three channels, none +// of them used with a blocking transact. +// verifier — ServiceLoop; answers a verify request after a delay the +// request names (defer/complete). +// devices — ServiceLoop with two channels: +// reset: release / assert / restore, answered at once +// events: a long-poll; one request parked until a +// simulated device boots. +// +// Wire frames are private to this test (see the three *_main.rs files). +// +// Flash: app images load contiguously from the end of kernel flash +// (0x20300), and kernel RAM starts at 0x60000. Three 128 KB apps would run +// into kernel RAM, so each app gets 64 KB. +{ + arch: { + type: "armv7m", + vector_table_start_address: 0x00000000, + vector_table_size_bytes: 1280, + }, + kernel: { + flash_start_address: 0x00000500, + flash_size_bytes: 129792, + ram_start_address: 0x00060000, + ram_size_bytes: 131072, + }, + apps: [ + { + name: "orchestrator", + flash_size_bytes: 65536, + processes: [ + { + name: "orchestrator_process", + ram_size_bytes: 65536, + objects: [ + { name: "wg", type: "wait_group" }, + { + name: "verifier", + type: "channel_initiator", + handler_process: "verifier_process", + handler_object_name: "ipc", + }, + { + name: "reset", + type: "channel_initiator", + handler_process: "devices_process", + handler_object_name: "reset", + }, + { + name: "events", + type: "channel_initiator", + handler_process: "devices_process", + handler_object_name: "events", + }, + { + type: "thread", + name: "orchestrator_thread", + kernel_stack_size_bytes: 4096, + },], + }, + ], + }, + { + name: "verifier", + flash_size_bytes: 65536, + processes: [ + { + name: "verifier_process", + ram_size_bytes: 32768, + objects: [ + { name: "wg", type: "wait_group" }, + { name: "ipc", type: "channel_handler" }, + { + type: "thread", + name: "verifier_thread", + kernel_stack_size_bytes: 4096, + // Higher than the orchestrator (default, Level15): a + // service runs as soon as a request or its timer is + // ready, instead of waiting out the requester's slice. + priority: "Level20", + },], + }, + ], + }, + { + name: "devices", + flash_size_bytes: 65536, + processes: [ + { + name: "devices_process", + ram_size_bytes: 32768, + objects: [ + { name: "wg", type: "wait_group" }, + { name: "reset", type: "channel_handler" }, + { name: "events", type: "channel_handler" }, + { + type: "thread", + name: "devices_thread", + kernel_stack_size_bytes: 4096, + // Higher than the orchestrator (default, Level15): a + // service runs as soon as a request or its timer is + // ready, instead of waiting out the requester's slice. + priority: "Level20", + },], + }, + ], + }, + ], +}
diff --git a/target/ast10x0/tests/orchestrator/async_sim/target.rs b/target/ast10x0/tests/orchestrator/async_sim/target.rs new file mode 100644 index 0000000..61c848b --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/target.rs
@@ -0,0 +1,44 @@ +// Licensed under the Apache-2.0 license +// SPDX-License-Identifier: Apache-2.0 + +//! Kernel target for the async orchestration simulation. +//! +//! Pass/fail is communicated by the `orchestrator` app calling +//! `syscall::debug_shutdown(Ok(()) | Err(...))`, which lands here and writes +//! the UART sentinel picked up by qemu_runner.py. + +#![no_std] +#![no_main] + +use arch_arm_cortex_m::Arch; +use console_backend::console_backend_write_all; +use entry as _; +use target_common::{declare_target, TargetInterface}; + +pub struct Target {} + +impl TargetInterface for Target { + const NAME: &'static str = "AST10x0 async orchestration simulation"; + + fn main() -> ! { + codegen::start(); + // Sleep rather than spin: a spinning thread at the same priority as + // the app threads delays every wake-up behind it. + loop { + let _ = kernel::sleep_until(Arch, kernel::Instant::MAX); + } + } + + fn shutdown(code: u32) -> ! { + let sentinel: &[u8] = if code == 0 { + b"TEST_RESULT:PASS\n" + } else { + b"TEST_RESULT:FAIL\n" + }; + let _ = console_backend_write_all(sentinel); + #[expect(clippy::empty_loop)] + loop {} + } +} + +declare_target!(Target);
diff --git a/target/ast10x0/tests/orchestrator/async_sim/verifier_main.rs b/target/ast10x0/tests/orchestrator/async_sim/verifier_main.rs new file mode 100644 index 0000000..3e2891d --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/verifier_main.rs
@@ -0,0 +1,143 @@ +// Licensed under the Apache-2.0 license +// SPDX-License-Identifier: Apache-2.0 + +//! Simulated verifier service. +//! +//! How long a check takes is the service's own business, fixed when the image +//! is built: each component has a latency per attempt, and the orchestrator +//! only names the component. The service counts the attempts itself. +//! +//! Request: `[OP_VERIFY, id]` +//! Response: `[status, id]` (status 0 is "authenticated") +//! +//! The reply is deferred. While it is parked the loop keeps running, but the +//! channel is out of the WaitGroup, so a client that cancels still cannot +//! have a new request read until the deferred reply is completed. The +//! scenario's timings account for that. +//! +//! This is a single-channel `ServiceLoop`: while the reply is parked the +//! WaitGroup is empty, and `next` sleeps until the reply is due. + +#![no_main] +#![no_std] + +use app_verifier::handle; +use pw_status::Result; +use userspace::entry; +use userspace::time::{Clock, Duration, Instant, SystemClock}; +use util_service::{Channel, Event, ServiceLoop}; + +const OP_VERIFY: u8 = 1; +const KEY: u32 = 1; +const COMPONENTS: usize = 3; + +/// How long each component's nth verify takes, in milliseconds. The last +/// entry repeats once the list runs out. +/// +/// C1's first verify is slow (210 ms), long enough that C0's 110 ms boot +/// watchdog fires while it is still running, and short enough to finish inside +/// the orchestrator's 300 ms verify deadline. The three gaps are laid out in +/// orchestrator_main.rs, beside `BOOT_WINDOW`. +#[cfg(flaky_boot)] +const LATENCY_MS: [&[u64]; COMPONENTS] = [&[10], &[210, 10], &[10]]; +/// C2's first verify takes 450 ms, past the orchestrator's 300 ms deadline. +/// +/// Two margins decide whether this scenario is steady, with D the deadline +/// (300), H this latency (450) and 10 ms for a normal verify: +/// +/// - `H - D` = 150 ms: the orchestrator must give up and cancel before this +/// reply lands. If it wakes later than that, the reply wins and no verify is +/// ever abandoned. +/// - `2D - (H + 10)` = 140 ms: after the cancel this service cannot read the +/// next request until it finishes this one, so the re-walk's first verify, +/// which starts about D in and has its own D to finish, is answered at +/// H + 10. If H grew past about 2D the restart would time out too. +/// +/// So H has to sit between D and 2D, and widening only one side breaks the +/// other. Scale D and H together to buy more margin. +#[cfg(verify_hangs)] +const LATENCY_MS: [&[u64]; COMPONENTS] = [&[10], &[10], &[450, 10]]; +#[cfg(not(any(flaky_boot, verify_hangs)))] +const LATENCY_MS: [&[u64]; COMPONENTS] = [&[10], &[10], &[10]]; + +/// Milliseconds the `n`th verify of component `id` takes. +fn latency_ms(id: u8, n: u32) -> u64 { + LATENCY_MS + .get(usize::from(id)) + .and_then(|row| { + usize::try_from(n) + .ok() + .and_then(|i| row.get(i)) + .or_else(|| row.last()) + }) + .copied() + .unwrap_or(10) +} + +fn serve() -> Result<()> { + let mut channels = [Channel::new(handle::IPC)]; + let mut service = ServiceLoop::new(handle::WG, &mut channels)?; + let mut buf = [0u8; 8]; + // How many verifies each component has been asked for. + let mut calls = [0u32; COMPONENTS]; + // When the parked reply is due, and for which component. + let mut parked: Option<(Instant, u8)> = None; + + loop { + let deadline = parked.map_or(Instant::MAX, |(due, _)| due); + match service.next(deadline, &mut buf)? { + Event::Request { channel, len } => { + let req = buf.get(..len).unwrap_or(&[]); + if let [OP_VERIFY, id] = *req { + match service.defer(channel, KEY) { + Ok(()) => { + // One millisecond late on purpose: the wake must + // not land a tick before the deadline. + let n = calls.get_mut(usize::from(id)).map_or(0, |c| { + let was = *c; + *c = c.saturating_add(1); + was + }); + let after = Duration::from_millis(latency_ms(id, n).saturating_add(1)); + let due = SystemClock::now() + .checked_add_duration(after) + .unwrap_or(Instant::MAX); + parked = Some((due, id)); + } + Err(_) => { + pw_log::error!("verifier: defer failed"); + let _ = service.reply(channel, &[1, id]); + } + } + } else { + let _ = service.reply(channel, &[1, 0]); + } + } + Event::Deadline => { + if let Some((due, id)) = parked + && SystemClock::now() >= due + { + parked = None; + // Fails if the client gave up (cancelled) while the reply + // was parked. The channel is served again either way. + if service.complete(KEY, &[0, id]).is_err() { + pw_log::warn!("verifier: late reply dropped (client cancelled)"); + } + } + } + Event::Source(_) => {} + } + } +} + +#[entry] +fn entry() { + if let Err(e) = serve() { + pw_log::error!("verifier: loop exited, status {}", e as u32); + } +} + +#[panic_handler] +fn panic(_info: &core::panic::PanicInfo) -> ! { + loop {} +}
diff --git a/target/ast10x0/tests/orchestrator/async_sim/verify_hangs/BUILD.bazel b/target/ast10x0/tests/orchestrator/async_sim/verify_hangs/BUILD.bazel new file mode 100644 index 0000000..cfa704e --- /dev/null +++ b/target/ast10x0/tests/orchestrator/async_sim/verify_hangs/BUILD.bazel
@@ -0,0 +1,19 @@ +# Licensed under the Apache-2.0 license +# SPDX-License-Identifier: Apache-2.0 + +# C2's first verify outlives the orchestrator's deadline. Passes when the +# executor cancels it and reports a failed verification, the verifier's late +# reply is dropped, and the re-walk ends in Ready. +# +# bazel test --config=virt_ast10x0 //target/ast10x0/tests/orchestrator/async_sim/verify_hangs:verify_hangs_qemu_test + +load("//target/ast10x0/tests/orchestrator/async_sim:apps.bzl", "APPS", "SYSTEM_CONFIG", "TARGET_SRC") +load("//target/ast10x0/tests/orchestrator/async_sim:scenario.bzl", "qemu_scenario") + +qemu_scenario( + name = "verify_hangs_qemu", + apps = APPS, + cfgs = ["verify_hangs"], + system_config = SYSTEM_CONFIG, + target_src = TARGET_SRC, +)