orchestrator: Drop verdicts for a component already isolated A failure verdict or a corruption report can be in flight from before the cascade gated its component. VerificationFailed and CorruptionDetected now return early when the component is gated. Recovering it re-walks the chain for a device that stays held, and on exhaustion a Required one locks the platform down over a cascade that was already contained, which contradicts the rule that a non-Required cascade never reaches lockdown. property_isolation_is_sticky_under_random_sequences guards both: after ReportIsolated(id), nothing in the rest of the run emits ReleaseReset or RecoverComponent for that id. The cursor invariant the walk depends on is written down on the field, since that property is what guards it. Assisted-by: Claude:claude-opus-5 Signed-off-by: Christina Quast <christina.quast@9elements.com>
diff --git a/services/orchestrator/sm/src/lib.rs b/services/orchestrator/sm/src/lib.rs index 1e1e85a..4ef2570 100644 --- a/services/orchestrator/sm/src/lib.rs +++ b/services/orchestrator/sm/src/lib.rs
@@ -186,6 +186,22 @@ /// default. `E` must be at least `2 * N + 2` (enforced in [`Rot::new`]). pub struct Rot<const N: usize, const E: usize> { chain: heapless::Vec<(ComponentId, ComponentAttrs), N>, + /// Index into `chain` of the component currently under verification, or the + /// past-the-end sentinel `chain.len()` once the walk is done. Only + /// `chain[cursor]` can be released: a `VerificationPassed` for any other id + /// is stale or out of turn and is dropped. + /// + /// While the walk runs (`PreSupervision` and `AwaitingReady`) the cursor + /// never points at a gated component. Gating the component under + /// verification therefore has to move the cursor past it, which + /// [`handle_corruption_advancing`](Self::handle_corruption_advancing) does: + /// a verdict already in flight then fails the `chain[cursor]` check and is + /// dropped instead of releasing a component the cascade just isolated. + /// `property_isolation_is_sticky_under_random_sequences` guards this. + /// + /// `Recovering` is the exception: `VerificationFailed` leaves the cursor on + /// the failed component and a corruption report can gate it there. Entry to + /// `PreSupervision` re-walks from 0, which restores the invariant. cursor: u8, /// One record per chain component (parallel to `chain` by index). Each /// [`ComponentStatus`] holds the component's service `lifecycle` (`Isolated` @@ -432,18 +448,24 @@ /// already found corrupt; `Required` → recover first (the halt-on-exhaustion /// decision happens later in `Recovering`). fn handle_corruption(&mut self, id: ComponentId, ctx: &mut Sink<E>) -> Outcome { + // Already isolated: it is held in reset and was reported. Recovering + // it restores a component the re-walk skips, and on exhaustion a + // `Required` one locks the platform down over a cascade that was + // already contained. + if self.is_gated(id) { + return Outcome::Handled; + } match self.gate_by_policy(ctx, id) { Gating::Gated => Outcome::Handled, Gating::NotGated => Outcome::Transition(State::Recovering(id)), } } - /// `CorruptionDetected` for the two states that release off `chain[cursor]`, - /// `PreSupervision` and `AwaitingReady`. Gates by policy, then moves the - /// cursor off the component under verification if the cascade gated it, so a - /// verdict already in flight is a mismatch and gets dropped instead of - /// releasing an isolated component. A `Required` corruption gates nothing - /// and returns `Transition(Recovering)` unchanged. + /// `CorruptionDetected` for `PreSupervision` and `AwaitingReady`, the two + /// states that release off `chain[cursor]`. Gates by policy, then keeps the + /// `cursor` invariant by moving it past the component under verification + /// when the cascade gated it. A `Required` corruption gates nothing and + /// returns `Transition(Recovering)` unchanged. /// /// Not called from `handle_supervising`: `Recovering`'s cursor is stale /// (`VerificationFailed` left it on the failed component), so advancing @@ -554,6 +576,12 @@ } } Event::VerificationFailed(id) => { + // A verdict from before the gating, for a component the + // cascade has since isolated: recovering it re-walks the + // chain for a device that stays held. + if self.is_gated(*id) { + return Outcome::Handled; + } // Recovery is attempted first for every failure, regardless // of the component's recovery-failure policy (CSA: recover // first, classify only once retries are exhausted). @@ -646,6 +674,11 @@ } } Event::VerificationFailed(id) => { + // Same in-flight verdict as above: an isolated component + // does not enter recovery. + if self.is_gated(*id) { + return Outcome::Handled; + } // Recovery is attempted first for every failure, regardless // of the component's recovery-failure policy. Outcome::Transition(State::Recovering(*id))
diff --git a/services/orchestrator/sm/src/tests.rs b/services/orchestrator/sm/src/tests.rs index d087fdd..ce0b0ec 100644 --- a/services/orchestrator/sm/src/tests.rs +++ b/services/orchestrator/sm/src/tests.rs
@@ -1070,7 +1070,6 @@ assert!(effects.contains(&Effect::ReportIsolated(id))); assert!(!effects.contains(&Effect::ReleaseReset(id))); } - // The walk carries on past the isolated pair. assert!(effects.contains(&Effect::ReadFirmware(C3))); assert!(effects.contains(&Effect::VerifyFirmware(C3))); } @@ -1181,16 +1180,16 @@ assert_eq!(state, State::Ready); // The cursor never left C1, so its verdict still counts. assert!(effects.contains(&Effect::ReleaseReset(C1))); - // C2 is gated before its turn and the walk skips it. assert!(effects.contains(&Effect::ReportIsolated(C2))); assert!(!effects.contains(&Effect::ReleaseReset(C2))); assert!(!effects.contains(&Effect::ReadFirmware(C2))); } /// The cascade gates the component the `AwaitingReady` slot waits on. The slot -/// keeps naming it, which is harmless: `gate_one` cleared its `awaiting_boot`, -/// a late `ComponentReady` releases nothing, and the walk reaches `Ready` -/// through the cursor rather than through readiness. +/// keeps naming it, which changes nothing: `gate_one` cleared its +/// `awaiting_boot`, a late `ComponentReady` releases nothing, and the walk +/// reaches `Ready` when the cursor reaches the end of the chain, not when +/// C0's `ComponentReady` arrives. #[test] fn gating_the_awaited_component_does_not_stall_the_walk() { let (effects, state) = drive( @@ -1217,11 +1216,86 @@ assert!(effects.contains(&Effect::ReportIsolated(id))); } assert!(!effects.contains(&Effect::ReleaseReset(C1))); - // The walk moved past the isolated pair and finished on C2. assert!(effects.contains(&Effect::ReadFirmware(C2))); assert!(effects.contains(&Effect::ReleaseReset(C2))); } +/// A failure verdict in flight when the cascade gated the component does not +/// put it into recovery. +#[test] +fn late_verification_failed_for_gated_component_is_dropped() { + let (effects, state) = drive( + chain(&[ + (C0, ComponentAttrs::active_required()), + (C1, ComponentAttrs::passive_cascading()), + (C2, ComponentAttrs::passive_required().with_depends_on(C1)), + (C3, ComponentAttrs::passive_required()), + ]), + &[ + BOOT, + Event::VerificationPassed(C0), // releases C0, cursor on C1 + Event::CorruptionDetected(C1), // gates C1 -> C2, cursor moves to C3 + Event::VerificationFailed(C1), // in flight before the gating + ], + ); + assert_ne!(state, State::Recovering(C1), "isolated C1 entered recovery"); + assert!( + !effects.contains(&Effect::RecoverComponent { id: C1, attempt: 0 }), + "RecoverComponent fired for isolated C1" + ); +} + +/// A contained cascade does not lock the platform down through the `Required` +/// component it held. Three corruption reports for the isolated C1 would +/// otherwise exhaust its retries, and `gate_by_policy` would read C1's own +/// `Required` policy and escalate. MAX_RETRY is 3. +#[test] +fn contained_cascade_does_not_lock_down_via_its_required_dependent() { + let (effects, state) = drive( + chain(&[ + (C0, ComponentAttrs::passive_cascading()), + (C1, ComponentAttrs::passive_required().with_depends_on(C0)), + ]), + &[ + BOOT, + Event::CorruptionDetected(C0), // gates C0, cascade-holds C1 + Event::CorruptionDetected(C1), // isolated C1 into recovery, attempt 0 + Event::Restored(C1), + Event::CorruptionDetected(C1), // attempt 1 + Event::Restored(C1), + Event::CorruptionDetected(C1), // attempt 2 + Event::Restored(C1), // retries exhausted + ], + ); + assert!( + !effects.contains(&Effect::LatchLockdown), + "contained cascade reached lockdown, state {state:?}" + ); +} + +/// A corruption report for a component the cascade already isolated is +/// dropped. +#[test] +fn corruption_report_for_an_isolated_component_is_dropped() { + let (effects, state) = drive( + chain(&[ + (C0, ComponentAttrs::passive_cascading()), + (C1, ComponentAttrs::passive_required().with_depends_on(C0)), + ]), + &[ + BOOT, + Event::CorruptionDetected(C0), // gates C0, cascade-holds C1 + Event::CorruptionDetected(C1), // C1 is already isolated + ], + ); + assert!(effects.contains(&Effect::ReportIsolated(C1))); + assert_ne!(state, State::Recovering(C1), "isolated C1 entered recovery"); + assert!( + !effects.contains(&Effect::RecoverComponent { id: C1, attempt: 0 }), + "RecoverComponent fired for cascade-held C1" + ); +} + /// Runtime corruption under a non-`Required` policy reports too. This path /// never enters recovery at all, so without its own report the isolation would /// be silent. @@ -2193,6 +2267,50 @@ } } +/// Isolation is sticky: once a component is reported isolated, nothing in the +/// rest of the run takes it out of reset or hands it to recovery. The +/// verify-before-release property misses the recovery half of that, because +/// recovery re-verifies before releasing. +#[test] +fn property_isolation_is_sticky_under_random_sequences() { + const RUNS: u64 = 20_000; + const MAX_LEN: u32 = 24; + + let palette = [C0, C1, C2, C3]; + + for seed in 0..RUNS { + let mut rng = SplitMix64(seed.wrapping_mul(0xD1B5_4A32_D192_ED03).wrapping_add(1)); + + let ch = random_chain(&mut rng); + let mut orch = + Orchestrator::<CAPACITY, ECAP>::new(ch.try_into().expect("valid chain"), MAX_RETRY); + let mut platform = Recorder::new(); + + orch.dispatch(&mut platform, BOOT); + let len = 1 + rng.below(MAX_LEN); + for _ in 0..len { + let event = random_event(&mut rng, &palette); + orch.dispatch(&mut platform, event); + } + + let mut isolated = [false; CAPACITY]; + for effect in &platform.recorded { + match effect { + Effect::ReportIsolated(id) => isolated[id.get() as usize] = true, + Effect::ReleaseReset(id) => assert!( + !isolated[id.get() as usize], + "seed {seed}: released {id:?} after reporting it isolated", + ), + Effect::RecoverComponent { id, .. } => assert!( + !isolated[id.get() as usize], + "seed {seed}: recovered {id:?} after reporting it isolated", + ), + _ => {} + } + } + } +} + #[test] fn property_verify_before_release_holds_under_random_sequences() { const RUNS: u64 = 20_000;