docs: add orchestrator state machine walkthrough and transitions Add two new documents — a narrative walkthrough of the orchestrator state machine (orchestrator-sm-walkthru.md) and a reference transitions table (orchestrator-sm-transitions.md) — and expand the existing state machine document with superstate handler descriptions. Register both new pages in SUMMARY.md.
diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index b7c13b5..2aae5a3 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md
@@ -35,3 +35,5 @@ * [Orchestrator](./design/orchestrator/orchestrator-overview.md) * [Verification Model](./design/orchestrator/orchestrator-model.md) * [State Machine](./design/orchestrator/orchestrator-machine.md) + * [State Machine Walkthrough](./design/orchestrator/orchestrator-sm-walkthru.md) + * [State Machine Transitions](./design/orchestrator/orchestrator-sm-transitions.md)
diff --git a/docs/src/design/orchestrator/orchestrator-machine.md b/docs/src/design/orchestrator/orchestrator-machine.md index 68c7f0c..bac9f44 100644 --- a/docs/src/design/orchestrator/orchestrator-machine.md +++ b/docs/src/design/orchestrator/orchestrator-machine.md
@@ -15,15 +15,12 @@ VerifyingPlatform --> VerifyingPlatform : VerificationPassed [more, Passive]<br/>/ ReleaseReset · ReadFirmware · VerifyFirmware VerifyingPlatform --> AwaitingReady : VerificationPassed [more, Active]<br/>/ ReleaseReset · ReadFirmware · VerifyFirmware VerifyingPlatform --> Ready : VerificationPassed [chain done]<br/>/ ReleaseReset - VerifyingPlatform --> VerifyingPlatform : VerificationFailed [Isolable or Cascading]<br/>(skip — held in reset) - VerifyingPlatform --> Recovering : VerificationFailed [Required]<br/>/ RestoreGoldenImage + VerifyingPlatform --> Recovering : VerificationFailed (any policy)<br/>/ RestoreGoldenImage AwaitingReady --> AwaitingReady : VerificationPassed [more]<br/>/ ReleaseReset · ReadFirmware · VerifyFirmware AwaitingReady --> Ready : ComponentReady [chain done or cursor past end] AwaitingReady --> AwaitingReady : ComponentReady [more] - AwaitingReady --> AwaitingReady : VerificationFailed [Isolable or Cascading, iRoT pending] - AwaitingReady --> Ready : VerificationFailed [Isolable or Cascading, no iRoT pending, chain done] - AwaitingReady --> Recovering : VerificationFailed [Required]<br/>/ RestoreGoldenImage + AwaitingReady --> Recovering : VerificationFailed (any policy)<br/>/ RestoreGoldenImage AwaitingReady --> Recovering : Timeout(id) [id == awaiting]<br/>/ RestoreGoldenImage state Operational { @@ -36,8 +33,9 @@ AwaitingReady --> Recovering : CorruptionDetected<br/>/ RestoreGoldenImage } - Recovering --> VerifyingPlatform : Restored [retry < max_retry] - Recovering --> Locked : Restored [retry ≥ max_retry]<br/>(self-emits RecoveryFailed)<br/>/ LatchLockdown + Recovering --> VerifyingPlatform : Restored [retry < max_retry]<br/>(re-verify) + Recovering --> VerifyingPlatform : Restored [retry ≥ max_retry, Isolable/Cascading]<br/>/ AssertReset (skip — held) + Recovering --> Locked : Restored [retry ≥ max_retry, PlatformHalt]<br/>(self-emits RecoveryFailed) / LatchLockdown Locked --> Locked : (terminal — all events ignored) ``` @@ -53,11 +51,11 @@ | Field | Type | Purpose | |---|---|---| | `chain` | `Vec<(ComponentId, ComponentAttrs), N>` | Ordered trust chain, supplied by the shell at construction time. Never mutated after build. | -| `cursor` | `u8` | Index of the component currently under verification. Reset to 0 on every `VerifyingPlatform` entry. Advances on each `VerificationPassed` (and on `Isolable`/`Cascading` `VerificationFailed`, or cascade-skip) via `Outcome::Handled`. | -| `held` | `Vec<ComponentId, N>` | Set of component IDs held in reset due to `Cascading` failure or cascade-skip (`Isolable` failures do not populate this set). Checked before emitting `ReadFirmware` for each new component to evaluate `depends_on`. Cleared on `VerifyingPlatform` entry alongside `cursor`. | -| `failed` | `Option<ComponentId>` | The component that triggered the current recovery episode; `None` while healthy. Set on `Required` `VerificationFailed`, `Timeout`, or `CorruptionDetected`. | -| `retry_count` | `u8` | Number of consecutive failed restore attempts. Cleared to 0 in `Ready`'s entry action — consecutive only (INV7). | -| `max_retry` | `u8` | Shell-chosen ceiling for `retry_count`. When `retry_count >= max_retry` the machine self-emits `RecoveryFailed` instead of re-walking the chain. | +| `cursor` | `u8` | Index of the component currently under verification. Reset to 0 on every `VerifyingPlatform` entry. Advances on each `VerificationPassed`, and past any component in `held` (skipped without verification), via `Outcome::Handled`. | +| `held` | `Vec<ComponentId, N>` | Components skipped because their recovery was **exhausted**: an `Isolable` component, or a `Cascading` component plus its `depends_on` dependents. Not verified during the walk — held in reset, cursor advances past them. Populated in `Recovering` when `retry_count` reaches `max_retry`; persists across re-walks; cleared on `Ready` entry. | +| `failed` | `Option<ComponentId>` | The component whose recovery episode is in progress; `None` while healthy. Set on any `VerificationFailed`, `Timeout`, or `CorruptionDetected` of a managed component. | +| `retry_count` | `u8` | Number of consecutive failed restore attempts in the current recovery episode. Cleared to 0 in `Ready`'s entry action — consecutive only (INV7). | +| `max_retry` | `u8` | Shell-chosen ceiling for `retry_count`. When `retry_count >= max_retry` recovery is **exhausted** and the failed component's recovery-failure policy (`Isolable`/`Cascading`/`PlatformHalt`) is applied. | | `awaiting` | `Option<ComponentId>` | The `Active` component whose iRoT readiness is currently outstanding. `Some` only while in `AwaitingReady`; `None` everywhere else (INV9). | The effect buffer is deliberately **absent** from `Rot`. Effects flow through the @@ -99,18 +97,24 @@ rather than a self-transition — a self-transition would re-run the entry action and reset the cursor. -**Entry action**: reset `cursor` to 0, `awaiting` to `None`, emit -`ReadFirmware(chain[0])` + `VerifyFirmware(chain[0])`. +**Entry action**: reset `cursor` to 0, set `awaiting` to `None`, emit +`ReadFirmware` + `VerifyFirmware` for the first component **not** in `held` +(`held` components stay in reset and are skipped). `held` is *not* cleared here — +it persists across re-walks so exhausted components are not re-verified. | Event | Guard | Effects | Next state | |---|---|---|---| | `VerificationPassed(id)` | more, current `Passive` | `ReleaseReset` · `ReadFirmware(next)` · `VerifyFirmware(next)` | `Handled` (cursor ++) | | `VerificationPassed(id)` | more, current `Active` | `ReleaseReset` · `ReadFirmware(next)` · `VerifyFirmware(next)` | `AwaitingReady` (awaiting = Some(id)) | | `VerificationPassed(id)` | chain done | `ReleaseReset(id)` | `Ready` | -| `VerificationFailed(id)` | `attrs.failure_policy == Required` | — | `Recovering` (failed = Some(id)) | -| `VerificationFailed(id)` | `Isolable` | — | `Handled` (skip; cursor ++; if chain done → `Ready`) | -| `VerificationFailed(id)` | `Cascading` | — | `Handled` (add to `held`; cascade-skip dependents; cursor ++; if chain done → `Ready`) | -| anything else | — | — | `Outcome::Super` → `Operational` | +| `VerificationFailed(id)` | — | — | `Recovering` (failed = Some(id)) — recovery is attempted first, regardless of the component's recovery-failure policy | +| anything else | — | — | `Outcome::Super` (top level — discarded) | + +When advancing the cursor, any component in `held` is skipped without +verification — it stays in reset and no `ReadFirmware`/`VerifyFirmware` is emitted +for it. The recovery-failure policy (`Isolable`/`Cascading`/`PlatformHalt`) is +**not** consulted here; it is applied later, in `Recovering`, only if the restore +attempts are exhausted. --- @@ -133,11 +137,7 @@ | `VerificationPassed(id)` | chain done | `ReleaseReset(id)` | `Ready` | | `Timeout(id)` | `id == awaiting` | — | `Recovering` (failed = Some(id), awaiting = None) | | `Timeout(id)` | `id != awaiting` | — | `Handled` (stale — ignore) | -| `VerificationFailed(id)` | `attrs.failure_policy == Required` | — | `Recovering` (failed = Some(id), awaiting = None) | -| `VerificationFailed(id)` | `Isolable`, iRoT pending | — | `Handled` (skip; cursor ++) | -| `VerificationFailed(id)` | `Isolable`, no iRoT pending, chain done | — | `Ready` | -| `VerificationFailed(id)` | `Cascading`, iRoT pending | — | `Handled` (add to `held`; cascade-skip dependents; cursor ++) | -| `VerificationFailed(id)` | `Cascading`, no iRoT pending, chain done | — | `Ready` | +| `VerificationFailed(id)` | — | — | `Recovering` (failed = Some(id), awaiting = None) — recovery attempted first | | anything else | — | — | `Outcome::Super` → `Operational` | `ComponentReady` and `VerificationPassed` are independent and may arrive in @@ -154,7 +154,8 @@ and corruption events. **Entry action**: reset `retry_count` to 0 (makes the cap count *consecutive* -failures — INV7). +failures — INV7), and clear `held` and `failed` (a clean boot ends any recovery +episode and the skip set). | Event | Guard | Effects | Next state | |---|---|---|---| @@ -189,18 +190,42 @@ | Event | Guard | Effects | Next state | |---|---|---|---| -| `Restored(_)` | `retry_count + 1 < max_retry` | — | `VerifyingPlatform` (re-walk from top) | -| `Restored(_)` | `retry_count + 1 >= max_retry` | `Effect::Emit(RecoveryFailed)` | `Handled` (orchestrator queues `RecoveryFailed` next — INV7) | +| `Restored(_)` | `retry_count + 1 < max_retry` | — | `VerifyingPlatform` (re-verify — the restored image may pass) | +| `Restored(_)` | cap reached, `failed` `Isolable` | `AssertReset(failed)` | `VerifyingPlatform` (recovery exhausted: add `failed` to `held`, clear `failed`; the re-walk skips it) | +| `Restored(_)` | cap reached, `failed` `Cascading` | `AssertReset(failed)` · `AssertReset(dependent…)` | `VerifyingPlatform` (recovery exhausted: add `failed` + `depends_on` dependents to `held`, clear `failed`) | +| `Restored(_)` | cap reached, `failed` `PlatformHalt` | `Effect::Emit(RecoveryFailed)` | `Handled` (orchestrator queues `RecoveryFailed` next — INV7) | | `RecoveryFailed` | — | — | `Locked` | | anything else | — | — | `Outcome::Super` → `Operational` | +("cap reached" = `retry_count + 1 >= max_retry`.) + +**Two-stage recovery (CSA-aligned).** A verification failure never skips a +component outright. Every failure — during initial boot or a re-walk — first +brings the machine here, to `Recovering`, which restores the failed component's +recovery region and re-verifies. Only when the restore attempts are *exhausted* +(`retry_count` reaches `max_retry` and the restored image still fails) does the +component's **recovery-failure policy** decide what happens next: + +- `Isolable` — skip just this component; hold it in reset (`held`) and continue + booting the rest of the platform. +- `Cascading` — skip this component *and* its `depends_on` dependents; continue + booting the remainder. +- `PlatformHalt` — stop entirely: self-emit `RecoveryFailed`, which drives the + machine to `Locked`. + +This mirrors the CSA Boot Sequence **Recovery Policy**: recovery (region restore) +is attempted for *every* failed device first, and the `Isolable`/`Cascading`/ +`Platform-halt` classification applies only *after* a recovery attempt itself +fails. + `Effect::Emit(RecoveryFailed)` is the *feedback-as-data* mechanism. It is easiest to understand by asking why the machine doesn't just jump straight to `Locked` -when the retry cap is hit. +when a `PlatformHalt` component's retry cap is hit. -When the last restore attempt fails, the machine has a decision to make: give up -and lock down. It could act on that decision silently, transitioning directly -from `Recovering` to `Locked` inside the handler. Instead it does something that +When a `PlatformHalt` component's last restore attempt fails, the machine has a +decision to make: give up and lock down. It could act on that decision silently, +transitioning directly from `Recovering` to `Locked` inside the handler. Instead +it does something that looks indirect at first: it emits `RecoveryFailed` as an *effect* — a piece of data saying "a follow-up event named `RecoveryFailed` should happen next" — and returns. The orchestrator sees that effect, puts `RecoveryFailed` at the front of @@ -223,9 +248,10 @@ (NIST SP 800-193) — requires that trust be re-established end-to-end before the platform is considered healthy again. The CSA document does not prescribe the exact recovery sequencing, but the re-walk implements the spirit of that -principle. `Isolable` and `Cascading` components that fail during the re-walk are skipped -(held in reset) as during initial boot; they are re-released only if they pass -`VerificationPassed` in the new walk. +principle. Components already in `held` — those whose recovery was exhausted +under an `Isolable` or `Cascading` policy — are skipped during the re-walk: they +stay in reset and are not re-verified. Every other component is re-verified from +scratch, and a fresh failure restarts the two-stage recovery for that component. --- @@ -243,9 +269,9 @@ `Ready`, `Updating`, `Recovering`, and `AwaitingReady` share this superstate. When a leaf state returns `Outcome::Super`, `statig` calls the superstate handler. -| Event | Effects | Next state | -|---|---|---| -| `AttestationChallenge` | `SignAttestation` | `Handled` (no transition — INV6) | +| Event | Guard | Effects | Next state | +|---|---|---|---| +| `AttestationChallenge` | — | `SignAttestation` | `Handled` (no transition — INV6) | | `CorruptionDetected(id)` | `attrs.required == true` | — | `Recovering` (failed = Some(id) — INV5) | | `CorruptionDetected(id)` | `attrs.required == false` | `AssertReset(id)` | `Handled` (component gated; machine stays in current state) | | anything else | — | — | `Outcome::Super` (discarded) |
diff --git a/docs/src/design/orchestrator/orchestrator-sm-transitions.md b/docs/src/design/orchestrator/orchestrator-sm-transitions.md new file mode 100644 index 0000000..96c68e3 --- /dev/null +++ b/docs/src/design/orchestrator/orchestrator-sm-transitions.md
@@ -0,0 +1,309 @@ +# State Machine Transitions + +This document narrates **every transition** in the orchestrator state machine, +one small section at a time. Where the [State Machine](./orchestrator-machine.md) +reference presents the transitions as tables and the +[Walkthrough](./orchestrator-sm-walkthru.md) tells the story end to end, this +document is the per-edge companion: each transition gets its own short prose +entry describing what triggers it, what guard it depends on, what effects it +emits, and where it lands. + +Transitions are grouped by their **source state**. Within each state, the +"anything else" fall-through is described last. Throughout, *the platform* is the +surrounding runtime that executes the core's effects and delivers events back to +it; the core itself never touches hardware. + +--- + +## From `PowerOnReset` + +The machine's initial state. It waits for the platform's first event, which is +always `PowerGood`, carrying the result of the eRoT's power-on self-check. + +### `PowerGood(Provisioned)` → `VerifyingPlatform` + +The eRoT is provisioned and passed its own self-verification, so it is entitled +to vouch for the rest of the platform. The machine leaves the gate and begins +walking the trust chain. No effects are emitted by the transition itself; the +`VerifyingPlatform` entry action starts the first verification. + +### `PowerGood(Unprovisioned)` → `Locked` + +The eRoT has no provisioning, so there is no reference to verify components +against. Rather than proceed blindly, the machine goes straight to `Locked` and +latches lockdown. This is a permanent dead end for this power cycle. + +### `PowerGood(SelfVerificationFailed)` → `Locked` + +The eRoT's own integrity check failed. Because the entire chain of trust hangs +off the anchor, an untrustworthy anchor must not release anything downstream. The +machine locks down immediately. + +### anything else → discarded + +`PowerOnReset` sits outside the `Operational` superstate, so any event other than +`PowerGood` falls through to the top level and is discarded. The machine does not +answer attestation or act on corruption before it has even begun verifying. + +--- + +## From `VerifyingPlatform` + +Walks the trust chain component by component. The entry action points the cursor +at the first component not already in `held` and asks the platform to read and +verify its firmware. The transitions below react to the platform's verdicts. + +### `VerificationPassed` [more components, current is `Passive`] → `VerifyingPlatform` (self) + +The current component is a symbiont device with no root of trust of its own, and +its single eRoT-side check just passed. The machine releases it (`ReleaseReset`), +asks the platform to read and verify the next component +(`ReadFirmware` · `VerifyFirmware`), and advances the cursor — all while staying +in `VerifyingPlatform`. This is the self-loop that rolls the walk forward through +symbiont devices. It uses `Outcome::Handled` rather than a real self-transition so +the entry action does not re-run and reset the cursor. + +### `VerificationPassed` [more components, current is `Active`] → `AwaitingReady` + +The current component is a SoC with its own integrated iRoT, and its eRoT-side +check passed. The machine releases it, speculatively starts the *next* +component's eRoT check, records the component in `awaiting`, and moves to +`AwaitingReady` to wait for the component's own root of trust to report in. The +release plus the next read/verify are emitted here; the wait happens in the +destination state. + +### `VerificationPassed` [chain done] → `Ready` + +The component that just passed was the last one in the chain. The machine +releases it and transitions to `Ready`: every component has been verified and +released, so the platform is up and the trust chain is established. + +### `VerificationFailed` (any policy) → `Recovering` + +A component's firmware failed its eRoT-side check. Regardless of the component's +recovery-failure policy, the machine records it in `failed` and enters +`Recovering` to attempt restoration. The component is **not** skipped here — it is +held in reset (never released, so it never runs unverified code) and given a +recovery attempt first. The `Isolable`/`Cascading`/`PlatformHalt` decision is +deferred until recovery has actually failed. + +### held components → skipped (no transition) + +Not an event-driven edge: while advancing the cursor, any component already in +`held` (one whose recovery was previously exhausted) is skipped without +verification. No `ReadFirmware`/`VerifyFirmware` is emitted for it and it stays in +reset; the cursor simply moves past it to the next candidate. + +### anything else → discarded + +`VerifyingPlatform` also sits outside `Operational`, so unrelated events fall +through to the top level and are discarded. Attestation challenges and corruption +reports are not serviced during the initial chain walk. + +--- + +## From `AwaitingReady` + +Reached when an `Active` component clears its eRoT check. The machine waits here +for the component's iRoT to signal readiness, while the next component's +speculative eRoT check may still be in flight. Two things must resolve — the +awaited `ComponentReady` and the pending `VerificationPassed` — and they can +arrive in either order. + +### `ComponentReady` [id ≠ `awaiting`] → `AwaitingReady` (self) + +The readiness signal is for some component other than the one being awaited — a +stale or spurious report. The machine ignores it (`Outcome::Handled`) and stays +put, so a late or duplicated signal can never push the walk forward incorrectly. + +### `ComponentReady` [id = `awaiting`, cursor in bounds] → `AwaitingReady` (self) + +The awaited component's iRoT has come up. The machine clears `awaiting` to record +that the readiness gate is satisfied, but stays in `AwaitingReady` because the +next component's eRoT verdict is still outstanding. + +### `ComponentReady` [id = `awaiting`, cursor past end] → `Ready` + +The awaited component's iRoT came up and there is nothing left to verify — the +cursor has already advanced past the end of the chain (the last component was +skipped or resolved). With both gates now clear, the machine transitions to +`Ready`. + +### `VerificationPassed` [more components] → `AwaitingReady` (self) + +The speculative eRoT check for the next component passed. The machine releases +that component, starts reading and verifying the one after it, and advances the +cursor — mirroring the `VerifyingPlatform` walk — while remaining in +`AwaitingReady` because it may still be waiting on an iRoT readiness signal. + +### `VerificationPassed` [chain done] → `Ready` + +The speculative check resolved and it was the last component in the chain. The +machine releases it and transitions to `Ready`; the walk is complete. + +### `Timeout` [id = `awaiting`] → `Recovering` + +The platform's boot-progress watchdog fired: the awaited component did not report +readiness within its window. The machine treats this as a verification failure — +records the component in `failed`, clears `awaiting`, and enters `Recovering`. +This realizes the CSA boot-progress checkpointing mechanism, where a missed +checkpoint is treated as a failure that initiates recovery. + +### `Timeout` [id ≠ `awaiting`] → `AwaitingReady` (self) + +A watchdog fired for a component the machine is not currently awaiting — stale. +The machine ignores it and stays put. + +### `VerificationFailed` (any policy) → `Recovering` + +The speculative eRoT check for the next component failed. As during the initial +walk, recovery is attempted first: the machine records the component in `failed`, +clears `awaiting` (abandoning the in-flight readiness wait), and enters +`Recovering`. + +### anything else → `Operational` + +`AwaitingReady` is one of the four operational states, so unrelated events fall +through to the `Operational` superstate — which answers attestation challenges +and acts on corruption reports even while the platform is still coming up. + +--- + +## From `Ready` + +Steady state: the whole chain is verified and released. The entry action clears +the recovery bookkeeping (`retry_count`, `held`, `failed`), since arriving here +means the platform booted clean. + +### `UpdateRequest` → `Updating` + +The platform has requested a firmware update. The machine transitions to +`Updating`, whose entry action begins authenticating and staging the new image. + +### anything else → `Operational` + +Everything else `Ready` does — answering attestation, handling corruption — is +inherited from the `Operational` superstate via fall-through. + +--- + +## From `Updating` + +An update is in progress. The entry action emits `AuthenticateUpdate` and +`StageUpdate`; the machine then waits for the platform's verdict. + +### `UpdateVerified` → `Ready` + +The staged image authenticated successfully. The machine emits `ActivateUpdate` +to switch to the new image and returns to `Ready`. + +### `UpdateRejected` → `Ready` + +The staged image failed authentication. The machine emits `DiscardStaged` to +throw it away and returns to `Ready`, continuing to run the image it already had. +A rejected update is deliberately **not** treated as corruption — nothing trusted +was damaged, so there is no reason to enter recovery. + +### anything else → `Operational` + +Attestation and corruption handling during an update come from the `Operational` +superstate. + +--- + +## From `Recovering` + +Attempting to restore a failed or corrupted component. The entry action emits +`RestoreGoldenImage(failed)`, which the platform applies to the failed +component's entire recovery region (all components sharing its `RegionId`). The +transitions below fire on `Restored` and branch on how many attempts remain and, +once exhausted, on the component's recovery-failure policy. + +### `Restored` [`retry_count + 1 < max_retry`] → `VerifyingPlatform` + +The restore completed and attempts remain. The machine re-walks the chain from +the top to re-verify — the restored image may now pass. Re-verifying end to end +(rather than resuming at the failed component) re-establishes trust across the +whole platform, which is the conservative reading of the "no component executes +unverified firmware" principle. + +### `Restored` [cap reached, `failed` is `Isolable`] → `VerifyingPlatform` + +Restore attempts are exhausted and the recovery image still fails, and the +component's policy is `Isolable`. The machine gives up on this one component +only: it emits `AssertReset(failed)` to keep it held, adds it to `held`, clears +`failed`, and re-walks to continue booting the rest of the platform. The re-walk +skips the now-`held` component. + +### `Restored` [cap reached, `failed` is `Cascading`] → `VerifyingPlatform` + +As with `Isolable`, but the failed component's dependents go down with it. The +machine emits `AssertReset` for the component and each component whose +`depends_on` names it, adds them all to `held`, clears `failed`, and re-walks to +continue booting the remainder. + +### `Restored` [cap reached, `failed` is `PlatformHalt`] → `Recovering` (self, then `Locked`) + +Restore attempts are exhausted and the component's policy is `PlatformHalt`, +meaning the platform cannot safely continue without it. Rather than jump straight +to lockdown, the machine emits `Effect::Emit(RecoveryFailed)` — a follow-up event +— and returns `Handled`. The orchestrator re-dispatches `RecoveryFailed` +immediately (see the next transition), so lockdown appears in the effect trace as +a discrete event rather than a hidden jump. + +### `RecoveryFailed` → `Locked` + +The follow-up event emitted above (or any `RecoveryFailed`) drives the machine to +`Locked`. Routing lockdown through this single event means `Locked` is only ever +entered one way, no matter where the give-up decision originated. + +### anything else → `Operational` + +`Recovering` is an operational state, so attestation and corruption events fall +through to the `Operational` superstate and are handled even mid-recovery. + +--- + +## From `Locked` + +Terminal. The entry action emits `LatchLockdown`, instructing the platform to +hold every component in reset permanently. + +### any event → discarded (self) + +`Locked` handles nothing and has no superstate, so every event falls through to +the top level and is discarded. The machine remains in `Locked` for the rest of +the power cycle. + +--- + +## From the `Operational` superstate + +`Ready`, `Updating`, `Recovering`, and `AwaitingReady` share this parent. When +one of them returns `Outcome::Super`, these handlers run. Centralizing them here +guarantees the two platform-wide behaviors apply identically in all four states. + +### `AttestationChallenge` → `Operational` (no transition) + +The machine emits `SignAttestation` to answer the challenge and stays exactly +where it is. Answering an attestation challenge never changes state, so it is safe +to service from any operational state — including mid-boot (`AwaitingReady`) and +mid-recovery (`Recovering`). + +### `CorruptionDetected` [component required] → `Recovering` + +A trusted-critical component was reported corrupt at runtime. The machine records +it in `failed` and drops into `Recovering`, re-entering the same two-stage +recovery flow used at boot. + +### `CorruptionDetected` [component not required] → `Operational` (no transition) + +The corrupt component is not required for the platform to run. Rather than tear +down the platform, the machine emits `AssertReset` to gate the component (put it +back in reset) and stays in its current state. The component is no longer trusted, +but the platform keeps operating. + +### anything else → discarded + +Any event the superstate does not recognize falls through to the top level and is +discarded.
diff --git a/docs/src/design/orchestrator/orchestrator-sm-walkthru.md b/docs/src/design/orchestrator/orchestrator-sm-walkthru.md new file mode 100644 index 0000000..0cd6463 --- /dev/null +++ b/docs/src/design/orchestrator/orchestrator-sm-walkthru.md
@@ -0,0 +1,358 @@ +# State Machine Walkthrough + +This document is a narrative tour of the orchestrator state machine. Where the +[State Machine](./orchestrator-machine.md) reference gives the precise states, +guards, and transition tables, this walkthrough reads the machine as a story — +following the platform from cold power through steady-state operation, recovery, +and lockdown — and ties each step back to the **CSA Boot Sequence** chapter it +implements. + +Throughout, *CSA* refers to the Composable Security Architecture *Boot Sequence* +chapter (the "Single Node Compute" flow and its Key Principles, Recovery Policy, +and Example Mechanisms sections). The orchestrator is the concrete state-machine +encoding of that flow for a discrete eRoT running OpenPRoT. + +> **How to read this alongside the reference.** Every state named here +> (`PowerOnReset`, `VerifyingPlatform`, …) has a full entry in +> [State Machine](./orchestrator-machine.md) with its entry action and transition +> table. This document explains *why* the transitions are shaped the way they are +> and *which CSA guarantee* each one upholds. When a claim needs the exact guard, +> follow the link. + +--- + +## The shape of the journey + +At the highest level the machine moves through three phases: + +```mermaid +stateDiagram-v2 + direction LR + [*] --> Boot + state Boot { + PowerOnReset --> VerifyingPlatform + VerifyingPlatform --> AwaitingReady + AwaitingReady --> VerifyingPlatform + } + Boot --> Operate : chain verified + state Operate { + Ready --> Updating + Updating --> Ready + } + Boot --> Recover : verification / readiness failure + Operate --> Recover : corruption + Recover --> Boot : restored (re-verify) + Recover --> Operate : restored (re-verify) + Boot --> Halt : unprovisioned / self-check failed + Recover --> Halt : recovery exhausted (PlatformHalt) + Halt --> [*] +``` + +- **Boot** establishes the trust chain: verify each component, release it, and — + for components with their own root of trust — wait for it to come up. +- **Operate** is steady state: answer attestation challenges, apply firmware + updates, and watch for corruption. +- **Recover / Halt** handle failure: restore a component and re-verify, or, when + restoration is hopeless, stop. + +The rest of this document walks each phase. + +--- + +## Phase 1 — Power-on and the provisioning gate + +**State: `PowerOnReset`.** + +> **CSA:** *"The eRoT is the first component to execute after standby power is +> applied. It is the trust anchor for the entire boot sequence."* + +The machine starts in `PowerOnReset` and does nothing until the shell delivers +the first event, `PowerGood`, carrying the result of the eRoT's own power-on +self-check. That single event fans out three ways: + +- `PowerGood(Provisioned)` — the eRoT is provisioned and self-verified, so it is + entitled to vouch for others. The machine advances to `VerifyingPlatform` and + begins the chain walk. +- `PowerGood(Unprovisioned)` — there is nothing to verify against, so the machine + goes straight to `Locked`. +- `PowerGood(SelfVerificationFailed)` — the trust anchor cannot trust *itself*, + so it must not vouch for anything else; straight to `Locked`. + +The two failing branches encode the CSA premise directly: everything downstream +hangs off the eRoT's own integrity, so if that is in doubt the platform never +leaves the gate. + +--- + +## Phase 2 — Walking the trust chain + +**State: `VerifyingPlatform`.** + +> **CSA:** *"No downstream component boots until the eRoT has verified its +> firmware. The eRoT holds each downstream component in reset until verification +> is complete and then releases it."* and *"The eRoT's boot orchestration is +> device-agnostic: it walks an ordered list of managed devices … For each device +> in turn, the eRoT verifies (and measures, where applicable) its firmware, +> releases it from reset, then waits for that device's boot-progress signal +> before proceeding."* + +`VerifyingPlatform` walks the shell-supplied trust chain one component at a time. +On entry it points the cursor at the first component not already skipped and asks +the shell to read and verify that component's firmware (`ReadFirmware` + +`VerifyFirmware`). From then on it reacts to the shell's verdicts. + +The machine is **device-agnostic**, exactly as CSA requires: the chain is a list +of opaque `ComponentId`s with per-component `ComponentAttrs`. The core never +learns what a component *is* — only its boot order, whether it has its own root +of trust, and what to do if it fails. + +### Two kinds of component: symbiont vs. self-verifying + +> **CSA:** *"Each SoC with an integrated iRoT independently measures and verifies +> its own firmware before executing it."* and *"Devices without their own root of +> trust are symbiont devices"* (NIST SP 800-193 §3.4). + +A component's `ComponentKind` decides what "verified" means: + +- **`Passive`** — a *symbiont device* (e.g. a NIC): no root of trust of its own. + The eRoT's signature/SVN check is the only gate. On `VerificationPassed` the + machine releases it (`ReleaseReset`), immediately starts the next component's + read, advances the cursor, and stays in `VerifyingPlatform`. This is the + self-loop: the walk rolls forward through symbiont devices. +- **`Active`** — a *SoC with an integrated iRoT* (e.g. a BMC or CPU with + Caliptra): it must clear **two independent gates**. When it passes the eRoT + check the machine releases it, speculatively starts the *next* component's + read, and moves to `AwaitingReady` to wait for the component's own root of + trust to report in. + +This is the CSA "complementary guarantees" principle made concrete: + +> **CSA:** *"the eRoT controls whether a component is released from reset; the +> iRoT controls whether the component's own firmware executes."* + +`ReleaseReset` is the eRoT gate; `ComponentReady` (below) is the iRoT gate. + +### Reaching the end + +When the component that just passed is the last one in the chain, the walk is +done: the machine releases it and transitions to `Ready`. The platform is up and +every verified component has been released — the trust chain is established. + +### What happens on failure + +> **CSA:** *"No component executes unverified firmware; failed devices are +> recovered before they are released."* + +Any `VerificationFailed` — regardless of the component's recovery-failure policy +— sends the machine to `Recovering`. **Recovery is always attempted first.** The +component is *not* skipped on the spot; it is held in reset (never released, so it +never runs unverified code) and handed to the recovery phase. The decision about +whether to eventually *skip* it or *halt* is deferred until recovery has actually +been tried and failed — see [Phase 5](#phase-5--two-stage-recovery). This is the +crux of CSA compliance: the classification (`Isolable`/`Cascading`/`PlatformHalt`) +is a *recovery-failure* policy, not a first-failure policy. + +--- + +## Phase 3 — Waiting for the iRoT + +**State: `AwaitingReady`.** + +> **CSA:** *"the eRoT … releases it from reset, then waits for that device's +> boot-progress signal before proceeding to the next device."* and, on the +> watchdog: *"the device's own firmware must report each expected boot-progress +> signal within its window, or the eRoT treats the device as failed and initiates +> recovery."* + +`AwaitingReady` exists because an `Active` component has two gates that resolve +independently and in either order: + +- The **iRoT gate** — `ComponentReady`, meaning the component's own root of trust + finished its local self-verification and the component is operational (e.g. its + MCTP channel is up). +- The **next eRoT check** — the speculative `VerificationPassed` for the following + component, which the `VerifyingPlatform` handler kicked off on the way in. + +The `awaiting` field remembers which component's readiness is still outstanding; +the state itself remembers whether the next component's eRoT verdict is still +pending. Both must resolve before the walk moves on, which is why the machine can +loop back into `AwaitingReady` several times: + +- `ComponentReady` from the awaited component clears the readiness gate. If there + is still chain left it stays here (now waiting only on the next eRoT verdict); + if the chain is already complete it advances to `Ready`. +- `ComponentReady` for any *other* id is stale or spurious and is ignored — a + guard so a late or duplicated signal cannot push the walk forward incorrectly. +- `VerificationPassed` for the next component advances the walk the same way + `VerifyingPlatform` does. + +### The boot-progress watchdog + +CSA's *boot-progress checkpointing* mechanism — arm a watchdog when releasing a +device, treat a missed checkpoint as failure — maps to the `Timeout` event. If +`Timeout(id)` arrives for the component the machine is awaiting, it is treated as +a verification failure and sent to `Recovering` (dropping the readiness wait). A +`Timeout` for any other id is stale and ignored. A `VerificationFailed` during +this window behaves identically to the boot-time case: recovery first. + +> **Simplification worth noting.** CSA allows *multiple* boot-progress checkpoints +> per device ("how many checkpoints are expected"). The orchestrator models a +> single readiness signal (`ComponentReady`) plus a single `Timeout` per +> `Active` component. A platform that needs multi-checkpoint progress reporting +> would extend this. + +--- + +## Phase 4 — The operational regime + +**States: `Ready`, `Updating`, and the `Operational` superstate.** + +Once the whole chain is verified and released, the machine settles in `Ready`. +On entry it clears the recovery bookkeeping (`retry_count`, `held`, `failed`) — +reaching `Ready` means the platform booted clean, so any prior recovery episode +is over. + +`Ready` itself does only one state-changing thing: on `UpdateRequest` it moves to +`Updating`, which asks the shell to authenticate and stage the new image, then +waits for a verdict — `UpdateVerified` activates the staged image and returns to +`Ready`; `UpdateRejected` discards it and returns to `Ready`. A rejected update is +explicitly **not** treated as corruption: the platform simply keeps running the +image it already had. + +### The operational contract + +`Ready`, `Updating`, `Recovering`, and `AwaitingReady` all sit under one shared +parent, `Operational`, which handles the two events that must behave identically +no matter which of those states is active: + +- **`AttestationChallenge`** → the machine signs an attestation response and stays + put. This realizes the CSA attestation relationship: + + > **CSA:** *"Firmware measurements taken by each iRoT during this sequence form + > the basis of the platform's attestation evidence … the eRoT has collected or + > can collect measurements from all managed devices and can present aggregated + > attestation evidence to a remote verifier."* + +- **`CorruptionDetected`** → if the affected component is required, the machine + records it and drops into `Recovering`; if it is not required, the component is + gated (`AssertReset`) and the machine stays put. This is the runtime arm of the + CSA Protection/Recovery principle: corruption of a trusted-critical component + re-enters the recovery flow rather than being ignored. + +Because these live in the parent, they apply during boot-time waiting +(`AwaitingReady`) and during recovery (`Recovering`) just as much as in `Ready`. +They do **not** apply in `PowerOnReset` or `VerifyingPlatform`, which sit outside +`Operational`: the eRoT does not answer attestation challenges or act on +corruption reports while it is still establishing the chain. + +--- + +## Phase 5 — Two-stage recovery + +**State: `Recovering`.** + +This is where the orchestrator implements the CSA **Recovery Policy** verbatim, +and it is deliberately a *two-stage* process. + +> **CSA (stage 1 — recover):** *"When a device requires recovery (its firmware +> fails verification), the scope of that recovery operation is determined by its +> configured recovery region … all devices within the same region must be updated +> and/or recovered together."* +> +> **CSA (stage 2 — classify on recovery failure):** *"When a recovery attempt +> itself fails (the alternate/recovery image also fails verification), the +> platform applies one of the following policies … Isolable … Cascading … +> Platform halt."* + +### Stage 1 — always recover first + +On entry, `Recovering` emits `RestoreGoldenImage(failed)`. Per CSA this targets +the failed component's **recovery region**: all components sharing the same +`RegionId` are restored together, not just the one component and not the whole +platform. Then, on `Restored`, the machine re-verifies by re-walking the chain. + +> **Why re-walk from the top?** After a restore the machine re-verifies the whole +> chain rather than resuming at the failed component. A corruption may indicate a +> broader integrity problem, and the CSA/NIST SP 800-193 core principle — *no +> component executes unverified firmware* — is best served by re-establishing +> trust end-to-end. Components already in `held` (exhausted under a prior +> `Isolable`/`Cascading` decision) are skipped during the re-walk; everything else +> is re-verified, and any fresh failure restarts recovery for that component. + +If the restored image now passes, recovery succeeded and the walk continues. + +### Stage 2 — classify only when recovery is exhausted + +Recovery gets `max_retry` attempts. Only when those are exhausted (the recovery +image *keeps* failing) does the failed component's **recovery-failure policy** +decide the outcome — this is the CSA stage-2 classification: + +| Recovery-failure policy | Behaviour | CSA policy | +|---|---|---| +| `Isolable` | Skip just this component: hold it in reset (`held`), continue booting the rest. | *Isolable — skip the failed device and continue.* | +| `Cascading` | Skip this component **and** its `depends_on` dependents, then continue. | *Cascading — skip the failed device and any device configured as dependent on it.* | +| `PlatformHalt` | Stop entirely: self-emit `RecoveryFailed`, which drives the machine to `Locked`. | *Platform halt — stop the boot sequence entirely and enter manual/out-of-band recovery.* | + +The essential point — and the reason this matches CSA — is the ordering: +**recovery is attempted for every failed component first**, and `Isolable` / +`Cascading` / `PlatformHalt` are consulted **only after** a recovery attempt +itself fails. A component is never skipped without first being given a chance to +recover. + +### Lockdown as visible data + +When a `PlatformHalt` component exhausts recovery, the machine does not silently +jump to `Locked`. It emits `RecoveryFailed` as an *effect* — a follow-up event — +which the orchestrator re-dispatches immediately, and *that* event drives the +transition to `Locked`. The give-up decision therefore appears in the effect +trace at the exact moment the cap was reached, and `Locked` is only ever entered +by handling that one event, no matter where lockdown was triggered from. (This is +the "feedback as data" principle; see the +[`Recovering` state](./orchestrator-machine.md#recovering) for the full argument.) + +--- + +## Phase 6 — The dead end + +**State: `Locked`.** + +> **CSA:** *"Platform halt — stop the boot sequence entirely and enter a manual or +> out-of-band recovery mode."* + +`Locked` is terminal. On entry it instructs the shell to hold every component in +reset permanently (`LatchLockdown`), and from then on every event is ignored. The +machine reaches here from exactly three places, all meaning "no trustworthy state +could be established, so refuse to run one": + +1. Power-on with an unprovisioned eRoT. +2. Power-on with a failed eRoT self-check. +3. A `PlatformHalt` component whose recovery was exhausted. + +--- + +## CSA compliance at a glance + +| CSA principle / policy | Where the machine upholds it | +|---|---| +| eRoT is the trust anchor, first to execute | `PowerOnReset` + `PowerGood` self-check gate | +| No downstream boots until eRoT verifies it; held in reset until release | `VerifyingPlatform` emits `ReleaseReset` only on `VerificationPassed` | +| iRoT independently verifies; complementary eRoT/iRoT gates | `Active` → `AwaitingReady` on `ComponentReady`; `Passive` → immediate | +| Device-agnostic ordered walk | Opaque `ComponentId` chain with `ComponentAttrs` | +| Symbiont devices (NIST SP 800-193 §3.4) | `ComponentKind::Passive` | +| Boot-progress watchdog → treat as failed, recover | `Timeout(id)` → `Recovering` | +| Recovery scope = recovery region (restore together) | `RegionId`; `RestoreGoldenImage` restores the whole region | +| Recover first for every failed device | Any `VerificationFailed` → `Recovering` | +| Classify only after recovery fails (Isolable/Cascading/halt) | `Recovering` applies the policy when `retry_count` reaches `max_retry` | +| Platform halt on unrecoverable failure | `PlatformHalt` → `RecoveryFailed` → `Locked` | +| Measurements form attestation evidence | `AttestationChallenge` → `SignAttestation` in `Operational` | + +--- + +## See also + +- [State Machine](./orchestrator-machine.md) — the authoritative states, guards, + and transition tables this walkthrough narrates. +- [Verification Model](./orchestrator-model.md) — `ComponentKind`, + `FailurePolicy`, `RegionId`, `ComponentAttrs`, and the verification boundary. +- [Orchestrator Overview](./orchestrator-overview.md) — design principles and + applicability across the admissible architectures.