ci: code size comment from stored master snapshots, ESP included (#4055)
- Size pinned boards' examples (ESP-IDF and -DMA variants included) with membrowse 1.2.12 and store per-board snapshots as artifacts.
- Compare a PR against the stored master snapshots and post one sticky comment: each changed TinyUSB file with its min to max Flash/RAM delta, plus what was not compared; the full report goes to the job summary and an artifact.
- Retire the Membrowse PR comment.
- Add tools/code_size.py report/diff for local runs, and the code-size skill.
diff --git a/.claude/skills/code-size/SKILL.md b/.claude/skills/code-size/SKILL.md
index 11df41b..5c78c69 100644
--- a/.claude/skills/code-size/SKILL.md
+++ b/.claude/skills/code-size/SKILL.md
@@ -1,16 +1,11 @@
---
name: code-size
-description: Use when sizing TinyUSB examples per file, section or symbol (report), or diffing them between a base ref (master by default) and the working tree to evaluate the size impact of changes (diff) — one example on one board, all examples on one board, or every CI-pinned board combined — sized by membrowse, linkermap or bloaty.
+description: Use when sizing TinyUSB examples per file, section or symbol (report), diffing them against a base ref to judge a change's size impact (diff), or reading or debugging the PR's "Code size" comment (CI snapshot/compare).
---
# Code Size
-`tools/code_size.py` has two subcommands:
-
-- **`report`** builds the working tree, uncommitted changes included, and tabulates each elf as linkermap does: a row per file, a column per output section, size and % of that elf's filtered total.
-- **`diff`** builds a base ref (default `master`) in a temporary worktree and the working tree, pairs each base elf with the current elf of the same (board, elf path) and reports per-file deltas per pair: a Flash/RAM table and the same section table as signed deltas, changed rows only.
-
-Sizes are never averaged or summed across examples or boards. Pick the narrowest scope that exercises the change:
+`tools/code_size.py report` sizes the working tree, uncommitted changes included. `diff` builds a base ref (default `master`) in a temporary worktree and the working tree, pairs elfs by board and elf path, and reports per-file deltas. Pick the narrowest scope that exercises the change; sizes are never summed or averaged across examples or boards.
| Scope | Command |
|-----------------------------------|----------------------------------------------------------------|
@@ -19,41 +14,36 @@
| all examples, CI-pinned, combined | `python3 tools/code_size.py diff --ci` |
| one tree, no diff | `python3 tools/code_size.py report -b BOARD -e device/cdc_msc` |
-The base worktree symlinks this checkout's fetched dependencies, so a `tools/get_deps.py` pin bump's own size change is not in the diff.
+The base worktree symlinks this checkout's fetched dependencies, so a `tools/get_deps.py` pin bump's own size change is not in the diff. A local `report` or `diff` cannot size a `-D` variant build.
-## Arguments
+## Setup and choices
-- **`-b BOARD`**, repeatable. For an iterative check with no board named, pass `-b raspberry_pi_pico`.
-- **`-e <group>/<name>`**, repeatable; omit for all examples.
-- **`--engine`**, membrowse unless asked:
+Flags and syntax: `python3 tools/code_size.py <command> --help`. The choices:
- | Engine | Per-file sizes from | Needs |
- |-----------------------|---------------------------------------------------------------------|---------------------------------------------|
- | `membrowse` (default) | `membrowse report --json --all-symbols` symbols | `pip install membrowse` |
- | `linkermap` | the GNU ld map's input sections, by object path | `python3 tools/get_deps.py tools/linkermap` |
- | `bloaty` | `bloaty -d compileunits,sections,symbols` VM sizes, by compile unit | `bloaty` on PATH |
+- No board named for an iterative check: `-b raspberry_pi_pico`. `-f` only if asked.
+- `--engine` membrowse unless asked (`pip install membrowse`; `membrowse==1.2.12` matches CI's attribution); linkermap needs `python3 tools/get_deps.py tools/linkermap`, bloaty needs `bloaty` on PATH. Flash/RAM come from the elf's headers for every engine. Membrowse's all-symbol total can overlap aliases and omit padding, so compare whole-elf totals only within one engine.
+- `--symbols` when Flash/RAM cancel: a pair counts as changed when its sections or symbols moved.
+- `--ci` for "all boards" / "CI"; it needs the arm, riscv, msp430 and ft9xx toolchains, and ESP-IDF or docker.
+- An espressif board builds each example `tools/build.py` picks as its own ESP-IDF project: with the exported ESP-IDF (`. "$IDF_PATH/export.sh"`), else in CI's image through docker, as your user (once: `docker pull espressif/idf:v5.5.3 && docker tag espressif/idf:v5.5.3 espressif/idf:tinyusb`, about 8.4 GB).
- Every engine takes flash/RAM from the elf's headers (a section copied from flash counts in both). The whole-elf total counts different things per engine (membrowse's all-symbol sum overlaps aliases and omits padding), so compare it only within one.
-- **`--symbols`**: lists each file's symbols under it (`└ name`); without it the table stops at files. linkermap's rows are input sections (`.text.cdcd_open`), not symbols. A diff counts a pair changed when its sections or, with `--symbols`, symbols moved even if Flash/RAM cancel.
-- **`--json`**: also writes each report's raw sizes (paired, with the base SHA, for diff), filters and failures as `.json` beside the `.md`; symbols only with `--symbols`.
-- **`-f SUBSTRING`**, only if asked: replaces the default filter, each build's absolute `<checkout>/src/` path, which matches TinyUSB code and no vendored `src/`.
-- diff only:
- - **`--base-branch <ref>`**: any branch, tag or commit.
- - **`--combined`**: also one report over every `-b` board.
- - **`--ci`**, for "all boards" / "CI": adds the `.github/ci-pinned-boards.json` boards, covering every dcd/hcd driver not waived there, and implies `--combined`; needs the arm, riscv and msp430 toolchains.
- - **`--bloaty`**, with `-e` only: also prints bloaty's section and symbol diff to stdout.
+## Results
-## Outputs and timing
+Reports go under `cmake-code-size/<board>/`, combined diffs under `cmake-code-size/_combined/`. A local command exits nonzero on a build or report failure or when no TinyUSB file matched; `report` also when it sized no elf, `diff` when it compared no pair. A failed build prints an excerpt on the console; its report records the first compiler, linker or CMake error.
-Reports go to `cmake-code-size/<board>/{report,diff}[_<example>].md` and, when combined, `cmake-code-size/_combined/diff.md`. The exit code is nonzero on a failure, when no pair was compared (diff) or no elf of a scope matched a file. The console shows each phase's time, one result line per scope and, for one board and one `-e` example, its changed tables; a failed build prints an excerpt of its output there, and its report and JSON record its first compiler, linker or CMake error, otherwise a fallback message.
+Approximate times: one diff example ~30 s, one board ~1-1.5 min, an espressif board 10+ min, `--ci` 30+ min. A report is about half a diff. A diff rebuilds both trees every run, so ask for `--symbols` or `--bloaty` in the same run. Run a long sweep in the background.
-A report builds one tree, about half a diff's time. One diff example ~30 s; one board ~60-90 s; `--ci` ~7-8 min, boards built one after another — run it in the background, it nears the 10-minute command timeout.
+Each report opens with a coverage line. `INCOMPLETE` means a build, measurement or filter match failed, or an elf exists on one side only; those elfs are listed and excluded, never counted as zero. Show the coverage line and the relevant tables, then:
-## Reporting results
+- A many-elf report summarizes each elf, a many-pair diff each file; per-elf tables are in the `.md`'s `<details>`. Diff statistics cover compared pairs only: `Changed / present` is the pairs whose Flash or RAM of the file changed over the compared pairs containing it. min > 0 means growth in every such pair, max > 0 in at least one; name the worst-growth pair.
+- A whole-elf Δ with a zero TinyUSB Δ (`filtered` under a custom `-f`) is unfiltered code (inlined headers, example/BSP code) or an attribution difference; check the per-file table for rows that cancel.
+- Corroborate a surprising delta on its own board and example, not the whole sweep: `-b <board> -e <example> --engine linkermap`, plus `--bloaty` if bloaty is on PATH.
-Each report opens with a coverage line; `INCOMPLETE` means a build, report or filter match failed, or an elf exists on one side only (listed, outside every statistic). A many-pair diff counts each file whose Flash or RAM changed as `Changed / present` pairs with its min/max Δ naming the pair; a many-elf report lists each elf's totals. Per-elf tables are then in the `.md`'s `<details>` only.
+An ESP-IDF app is sized by its image: initialized IRAM/DRAM counts in both Flash and RAM, `.flash*` NOBITS reservations are not counted, and its TinyUSB files come from DWARF.
-Show the coverage line and the relevant tables, then:
-- A filtered delta suggests a TinyUSB size impact in that configuration. A whole-elf Δ with a zero filtered total is outside the filter (inlined headers, example/BSP code); check the per-file table for changes that cancel.
-- min > 0 means growth in every present pair, max > 0 in at least one; name the worst-growth pair.
-- Corroborate a surprising delta on its own board and example, not the whole sweep: `-b <board> -e <example> --engine linkermap` (~30 s; every run rebuilds both trees); add `--bloaty` if bloaty is on PATH to see the sections and symbols behind it.
+## CI comment
+
+CI never rebuilds for sizing. In a code-changing run, each selected pinned `cmake` build leg, and each `hil-build-esp` leg inside the ESP-IDF image, snapshots the boards it built and uploads `code-size-<toolchain>-<leg>`; it tries even after a failed build, which a missing membrowse does not stop it recording. A `-DMA` variant leg reports as a board of its own named by its `--build-name` (`espressif_s3_devkitm-DMA`), in the comment and on the Membrowse dashboard; reproducing its snapshot needs that name and the leg's `-D` defines, which can enable examples. A `code-size-scope` job uploads the legs and examples the run selected. Master push runs keep snapshots 90 days as baselines, PR runs 14.
+
+`pr_comment.yml`'s `code-size-comment` job runs the default branch's scripts. `.github/scripts/code_size_ci.py baseline` takes the master push run of the merge commit's first parent, else its nearest first-parent ancestor with snapshots (up to 30), labelled approximate: master changes after it count as the PR's. `code_size.py compare` produces the report, which the job publishes as its summary and the `code-size-report` artifact; the sticky `code-size` comment is posted only when `code_size_ci.py is-current` finds no newer push, Build run or attempt for the PR. Runs of other PRs on the same commit are ignored; two open PRs from one head branch can still mask each other. A comment that did not update: read that step's reason and any superseding run it names. Neither the job nor `compare` fails on `INCOMPLETE`.
+
+Reading the comment: it lists every changed TinyUSB file with its min to max Flash and RAM Δ across the builds containing it, so a small driver change shows beside a heavy one; per-build tables are in the full report. Its `Not compared:` list is the full report's `INCOMPLETE`, with one-sided elfs named `missing in this PR` and `new in this PR`. CI's `INCOMPLETE` adds a leg or board with no snapshot, a board with no baseline, and snapshots of mismatched commits. Compiler and membrowse differences are noted, not failed; baseline elfs of examples the PR did not build are outside coverage. To reproduce, `gh run download` both runs' `code-size-*` artifacts into two directories and run `python3 tools/code_size.py compare CUR BASE -o OUT --symbols`, adding `--baseline-info INFO` for the baseline annotations.
diff --git a/.github/actions/get_deps/action.yml b/.github/actions/get_deps/action.yml
index 96eee01..66deb77 100644
--- a/.github/actions/get_deps/action.yml
+++ b/.github/actions/get_deps/action.yml
@@ -26,7 +26,7 @@
NINJA_URL=https://github.com/ninja-build/ninja/releases/download/v1.13.1/ninja-linux.zip
wget $NINJA_URL -O ninja-linux.zip
unzip ninja-linux.zip -d ninja-bin
- pip install --only-binary :all: membrowse==1.2.9
+ pip install --only-binary :all: membrowse==1.2.12
echo >> $GITHUB_PATH "${{ github.workspace }}/ninja-bin"
shell: bash
diff --git a/.github/ci-pinned-boards.json b/.github/ci-pinned-boards.json
index 0df5738..a6f09f3 100644
--- a/.github/ci-pinned-boards.json
+++ b/.github/ci-pinned-boards.json
@@ -1,34 +1,39 @@
{
"boards": [
- { "board": "adafruit_fruit_jam" },
- { "board": "ch32v307v_r1_1v0" },
- { "board": "ea4088_quickstart" },
- { "board": "ek_tm4c123gxl" },
- { "board": "feather_nrf52840_express" },
- { "board": "frdm_kl25z" },
- { "board": "lpcxpresso11u68" },
- { "board": "lpcxpresso55s28" },
- { "board": "max32666fthr" },
- { "board": "metro_m0_express" },
- { "board": "metro_m4_express" },
- { "board": "mimxrt1064_evk" },
- { "board": "msp_exp430f5529lp" },
- { "board": "nanoch32v203" },
- { "board": "ra6m5_ek" },
- { "board": "raspberry_pi_pico" },
- { "board": "raspberry_pi_pico2" },
- { "board": "same70_xplained" },
- { "board": "samg55_xplained" },
- { "board": "stm32f407disco" },
- { "board": "stm32g0b1nucleo" },
- { "board": "stm32h743eval" },
- { "board": "stm32l412nucleo" }
+ { "board": "adafruit_fruit_jam", "note": "rp2350 FS + pio-usb host" },
+ { "board": "ch32v307v_r1_1v0", "note": "ch32 usbhs HS" },
+ { "board": "da14695_dk_usb", "note": "da146xx FS" },
+ { "board": "ea4088_quickstart", "note": "lpc17_40 FS, ohci host" },
+ { "board": "ek_tm4c123gxl", "note": "musb FS" },
+ { "board": "espressif_p4_function_ev", "note": "dwc2 4.00a HS, UTMI+, host; built by hil-build-esp with its -DMA variant, not the cmake job" },
+ { "board": "espressif_s3_devkitm", "note": "dwc2 4.00a FS, host; built by hil-build-esp with its -DMA variant, not the cmake job" },
+ { "board": "feather_nrf52840_express", "note": "nrf5x FS" },
+ { "board": "fomu", "note": "eptri FS" },
+ { "board": "frdm_k64f", "note": "ci_fs FS, host" },
+ { "board": "lpcxpresso11u68", "note": "ip3511 FS" },
+ { "board": "lpcxpresso55s28", "note": "ip3511 HS, ohci host" },
+ { "board": "max32666fthr", "note": "musb HS" },
+ { "board": "metro_m0_express", "note": "samd21 FS, host" },
+ { "board": "metro_m4_express", "note": "samd51 FS, host" },
+ { "board": "mimxrt1064_evk", "note": "ci_hs HS, ehci host" },
+ { "board": "mm900evxb", "note": "ft9xx HS" },
+ { "board": "msp_exp430f5529lp", "note": "msp430x5xx FS" },
+ { "board": "nanoch32v203", "note": "ch32 usbfs + fsdev FS, host" },
+ { "board": "ra6m5_ek", "note": "rusb2 HS, host" },
+ { "board": "raspberry_pi_pico", "note": "rp2040 FS, host" },
+ { "board": "raspberry_pi_pico2", "note": "rp2350 FS, host" },
+ { "board": "same70_xplained", "note": "samx7x HS" },
+ { "board": "samg55_xplained", "note": "samg FS" },
+ { "board": "stm32f303disco", "note": "fsdev 512B, 1x16-bit" },
+ { "board": "stm32f407disco", "note": "dwc2 2.81a FS, host" },
+ { "board": "stm32g0b1nucleo", "note": "fsdev 2KB, 32-bit, host" },
+ { "board": "stm32h743eval", "note": "dwc2 3.30a HS, ULPI, host" },
+ { "board": "stm32h7s3nucleo", "note": "dwc2 4.11a HS, UTMI+, host" },
+ { "board": "stm32l412nucleo", "note": "fsdev 1KB, 2x16-bit" },
+ { "board": "stm32n657nucleo", "note": "dwc2 4.11a HS, UTMI+, host" }
],
"uncovered": {
- "dcd_cxd56": "spresense unpinned - no board on either HIL roster and no other CI board carries dcd_cxd56; build-only size tracking with no hardware behind it",
- "dcd_da146xx": "da1469x_dk_pro unpinned - no board on either HIL roster and no other CI board carries dcd_da146xx; build-only size tracking with no hardware behind it",
- "dcd_eptri": "fomu unpinned - no board on either HIL roster and no other CI board carries dcd_eptri; build-only size tracking with no hardware behind it",
- "dcd_ft9xx": "mm900evxb unpinned - no board on either HIL roster and no other CI board carries dcd_ft9xx; build-only size tracking with no hardware behind it",
+ "dcd_cxd56": "cxd56 has no CI family lane (not in ci_set_matrix.family_list), so no CI board can build it",
"dcd_mm32f327x_otg": "mm32 unpinned - maintainer trimmed the pinned set",
"dcd_nuc120": "nuvoton unpinned - maintainer trimmed the pinned set",
"dcd_nuc121": "nuvoton unpinned - maintainer trimmed the pinned set",
diff --git a/.github/membrowse_pr_message.j2 b/.github/membrowse_pr_message.j2
deleted file mode 100644
index fbba1ee..0000000
--- a/.github/membrowse_pr_message.j2
+++ /dev/null
@@ -1,38 +0,0 @@
-{#- Top 10 targets with biggest memory changes + project dashboard link -#}
-{% set section_columns = ['.text', '.rodata', '.data', '.bss'] -%}
-{#- --- Compute per-target total absolute delta and collect changed targets --- -#}
-{% set changed = [] -%}
-{% for target in targets -%}
-{% if target.has_changes -%}
-{% set ns = namespace(total_delta=0, total_current=0) -%}
-{% for region in target.regions -%}
-{% set ns.total_delta = ns.total_delta + region.delta -%}
-{% set ns.total_current = ns.total_current + region.used_size -%}
-{% endfor -%}
-{% set total_old = ns.total_current - ns.total_delta -%}
-{% set pct = (ns.total_delta / total_old * 100) if total_old > 0 else 0 -%}
-{% set abs_pct = (ns.total_delta | abs) if total_old == 0 else (pct | abs) -%}
-{% set _ = changed.append({'target': target, 'total_current': ns.total_current, 'total_old': total_old, 'total_delta': ns.total_delta, 'pct': pct, 'abs_pct': abs_pct}) -%}
-{% endif -%}
-{% endfor -%}
-{#- --- Sort by absolute percentage change descending and take top 10 --- -#}
-{% set sorted_changed = changed | sort(attribute='abs_pct', reverse=true) -%}
-{% set top10 = sorted_changed[:10] -%}
-{#- --- Render --- -#}
-{% if top10 %}
-### Top {{ top10 | length }} targets by memory change (%) (out of {{ targets | length }} targets) {% if dashboard_url %} [View Project Dashboard →]({{ dashboard_url }}){% endif %}
-
-| target | .text | .rodata | .data | .bss | total | % diff |
-|--------|-------|---------|-------|------|-------|--------|
-{% for info in top10 -%}
-{% set target = info.target -%}
-{% set section_map = {} -%}
-{% for section in target.sections -%}
-{% set _ = section_map.update({section.name: section}) -%}
-{% endfor -%}
-| {% if target.comparison_url %}[{{ target.name }}]({{ target.comparison_url }}){% else %}{{ target.name }}{% endif %} |
-{%- for col in section_columns %} {% if col in section_map %}{{ "{:,}".format(section_map[col].old.size) }} → {{ "{:,}".format(section_map[col].size) }} ({{ section_map[col].delta_str }}){% else %}—{% endif %} |{% endfor %} {{ "{:,}".format(info.total_old) }} → {{ "{:,}".format(info.total_current) }} ({% if info.total_delta >= 0 %}+{{ "{:,}".format(info.total_delta) }}{% else %}{{ "{:,}".format(info.total_delta) }}{% endif %}) | {% if info.total_old > 0 %}{% if info.pct >= 0 %}+{% endif %}{{ "%.1f" | format(info.pct) }}%{% else %}N/A{% endif %} |
-{% endfor %}
-{% else %}
-No memory changes detected across {{ targets | length }} target{{ 's' if targets | length != 1 else '' }}.{% if dashboard_url %} [View Project Dashboard →]({{ dashboard_url }}){% endif %}
-{% endif -%}
diff --git a/.github/scripts/code_size_ci.py b/.github/scripts/code_size_ci.py
new file mode 100644
index 0000000..4514311
--- /dev/null
+++ b/.github/scripts/code_size_ci.py
@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+"""pr_comment.yml's code-size comment mechanics. Needs `gh` with a token that can read Actions.
+
+ code_size_ci.py baseline --repo OWNER/NAME --current CUR_DIR --info INFO.json
+
+finds the baseline of a PR's Build run: the master push Build run of the commit the PR
+was built against (its merge commit's first parent), else of its nearest first-parent
+ancestor with snapshots, labelled approximate. Never rebuilds; no baseline, a failed
+lookup included, is a result, not an error. CUR_DIR holds the PR run's downloaded
+code-size artifacts; INFO.json gets {sha, url, exact, note, run_id} for `tools/code_size.py
+compare --baseline-info`, sha and run_id None when there is no baseline. The workflow
+downloads run_id's `code-size-*` artifacts.
+
+ code_size_ci.py is-current --repo OWNER/NAME --pr N --run-id ID --attempt A --head-sha SHA
+
+exits 0 when that Build run attempt is still the PR's latest, 1 when a newer push, run
+or attempt supersedes it: its comment would overwrite a newer one.
+"""
+import argparse
+import json
+import os
+import subprocess
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), '..', '..', 'tools'))
+import code_size # noqa: E402 stdlib-only; its snapshot loader is the one compare uses
+
+WORKFLOW = 'build.yml'
+BRANCH = 'master'
+MAX_DEPTH = 30 # first-parent ancestors tried before giving up
+
+
+def gh(path, paginate=False):
+ """Parsed `gh api` GET of `path`; with `paginate`, the list of every page."""
+ cmd = ['gh', 'api', '-H', 'Accept: application/vnd.github+json'] + (['--paginate', '--slurp'] if paginate else [])
+ ret = subprocess.run(cmd + [path], capture_output=True, text=True, check=False)
+ if ret.returncode != 0:
+ raise RuntimeError(f'gh api {path}: {ret.stderr.strip()}')
+ return json.loads(ret.stdout)
+
+
+def gh_items(path, key):
+ """Every item of a paginated list endpoint whose items are under `key`."""
+ return [item for page in gh(path, paginate=True) for item in page[key]]
+
+
+def current_base_sha(current):
+ """The one base_sha the PR run's legs record, else None."""
+ shas = {code_size._commits(leg)[1] for leg in code_size.load_snapshots(current)['legs'].values()}
+ return shas.pop() if len(shas) == 1 else None
+
+
+def is_master_ancestor(repo, sha):
+ """Whether `sha` is on master's history, so a PR can never pick its own run as baseline."""
+ return gh(f'repos/{repo}/compare/{sha}...{BRANCH}')['status'] in ('ahead', 'identical')
+
+
+def snapshot_run(repo, sha):
+ """The latest completed master push Build run of `sha` with unexpired snapshots, else None."""
+ runs = gh_items(f'repos/{repo}/actions/workflows/{WORKFLOW}/runs?head_sha={sha}&event=push'
+ f'&branch={BRANCH}&status=completed', 'workflow_runs')
+ for run in sorted(runs, key=lambda r: r['run_number'], reverse=True):
+ if run['head_sha'] != sha or run['head_branch'] != BRANCH or run['event'] != 'push' \
+ or (run.get('head_repository') or {}).get('full_name') != repo:
+ continue
+ names = {a['name'] for a in gh_items(f'repos/{repo}/actions/runs/{run["id"]}/artifacts', 'artifacts')
+ if a['name'].startswith('code-size-') and not a['expired']}
+ # a docs-only commit uploads a scope that measured nothing: not a baseline
+ if 'code-size-scope' in names and len(names) > 1:
+ return run
+ return None
+
+
+def find_baseline(repo, sha):
+ """(run, depth) of the first first-parent ancestor of `sha`, itself at depth 0, with
+ snapshots, or None within MAX_DEPTH."""
+ for depth in range(MAX_DEPTH):
+ run = snapshot_run(repo, sha)
+ if run:
+ return run, depth
+ parents = gh(f'repos/{repo}/commits/{sha}')['parents']
+ if not parents:
+ return None
+ sha = parents[0]['sha']
+ return None
+
+
+def lookup(repo, current):
+ """The baseline description written to INFO.json."""
+ info = {'sha': None, 'url': None, 'exact': False, 'run_id': None}
+ sha = current_base_sha(current)
+ if sha is None:
+ return {**info, 'note': 'the PR snapshots record no single base commit'}
+ if not is_master_ancestor(repo, sha):
+ return {**info, 'note': f'base {sha[:10]} is not on {BRANCH}'}
+ found = find_baseline(repo, sha)
+ if found is None:
+ return {**info, 'note': f'no {BRANCH} Build run with snapshots within {MAX_DEPTH} commits of {sha[:10]}'}
+ run, depth = found
+ info = {'sha': run['head_sha'], 'url': run['html_url'], 'exact': depth == 0, 'run_id': run['id']}
+ note = (f'{depth} {BRANCH} commits before this build\'s base {sha[:10]}: their changes count as this PR\'s')
+ return {**info, 'note': note} if depth else info
+
+
+def baseline(args):
+ try:
+ info = lookup(args.repo, args.current)
+ except (RuntimeError, KeyError, TypeError, ValueError) as e: # an API failure or an unexpected reply
+ info = {'sha': None, 'url': None, 'exact': False, 'run_id': None, 'note': f'the baseline lookup failed: {e}'}
+ with open(args.info, 'w') as f:
+ json.dump(info, f, indent=1, sort_keys=True)
+ f.write('\n')
+ print(json.dumps(info))
+ return 0
+
+
+def is_run_of(run, pr, head):
+ """Whether a pull_request Build run may be PR `pr`'s, whose head is `head`: another PR can
+ share the commit. GitHub lists the open same-repo PRs of the run's head in pull_requests and
+ leaves it empty for a fork's, so an unlisted run counts when it built the PR's head branch."""
+ numbers = [p['number'] for p in run.get('pull_requests') or []]
+ if numbers:
+ return pr in numbers
+ return run.get('head_branch') == head.get('ref') and \
+ (run.get('head_repository') or {}).get('full_name') == (head.get('repo') or {}).get('full_name')
+
+
+def run_owner(run):
+ """Run `run`'s id and whose it is, so a stale verdict names what superseded it."""
+ numbers = [p['number'] for p in run.get('pull_requests') or []]
+ owner = ', '.join(f'#{n}' for n in numbers) if numbers else \
+ f"{(run.get('head_repository') or {}).get('full_name')}:{run.get('head_branch')}"
+ return f"run {run['id']} of {owner}"
+
+
+def is_current(args):
+ """Whether run `args.run_id` attempt `args.attempt` of the PR head is still the newest."""
+ why = None
+ head = gh(f'repos/{args.repo}/pulls/{args.pr}')['head']
+ if head['sha'] != args.head_sha:
+ why = 'the PR head moved on'
+ else:
+ runs = [r for r in gh_items(f'repos/{args.repo}/actions/workflows/{WORKFLOW}/runs?head_sha={args.head_sha}'
+ f'&event=pull_request', 'workflow_runs') if is_run_of(r, args.pr, head)]
+ latest = max(runs, key=lambda r: r['run_number'], default=None)
+ if latest is None:
+ why = 'no Build run of the PR head found'
+ elif latest['id'] != args.run_id:
+ why = f'a newer Build run exists: {run_owner(latest)}'
+ elif gh(f'repos/{args.repo}/actions/runs/{args.run_id}')['run_attempt'] != args.attempt:
+ why = 'a newer attempt exists'
+ print(f'stale: {why}' if why else 'current')
+ return 1 if why else 0
+
+
+def main():
+ top = argparse.ArgumentParser(description=__doc__.split('\n\n')[0])
+ sub = top.add_subparsers(dest='command', required=True)
+ p = sub.add_parser('baseline', help='find the baseline run of a PR run\'s snapshots')
+ p.add_argument('--repo', required=True, help='OWNER/NAME whose master runs are baselines')
+ p.add_argument('--current', required=True, help='The PR run\'s downloaded code-size artifacts')
+ p.add_argument('--info', required=True, help='Where to write the baseline description JSON')
+ p = sub.add_parser('is-current', help='exit 1 when a newer push, run or attempt supersedes this one')
+ p.add_argument('--repo', required=True)
+ p.add_argument('--pr', required=True, type=int)
+ p.add_argument('--run-id', required=True, type=int)
+ p.add_argument('--attempt', required=True, type=int)
+ p.add_argument('--head-sha', required=True)
+ args = top.parse_args()
+ return baseline(args) if args.command == 'baseline' else is_current(args)
+
+
+if __name__ == '__main__':
+ sys.exit(main())
diff --git a/.github/scripts/hil_ci_set_matrix.py b/.github/scripts/hil_ci_set_matrix.py
index e465c64..f973cc1 100644
--- a/.github/scripts/hil_ci_set_matrix.py
+++ b/.github/scripts/hil_ci_set_matrix.py
@@ -120,6 +120,9 @@
except ValueError as e:
raise SystemExit(f'{config_file}: {e}')
for v in variants:
+ if toolchain == 'esp-idf' and v['name'] == name and (v['defines'] or v['flags']):
+ # hil-build-esp sizes and uploads every leg as the board its --build-name names
+ raise SystemExit(f'{config_file}: {name}: an esp-idf variant needs a name of its own')
arg = build_board
if v['name'] != name:
arg += f' --build-name {v["name"]}'
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index ad975d2..7d4a319 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -42,6 +42,7 @@
- 'tools/get_deps.py'
- 'tools/rtt.py'
- 'tools/membrowse_cli.py'
+ - 'tools/code_size.py'
- '.github/ci-pinned-boards.json'
- '.github/actions/**'
- '.github/workflows/build.yml'
@@ -51,6 +52,7 @@
set-matrix:
runs-on: ubuntu-latest
outputs:
+ cmake_toolchains: ${{ steps.set-matrix-json.outputs.cmake_toolchains }}
pinned_json: ${{ steps.set-matrix-json.outputs.pinned_matrix }}
hil_json: ${{ steps.set-matrix-json.outputs.hil_matrix }}
example_map: ${{ steps.set-matrix-json.outputs.example_map }}
@@ -178,6 +180,9 @@
echo "pinned_matrix=$PINNED_JSON"
echo "pinned_matrix=$PINNED_JSON" >> $GITHUB_OUTPUT
echo "example_map=$EXAMPLE_MAP" >> $GITHUB_OUTPUT
+ # the `cmake` job's toolchains, one list for its matrix and the code-size scope
+ # (arm-clang and esp-idf are not built there: see that job's comment)
+ echo 'cmake_toolchains=["aarch64-gcc", "arm-gcc", "ft9xx-gcc", "msp430-gcc", "riscv-gcc"]' >> $GITHUB_OUTPUT
# HIL matrix (merged from tinyusb + hifiphile configs), scoped on PRs.
# Scoping is best-effort too: fall back to the unscoped (full) matrix.
@@ -194,6 +199,45 @@
echo "hil_matrix=$HIL_MATRIX_JSON"
echo "hil_matrix=$HIL_MATRIX_JSON" >> $GITHUB_OUTPUT
+ # The legs this run expects a code-size snapshot from, for pr_comment.yml's compare: a
+ # leg that dies before its snapshot step leaves no artifact, which only this manifest
+ # can tell apart from a leg that was never selected: the cmake job's pinned families and
+ # hil-build-esp's legs (a -DMA variant reports as a board of its own), under that job's own
+ # gate. No legs when no code changed.
+ # Nothing needs it, so it stays off the critical path; measurement only, never red.
+ code-size-scope:
+ needs: [ check-paths, set-matrix ]
+ if: github.event_name == 'pull_request' || github.event_name == 'push'
+ runs-on: ubuntu-latest
+ continue-on-error: true
+ steps:
+ - name: Code size scope
+ env:
+ TOOLCHAINS: ${{ needs.set-matrix.outputs.cmake_toolchains }}
+ PINNED_JSON: ${{ needs.set-matrix.outputs.pinned_json }}
+ EXAMPLE_MAP: ${{ needs.set-matrix.outputs.example_map }}
+ CODE_CHANGED: ${{ needs.check-paths.outputs.code_changed }}
+ HIL_JSON: ${{ needs.set-matrix.outputs.hil_json }}
+ OWNER: ${{ github.repository_owner }}
+ run: |
+ mkdir -p code-size-scope
+ jq -n --argjson tc "$TOOLCHAINS" --argjson pinned "$PINNED_JSON" --argjson ex "$EXAMPLE_MAP" \
+ --argjson hil "$HIL_JSON" --arg changed "$CODE_CHANGED" --arg owner "$OWNER" \
+ '{schema: 1, code_changed: ($changed == "true"), family_examples: $ex,
+ legs: (if $changed == "true" then [$tc[] as $t | ($pinned[$t] // [])[] | {toolchain: $t, arg: .}]
+ + [if $owner == "hathach" then ($hil["esp-idf"] // [])[] else empty end
+ | {toolchain: "esp-idf", arg: .}]
+ else [] end)}' \
+ > code-size-scope/scope.json
+ cat code-size-scope/scope.json
+
+ - name: Upload code size scope
+ uses: actions/upload-artifact@v7
+ with:
+ name: code-size-scope
+ path: code-size-scope/scope.json
+ retention-days: ${{ github.event_name == 'push' && 90 || 14 }}
+
# ------------------------------------------------------------------------------
# CMake build: the families with a membrowse CI board, building that board - the only
# boards this job exists for, since CircleCI builds every board of every family. A
@@ -209,14 +253,8 @@
strategy:
fail-fast: false
matrix:
- toolchain:
- - 'aarch64-gcc'
- #- 'arm-clang'
- - 'arm-gcc'
- #- 'esp-idf'
- - 'ft9xx-gcc'
- - 'msp430-gcc'
- - 'riscv-gcc'
+ # arm-clang and esp-idf stay out: esp-idf would double-build hil-build-esp's boards
+ toolchain: ${{ fromJSON(needs.set-matrix.outputs.cmake_toolchains) }}
with:
build-system: 'cmake'
toolchain: ${{ matrix.toolchain }}
@@ -344,9 +382,9 @@
persist-credentials: false
- name: Install membrowse
- # pinned and wheel-only: an unpinned sdist would run its setup scripts on
- # a runner that holds MEMBROWSE_API_KEY
- run: 'pip install --only-binary :all: membrowse==1.2.9'
+ # this runner holds MEMBROWSE_API_KEY: wheel-only runs no sdist setup script,
+ # pinned runs no new release unreviewed
+ run: 'pip install --only-binary :all: membrowse==1.2.12'
- name: Membrowse identical upload (espressif, no code change)
continue-on-error: true
@@ -359,15 +397,7 @@
python3 .github/scripts/hil_ci_set_matrix.py test/hil/tinyusb.json |
python3 -c 'import json, sys; [print(a) for a in json.load(sys.stdin)["esp-idf"]]' |
while IFS= read -r arg; do
- # same variant guard as build_util.yml's Membrowse Upload step: a
- # -DMA variant leg configures with the same -DBOARD=<name> as the
- # plain leg, and membrowse's target-name ({board}/{example}) is
- # derived from the board alone - uploading both would write two
- # --identical rows under the same target every no-code-change push
- case " $arg " in
- *' --build-name '*|*' -D'*|*' --cflag='*)
- echo "skip membrowse upload: variant build ($arg)"; continue ;;
- esac
+ # a -DMA variant uploads as <build-name>/<example> (tools/build.py)
python3 tools/build.py -s cmake --target examples-membrowse-upload -j 1 $arg || true
done
diff --git a/.github/workflows/build_util.yml b/.github/workflows/build_util.yml
index a3f09b7..5dcf92c 100644
--- a/.github/workflows/build_util.yml
+++ b/.github/workflows/build_util.yml
@@ -100,6 +100,7 @@
echo "EX_ARGS=$EX_ARGS" >> $GITHUB_ENV
- name: Build
+ id: build
if: ${{ inputs.code-changed }}
env:
IAR_LMS_BEARER_TOKEN: ${{ secrets.IAR_LMS_BEARER_TOKEN }}
@@ -116,6 +117,31 @@
fi
shell: bash
+ - name: Code size snapshot
+ id: code-size
+ # measurement only: it also runs after a failed Build and records that in the
+ # snapshot, and a sizing failure never fails the leg. Before the membrowse upload,
+ # so a membrowse service error cannot cost the local measurement.
+ if: ${{ !cancelled() && inputs.upload-membrowse && inputs.code-changed }}
+ continue-on-error: true
+ env:
+ ARG: ${{ matrix.arg }}
+ BUILD_OUTCOME: ${{ steps.build.outcome }}
+ TOOLCHAIN: ${{ inputs.toolchain }}
+ run: |
+ if [ "$TOOLCHAIN" != esp-idf ]; then
+ python tools/code_size.py snapshot --symbols --build-outcome "$BUILD_OUTCOME" -o code-size $ARG $EX_ARGS
+ exit
+ fi
+ # in the build's image: membrowse reads the /project link commands and linker scripts.
+ # The command is constant (PR-derived values arrive as env), and the container's root
+ # does not own the runner's checkout, which git refuses without safe.directory.
+ docker run --rm -e ARG -e EX_ARGS -e BUILD_OUTCOME -e GITHUB_EVENT_NAME -v "$PWD:/project" -w /project \
+ espressif/idf:tinyusb bash -c 'git config --global --add safe.directory /project &&
+ pip install --only-binary :all: membrowse==1.2.12 >/dev/null &&
+ python tools/code_size.py snapshot --symbols --build-outcome "$BUILD_OUTCOME" -o code-size $ARG $EX_ARGS'
+ shell: bash
+
- name: Membrowse Upload
if: inputs.upload-membrowse == true
continue-on-error: true
@@ -123,18 +149,6 @@
MEMBROWSE_API_KEY: ${{ secrets.MEMBROWSE_API_KEY }}
ARG: ${{ matrix.arg }}
run: |
- # A HIL variant leg (--build-name, an extra -D define, or --cflag beyond
- # the bare board) still configures with the same -DBOARD=<name> as the
- # plain build - membrowse's target-name is <board>/<example>, derived
- # from the board alone - so uploading it would overwrite the plain
- # build's (or another variant's) history under that same name. Only a
- # bare `-b <board> [-e ...]` leg, the plain build, uploads here; a
- # variant board still gets built (for the rig) by this same job, just
- # not uploaded from this leg.
- case " $ARG " in
- *' --build-name '*|*' -D'*|*' --cflag='*)
- echo "skip membrowse upload: variant build ($ARG)"; exit 0 ;;
- esac
# code-changed false -> no elf -> membrowse uploads with --identical.
# --ci-pinned-boards-only: families without a CI board were built as compile
# smoke-checks only and must not upload. -e (in $EX_ARGS or $ARG) only picks
@@ -154,14 +168,14 @@
-e GITHUB_EVENT_PATH="$GITHUB_EVENT_PATH")
DOCKER_MOUNTS=(-v $PWD:/project)
[ -n "$GITHUB_EVENT_PATH" ] && DOCKER_MOUNTS+=(-v "$GITHUB_EVENT_PATH:$GITHUB_EVENT_PATH:ro")
- docker run --rm "${DOCKER_ENV[@]}" "${DOCKER_MOUNTS[@]}" -w /project espressif/idf:tinyusb bash -c "pip install --only-binary :all: membrowse==1.2.9 >/dev/null && $CMD"
+ docker run --rm "${DOCKER_ENV[@]}" "${DOCKER_MOUNTS[@]}" -w /project espressif/idf:tinyusb bash -c "pip install --only-binary :all: membrowse==1.2.12 >/dev/null && $CMD"
else
$CMD
fi
shell: bash
- name: Artifact name
- if: inputs.upload-artifacts == true
+ if: ${{ !cancelled() && (inputs.upload-artifacts || steps.code-size.outcome == 'success' || steps.code-size.outcome == 'failure') }}
env:
ARG: ${{ matrix.arg }}
run: |
@@ -196,3 +210,15 @@
cmake-build/cmake-build-*/*/*/config.env
cmake-build/cmake-build-*/*/*/flash_args
cmake-build/hw/mcu/**/*.ld
+
+ - name: Upload code size snapshot
+ if: ${{ !cancelled() && (steps.code-size.outcome == 'success' || steps.code-size.outcome == 'failure') }}
+ continue-on-error: true
+ uses: actions/upload-artifact@v7
+ with:
+ # the toolchain too: the same family could be a leg of two toolchains
+ name: code-size-${{ inputs.toolchain }}-${{ env.ARTIFACT_TAG }}
+ path: code-size/*.json
+ # master's snapshots are the baselines PRs compare against
+ retention-days: ${{ github.event_name == 'push' && 90 || 14 }}
+ if-no-files-found: ignore
diff --git a/.github/workflows/membrowse-comment.yml b/.github/workflows/membrowse-comment.yml
deleted file mode 100644
index 952d8ba..0000000
--- a/.github/workflows/membrowse-comment.yml
+++ /dev/null
@@ -1,33 +0,0 @@
-name: Membrowse Comment
-
-on:
- workflow_run:
- workflows: ["Build"]
- types:
- - completed
-
-jobs:
- post-comment:
- runs-on: ubuntu-latest
- # Run the comment job even if some of the builds fail
- if: >
- github.event.workflow_run.event == 'pull_request' &&
- github.event.workflow_run.conclusion != 'cancelled'
- permissions:
- contents: read
- actions: read
- pull-requests: write
- steps:
- - name: Checkout repository
- uses: actions/checkout@v6
-
- - name: Post Membrowse PR comment
- if: ${{ env.MEMBROWSE_API_KEY != '' }}
- uses: membrowse/membrowse-action/comment-action@v1
- with:
- api_key: ${{ secrets.MEMBROWSE_API_KEY }}
- commit: ${{ github.event.workflow_run.head_sha }}
- comment_template: .github/membrowse_pr_message.j2
- env:
- MEMBROWSE_API_KEY: ${{ secrets.MEMBROWSE_API_KEY }}
- GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/pr_comment.yml b/.github/workflows/pr_comment.yml
index 1393e13..0f4df77 100644
--- a/.github/workflows/pr_comment.yml
+++ b/.github/workflows/pr_comment.yml
@@ -106,8 +106,106 @@
- name: Post HIL report as sticky PR comment
if: steps.combine.outputs.found == 'true'
- uses: marocchino/sticky-pull-request-comment@v2
+ uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
with:
header: hil-report
path: hil_combined.md
number: ${{ needs.pr_number.outputs.number }}
+
+ # ---------------------------------------
+ # Code size of the PR's CI-pinned boards against the master push run of the commit it was
+ # built against (code_size_ci.py baseline): no base build, only the snapshots both Build
+ # runs uploaded. Artifacts are PR-built data, parsed only by the default branch's scripts
+ # and rendered escaped. Report only: this job never fails the PR.
+ # ---------------------------------------
+ code-size-comment:
+ needs: pr_number
+ if: >
+ github.event.workflow_run.conclusion != 'cancelled' &&
+ needs.pr_number.outputs.number != ''
+ runs-on: ubuntu-latest
+ continue-on-error: true
+ permissions:
+ contents: read
+ actions: read
+ pull-requests: write
+ env:
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ REPO: ${{ github.repository }}
+ steps:
+ - name: Checkout default branch scripts
+ uses: actions/checkout@v6
+ with:
+ persist-credentials: false
+
+ # every step up to the freshness check must succeed, even with nothing to compare:
+ # a failed one would skip the post and leave an earlier push's table looking current
+ - name: Download code size snapshots
+ continue-on-error: true
+ uses: actions/download-artifact@v5
+ with:
+ run-id: ${{ github.event.workflow_run.id }}
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ pattern: code-size-*
+ path: ${{ runner.temp }}/code-size/current
+
+ - name: Find the baseline
+ id: baseline
+ run: |
+ CS="$RUNNER_TEMP/code-size"
+ mkdir -p "$CS/current" "$CS/out"
+ # it reports its own failures in the info; a crash still must not skip the post
+ python3 .github/scripts/code_size_ci.py baseline --repo "$REPO" --current "$CS/current" \
+ --info "$CS/baseline.json" || echo "::warning::code_size_ci.py baseline crashed"
+ echo "run_id=$(jq -r '.run_id // empty' "$CS/baseline.json")" >> "$GITHUB_OUTPUT"
+
+ - name: Download baseline snapshots
+ if: steps.baseline.outputs.run_id != ''
+ continue-on-error: true
+ uses: actions/download-artifact@v5
+ with:
+ run-id: ${{ steps.baseline.outputs.run_id }}
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ pattern: code-size-*
+ path: ${{ runner.temp }}/code-size/base
+
+ - name: Compare against the baseline
+ env:
+ RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+ run: |
+ CS="$RUNNER_TEMP/code-size"
+ if ! python3 tools/code_size.py compare "$CS/current" "$CS/base" -o "$CS/out" --symbols \
+ --baseline-info "$CS/baseline.json"; then
+ printf '## Code size\n\n_The comparison failed._\n' > "$CS/out/comment.md"
+ fi
+ echo "_[Full report]($RUN_URL)_" >> "$CS/out/comment.md"
+ # the job summary caps at 1 MiB
+ if [ -f "$CS/out/code-size.md" ]; then head -c 1000000 "$CS/out/code-size.md" >> "$GITHUB_STEP_SUMMARY"; fi
+
+ - name: Upload full report
+ continue-on-error: true
+ uses: actions/upload-artifact@v7
+ with:
+ name: code-size-report
+ path: ${{ runner.temp }}/code-size/out/code-size.*
+
+ - name: Skip a superseded run
+ id: current
+ env:
+ PR: ${{ needs.pr_number.outputs.number }}
+ RUN_ID: ${{ github.event.workflow_run.id }}
+ ATTEMPT: ${{ github.event.workflow_run.run_attempt }}
+ HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
+ run: |
+ if python3 .github/scripts/code_size_ci.py is-current --repo "$REPO" --pr "$PR" \
+ --run-id "$RUN_ID" --attempt "$ATTEMPT" --head-sha "$HEAD_SHA"; then
+ echo "post=true" >> "$GITHUB_OUTPUT"
+ fi
+
+ - name: Post code size as sticky PR comment
+ if: steps.current.outputs.post == 'true'
+ uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2.9.4
+ with:
+ header: code-size
+ path: ${{ runner.temp }}/code-size/out/comment.md
+ number: ${{ needs.pr_number.outputs.number }}
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index c67d733..059ec1f 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -106,11 +106,11 @@
- id: drivers-coverage
name: drivers-coverage
- files: ^(\.github/ci-pinned-boards\.json$|\.github/scripts/ci_set_matrix\.py$|tools/drivers_coverage_check\.py$|tools/ci_select\.py$|tools/build_utils\.py$|hw/bsp/|src/portable/|examples/|test/hil/test/test_drivers_coverage\.py$|tools/build\.py$|test/hil/test/test_ci_boards\.py$|tools/code_size\.py$|test/hil/test/test_code_size\.py$|tools/membrowse_cli\.py$|test/hil/test/test_membrowse_cli\.py$|test/hil/tinyusb\.json$|test/hil/hfp\.json$)
- entry: sh -c "cd test/hil/test && python3 -m unittest -q test_drivers_coverage test_ci_boards test_code_size test_membrowse_cli"
+ files: ^(\.github/ci-pinned-boards\.json$|\.github/scripts/ci_set_matrix\.py$|tools/drivers_coverage_check\.py$|tools/ci_select\.py$|tools/build_utils\.py$|hw/bsp/|src/portable/|examples/|test/hil/test/test_drivers_coverage\.py$|tools/build\.py$|test/hil/test/test_ci_boards\.py$|tools/code_size\.py$|test/hil/test/test_code_size\.py$|\.github/scripts/code_size_ci\.py$|test/hil/test/test_code_size_ci\.py$|tools/membrowse_cli\.py$|test/hil/test/test_membrowse_cli\.py$|test/hil/tinyusb\.json$|test/hil/hfp\.json$)
+ entry: sh -c "cd test/hil/test && python3 -m unittest -q test_drivers_coverage test_ci_boards test_code_size test_code_size_ci test_membrowse_cli"
pass_filenames: false
language: python
- additional_dependencies: [pyyaml]
+ additional_dependencies: [pyyaml, pyelftools]
# - id: build-fuzzer
# name: build-fuzzer
diff --git a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md
index 17769e1..ee189ce 100644
--- a/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md
+++ b/docs/superpowers/specs/2026-08-19-ci-build-family-filter-design.md
@@ -51,9 +51,9 @@
| 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — |
| 1b | `.gitignore`, `.clang-format`, `.agents`, `.codex/**`, `.idea/**`, `test/{fuzz,unit-test}/**`, `test/hil/test/**`, non-build `.github/**`, packaging manifests | — | — | — |
| 2 | `test/hil/**` (not `test/hil/test/**`) | — | — | all boards → all tests |
-| 2b | `tools/code_size.py`, `tools/drivers_coverage_check.py` | — (local-only tooling, no CI build runs it) | — | — (nothing on the rig runs it) |
+| 2b | `tools/drivers_coverage_check.py` | — (local-only tooling, no CI build runs it) | — | — (nothing on the rig runs it) |
| 2c | `.github/ci-pinned-boards.json` | `ALL` | `ALL` | — (CI board data; no rig board's behaviour depends on it) |
-| 2d | `tools/membrowse_cli.py` | `ALL` | `ALL` | — (build-time script invoked from family_support.cmake; no rig board runs it) |
+| 2d | `tools/membrowse_cli.py`, `tools/code_size.py` | `ALL` | `ALL` | — (size scripts the pinned build legs run; no rig board runs them) |
| 3 | `src/portable/<port>/dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests |
| 4 | `src/portable/<port>/hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests |
| 5 | `src/portable/<port>/**` (anything else) | `FAM` | `ALL` | `FAM`'s boards → all their tests |
diff --git a/hw/bsp/family_support.cmake b/hw/bsp/family_support.cmake
index 215683a..04d12d4 100644
--- a/hw/bsp/family_support.cmake
+++ b/hw/bsp/family_support.cmake
@@ -285,6 +285,10 @@
else ()
set(ELF_TARGET ${TARGET})
endif ()
+ # a variant build (build.py --build-name) reports as a board of its own
+ if (NOT DEFINED MEMBROWSE_BOARD)
+ set(MEMBROWSE_BOARD ${BOARD})
+ endif ()
# For Ninja generator, `tools/membrowse_cli.py report` extracts linker scripts (with
# INCLUDE resolution) and --defsym symbols from the ninja build graph, then runs
@@ -299,7 +303,7 @@
--ninja ${CMAKE_MAKE_PROGRAM}
--elf ${TARGET_ELF_PATH}
# membrowse history is keyed on this name (pinned by test_membrowse_cli): never rename
- --target-name ${BOARD}/${TARGET}
+ --target-name ${MEMBROWSE_BOARD}/${TARGET}
)
if (DEFINED MEMBROWSE_OPTION)
list(APPEND MEMBROWSE_ARGS "--option=${MEMBROWSE_OPTION}")
diff --git a/test/hil/test/test_ci_metrics.py b/test/hil/test/test_ci_metrics.py
index 3db5fb6..faf605f 100644
--- a/test/hil/test/test_ci_metrics.py
+++ b/test/hil/test/test_ci_metrics.py
@@ -209,6 +209,122 @@
# (tools/build.py sets build_examples=None) and compile every example
self.assertNotIn('--ci-pinned-boards-only', self.jobs['cmake'])
+ def test_the_code_size_scope_lists_every_cmake_leg_and_esp_leg(self):
+ # pr_comment.yml's compare expects a snapshot artifact per listed leg: the list
+ # must be the cmake job's matrix, from the one toolchain list both read, plus every
+ # hil-build-esp leg (a -DMA variant is a board of its own) under that job's owner gate
+ import subprocess, tempfile
+ self.assertIn('fromJSON(needs.set-matrix.outputs.cmake_toolchains)', self.jobs['cmake'])
+ self.assertIn("github.repository_owner == 'hathach'", self.jobs['hil-build-esp'])
+ m = re.search(r"echo 'cmake_toolchains=(\[.*\])' >> \$GITHUB_OUTPUT", self.build)
+ toolchains = json.loads(m.group(1))
+ self.assertNotIn('esp-idf', toolchains)
+ i = self.build.index('mkdir -p code-size-scope')
+ i = self.build.rindex('\n', 0, i) + 1
+ j = self.build.index('cat code-size-scope/scope.json', i)
+ block = re.sub(r'^ {10}', '', self.build[i:j], flags=re.M)
+ pinned = {'arm-gcc': ['stm32f4', 'imxrt'], 'riscv-gcc': ['fomu'], 'esp-idf': ['espressif']}
+ hil = {'arm-gcc': ['-b x'], 'esp-idf': [
+ '-b espressif_s3_devkitm -e device/cdc_msc_freertos',
+ '-b espressif_s3_devkitm --build-name espressif_s3_devkitm-DMA --cflag=-DCFG_TUD_DWC2_DMA_ENABLE=1']}
+
+ def scope(changed, owner='hathach'):
+ with tempfile.TemporaryDirectory() as d:
+ subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', block], cwd=d, capture_output=True, check=True,
+ env={**os.environ, 'TOOLCHAINS': m.group(1), 'PINNED_JSON': json.dumps(pinned),
+ 'EXAMPLE_MAP': '{"stm32f4": ["device/cdc_msc"]}', 'CODE_CHANGED': changed,
+ 'HIL_JSON': json.dumps(hil), 'OWNER': owner})
+ with open(os.path.join(d, 'code-size-scope', 'scope.json')) as fh:
+ return json.load(fh)
+ cmake = [{'toolchain': 'arm-gcc', 'arg': 'stm32f4'}, {'toolchain': 'arm-gcc', 'arg': 'imxrt'},
+ {'toolchain': 'riscv-gcc', 'arg': 'fomu'}]
+ s = scope('true')
+ self.assertEqual(s['legs'], cmake + [{'toolchain': 'esp-idf', 'arg': a} for a in hil['esp-idf']])
+ self.assertEqual((s['code_changed'], s['family_examples']), (True, {'stm32f4': ['device/cdc_msc']}))
+ self.assertEqual(scope('true', owner='fork')['legs'], cmake) # hil-build-esp does not run there
+ self.assertEqual(scope('false')['legs'], []) # no code change: nothing to measure
+
+ def test_the_esp_snapshot_runs_in_the_idf_image_with_the_leg_as_data(self):
+ import subprocess, tempfile
+ with open(os.path.join(REPO, '.github', 'workflows', 'build_util.yml')) as f:
+ text = f.read()
+ def step_script(name):
+ i = text.index('run: |\n', text.index(f'- name: {name}\n')) + len('run: |\n')
+ return re.sub(r'^ {10}', '', text[i:text.index(' shell: bash', i)], flags=re.M)
+ block = step_script('Code size snapshot')
+ with tempfile.TemporaryDirectory() as d:
+ # docker runs its inner command here with the env it was told to pass; git, pip
+ # and python only record their argv
+ stubs = {'docker': 'echo "docker ${*:1:$#-1}" >> "$LOG"; exec bash -c "${!#}"',
+ 'git': 'echo "git $*" >> "$LOG"', 'pip': 'echo "pip $*" >> "$LOG"',
+ 'python': '{ printf "python"; printf " [%s]" "$@"; echo; } >> "$LOG"'}
+ for name, body in stubs.items():
+ with open(os.path.join(d, name), 'w') as f:
+ f.write(f'#!/bin/bash\n{body}\n')
+ os.chmod(os.path.join(d, name), 0o755)
+ env = {**os.environ, 'PATH': d + os.pathsep + os.environ['PATH'], 'LOG': os.path.join(d, 'log'),
+ 'TOOLCHAIN': 'esp-idf', 'BUILD_OUTCOME': 'success', 'EX_ARGS': '',
+ 'GITHUB_EVENT_NAME': 'pull_request'}
+
+ def run(arg, toolchain='esp-idf'):
+ open(env['LOG'], 'w').close()
+ r = subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', block], cwd=d, capture_output=True,
+ text=True, env={**env, 'ARG': arg, 'TOOLCHAIN': toolchain})
+ self.assertEqual(r.returncode, 0, r.stderr)
+ with open(env['LOG']) as f:
+ return f.read().splitlines()
+ log = run('-b espressif_s3_devkitm $(touch pwned) ;id')
+ self.assertFalse(os.path.exists(os.path.join(d, 'pwned')))
+ self.assertTrue(log[0].startswith('docker run --rm -e ARG -e EX_ARGS -e BUILD_OUTCOME -e GITHUB_EVENT_NAME '
+ f'-v {d}:/project -w /project espressif/idf:tinyusb bash -c'))
+ self.assertEqual(log[1:3], ['git config --global --add safe.directory /project',
+ 'pip install --only-binary :all: membrowse==1.2.12'])
+ self.assertEqual(log[3], 'python [tools/code_size.py] [snapshot] [--symbols] [--build-outcome] [success] '
+ '[-o] [code-size] [-b] [espressif_s3_devkitm] [$(touch] [pwned)] [;id]')
+ self.assertEqual(run('stm32f4', toolchain='arm-gcc'), # every other toolchain sizes on the runner
+ ['python [tools/code_size.py] [snapshot] [--symbols] [--build-outcome] [success] '
+ '[-o] [code-size] [stm32f4]'])
+ # a variant leg is sized too, as the board its --build-name names
+ self.assertEqual(run('-b espressif_s3_devkitm --build-name x-DMA --cflag=-DY=1')[-1],
+ 'python [tools/code_size.py] [snapshot] [--symbols] [--build-outcome] [success] '
+ '[-o] [code-size] [-b] [espressif_s3_devkitm] [--build-name] [x-DMA] [--cflag=-DY=1]')
+
+ def test_the_code_size_comment_is_written_even_without_usable_snapshots(self):
+ # a run without usable snapshots, and a baseline lookup that fails, must still
+ # leave a comment.md, so the post replaces an earlier push's table
+ import subprocess, tempfile
+ with open(os.path.join(REPO, '.github', 'workflows', 'pr_comment.yml')) as f:
+ text = f.read()
+
+ def step_script(name):
+ i = text.index('run: |\n', text.index(f'- name: {name}\n')) + len('run: |\n')
+ j = re.search(r'^\s*$', text[i:], re.M).start() + i # the block ends at the first blank line
+ return re.sub(r'^ {10}', '', text[i:j], flags=re.M)
+ with tempfile.TemporaryDirectory() as d:
+ env = {**os.environ, 'RUNNER_TEMP': d, 'REPO': 'x/y', 'RUN_URL': 'https://example/run',
+ 'GITHUB_STEP_SUMMARY': os.path.join(d, 'summary.md'), 'GITHUB_OUTPUT': os.path.join(d, 'out'),
+ 'PATH': d + os.pathsep + os.environ['PATH']}
+ with open(os.path.join(d, 'gh'), 'w') as f: # any API call fails
+ f.write('#!/bin/sh\nexit 1\n')
+ os.chmod(os.path.join(d, 'gh'), 0o755)
+ leg = os.path.join(d, 'code-size', 'current', 'code-size-arm-gcc-x')
+ os.makedirs(leg)
+ with open(os.path.join(leg, 'leg.json'), 'w') as f:
+ json.dump({'schema': 1, 'boards': [], 'examples': None, 'build_outcome': 'success',
+ 'sha': 'a' * 40, 'base_sha': 'a' * 40, 'head_sha': 'a' * 40}, f)
+ for name in ('Find the baseline', 'Compare against the baseline'):
+ r = subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', step_script(name)], cwd=REPO,
+ capture_output=True, text=True, env=env)
+ self.assertEqual(r.returncode, 0, f'{name}: {r.stderr}')
+ with open(os.path.join(d, 'out')) as f:
+ self.assertEqual(f.read(), 'run_id=\n') # no baseline download
+ with open(os.path.join(d, 'code-size', 'out', 'comment.md')) as f:
+ comment = f.read()
+ self.assertTrue(comment.startswith('## Code size'))
+ self.assertIn('Baseline: unavailable - the baseline lookup failed', comment)
+ self.assertIn('no usable scope manifest', comment)
+ self.assertTrue(comment.endswith('_[Full report](https://example/run)_\n'))
+
def _run_matrix_step(self, sel, fail_pinned=False):
"""Run the whole 'Generate matrix json' step for real, optionally with the
SCOPED --pinned invocation failing (the unscoped fallback still works, as a
diff --git a/test/hil/test/test_ci_select.py b/test/hil/test/test_ci_select.py
index a5ef682..35e7a8e 100644
--- a/test/hil/test/test_ci_select.py
+++ b/test/hil/test/test_ci_select.py
@@ -1249,6 +1249,7 @@
'test/hil/test/test_ci_metrics.py',
'test/hil/test/test_ci_select.py',
'test/hil/test/test_code_size.py',
+ 'test/hil/test/test_code_size_ci.py',
'test/hil/test/test_drivers_coverage.py',
'test/hil/test/test_family_json.py',
'test/hil/test/test_hil_args.py',
@@ -1874,7 +1875,7 @@
'.clang-format', '.idea/misc.xml', 'version.yml', 'library.json',
'examples/CMakePresets.json', 'test/fuzz/fuzz.cc',
'test/unit-test/project.yml', '.github/workflows/pr_comment.yml',
- '.claude/skills/build-doc/scripts/gen_doc.py', '.agents'):
+ '.github/scripts/code_size_ci.py', '.claude/skills/build-doc/scripts/gen_doc.py', '.agents'):
s = self.b([p])
self.assertFalse(s['full'], p)
self.assertEqual(s['families'], [], p)
@@ -1992,9 +1993,8 @@
whole build matrix plus an exclusive full-rig sweep - where master ran nothing."""
def test_local_tooling_contributes_nothing_on_either_axis(self):
- # no CI build and no rig board runs any of these (drivers_coverage_check.py is
- # pre-commit only)
- for p in ('tools/code_size.py', 'tools/drivers_coverage_check.py'):
+ # no CI build and no rig board runs it (pre-commit only)
+ for p in ('tools/drivers_coverage_check.py',):
h = sel([p])
self.assertFalse(h['full'], p)
self.assertEqual(h['boards'], {}, p)
@@ -2010,13 +2010,14 @@
self.assertEqual(h['boards'], {}, p)
self.assertTrue(ci_select.classify_build([p], REPO)['full'], p)
- def test_membrowse_cli_is_a_full_build_matrix_without_hil(self):
- # run by family_add_membrowse() for every pinned board (rule 2d)
- p = 'tools/membrowse_cli.py'
- h = sel([p])
- self.assertFalse(h['full'], p)
- self.assertEqual(h['boards'], {}, p)
- self.assertTrue(ci_select.classify_build([p], REPO)['full'], p)
+ def test_ci_size_scripts_are_a_full_build_matrix_without_hil(self):
+ # run by every pinned build leg (rule 2d): membrowse_cli.py from
+ # family_add_membrowse(), code_size.py as the CI size snapshot
+ for p in ('tools/membrowse_cli.py', 'tools/code_size.py'):
+ h = sel([p])
+ self.assertFalse(h['full'], p)
+ self.assertEqual(h['boards'], {}, p)
+ self.assertTrue(ci_select.classify_build([p], REPO)['full'], p)
def test_typec_example_builds_but_runs_nothing(self):
# examples/typec is compiled by the build matrix and run by no rig board; the
@@ -2305,6 +2306,25 @@
self.assertEqual(r.returncode, 0, r.stderr)
return r.stdout
+ def test_an_esp_idf_variant_must_name_itself(self):
+ # hil-build-esp sizes and uploads every leg under its --build-name: an unnamed
+ # esp-idf variant would collide with the plain board; other buckets never upload
+ import tempfile
+ other = {'name': 'metro_m4_express', 'uid': 'm', 'flasher': {'name': 'jlink'},
+ 'variant': [{'name': 'metro_m4_express', 'defines': ['MAX3421_HOST=1']}]}
+ for vname, ok in (('espressif_s3_devkitm', False), ('espressif_s3_devkitm-X', True)):
+ esp = {'name': 'espressif_s3_devkitm', 'uid': 'u', 'flasher': {'name': 'esptool'},
+ 'variant': [{'name': vname, 'defines': ['X=1']}]}
+ with tempfile.NamedTemporaryFile('w', suffix='.json') as f:
+ json.dump({'boards': [esp, other]}, f)
+ f.flush()
+ r = subprocess.run([sys.executable, HIL_SET_MATRIX, f.name], capture_output=True, text=True)
+ self.assertEqual(r.returncode == 0, ok, r.stderr)
+ if ok:
+ self.assertEqual(json.loads(r.stdout)['arm-gcc'], ['-b metro_m4_express -DMAX3421_HOST=1'])
+ else:
+ self.assertIn('an esp-idf variant needs a name of its own', r.stderr)
+
def test_no_hil_examples_is_byte_identical(self):
plain = self.run_matrix()
sel = json.dumps({'full': True, 'boards': {}})
@@ -2437,6 +2457,36 @@
self.assertEqual(cmd[cmd.index('--elf') + 1],
'cmake-build/cmake-build-espressif_s3_devkitc/device/cdc_msc_freertos/cdc_msc_freertos.elf')
+ def test_a_variant_is_a_membrowse_board_named_by_its_build_name(self):
+ from unittest import mock
+ calls = []
+ def fake_run(cmd):
+ calls.append(cmd)
+ return types.SimpleNamespace(returncode=0)
+ real_isdir = os.path.isdir
+ no_build_dir = lambda p: False if str(p).startswith('cmake-build/') else real_isdir(p)
+ with mock.patch.object(self.build, 'run_cmd', fake_run), \
+ mock.patch.object(self.build.os.path, 'isdir', no_build_dir):
+ self.build.cmake_board('espressif_s3_devkitc', [], 'espressif_s3_devkitc-DMA', ['-DX=1'], ['all'],
+ examples=['device/cdc_msc_freertos'])
+ self.build.cmake_board('espressif_s3_devkitc', [], 'espressif_s3_devkitc-DMA', [],
+ ['examples-membrowse-upload'], examples=['device/cdc_msc_freertos'])
+ idf, identical = calls
+ self.assertIn('-DMEMBROWSE_BOARD=espressif_s3_devkitc-DMA', idf)
+ self.assertIn('cmake-build/cmake-build-espressif_s3_devkitc-DMA/device/cdc_msc_freertos', idf)
+ self.assertEqual(identical[identical.index('--target-name') + 1], 'espressif_s3_devkitc-DMA/cdc_msc_freertos')
+
+ def test_an_unnamed_variant_never_uploads_under_the_plain_board(self):
+ from unittest import mock
+ for argv, refused in ((['-b', 'b1', '--cflag=-DX=1'], True), (['-b', 'b1', '-DX=1'], True),
+ (['-b', 'b1', '--build-name', 'b1-X', '--cflag=-DX=1'], False), (['-b', 'b1'], False)):
+ with mock.patch.object(self.build, 'build_boards_list', return_value=[1, 0, 0]) as build, \
+ mock.patch.object(sys, 'argv', ['build.py', '-T', 'examples-membrowse-upload'] + argv), \
+ contextlib.redirect_stdout(io.StringIO()) as out:
+ ret = self.build.main()
+ self.assertEqual((ret, build.called), (1, False) if refused else (0, True), argv)
+ self.assertEqual('requires --build-name' in out.getvalue(), refused)
+
def test_make_one_example_uses_make_semantics(self):
# F1 end to end: the make path must ask skip_example with build_system='make',
# or lpc54's cmake-only FAMILY_MCUS un-skips a host example whose make build
diff --git a/test/hil/test/test_code_size.py b/test/hil/test/test_code_size.py
index 7c9812a..5414d18 100644
--- a/test/hil/test/test_code_size.py
+++ b/test/hil/test/test_code_size.py
@@ -64,6 +64,21 @@
self.assertEqual(cmd[cmd.index('--def') + 1], 'X=1')
self.assertEqual(run.call_args.kwargs['cwd'], build)
+ def test_an_esp_idf_build_uses_its_generated_scripts(self):
+ with tempfile.TemporaryDirectory() as build:
+ open(os.path.join(build, 'build.ninja'), 'w').close()
+ ld = os.path.join(build, 'esp-idf', 'esp_system', 'ld')
+ os.makedirs(ld)
+ commands = 'cc -T memory.ld -T sections.ld -T esp32s3.rom.ld -Wl,--defsym=X=1 -o x.elf\n'
+ open(os.path.join(ld, 'memory.ld'), 'w').close()
+ with mock.patch.object(sd, 'link_command', return_value=commands), \
+ self.assertRaisesRegex(RuntimeError, 'lacks one of'):
+ sd._link_settings(os.path.join(build, 'x.elf'))
+ open(os.path.join(ld, 'sections.ld'), 'w').close()
+ with mock.patch.object(sd, 'link_command', return_value=commands):
+ self.assertEqual(sd._link_settings(os.path.join(build, 'x.elf')),
+ (build, [os.path.join(ld, 'memory.ld'), os.path.join(ld, 'sections.ld')], ['X=1']))
+
def test_malformed_output_is_a_runtime_error(self):
done = subprocess.CompletedProcess([], 0, 'not json', '')
with mock.patch.object(sd, '_link_settings', return_value=('/b', ['m.ld'], [])), \
@@ -167,7 +182,7 @@
a, = pair_ids(1)
md = self.report({a: elf({'x.c': (60, 20), 'y.c': (20, 0)}, all_syms=(500, 40))}, 'membrowse')
self.assertIn('**Coverage (complete, membrowse):** 1 of 1 elfs sized', md)
- self.assertIn('filtered Flash 80, RAM 20; all symbols Flash 500, RAM 40', md)
+ self.assertIn('TinyUSB Flash 80, RAM 20; all symbols Flash 500, RAM 40', md)
self.assertIn('| File | .text | .bss | size | % |', md)
self.assertLess(md.index('| x.c | 60 | 20 | 80 | 80.0% |'), md.index('| y.c | 20 | 0 | 20 | 20.0% |'))
self.assertIn('| TOTAL | 80 | 20 | 100 | 100.0% |', md)
@@ -461,6 +476,44 @@
b = self.buckets([('.init_array', INIT_ARRAY, WA, 0x54a4, 4)], [(0x200, 0x200, 0x6000)])
self.assertEqual(b['.init_array'], {'flash'})
+ def test_an_esp_idf_app_is_split_by_its_flash_mapped_sections(self):
+ # every section runs where it loads; .flash* is flash-mapped, the rest the bootloader
+ # copies from the image into RAM (real S3 names, incl. writable .flash.rodata)
+ sections = [('.flash.appdesc', PROGBITS, A, 0x3c020020, 0x100),
+ ('.flash.rodata', PROGBITS, WA, 0x3c020120, 0x10),
+ ('.flash.text', PROGBITS, AX, 0x42000020, 0x10),
+ ('.flash_rodata_dummy', NOBITS, WA, 0x3c000020, 0x20000),
+ ('.iram0.text', PROGBITS, AX, 0x40374404, 0x10),
+ ('.dram0.data', PROGBITS, WA, 0x3fc88000, 0x10),
+ ('.rtc.force_slow', PROGBITS, WA, 0x50000000, 0x10),
+ ('.dram0.bss', NOBITS, WA, 0x3fc96d20, 0x10)]
+ with tempfile.TemporaryDirectory() as tmp:
+ elf = os.path.join(tmp, 'x.elf')
+ write_elf(elf, sections, [(0, 0, 0xffffffff)])
+ b = sd.section_buckets(elf, os.path.join(tmp, 'no.map')) # an IDF app never reads the map
+ self.assertEqual({n: set(v) for n, v in b.items() if n != '.shstrtab'},
+ {'.flash.appdesc': {'flash'}, '.flash.rodata': {'flash'}, '.flash.text': {'flash'},
+ '.flash_rodata_dummy': set(), '.iram0.text': {'flash', 'ram'},
+ '.dram0.data': {'flash', 'ram'}, '.rtc.force_slow': {'flash', 'ram'},
+ '.dram0.bss': {'ram'}})
+
+ def test_overlapping_flash_and_ram_regions_still_fail_outside_esp_idf(self):
+ regions = MAP_REGIONS.replace('m_text 0x00000000 0x00010000 xr',
+ 'drom0_0_seg 0x3c000020 0x01ffffe0 r\n'
+ 'extern_ram_seg 0x3c000020 0x01ffffe0 xrw')
+ with self.assertRaisesRegex(RuntimeError, 'no map region recognized'):
+ self.buckets([('.ext_ram.data', PROGBITS, WA, 0x3c040020, 0x10)],
+ [(0x3c040020, 0x3c040020, 0x200)], regions)
+
+ def test_an_esp_idf_map_is_named_after_the_project(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ elf = os.path.join(tmp, 'x.elf')
+ self.assertEqual(sd._map_path(elf), elf + '.map') # neither: the usual name, reported missing
+ open(os.path.join(tmp, 'x.map'), 'w').close()
+ self.assertEqual(sd._map_path(elf), os.path.join(tmp, 'x.map'))
+ open(elf + '.map', 'w').close()
+ self.assertEqual(sd._map_path(elf), elf + '.map')
+
def test_writable_data_in_place_takes_its_map_region(self):
# RAM-only image (raspberrypi_zero): .data runs where it loads, in RAM
b = self.buckets([('.data', PROGBITS, WA, 0x20000000, 0x10)], [(0x20000000, 0x20000000, 0x10)])
@@ -580,6 +633,42 @@
[(0x10000000, 0x10000000, 0x400), (0x20000000, 0x10000400, 0x30)])
+class DwarfSources(unittest.TestCase):
+ @unittest.skipUnless(shutil.which('gcc'), 'needs gcc to build an elf with DWARF')
+ def test_compile_units_are_found_by_basename(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ for rel, code in (('src/device/usbd.c', 'int usbd(void) { return 1; }\n'),
+ ('main.c', 'int usbd(void);\nint main(void) { return usbd(); }\n')):
+ os.makedirs(os.path.dirname(os.path.join(tmp, rel)), exist_ok=True)
+ with open(os.path.join(tmp, rel), 'w') as f:
+ f.write(code)
+ subprocess.run(['gcc', '-g', '-o', 'x.elf', 'main.c', 'src/device/usbd.c'], cwd=tmp, check=True)
+ sources = sd._dwarf_sources(os.path.join(tmp, 'x.elf'))
+ self.assertEqual(sources['usbd.c'], {os.path.join(os.path.realpath(tmp), 'src', 'device', 'usbd.c')})
+
+
+class SourcePath(unittest.TestCase):
+ def path(self, sym, sources):
+ with mock.patch.object(sd, '_dwarf_sources', return_value=sources):
+ return sd._source_path(sym, 'x.elf', ['/c/src/'])
+
+ def test_an_object_path_or_full_source_is_kept(self):
+ self.assertEqual(self.path({'object_file': '/c/src/tusb.c.obj', 'source_file': 'tusb.c'}, {}),
+ '/c/src/tusb.c.obj')
+ self.assertEqual(self.path({'source_file': '/c/src/tusb.c'}, {}), '/c/src/tusb.c')
+
+ def test_a_bare_source_name_takes_its_one_compile_unit(self):
+ # ESP-IDF archive member: membrowse keeps only the basename
+ self.assertEqual(self.path({'source_file': 'usbd.c'}, {'usbd.c': {'/c/src/device/usbd.c'}}),
+ '/c/src/device/usbd.c')
+
+ def test_an_ambiguous_name_fails_only_when_one_candidate_is_filtered(self):
+ self.assertEqual(self.path({'source_file': 'port.c'}, {'port.c': {'/idf/a/port.c', '/idf/b/port.c'}}),
+ 'port.c')
+ with self.assertRaisesRegex(RuntimeError, 'several compile units'):
+ self.path({'source_file': 'x.c', 'name': 'f'}, {'x.c': {'/c/src/x.c', '/idf/x.c'}})
+
+
class MembrowseSizes(unittest.TestCase):
def sizes(self, symbols, filters=('/co/src/',)):
with tempfile.TemporaryDirectory() as tmp:
@@ -622,6 +711,27 @@
with self.assertRaisesRegex(RuntimeError, r'section \.gone, not in'):
self.sizes([{'name': 'g', 'size': 4, 'section': '.gone', 'object_file': 'o/co/src/g.c.obj'}])
+ @staticmethod
+ def pins(text):
+ return re.findall(r'pip3? install [^\n]*?\bmembrowse\b(==[^\s\'"]+)?', text.replace('\\\n', ' '))
+
+ def test_pins_reports_unpinned_installs_in_every_shell_form(self):
+ self.assertEqual(self.pins('pip3 install membrowse\n'
+ 'pip install \\\n --only-binary :all: \\\n membrowse\n'
+ 'python -m pip install membrowse\n'), ['', '', ''])
+
+ def test_every_ci_install_pins_the_version_the_install_hint_names(self):
+ self.assertEqual(self.pins("pip install membrowse==1.2.12rc1 'x'; pip install membrowse >/dev/null"),
+ ['==1.2.12rc1', ''])
+ installs = []
+ for root, _, files in os.walk(os.path.join(REPO, '.github')):
+ for name in files:
+ with open(os.path.join(root, name), errors='replace') as f:
+ installs += self.pins(f.read())
+ self.assertGreaterEqual(len(installs), 4)
+ self.assertEqual(set(installs), {f'=={sd.MEMBROWSE_CI_VERSION}'})
+ self.assertIn(f'membrowse=={sd.MEMBROWSE_CI_VERSION}', sd.ENGINES['membrowse'].install)
+
class BloatySizes(unittest.TestCase):
def sizes(self, csv_text, returncode=0):
@@ -706,6 +816,15 @@
self.assertIn('another --engine', errors[0][1])
self.assertEqual(out, '')
+ def test_fetched_tools_under_deps_are_not_examples(self):
+ # picotool's enc_bootloader.elf has no linker map, so sizing it fails every rp2 board
+ sizer = mock.Mock(return_value=_elf(7))
+ with tempfile.TemporaryDirectory() as build:
+ _touch(build, 'device/ex/ex.elf', '_deps/picotool/enc_bootloader.elf')
+ with _engine('bloaty', sizer):
+ sizes, errors = sd.generate_sizes(build, ['src/'], engine='bloaty')
+ self.assertEqual((list(sizes), errors), (['device/ex/ex.elf'], []))
+
def test_the_selected_engine_sizes_each_elf(self):
bloaty = mock.Mock(return_value=_elf(7))
with mock.patch('glob.glob', return_value=['/fake/build/ex/ex.elf']), \
@@ -881,6 +1000,105 @@
sd.build_board(tmp, os.path.join(tmp, 'b'), 'b', 'device/ex', 'build')
self.assertEqual(run.call_args, mock.call(['ninja', '-C', os.path.join(tmp, 'b'), 'ex'], timeout=600))
+ def _esp_src(self, tmp):
+ """A checkout with one espressif board, one example and a dependency linked in two
+ hops, as a base worktree's: src -> checkout -> main checkout."""
+ src = os.path.join(tmp, 'src')
+ os.makedirs(os.path.join(src, 'hw', 'bsp', 'espressif', 'boards', 'esp'))
+ # an ESP-IDF component, which get_examples('espressif') requires
+ _touch(src, 'examples/device/a_freertos/src/CMakeLists.txt')
+ os.makedirs(os.path.join(src, 'tools'))
+ os.makedirs(os.path.join(src, 'lib'))
+ os.makedirs(os.path.join(tmp, 'main', 'lib', 'dep'))
+ os.makedirs(os.path.join(tmp, 'checkout', 'lib'))
+ os.symlink(os.path.join(tmp, 'main', 'lib', 'dep'), os.path.join(tmp, 'checkout', 'lib', 'dep'))
+ os.symlink(os.path.join(tmp, 'checkout', 'lib', 'dep'), os.path.join(src, 'lib', 'dep'))
+ with open(os.path.join(src, 'tools', 'get_deps.py'), 'w') as f:
+ f.write("deps_all = {'lib/dep': None, 'lib/absent': None}\n")
+ return src
+
+ def _build_esp(self, tmp, example='device/a_freertos', which=('idf.py',), inspect=0, rc=0, runs=None):
+ """(build_board's error, its run() calls but the image check, also into `runs`); `rc`
+ an exception class raises it from the build."""
+ runs = [] if runs is None else runs
+ def run(cmd, timeout=None):
+ if cmd[:3] == ['docker', 'image', 'inspect']:
+ return subprocess.CompletedProcess([], inspect, '', '')
+ runs.append((cmd, timeout))
+ if isinstance(rc, type) and 'app' in cmd:
+ raise rc
+ return subprocess.CompletedProcess([], rc, '', 'x')
+ with mock.patch.object(sd, 'run', run), mock.patch.object(sd, 'CODE_SIZE_DIR', tmp), \
+ mock.patch.object(sd.shutil, 'which', lambda name: f'/bin/{name}' if name in which else None), \
+ mock.patch.object(sd.build_utils, 'skip_example', return_value=False), \
+ contextlib.redirect_stdout(io.StringIO()):
+ error = sd.build_board(self._esp_src(tmp), os.path.join(tmp, 'b'), 'esp', example, 'build')
+ return error, runs
+
+ def test_an_espressif_board_builds_each_examples_app_as_an_idf_project(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ error, runs = self._build_esp(tmp)
+ self.assertIsNone(error)
+ self.assertEqual(runs, [(['idf.py', '-C', os.path.join(tmp, 'src', 'examples', 'device', 'a_freertos'),
+ '-B', os.path.join(tmp, 'b', 'device', 'a_freertos'), '-GNinja', '-DBOARD=esp',
+ 'app'], 600)])
+
+ def test_without_an_exported_idf_it_builds_in_cis_image_as_this_user(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ error, runs = self._build_esp(tmp, which=('docker',))
+ dep, cache = os.path.realpath(os.path.join(tmp, 'main', 'lib', 'dep')), os.path.join(tmp, '_ccache')
+ mounts = [f'{os.path.realpath(tmp)}/_ccache:{cache}', f'{os.path.realpath(tmp)}/b:{tmp}/b',
+ f'{dep}:{tmp}/checkout/lib/dep', f'{dep}:{tmp}/main/lib/dep',
+ f'{os.path.realpath(tmp)}/src:{tmp}/src']
+ self.assertTrue(os.path.isdir(cache))
+ self.assertIsNone(error)
+ cmd = runs[0][0]
+ self.assertEqual(cmd[:11], ['docker', 'run', '--rm', '--name', f'tinyusb-code-size-{os.getpid()}',
+ '--user', f'{os.getuid()}:{os.getgid()}', '-e', 'HOME=/tmp',
+ '-e', f'CCACHE_DIR={cache}'])
+ self.assertEqual([cmd[i + 1] for i, a in enumerate(cmd) if a == '-v'], mounts)
+ self.assertEqual(cmd[cmd.index(sd.ESP_IDF_IMAGE):][:3], [sd.ESP_IDF_IMAGE, 'idf.py', '-C'])
+
+ def test_without_idf_or_its_image_the_espressif_build_fails_saying_how_to_get_one(self):
+ for which, inspect in (((), 0), (('docker',), 1)):
+ with tempfile.TemporaryDirectory() as tmp:
+ error, runs = self._build_esp(tmp, which=which, inspect=inspect)
+ self.assertEqual(runs, [])
+ self.assertIn('esp needs ESP-IDF: source $IDF_PATH/export.sh', error)
+
+ def test_without_idf_report_and_diff_refuse_an_espressif_board_before_building(self):
+ pinned = ['stm32f407disco', 'espressif_s3_devkitm']
+ for argv in (['report', '-b', pinned[0], '-b', pinned[1]], ['diff', '-b', pinned[0], '-b', pinned[1]],
+ ['diff', '--ci']):
+ err = io.StringIO()
+ with mock.patch.object(sys, 'argv', ['code_size.py'] + argv), \
+ mock.patch.object(sd, 'ci_pinned_boards', return_value=pinned), \
+ mock.patch.object(sd, 'engine_missing', return_value=False), \
+ mock.patch.object(sd.shutil, 'which', return_value=None), \
+ mock.patch.object(sd, 'run') as run, contextlib.redirect_stderr(err):
+ with self.assertRaises(SystemExit):
+ sd.main()
+ run.assert_not_called()
+ self.assertIn('espressif_s3_devkitm need ESP-IDF: source $IDF_PATH/export.sh', err.getvalue())
+
+ def test_an_example_the_tree_does_not_build_for_espressif_fails_the_build(self):
+ # device/board_test: an espressif example absent from this tree
+ for example in ('device/cdc_msc', 'device/board_test'):
+ with tempfile.TemporaryDirectory() as tmp:
+ error, runs = self._build_esp(tmp, example=example)
+ self.assertEqual((error, runs), (f'esp builds no {example}', []))
+
+ def test_a_timed_out_or_interrupted_docker_build_removes_its_container(self):
+ rm = ['docker', 'rm', '-f', f'tinyusb-code-size-{os.getpid()}']
+ for which, rc, removed in ((('docker',), 124, True), (('docker',), 2, False), (('idf.py',), 124, False)):
+ with tempfile.TemporaryDirectory() as tmp:
+ _error, runs = self._build_esp(tmp, which=which, rc=rc)
+ self.assertEqual(runs[-1][0] == rm, removed, (which, rc))
+ runs = []
+ with tempfile.TemporaryDirectory() as tmp, self.assertRaises(KeyboardInterrupt):
+ self._build_esp(tmp, which=('docker',), rc=KeyboardInterrupt, runs=runs)
+ self.assertEqual(runs[-1][0], rm)
+
def test_a_timeout_returns_124_and_keeps_the_output_as_text(self):
ret = sd.run(['sh', '-c', 'printf err >&2; echo out; exec sleep 30'], timeout=1)
self.assertEqual(ret.returncode, 124)
@@ -1095,7 +1313,7 @@
with open(os.path.join(tmp, 'b', 'diff.md')) as f:
self.assertIn('| x.c | 1 → 3 | +2 | 0 → 0 | 0 |', unpad(f.read()))
# no -e: the console gets the summary line, the tables stay in the .md
- self.assertIn(' 1 pair, 1 changed; filtered Flash Δ +2, RAM Δ 0\n', out)
+ self.assertIn(' 1 pair, 1 changed; TinyUSB Flash Δ +2, RAM Δ 0\n', out)
self.assertNotIn('| x.c |', out)
def test_a_filter_mismatch_fails_the_size_and_compare_phase(self):
@@ -1192,7 +1410,7 @@
rc, out = self._run_main(tmp, ['-b', 'b', '-e', 'ex'], build, lambda *_a, **_k: next(side_sizes))
self.assertEqual(rc, 0)
self.assertRegex(out, r'^diff master \(c0ffee\) vs working tree \(c0ffee\) · membrowse\n\[1/1\] b / ex\n'
- r' size and compare… \d+\.\ds\n 1 pair, 1 changed; filtered Flash Δ \+2')
+ r' size and compare… \d+\.\ds\n 1 pair, 1 changed; TinyUSB Flash Δ \+2')
# indented tables; unpad() folds their indent to one space
self.assertIn('\n | x.c | 1 → 3 | +2 | 0 → 0 | 0 |', out)
self.assertIn('\n | x.c | +2 | +2 |', out)
@@ -1223,10 +1441,19 @@
rc, out = self._main(tmp, sizes=({'ex/ex.elf': _elf(8)}, []),
cur_sizes=({'ex/ex.elf': empty}, []))
self.assertEqual(rc, 0)
- self.assertIn('1 pair, 1 changed; filtered Flash Δ -8', out)
+ self.assertIn('1 pair, 1 changed; TinyUSB Flash Δ -8', out)
with open(os.path.join(tmp, 'b', 'diff.md')) as f:
self.assertIn('| x.c | 8 → 0 | -8 | 0 → 0 | 0 |', unpad(f.read()))
+ def test_a_custom_filter_is_not_called_tinyusb(self):
+ def build(_src, _build_dir, board, *_args, **_kwargs):
+ os.makedirs(os.path.join(tmp, board), exist_ok=True)
+ side_sizes = iter([({'ex/ex.elf': _elf(1)}, []), ({'ex/ex.elf': _elf(3)}, [])])
+ with tempfile.TemporaryDirectory() as tmp:
+ rc, out = self._run_main(tmp, ['-b', 'b', '-f', 'lib/'], build, lambda *_a, **_k: next(side_sizes))
+ self.assertEqual(rc, 0)
+ self.assertIn('1 pair, 1 changed; filtered Flash Δ +2', out)
+
def _main_combined(self, tmp, sizes, build_ok=('b1', 'b2'), example=None):
"""main() with -b b1 -b b2 --combined. `sizes` maps
(board, side) to what generate_sizes() returns; boards not in
@@ -1558,7 +1785,7 @@
self.assertIn('Coverage (complete, bloaty)', md)
self.assertIn('| x.c | 4 | 4 | 100.0% |', md)
self.assertRegex(self.out, r'^report working tree · bloaty\n\[1/1\] b / device/cdc_msc\n'
- r' size… \d+\.\ds\n 1 of 1 elfs sized; filtered Flash 4, RAM 0\n')
+ r' size… \d+\.\ds\n 1 of 1 elfs sized; TinyUSB Flash 4, RAM 0\n')
self.assertIn('\n | x.c | 4 | 4 | 100.0% |', self.out)
def test_an_examples_trailing_slash_is_dropped(self):
@@ -1708,6 +1935,530 @@
'"uncovered": []}')
self.assertEqual(built, ['extra', 'b1', 'b2'])
+ def test_ci_builds_pinned_espressif_boards_too(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ built = self._boards_built(tmp, '{"boards": [{"board": "b1"}, '
+ '{"board": "espressif_s3_devkitm"}], "uncovered": []}')
+ self.assertEqual(built, ['extra', 'b1', 'espressif_s3_devkitm'])
+
+
+def _touch(root, *rels):
+ for rel in rels:
+ os.makedirs(os.path.dirname(os.path.join(root, rel)), exist_ok=True)
+ open(os.path.join(root, rel), 'w').close()
+
+
+class Snapshot(unittest.TestCase):
+ SHA = [c * 40 for c in 'abc']
+
+ def test_a_missing_build_dir_is_a_build_failure(self):
+ elfs, failures = sd.board_snapshot('b1', '/nonexistent/cmake-build-b1', None, ['src/'])
+ self.assertEqual(elfs, {})
+ self.assertEqual([(f['elf'], f['stage']) for f in failures], [(None, 'build')])
+
+ def test_sizes_every_elf_and_records_failed_ones(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ _touch(tmp, 'device/a/a.elf', 'device/a/loader.elf', 'host/bad/bad.elf')
+
+ def sizes(path, _filters):
+ if 'bad' in path:
+ raise RuntimeError('membrowse report failed')
+ return _elf(5)
+ with _engine('membrowse', sizes):
+ elfs, failures = sd.board_snapshot('b1', tmp, None, ['src/'])
+ self.assertEqual(sorted(elfs), ['device/a/a.elf', 'device/a/loader.elf'])
+ self.assertEqual([(f['elf'], f['stage']) for f in failures], [('host/bad/bad.elf', 'report')])
+
+ def test_scoped_examples_skip_what_the_board_skips_and_fail_a_missing_one(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ _touch(tmp, 'device/a/a.elf')
+ with _engine('membrowse', lambda _p, _f: _elf(5)), \
+ mock.patch.object(sd.build_utils, 'skip_example', side_effect=lambda e, b, _d: e == 'device/skipped'):
+ elfs, failures = sd.board_snapshot('b1', tmp, ['device/a', 'device/skipped', 'device/gone'],
+ ['src/'])
+ self.assertEqual(list(elfs), ['device/a/a.elf'])
+ self.assertEqual(len(failures), 1)
+ self.assertEqual(failures[0]['stage'], 'build')
+ self.assertIn('device/gone', failures[0]['message'])
+
+ def test_no_tinyusb_file_in_any_elf_fails_the_filter(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ _touch(tmp, 'device/a/a.elf')
+ with _engine('membrowse', lambda _p, _f: elf({})):
+ elfs, failures = sd.board_snapshot('b1', tmp, None, ['src/'])
+ self.assertEqual(list(elfs), ['device/a/a.elf'])
+ self.assertEqual([f['stage'] for f in failures], ['filter'])
+
+ def test_compiler_comes_from_cmake_not_the_host(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ os.makedirs(os.path.join(tmp, 'CMakeFiles', '4.1.2'))
+ with open(os.path.join(tmp, 'CMakeFiles', '4.1.2', 'CMakeCCompiler.cmake'), 'w') as f:
+ f.write('set(CMAKE_C_COMPILER "/opt/arm/bin/arm-none-eabi-gcc")\n'
+ 'set(CMAKE_C_COMPILER_ID "GNU")\nset(CMAKE_C_COMPILER_VERSION "13.3.1")\n')
+ with open(os.path.join(tmp, 'CMakeCache.txt'), 'w') as f:
+ f.write('CMAKE_BUILD_TYPE:STRING=MinSizeRel\n')
+ self.assertEqual(sd._cmake_compiler(tmp), {'id': 'GNU', 'version': '13.3.1',
+ 'name': 'arm-none-eabi-gcc', 'build_type': 'MinSizeRel'})
+ self.assertEqual(sd._cmake_compiler('/nonexistent'), {'id': '', 'version': '', 'name': '', 'build_type': ''})
+
+ def test_an_esp_idf_board_reads_its_first_example_project(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ for ex, ver in (('device/b', '14.2.0'), ('device/a', '14.2.0'), ('device/a/bootloader', '1.0')):
+ os.makedirs(os.path.join(tmp, ex, 'CMakeFiles', '3.30'))
+ with open(os.path.join(tmp, ex, 'CMakeFiles', '3.30', 'CMakeCCompiler.cmake'), 'w') as f:
+ f.write(f'set(CMAKE_C_COMPILER "/idf/xtensa-esp32s3-elf-gcc")\nset(CMAKE_C_COMPILER_VERSION "{ver}")\n')
+ with open(os.path.join(tmp, 'device', 'a', 'CMakeCache.txt'), 'w') as f:
+ f.write('CMAKE_BUILD_TYPE:STRING=\n')
+ self.assertEqual(sd._cmake_compiler(tmp), {'id': '', 'version': '14.2.0',
+ 'name': 'xtensa-esp32s3-elf-gcc', 'build_type': ''})
+
+ def test_a_pull_request_reads_base_and_head_from_the_merge_commit(self):
+ ret = subprocess.CompletedProcess([], 0, '\n'.join(self.SHA) + '\n', '')
+ with mock.patch.object(sd, 'run', return_value=ret) as run:
+ self.assertEqual(sd._git_shas('pull_request'), tuple(self.SHA))
+ self.assertEqual(run.call_args[0][0][-3:], ['HEAD', 'HEAD^1', 'HEAD^2'])
+ with mock.patch.object(sd, 'run', return_value=subprocess.CompletedProcess([], 0, self.SHA[0], '')):
+ self.assertEqual(sd._git_shas('push'), (self.SHA[0],) * 3)
+
+ def test_unresolvable_commits_raise(self):
+ # a shallow pull_request checkout has no HEAD^2: never a snapshot with a guessed base
+ for ret in (subprocess.CompletedProcess([], 128, '', 'unknown revision HEAD^2'),
+ subprocess.CompletedProcess([], 0, 'HEAD^2\n', '')):
+ with mock.patch.object(sd, 'run', return_value=ret), self.assertRaises(RuntimeError):
+ sd._git_shas('pull_request')
+
+ def test_boards_are_the_pinned_ones_of_the_leg(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ pinned = os.path.join(tmp, 'pinned.json')
+ with open(pinned, 'w') as f:
+ f.write('{"boards": [{"board": "p1"}, {"board": "p2"}]}')
+ import build
+ with mock.patch.object(sd, 'CI_PINNED_BOARDS', pinned), \
+ mock.patch.object(build, 'builds_any', return_value=True), \
+ mock.patch.object(build, 'resolve_ci_boards', return_value=['p2']) as resolve:
+ boards = sd.snapshot_boards(['fam'], ['p1', 'notpinned', 'p2'], ['device/a'])
+ self.assertEqual(boards, ['p1', 'p2'])
+ resolve.assert_called_once_with(pinned, 'fam', True, ['device/a'], extra_defines=())
+
+ def test_a_pinned_board_that_builds_none_of_the_examples_is_not_expected(self):
+ # build.py skips it before configuring: no build dir, and no failure either
+ with tempfile.TemporaryDirectory() as tmp:
+ pinned = os.path.join(tmp, 'pinned.json')
+ with open(pinned, 'w') as f:
+ f.write('{"boards": [{"board": "p1"}, {"board": "p2"}]}')
+ import build
+ with mock.patch.object(sd, 'CI_PINNED_BOARDS', pinned), \
+ mock.patch.object(build, 'builds_any', side_effect=lambda b, _e, _d: b == 'p2'):
+ self.assertEqual(sd.snapshot_boards([], ['p1', 'p2'], ['host/device_info']), ['p2'])
+
+ def test_non_integer_sizes_fail_the_elf(self):
+ for bad in (1.5, True):
+ sizes = _elf(5)
+ sizes['files']['x.c']['flash'] = bad
+ with tempfile.TemporaryDirectory() as tmp:
+ _touch(tmp, 'device/a/a.elf')
+ with _engine('membrowse', lambda _p, _f, s=sizes: s):
+ elfs, failures = sd.board_snapshot('b1', tmp, None, ['src/'])
+ self.assertEqual(elfs, {})
+ self.assertEqual([(f['elf'], f['stage']) for f in failures], [('device/a/a.elf', 'report')])
+
+ def test_a_repeated_example_is_sized_once(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ _touch(tmp, 'device/a/a.elf')
+ sizer = mock.Mock(return_value=_elf(5))
+ with _engine('membrowse', sizer), mock.patch.object(sd.build_utils, 'skip_example', return_value=False):
+ elfs, failures = sd.board_snapshot('b1', tmp, ['device/a', 'device/a'], ['src/'])
+ self.assertEqual((list(elfs), failures, sizer.call_count), (['device/a/a.elf'], [], 1))
+
+ def test_writes_one_json_per_board_symbols_only_on_request(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ build_root, out = os.path.join(tmp, 'cmake-build'), os.path.join(tmp, 'out')
+ _touch(build_root, 'cmake-build-b1/device/a/a.elf')
+ for symbols in (False, True):
+ args = mock.Mock(event='push', families=[], board=['b1'], example=None, output=out,
+ build_root=build_root, build_outcome='failure', symbols=symbols, filter=None,
+ build_name=None, define_symbol=[])
+ with _engine('membrowse', lambda _p, _f: _elf(5)), \
+ mock.patch.object(sd, '_git_shas', return_value=(self.SHA[0],) * 3), \
+ mock.patch.object(sd, 'snapshot_boards', return_value=['b1']), \
+ contextlib.redirect_stdout(io.StringIO()):
+ self.assertEqual(sd.run_snapshot(args), 0)
+ with open(os.path.join(out, 'code-size-b1.json')) as f:
+ data = json.load(f)
+ self.assertEqual((data['schema'], data['board']), (sd.SNAPSHOT_SCHEMA, 'b1'))
+ self.assertEqual(data['elfs']['device/a/a.elf']['files'], {'x.c': {'flash': 5, 'ram': 0}})
+ self.assertEqual('symbols' in data['elfs']['device/a/a.elf'], symbols)
+ with open(os.path.join(out, 'leg.json')) as f:
+ leg = json.load(f)
+ self.assertEqual((leg['boards'], leg['sha'], leg['build_outcome']), (['b1'], self.SHA[0], 'failure'))
+
+ def test_a_leg_without_pinned_boards_still_leaves_its_marker(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ args = mock.Mock(event='push', families=['fam'], board=[], example=['device/a'], output=tmp,
+ build_root=tmp, build_outcome='success', symbols=False, filter=None, build_name=None,
+ define_symbol=[])
+ with mock.patch.object(sd, '_git_shas', return_value=(self.SHA[0],) * 3), \
+ mock.patch.object(sd, 'snapshot_boards', return_value=[]), \
+ contextlib.redirect_stdout(io.StringIO()):
+ self.assertEqual(sd.run_snapshot(args), 0)
+ self.assertEqual(os.listdir(tmp), ['leg.json'])
+ with open(os.path.join(tmp, 'leg.json')) as f:
+ leg = json.load(f)
+ self.assertEqual((leg['boards'], leg['examples']), ([], ['device/a']))
+
+ def test_a_variant_reports_its_board_under_the_build_name(self):
+ with tempfile.TemporaryDirectory() as tmp:
+ build_root, out = os.path.join(tmp, 'cmake-build'), os.path.join(tmp, 'out')
+ _touch(build_root, 'cmake-build-b1/device/a/a.elf', 'cmake-build-b1-DMA/device/a/a.elf')
+ args = mock.Mock(event='push', families=[], board=['b1'], example=['device/a'], output=out,
+ build_root=build_root, build_outcome='success', symbols=False, filter=None,
+ build_name='b1-DMA', define_symbol=['MAX3421_HOST=1'])
+ sized = []
+ with _engine('membrowse', lambda p, _f: sized.append(p) or _elf(5)), \
+ mock.patch.object(sd, '_git_shas', return_value=(self.SHA[0],) * 3), \
+ mock.patch.object(sd, 'snapshot_boards', return_value=['b1']) as boards, \
+ mock.patch.object(sd.build_utils, 'skip_example', return_value=False) as skip, \
+ contextlib.redirect_stdout(io.StringIO()):
+ self.assertEqual(sd.run_snapshot(args), 0)
+ self.assertEqual(sorted(os.listdir(out)), ['code-size-b1-DMA.json', 'leg.json'])
+ with open(os.path.join(out, 'leg.json')) as f:
+ self.assertEqual(json.load(f)['boards'], ['b1-DMA'])
+ with open(os.path.join(out, 'code-size-b1-DMA.json')) as f:
+ self.assertEqual(json.load(f)['board'], 'b1-DMA')
+ self.assertEqual([os.path.relpath(p, build_root) for p in sized], ['cmake-build-b1-DMA/device/a/a.elf'])
+ # the board's own skip rules, with the leg's defines as build.py applied them
+ boards.assert_called_once_with([], ['b1'], ['device/a'], ('MAX3421_HOST=1',))
+ skip.assert_called_with('device/a', 'b1', ('MAX3421_HOST=1',))
+
+ def test_a_variant_needs_one_board_and_a_build_name(self):
+ for argv in (['-b', 'b1', '-b', 'b2', '--build-name', 'x'], ['fam', '--build-name', 'x'],
+ ['-b', 'b1', '--cflag=-DX=1'], ['-b', 'b1', '-DX=1'], ['-b', 'b1', '--build-name', '../x']):
+ with mock.patch.object(sd, 'engine_missing', return_value=False), \
+ mock.patch.object(sd, 'run_snapshot') as run, \
+ mock.patch.object(sys, 'argv', ['code_size.py', 'snapshot', '-o', 'out'] + argv), \
+ contextlib.redirect_stderr(io.StringIO()), self.assertRaises(SystemExit):
+ sd.main()
+ run.assert_not_called()
+
+ def test_a_variant_takes_build_py_defines(self):
+ argv = ['-b', 'b1', '--build-name', 'b1-X', '-DX=1', '--define-symbol', 'Y=2', '--cflag=-DZ=3']
+ with mock.patch.object(sd, 'engine_missing', return_value=False), \
+ mock.patch.object(sd, 'run_snapshot', return_value=0) as run, \
+ mock.patch.object(sys, 'argv', ['code_size.py', 'snapshot', '-o', 'out'] + argv):
+ self.assertEqual(sd.main(), 0)
+ self.assertEqual((run.call_args[0][0].define_symbol, run.call_args[0][0].cflag), (['X=1', 'Y=2'], ['-DZ=3']))
+
+
+def _shard(board, elfs, sha='a', base='b', head='c', failures=(), examples=None, compiler='gcc 14'):
+ return {'schema': 1, 'board': board, 'engine': 'membrowse', 'membrowse_version': '1.2.9',
+ 'compiler': compiler, 'sha': sha * 40, 'base_sha': base * 40, 'head_sha': head * 40,
+ 'build_outcome': 'success', 'examples': examples, 'elfs': elfs, 'failures': list(failures)}
+
+
+def _write(root, rel, data):
+ os.makedirs(os.path.dirname(os.path.join(root, rel)), exist_ok=True)
+ with open(os.path.join(root, rel), 'w') as f:
+ json.dump(data, f)
+
+
+LEG_KEYS = ('sha', 'base_sha', 'head_sha', 'examples', 'build_outcome')
+
+
+def _run_dir(root, shards, legs=None, scope=None):
+ """A downloaded run: one artifact dir per leg ({artifact: [boards]}), each leg taking its
+ commits, examples and build outcome from its first board's _shard(), and the shards,
+ without those, spread over them."""
+ legs = legs if legs is not None else {'code-size-arm-gcc-fam': [s['board'] for s in shards]}
+ by_board = {s['board']: s for s in shards}
+ for artifact, boards in legs.items():
+ first = by_board.get(boards[0], _shard('x', {})) if boards else _shard('x', {})
+ _write(root, f'{artifact}/leg.json', {'schema': 1, 'boards': boards, **{k: first[k] for k in LEG_KEYS}})
+ for b in boards:
+ if b in by_board:
+ _write(root, f'{artifact}/code-size-{b}.json',
+ {k: v for k, v in by_board[b].items() if k not in LEG_KEYS})
+ if scope is not None:
+ _write(root, 'code-size-scope/scope.json', {'schema': 1, **scope})
+ return sd.load_snapshots(root)
+
+
+class Compare(unittest.TestCase):
+ SCOPE = {'code_changed': True, 'legs': [{'toolchain': 'arm-gcc', 'arg': 'fam'}]}
+
+ def compare(self, base_shards, cur_shards, cur_legs=None, scope=SCOPE, baseline=None, symbols=False):
+ with tempfile.TemporaryDirectory() as b, tempfile.TemporaryDirectory() as c:
+ return sd.compare_runs(_run_dir(b, base_shards, scope=self.SCOPE),
+ _run_dir(c, cur_shards, cur_legs, scope), baseline, symbols)
+
+ def failed(self, data):
+ return {(f['board'], f['side'], f['stage']) for f in data['failures']}
+
+ def test_pairs_boards_and_reports_the_change(self):
+ md, comment, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(10)}, sha='b', head='b')],
+ [_shard('b1', {'device/a/a.elf': _elf(14)})])
+ self.assertEqual(data['status'], 'complete')
+ self.assertIn('+4', md)
+ self.assertTrue(comment.startswith('## Code size'))
+ self.assertIn('bbbbbbbbbb', md) # sides labelled by the merge commit's parents
+
+ def test_base_elfs_of_examples_the_pr_did_not_build_are_outside_coverage(self):
+ base = _shard('b1', {'device/a/a.elf': _elf(10), 'device/z/z.elf': _elf(3)})
+ cur = _shard('b1', {'device/a/a.elf': _elf(10)}, examples=['device/a'])
+ scope = {**self.SCOPE, 'family_examples': {'fam': ['device/a']}}
+ _md, _c, data = self.compare([base], [cur], scope=scope)
+ self.assertEqual(data['status'], 'complete')
+ self.assertEqual(data['outside'], 1)
+ self.assertEqual(data['base_only'], [])
+
+ def test_a_leg_measuring_other_examples_than_the_scope_fails(self):
+ cur = _shard('b1', {'device/z/z.elf': _elf(1)}, examples=['device/z'])
+ scope = {**self.SCOPE, 'family_examples': {'fam': ['device/a']}}
+ _md, _c, data = self.compare([_shard('b1', {'device/z/z.elf': _elf(1)})], [cur], scope=scope)
+ self.assertIn(('code-size-arm-gcc-fam', 'current', 'scope'), self.failed(data))
+
+ def test_an_esp_leg_matches_its_artifact_and_its_own_examples(self):
+ arg = '-b espressif_s3_devkitm -e device/cdc_msc_freertos'
+ scope = {'code_changed': True, 'legs': [{'toolchain': 'esp-idf', 'arg': arg}]}
+ legs = {'code-size-esp-idf--b espressif_s3_devkitm': ['espressif_s3_devkitm']}
+ elfs = {'device/cdc_msc_freertos/cdc_msc_freertos.elf': _elf(1)}
+ _md, _c, data = self.compare([_shard('espressif_s3_devkitm', elfs)],
+ [_shard('espressif_s3_devkitm', elfs, examples=['device/cdc_msc_freertos'])],
+ cur_legs=legs, scope=scope)
+ self.assertEqual(data['status'], 'complete')
+ _md, _c, data = self.compare([], [_shard('espressif_s3_devkitm', elfs, examples=['device/other'])],
+ cur_legs=legs, scope=scope)
+ self.assertIn(('code-size-esp-idf--b espressif_s3_devkitm', 'current', 'scope'), self.failed(data))
+
+ def test_a_leg_outside_the_scope_and_a_shard_outside_its_leg_fail(self):
+ shards = [_shard('b1', {'device/a/a.elf': _elf(1)}), _shard('b2', {'device/a/a.elf': _elf(1)})]
+ with tempfile.TemporaryDirectory() as b, tempfile.TemporaryDirectory() as c:
+ cur = _run_dir(c, shards, {'code-size-arm-gcc-fam': ['b1'], 'code-size-arm-gcc-extra': ['b2']},
+ self.SCOPE)
+ _write(c, 'code-size-arm-gcc-fam/code-size-b3.json', _shard('b3', {'device/a/a.elf': _elf(1)}))
+ _md, _c, data = sd.compare_runs(_run_dir(b, shards + [_shard('b3', {})]), sd.load_snapshots(c))
+ self.assertIn(('code-size-arm-gcc-extra', 'current', 'scope'), self.failed(data))
+ self.assertIn(('b3', 'current', 'snapshot'), self.failed(data))
+ self.assertEqual(data['boards'], ['b1', 'b2'])
+
+ def test_a_new_elf_is_current_only(self):
+ md, comment, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(10), 'device/o/o.elf': _elf(1)})],
+ [_shard('b1', {'device/a/a.elf': _elf(10), 'device/n/n.elf': _elf(1)})])
+ self.assertEqual(data['current_only'], [{'board': 'b1', 'elf': 'device/n/n.elf'}])
+ self.assertEqual(data['status'], 'INCOMPLETE')
+ self.assertIn('- missing in this PR: `b1: device/o/o.elf`\n'
+ '- new in this PR (no master size yet): `b1: device/n/n.elf`\n', comment)
+ self.assertIn('current-only: `b1: device/n/n.elf`', md) # the full report keeps its terms
+
+ def test_a_board_without_baseline_leaves_coverage_incomplete(self):
+ md, comment, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(10)})],
+ [_shard('b1', {'device/a/a.elf': _elf(10)}), _shard('b2', {'device/a/a.elf': _elf(1)})])
+ self.assertEqual(data['no_baseline'], ['b2'])
+ self.assertEqual(data['status'], 'INCOMPLETE')
+ self.assertIn('FAILED `b2` base snapshot: no baseline snapshot', md)
+ self.assertEqual(len(data['pairs']), 1)
+ self.assertIn('1 builds on 1 boards compared, 0 changed\n\nNot compared:\n\n'
+ '- FAILED `b2` master snapshot: no baseline snapshot\n', comment)
+ self.assertNotIn('INCOMPLETE', comment)
+
+ def test_a_missing_scope_or_a_failed_build_leaves_coverage_incomplete(self):
+ broken = {**_shard('b1', {'device/a/a.elf': _elf(1)}), 'build_outcome': 'failure'}
+ _md, _c, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(1)})], [broken], scope=None)
+ self.assertEqual(self.failed(data), {('scope', 'current', 'snapshot'), ('b1', 'current', 'build')})
+
+ def test_a_leg_or_board_that_left_no_snapshot_fails(self):
+ scope = {'code_changed': True, 'legs': [{'toolchain': 'arm-gcc', 'arg': 'fam -e device/a'},
+ {'toolchain': 'riscv-gcc', 'arg': 'other'}]}
+ _md, _c, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(1)})],
+ [_shard('b1', {'device/a/a.elf': _elf(1)}, examples=['device/a'])],
+ cur_legs={'code-size-arm-gcc-fam': ['b1', 'b2']}, scope=scope)
+ self.assertEqual(self.failed(data), {('code-size-riscv-gcc-other', 'current', 'snapshot'),
+ ('b2', 'current', 'snapshot')})
+
+ def test_a_failed_baseline_elf_is_a_base_failure_not_a_new_elf(self):
+ base = _shard('b1', {'device/a/a.elf': _elf(1)},
+ failures=[{'elf': 'device/n/n.elf', 'stage': 'report', 'message': 'boom'}])
+ _md, _c, data = self.compare([base], [_shard('b1', {'device/a/a.elf': _elf(1), 'device/n/n.elf': _elf(2)})])
+ self.assertEqual(data['current_only'], [])
+ self.assertEqual([(f['elf'], f['side']) for f in data['failures']], [('device/n/n.elf', 'base')])
+
+ def test_a_compiler_mismatch_is_a_warning(self):
+ md, _c, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(1)}, compiler='gcc 13')],
+ [_shard('b1', {'device/a/a.elf': _elf(1)})])
+ self.assertEqual(data['status'], 'complete')
+ self.assertIn('compiler differs', md)
+
+ def test_current_snapshots_of_different_commits_are_not_compared(self):
+ _md, _c, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(1)}), _shard('b2', {'device/a/a.elf': _elf(1)})],
+ [_shard('b1', {'device/a/a.elf': _elf(1)}),
+ _shard('b2', {'device/a/a.elf': _elf(1)}, head='d')],
+ cur_legs={'code-size-arm-gcc-fam': ['b1'], 'code-size-arm-gcc-x': ['b2']},
+ scope={'code_changed': True, 'legs': [{'toolchain': 'arm-gcc', 'arg': 'fam'},
+ {'toolchain': 'arm-gcc', 'arg': 'x'}]})
+ self.assertIn(('commits', 'current', 'snapshot'), self.failed(data))
+ self.assertEqual(data['pairs'], [])
+
+ def test_baseline_snapshots_of_another_commit_than_the_selected_run_are_dropped(self):
+ _md, _c, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(1)}, sha='d')],
+ [_shard('b1', {'device/a/a.elf': _elf(1)})],
+ baseline={'sha': 'b' * 40, 'url': 'u', 'exact': True})
+ self.assertIn(('commits', 'base', 'snapshot'), self.failed(data))
+ self.assertEqual(data['pairs'], [])
+
+ def test_a_run_with_no_leg_to_measure_says_so(self):
+ for changed in (False, True): # e.g. a HIL-harness-only PR selects no pinned leg
+ md, comment, data = self.compare([], [], cur_legs={}, scope={'code_changed': changed, 'legs': []})
+ self.assertEqual(data, {'status': 'nothing measured'})
+ self.assertTrue(comment.startswith('## Code size\n\nNothing to measure'))
+
+ def test_snapshots_a_scope_without_legs_did_not_expect_are_reported(self):
+ _md, _c, data = self.compare([_shard('b1', {'device/a/a.elf': _elf(1)})],
+ [_shard('b1', {'device/a/a.elf': _elf(41)})],
+ scope={'code_changed': True, 'legs': []})
+ self.assertEqual(data['status'], 'INCOMPLETE')
+ self.assertIn(('code-size-arm-gcc-fam', 'current', 'scope'), self.failed(data))
+
+ def test_the_comment_has_a_row_per_changed_file_however_small(self):
+ def msc(n, ram=0):
+ return elf({'class/msc/msc_device.c': (n, 0), 'device/usbd.c': (500, ram)})
+ base = [_shard('b1', {'device/msc/msc.elf': msc(4000), 'device/hid/hid.elf': elf({'class/hid/hid_device.c': (1200, 128)}),
+ 'device/n/n.elf': elf({'device/usbd.c': (500, 0)}, all_syms=(900, 9))}),
+ _shard('b2', {'device/msc/msc.elf': msc(3800)})]
+ cur = [_shard('b1', {'device/msc/msc.elf': msc(4200, 4), 'device/hid/hid.elf': elf({'class/hid/hid_device.c': (1224, 136)}),
+ 'device/n/n.elf': elf({'device/usbd.c': (500, 0)}, all_syms=(904, 9))}),
+ _shard('b2', {'device/msc/msc.elf': msc(3980)})]
+ md, comment, _data = self.compare(base, cur)
+ self.assertIn('4 builds on 2 boards compared, 4 changed. Whole firmware¹ Flash Δ +4 → +200, '
+ 'RAM Δ 0 → +8', comment)
+ rows = [line for line in comment.splitlines() if line.startswith('| ') and 'File' not in line]
+ self.assertEqual([[c.strip() for c in r.strip('|').split('|')] for r in rows],
+ [['class/msc/msc_device.c', '+180 → +200', '0'],
+ ['class/hid/hid_device.c', '+24', '+8'],
+ ['device/usbd.c', '0', '0 → +4']])
+ self.assertIn('1 other build changed without a TinyUSB file-size change.', comment)
+ self.assertNotIn('boards: ', comment)
+ self.assertNotIn('Not compared', comment)
+ self.assertNotIn('| b1: ', comment)
+ self.assertNotIn('<details>', comment)
+ self.assertIn('<details>', md)
+ self.assertIn('| b1: device/hid/hid.elf ', md) # the full report keeps every pair
+
+ def test_a_comment_over_the_limit_is_cut_at_a_line(self):
+ base = _shard('b1', {'device/a/a.elf': elf({f'f{j}.c': (10, 0) for j in range(50)})})
+ cur = _shard('b1', {'device/a/a.elf': elf({f'f{j}.c': (11, 0) for j in range(50)})})
+ with mock.patch.object(sd, 'COMMENT_LIMIT', 800):
+ _md, comment, _data = self.compare([base], [cur])
+ self.assertLessEqual(len(comment), 800)
+ self.assertIn('Whole firmware¹', comment)
+ self.assertTrue(comment.endswith('_Truncated: see the full report._\n\n'
+ '¹ Sum of all symbols measured by membrowse, not the exact image size.\n'))
+
+ def test_a_symbol_only_change_counts_under_symbols(self):
+ sizes = lambda a, b: elf({'x.c': (10, 0)}, symbols={'x.c': {'.text': {'f': a, 'g': b}}}) # noqa: E731
+ for symbols, head in ((False, '1 builds on 1 boards compared, 0 changed'),
+ (True, '1 builds on 1 boards compared, 1 changed')):
+ _md, comment, _data = self.compare([_shard('b1', {'device/a/a.elf': sizes(4, 6)})],
+ [_shard('b1', {'device/a/a.elf': sizes(6, 4)})], symbols=symbols)
+ self.assertIn(head, comment)
+ self.assertIn('1 other build changed without a TinyUSB file-size change.', comment)
+
+ def test_pr_controlled_names_cannot_inject_markdown(self):
+ evil = 'x`|<img src=x>@team\n\n# FORGED\r\n'
+ sizes = lambda n: elf({f'{evil}.c': (n, 0)}, symbols={f'{evil}.c': {evil: {evil: n}}}) # noqa: E731
+ fail = [{'elf': None, 'stage': 'build', 'message': '<script>@maintainers `x`'}]
+ base = _shard('b1', {'device/a/a.elf': sizes(1)})
+ with tempfile.TemporaryDirectory() as b, tempfile.TemporaryDirectory() as c:
+ md, comment, _data = sd.compare_runs(
+ _run_dir(b, [base]), _run_dir(c, [_shard('b1', {'device/a/a.elf': sizes(2)}, failures=fail)],
+ scope=self.SCOPE), symbols=True)
+ for text in (md, comment):
+ self.assertNotIn('<img', text)
+ self.assertNotIn('<script', text)
+ self.assertIsNone(re.search(r'@\w', text))
+ self.assertNotIn('`|', text)
+ self.assertIsNone(re.search(r'^# FORGED', text, re.M)) # stays inside its cell
+ self.assertIn('<img', md) # escaped, still shown
+
+ def test_names_that_escape_alike_stay_distinct(self):
+ files = lambda a, b: elf({'x@.c': (a, 0), 'x#.c': (b, 0)}) # noqa: E731
+ _md, _c, data = self.compare([_shard('b1', {'device/a/a.elf': files(1, 2)})],
+ [_shard('b1', {'device/a/a.elf': files(1, 5)})])
+ self.assertEqual(sorted(data['pairs'][0]['current']['files']), ['x#.c', 'x@.c'])
+
+ def test_invalid_shards_are_errors_not_data(self):
+ good = _shard('b0', {})
+ bad = [_shard('b1', {'../x.elf': _elf(1)}), _shard('b2', {'device/a/a.elf': {'files': {}}}),
+ {**_shard('b3', {}), 'engine': 'bloaty'}, {**_shard('b4', {}), 'schema': 2},
+ _shard('b5', {}, failures=[1]),
+ _shard('b7', {'device/a/a.elf': {**_elf(1), 'files': 1}}),
+ _shard('b8', {'device/a/a.elf': {**_elf(1), 'sections': {'x.c': {'.text': True}}}}),
+ _shard('b10', {}, failures=[{'stage': 'build', 'message': 'no elf key'}])]
+ with tempfile.TemporaryDirectory() as c:
+ run = _run_dir(c, [good] + bad)
+ _write(c, 'code-size-scope/scope.json', {'schema': 1, 'code_changed': True, 'legs': None})
+ run = sd.load_snapshots(c)
+ sd.compare_runs(run, run) # malformed input never crashes
+ self.assertEqual(list(run['shards']), ['b0'])
+ self.assertIsNone(run['scope'])
+ self.assertEqual(len(run['errors']), len(bad) + 1)
+
+ def test_invalid_legs_are_errors_not_data(self):
+ good = {'schema': 1, 'boards': [], 'examples': None, 'build_outcome': 'success',
+ 'sha': 'a' * 40, 'base_sha': 'a' * 40, 'head_sha': 'a' * 40}
+ bad = [{k: v for k, v in good.items() if k != 'examples'}, {**good, 'examples': 1},
+ {**good, 'sha': 'nothex'}, {**good, 'boards': None}, {**good, 'build_outcome': None}]
+ with tempfile.TemporaryDirectory() as c:
+ for i, leg in enumerate([good] + bad):
+ _write(c, f'code-size-arm-gcc-f{i}/leg.json', leg)
+ run = sd.load_snapshots(c)
+ self.assertEqual(list(run['legs']), ['code-size-arm-gcc-f0'])
+ self.assertEqual(len(run['errors']), len(bad))
+
+ def test_an_invalid_baseline_shard_of_a_board_the_pr_did_not_build_is_ignored(self):
+ with tempfile.TemporaryDirectory() as b, tempfile.TemporaryDirectory() as c:
+ base = _run_dir(b, [_shard('b1', {'device/a/a.elf': _elf(1)})])
+ _write(b, 'code-size-arm-gcc-fam/code-size-other.json', {'schema': 1})
+ _md, _c, data = sd.compare_runs(sd.load_snapshots(b), _run_dir(
+ c, [_shard('b1', {'device/a/a.elf': _elf(1)})], scope=self.SCOPE))
+ self.assertEqual(data['status'], 'complete')
+
+ def test_symbols_are_omitted_when_a_snapshot_lacks_them(self):
+ bare = {k: v for k, v in _elf(1).items() if k != 'symbols'}
+ with tempfile.TemporaryDirectory() as b, tempfile.TemporaryDirectory() as c:
+ md, _c, _d = sd.compare_runs(_run_dir(b, [_shard('b1', {'device/a/a.elf': bare})]),
+ _run_dir(c, [_shard('b1', {'device/a/a.elf': _elf(2)})], scope=self.SCOPE),
+ symbols=True)
+ self.assertIn('symbols omitted', md)
+
+ def test_a_board_in_two_artifacts_is_rejected(self):
+ with tempfile.TemporaryDirectory() as c:
+ run = _run_dir(c, [_shard('b1', {})], legs={'code-size-arm-gcc-x': ['b1'], 'code-size-riscv-gcc-y': ['b1']})
+ self.assertEqual(list(run['shards']), ['b1'])
+ self.assertEqual(len(run['errors']), 1)
+
+ def test_an_approximate_baseline_is_labelled_with_its_note(self):
+ md, _c, _d = self.compare([_shard('b1', {'device/a/a.elf': _elf(1)}, sha='d')],
+ [_shard('b1', {'device/a/a.elf': _elf(1)})],
+ baseline={'sha': 'd' * 40, 'url': 'https://github.com/x/y/actions/runs/1',
+ 'exact': False, 'note': '2 master commits before this build'})
+ self.assertIn('Baseline: dddddddddd (approximate) https://github.com/x/y/actions/runs/1 - 2 master '
+ 'commits before this build', md)
+
+ def test_an_unavailable_baseline_is_said_so(self):
+ md, _c, data = self.compare([], [_shard('b1', {'device/a/a.elf': _elf(1)})],
+ baseline={'sha': None, 'url': None, 'exact': False, 'note': 'base x is not on master'})
+ self.assertIn('Baseline: unavailable - base x is not on master', md)
+ self.assertEqual(data['no_baseline'], ['b1'])
+
+ def test_cli_writes_the_three_reports(self):
+ with tempfile.TemporaryDirectory() as b, tempfile.TemporaryDirectory() as c, \
+ tempfile.TemporaryDirectory() as out:
+ _run_dir(b, [_shard('b1', {'device/a/a.elf': _elf(1)})])
+ _run_dir(c, [_shard('b1', {'device/a/a.elf': _elf(2)})], scope=self.SCOPE)
+ with mock.patch.object(sys, 'argv', ['code_size.py', 'compare', c, b, '-o', out]), \
+ contextlib.redirect_stdout(io.StringIO()):
+ self.assertEqual(sd.main(), 0)
+ self.assertEqual(sorted(os.listdir(out)), ['code-size.json', 'code-size.md', 'comment.md'])
+
class SymlinkDeps(unittest.TestCase):
def test_links_each_dep_path_of_the_worktrees_own_manifest(self):
diff --git a/test/hil/test/test_code_size_ci.py b/test/hil/test/test_code_size_ci.py
new file mode 100644
index 0000000..7398010
--- /dev/null
+++ b/test/hil/test/test_code_size_ci.py
@@ -0,0 +1,186 @@
+#!/usr/bin/env python3
+"""Tests for .github/scripts/code_size_ci.py, against a stubbed GitHub API."""
+import contextlib
+import io
+import json
+import os
+import re
+import sys
+import tempfile
+import unittest
+from unittest import mock
+
+REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', '..', '..'))
+sys.path.insert(0, os.path.join(REPO, '.github', 'scripts'))
+import code_size_ci as ci # noqa: E402
+
+NAME = 'hathach/tinyusb'
+
+
+def sha(c):
+ return c * 40
+
+
+def run(run_id, commit, number=None, event='push', branch='master', repo=NAME):
+ return {'id': run_id, 'run_number': number or run_id, 'head_sha': commit, 'head_branch': branch,
+ 'event': event, 'head_repository': {'full_name': repo},
+ 'html_url': f'https://github.com/{NAME}/actions/runs/{run_id}'}
+
+
+def artifact(art_id, name, expired=False):
+ return {'id': art_id, 'name': name, 'expired': expired}
+
+
+class FakeGh:
+ """The API paths code_size_ci.py reads: first-parent history, runs per commit,
+ artifacts per run, the ancestry compare, a PR and a run. A paginated call gets
+ everything as one page."""
+
+ def __init__(self, parents, runs=None, artifacts=None, status='ahead'):
+ self.parents, self.runs, self.artifacts, self.status = parents, runs or {}, artifacts or {}, status
+ self.pr_head, self.pr_repo, self.attempts = None, NAME, {}
+
+ def __call__(self, path, paginate=False):
+ reply = self.reply(path)
+ return [reply] if paginate else reply
+
+ def reply(self, path):
+ if m := re.match(rf'repos/{NAME}/actions/workflows/build\.yml/runs\?head_sha=(\w+)&', path):
+ return {'workflow_runs': self.runs.get(m.group(1), [])}
+ if m := re.fullmatch(rf'repos/{NAME}/actions/runs/(\d+)/artifacts', path):
+ return {'artifacts': self.artifacts.get(int(m.group(1)), [])}
+ if m := re.fullmatch(rf'repos/{NAME}/actions/runs/(\d+)', path):
+ return {'run_attempt': self.attempts[int(m.group(1))]}
+ if m := re.fullmatch(rf'repos/{NAME}/commits/(\w+)', path):
+ parent = self.parents.get(m.group(1))
+ return {'parents': [{'sha': parent}] if parent else []}
+ if re.fullmatch(rf'repos/{NAME}/pulls/\d+', path):
+ return {'head': {'sha': self.pr_head, 'ref': 'feature', 'repo': {'full_name': self.pr_repo}}}
+ if re.fullmatch(rf'repos/{NAME}/compare/\w+\.\.\.master', path):
+ return {'status': self.status}
+ raise AssertionError(f'unexpected API call {path}')
+
+
+SNAPSHOTS = [artifact(1, 'code-size-scope'), artifact(2, 'code-size-arm-gcc-stm32f4')]
+
+
+def leg(base):
+ return {'schema': 1, 'boards': [], 'examples': None, 'build_outcome': 'success',
+ 'sha': sha('e'), 'base_sha': base, 'head_sha': sha('f')}
+
+
+class Baseline(unittest.TestCase):
+ def lookup(self, fake, base=sha('b'), legs=None):
+ with tempfile.TemporaryDirectory() as tmp:
+ cur, info = os.path.join(tmp, 'cur'), os.path.join(tmp, 'info.json')
+ for i, data in enumerate(legs or [leg(base)]):
+ os.makedirs(os.path.join(cur, f'code-size-leg{i}'))
+ with open(os.path.join(cur, f'code-size-leg{i}', 'leg.json'), 'w') as f:
+ json.dump(data, f)
+ args = ['code_size_ci.py', 'baseline', '--repo', NAME, '--current', cur, '--info', info]
+ with mock.patch.object(ci, 'gh', fake), mock.patch.object(sys, 'argv', args), \
+ contextlib.redirect_stdout(io.StringIO()):
+ self.assertEqual(ci.main(), 0)
+ with open(info) as f:
+ return json.load(f)
+
+ def test_the_base_commits_own_run_is_exact(self):
+ fake = FakeGh({}, {sha('b'): [run(7, sha('b'))]}, {7: SNAPSHOTS})
+ self.assertEqual(self.lookup(fake), {'sha': sha('b'), 'url': f'https://github.com/{NAME}/actions/runs/7',
+ 'exact': True, 'run_id': 7})
+
+ def test_a_commit_without_snapshots_falls_back_to_its_first_parent_as_approximate(self):
+ docs_only = [artifact(3, 'code-size-scope'), artifact(4, 'binaries-arm-gcc-x')]
+ fake = FakeGh({sha('b'): sha('a')}, {sha('b'): [run(8, sha('b'))], sha('a'): [run(7, sha('a'))]},
+ {8: docs_only, 7: SNAPSHOTS})
+ info = self.lookup(fake)
+ self.assertEqual((info['sha'], info['exact'], info['run_id']), (sha('a'), False, 7))
+ self.assertIn('1 master commits before', info['note'])
+
+ def test_expired_artifacts_do_not_count(self):
+ expired = [artifact(1, 'code-size-scope'), artifact(2, 'code-size-arm-gcc-x', expired=True)]
+ fake = FakeGh({sha('b'): sha('a')}, {sha('b'): [run(8, sha('b'))], sha('a'): [run(7, sha('a'))]},
+ {8: expired, 7: SNAPSHOTS})
+ self.assertEqual(self.lookup(fake)['sha'], sha('a'))
+
+ def test_the_latest_run_of_a_commit_wins(self):
+ fake = FakeGh({}, {sha('b'): [run(7, sha('b')), run(8, sha('b'))]}, {7: SNAPSHOTS, 8: SNAPSHOTS})
+ self.assertEqual(self.lookup(fake)['run_id'], 8)
+
+ def test_runs_of_other_events_branches_or_repos_are_not_baselines(self):
+ others = [run(5, sha('b'), event='pull_request'), run(6, sha('b'), branch='dev'),
+ run(7, sha('b'), repo='fork/tinyusb')]
+ fake = FakeGh({}, {sha('b'): others}, {5: SNAPSHOTS, 6: SNAPSHOTS, 7: SNAPSHOTS})
+ self.assertIsNone(self.lookup(fake)['run_id'])
+
+ def test_a_base_off_master_has_no_baseline(self):
+ fake = FakeGh({}, {sha('b'): [run(7, sha('b'))]}, {7: SNAPSHOTS}, status='diverged')
+ info = self.lookup(fake)
+ self.assertIsNone(info['run_id'])
+ self.assertIn('not on master', info['note'])
+
+ def test_the_search_stops_at_the_depth_cap(self):
+ history = [f'{i:040x}' for i in range(ci.MAX_DEPTH + 5)]
+ fake = FakeGh(dict(zip(history, history[1:])), {history[-1]: [run(7, history[-1])]}, {7: SNAPSHOTS})
+ info = self.lookup(fake, base=history[0])
+ self.assertIsNone(info['sha'])
+ self.assertIn(f'within {ci.MAX_DEPTH} commits', info['note'])
+
+ def test_legs_that_disagree_on_their_base_or_are_invalid_have_no_baseline(self):
+ for legs in ([leg(sha('b')), leg(sha('c'))], [leg(['x'])], [{'base_sha': sha('b')}]):
+ info = self.lookup(FakeGh({}), legs=legs)
+ self.assertIsNone(info['sha'])
+ self.assertIn('no single base commit', info['note'])
+
+ def test_a_failed_lookup_is_reported_not_raised(self):
+ def broken(path, paginate=False):
+ raise RuntimeError('gh api: HTTP 502')
+ info = self.lookup(broken)
+ self.assertEqual((info['sha'], info['run_id']), (None, None))
+ self.assertIn('the baseline lookup failed: gh api: HTTP 502', info['note'])
+
+
+def pr_run(run_id, prs=(5,), branch='feature', repo=NAME):
+ return {**run(run_id, sha('c'), event='pull_request', branch=branch, repo=repo),
+ 'pull_requests': [{'number': n} for n in prs]}
+
+
+class IsCurrent(unittest.TestCase):
+ def check(self, pr_head=sha('c'), runs=(pr_run(7),), attempts=None, pr_repo=NAME):
+ fake = FakeGh({}, {sha('c'): list(runs)})
+ fake.pr_head, fake.pr_repo, fake.attempts = pr_head, pr_repo, attempts or {7: 1}
+ args = ['code_size_ci.py', 'is-current', '--repo', NAME, '--pr', '5', '--run-id', '7',
+ '--attempt', '1', '--head-sha', sha('c')]
+ out = io.StringIO()
+ with mock.patch.object(ci, 'gh', fake), mock.patch.object(sys, 'argv', args), \
+ contextlib.redirect_stdout(out):
+ return ci.main(), out.getvalue().strip()
+
+ def test_the_latest_attempt_of_the_latest_run_of_the_head_is_current(self):
+ self.assertEqual(self.check(), (0, 'current'))
+
+ def test_a_moved_head_a_newer_run_or_attempt_is_stale(self):
+ self.assertEqual(self.check(pr_head=sha('d')), (1, 'stale: the PR head moved on'))
+ self.assertEqual(self.check(runs=(pr_run(7), pr_run(8))), (1, 'stale: a newer Build run exists: run 8 of #5'))
+ self.assertEqual(self.check(attempts={7: 2}), (1, 'stale: a newer attempt exists'))
+ self.assertEqual(self.check(runs=()), (1, 'stale: no Build run of the PR head found'))
+
+ def test_a_newer_run_of_another_pr_sharing_the_head_does_not_supersede(self):
+ self.assertEqual(self.check(runs=(pr_run(7), pr_run(8, prs=(6,)))), (0, 'current'))
+ self.assertEqual(self.check(runs=(pr_run(7), pr_run(8, prs=(5, 6)))),
+ (1, 'stale: a newer Build run exists: run 8 of #5, #6'))
+
+ def test_a_fork_prs_unlisted_runs_count_by_their_head_branch_and_repo(self):
+ fork = 'someone/tinyusb'
+ own = pr_run(7, prs=(), repo=fork)
+ self.assertEqual(self.check(runs=(own,), pr_repo=fork), (0, 'current'))
+ self.assertEqual(self.check(runs=(own, pr_run(8, prs=(), repo=fork)), pr_repo=fork),
+ (1, 'stale: a newer Build run exists: run 8 of someone/tinyusb:feature'))
+ self.assertEqual(self.check(runs=(own, pr_run(8, prs=(), branch='other', repo=fork)), pr_repo=fork),
+ (0, 'current'))
+ self.assertEqual(self.check(runs=(own, pr_run(8, prs=(), repo='other/tinyusb')), pr_repo=fork),
+ (0, 'current'))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/test/hil/test/test_membrowse_cli.py b/test/hil/test/test_membrowse_cli.py
index 7d7d61e..a752716 100644
--- a/test/hil/test/test_membrowse_cli.py
+++ b/test/hil/test/test_membrowse_cli.py
@@ -259,11 +259,12 @@
self.assertEqual(report.call_args[0][0].option, '--all-symbols')
def test_ci_uploads_use_the_same_target_name(self):
- # membrowse history is keyed on it: a rename orphans every series
+ # membrowse history is keyed on it: a rename orphans every series. A plain build
+ # is <board>/<example>; only build.py --build-name sets MEMBROWSE_BOARD.
with open(os.path.join(REPO, 'hw', 'bsp', 'family_support.cmake')) as f:
args = [line.split('#', 1)[0].strip() for line in f]
self.assertEqual([a for a in args if a.startswith('--target-name')],
- ['--target-name ${BOARD}/${TARGET}'])
+ ['--target-name ${MEMBROWSE_BOARD}/${TARGET}'])
def test_paths_and_build_script_are_repo_root_relative(self):
cmd = cli.compose('stm32f407disco', 'device/cdc_msc', 30, False, 'k', [])
diff --git a/tools/build.py b/tools/build.py
index 13690d2..f5fc8a8 100755
--- a/tools/build.py
+++ b/tools/build.py
@@ -148,6 +148,8 @@
build_flags = []
if build_cflags:
build_flags.append('-DCFLAGS_CLI=' + ' '.join(build_cflags))
+ if build_name: # a variant is a membrowse board of its own (family_add_membrowse)
+ build_flags.append(f'-DMEMBROWSE_BOARD={build_name}')
family = find_family(board)
if family == 'espressif':
@@ -182,7 +184,7 @@
sys.executable, os.path.join(os.path.dirname(__file__), 'membrowse_cli.py'), 'report',
'--build-dir', example_build_dir, '--ninja', 'ninja',
'--elf', f'{example_build_dir}/{name}.elf',
- '--target-name', f'{board}/{name}', '--upload',
+ '--target-name', f'{build_name or board}/{name}', '--upload',
])
ret[0 if rcmd.returncode == 0 else 1] += 1
elif not os.path.isdir(example_build_dir):
@@ -413,7 +415,7 @@
with open(boards_path) as f:
data = json.load(f)
# membership in this family's board dir, not find_family() per entry: that walks
- # every hw/bsp family for each of the 23 pinned boards to answer one family's question
+ # every hw/bsp family for each pinned board to answer one family's question
family_boards = {e.name for e in os.scandir(f'hw/bsp/{family}/boards') if e.is_dir()}
ci_boards = [t['board'] for t in data['boards']
if t['board'] in family_boards
@@ -492,7 +494,8 @@
# that existed before the configure); -e only selects targets, the cmake tree configures whole.
# The command line is only half of it - canonical_row() refuses the flags CFLAGS and
# its siblings add to a configure this test cannot see
- canonical = build_system == 'cmake' and not build_defines and not build_cflags and toolchain == 'gcc'
+ variant = bool(build_defines or build_cflags)
+ canonical = build_system == 'cmake' and not variant and toolchain == 'gcc'
for e in args.example:
if not EXAMPLE_RE.fullmatch(e):
@@ -520,6 +523,12 @@
print("--build-name requires exactly one board (-b) and no families")
return 1
+ # membrowse names an upload <board>/<example>: an unnamed variant would overwrite
+ # the plain build's history (family_add_membrowse)
+ if variant and not build_name and 'examples-membrowse-upload' in build_targets:
+ print("examples-membrowse-upload of a -D/--cflag variant requires --build-name")
+ return 1
+
print(build_separator)
print(build_format.format('Board', 'Target', '\033[39mResult\033[0m', 'Time'))
total_time = time.monotonic()
diff --git a/tools/ci_select.py b/tools/ci_select.py
index 17dfd89..e7e989e 100755
--- a/tools/ci_select.py
+++ b/tools/ci_select.py
@@ -25,9 +25,9 @@
| 1 | `docs/`, `.claude/`, `*.md`, `*.rst`, `LICENSE` | — | — | — |
| 1b | `.gitignore`, `.clang-format`, `.agents`, `.codex/**`, `.idea/**`, `test/{fuzz,unit-test}/**`, `test/hil/test/**`, non-build `.github/**`, packaging manifests | — | — | — |
| 2 | `test/hil/**` (not `test/hil/test/**`) | — | — | all boards → all tests |
-| 2b | `tools/code_size.py`, `tools/drivers_coverage_check.py` | — (local-only tooling, no CI build runs it) | — | — (nothing on the rig runs it) |
+| 2b | `tools/drivers_coverage_check.py` | — (local-only tooling, no CI build runs it) | — | — (nothing on the rig runs it) |
| 2c | `.github/ci-pinned-boards.json` | `ALL` | `ALL` | — (CI board data; no rig board's behaviour depends on it) |
-| 2d | `tools/membrowse_cli.py` | `ALL` | `ALL` | — (build-time script invoked from family_support.cmake; no rig board runs it) |
+| 2d | `tools/membrowse_cli.py`, `tools/code_size.py` | `ALL` | `ALL` | — (size scripts the pinned build legs run; no rig board runs them) |
| 3 | `src/portable/<port>/dcd_*`, `*_device.[ch]` | `FAM` | `DEV`+`DUAL` | `FAM`'s device-role boards → device+dual tests |
| 4 | `src/portable/<port>/hcd_*`, `*_host.[ch]` | `FAM` | `HOST`+`DUAL` | `FAM`'s host-role boards → host+dual tests |
| 5 | `src/portable/<port>/**` (anything else) | `FAM` | `ALL` | `FAM`'s boards → all their tests |
@@ -98,7 +98,8 @@
#
# Deliberately NOT here, and still full: .circleci/**, .github/workflows/build*.yml,
# .github/actions/**, .github/scripts/** - those decide what gets built. The line is
-# "does any Build step read this file", not "is it source".
+# "does any Build step read this file", not "is it source": code_size_ci.py is read by
+# pr_comment.yml only, so it is metadata.
#
# test/{fuzz,unit-test} have their own jobs (cifuzz.yml, the unit-test pre-commit hook
# and workflow); the Build matrix never compiles them, and test/hil is rule 2.
@@ -119,25 +120,25 @@
# The harness itself stays under _FULL_RE's test/hil/ prefix.
r'test/(fuzz|unit-test)/|test/hil/test/|'
# .github, minus the build machinery named in _FULL_RE and _CI_BOARDS_RE
- r'\.github/(FUNDING\.yml$|labeler\.yml$|membrowse_pr_message\.j2$|'
- r'ISSUE_TEMPLATE/|'
- r'workflows/(cifuzz|claude|claude-code-review|labeler|membrowse-comment|'
+ r'\.github/(FUNDING\.yml$|labeler\.yml$|'
+ r'ISSUE_TEMPLATE/|scripts/code_size_ci\.py$|'
+ r'workflows/(cifuzz|claude|claude-code-review|labeler|'
r'pr_comment|pre-commit|static_analysis|trigger)\.yml$)|'
# tools/ scripts no build invokes (tools/build*.py and local tooling are handled separately)
r'tools/(check_example_pids|file2carray|iar_gen|mksunxi|pcapng_to_corpus)\.py$|'
r'tools/iar_template\.ipcf$'
r')')
-# Local-only size/coverage tooling: no CI build and no rig board runs it (the
+# Local-only coverage tooling: no CI build and no rig board runs it (the
# drivers-coverage checker is pre-commit only), so no contribution on either axis.
-_LOCAL_TOOLING_RE = re.compile(
- r'^tools/(code_size|drivers_coverage_check)\.py$')
+_LOCAL_TOOLING_RE = re.compile(r'^tools/drivers_coverage_check\.py$')
# CI board data (tools/build.py --ci-pinned-boards): it decides which boards a family's
# build legs compile, so a bad edit can silently drop a family - full build matrix. No
# rig board depends on it.
_CI_BOARDS_RE = re.compile(r'^\.github/ci-pinned-boards\.json$')
-# Run by family_add_membrowse() for every family with a pinned board: which family it
-# breaks is data, not code, so full build matrix as rule 2c. No rig board runs it.
-_MEMBROWSE_SCRIPT_RE = re.compile(r'^tools/membrowse_cli\.py$')
+# Size scripts every pinned build leg runs (membrowse_cli.py from family_add_membrowse(),
+# code_size.py's CI snapshot): which family they break is data, not code, so full build
+# matrix as rule 2c. No rig board runs them.
+_MEMBROWSE_SCRIPT_RE = re.compile(r'^tools/(membrowse_cli|code_size)\.py$')
_FULL_RE = re.compile(
r'^(src/common/|src/osal/|src/tusb\.c$|src/tusb\.h$|src/tusb_option\.h$|'
# tools/rtt.py is part of the harness, not a standalone tool: hil_util imports it
@@ -659,13 +660,13 @@
s.reasons.append(f'{path}: non-code, no contribution')
return
if _LOCAL_TOOLING_RE.match(path): # rule 2b
- s.reasons.append(f'{path}: local size/coverage tooling, no HIL contribution')
+ s.reasons.append(f'{path}: local coverage tooling, no HIL contribution')
return
if _CI_BOARDS_RE.match(path): # rule 2c
s.reasons.append(f'{path}: CI board data, no HIL contribution')
return
if _MEMBROWSE_SCRIPT_RE.match(path): # rule 2d
- s.reasons.append(f'{path}: membrowse build-time script, no HIL contribution')
+ s.reasons.append(f'{path}: CI size script, no HIL contribution')
return
if _FULL_RE.match(path):
s.force_full(f'{path}: core/infra -> full matrix')
@@ -1181,13 +1182,13 @@
s.add(all_bsp_families(repo_root), exs, f'{path}: lib {lib} -> {sorted(exs)}')
return
if _LOCAL_TOOLING_RE.match(path): # rule 2b
- s.reasons.append(f'{path}: local size/coverage tooling, no build contribution')
+ s.reasons.append(f'{path}: local coverage tooling, no build contribution')
return
if _CI_BOARDS_RE.match(path): # rule 2c
s.force_full(f'{path}: CI board data changes which boards build -> full build matrix')
return
if _MEMBROWSE_SCRIPT_RE.match(path): # rule 2d
- s.force_full(f'{path}: membrowse build-time script -> full build matrix')
+ s.force_full(f'{path}: CI size script -> full build matrix')
return
if _FULL_RE.match(path): # rules 15-16
# attribution, not behaviour: these already reached `full` through the
diff --git a/tools/code_size.py b/tools/code_size.py
index bebc69f..798c749 100644
--- a/tools/code_size.py
+++ b/tools/code_size.py
@@ -35,6 +35,7 @@
import argparse
import collections
import concurrent.futures
+import contextlib
import csv
import functools
import glob
@@ -53,7 +54,7 @@
import time
import build_utils
-from membrowse_cli import extract_ld_scripts, extract_defsyms, link_command, report_inputs
+from membrowse_cli import IDF_LD_SCRIPTS, extract_ld_scripts, extract_defsyms, link_command, report_inputs
# resolved like tinyusb_src_filter(), so a symlinked checkout still matches its filter
TINYUSB_ROOT = os.path.dirname(os.path.dirname(os.path.realpath(__file__)))
@@ -84,6 +85,12 @@
raise RuntimeError(f'no build.ninja found above {elf_path} - cannot '
f'determine its linker scripts')
commands = link_command('ninja', build_dir, elf_path)
+ # ESP-IDF links its generated scripts by bare name via -L
+ idf_ld = [os.path.join(build_dir, p) for p in IDF_LD_SCRIPTS]
+ if any(map(os.path.isfile, idf_ld)):
+ if not all(map(os.path.isfile, idf_ld)):
+ raise RuntimeError(f'ESP-IDF build {build_dir} lacks one of {", ".join(idf_ld)}')
+ return build_dir, idf_ld, extract_defsyms(commands)
ld_scripts = extract_ld_scripts(commands)
if not ld_scripts:
raise RuntimeError(f'no linker script found in the ninja build graph '
@@ -175,6 +182,12 @@
return [(name(h[0]), h[1], h[2], h[3], h[5]) for h in headers[1:]], loads
+def _map_path(elf):
+ """The elf's GNU ld map: `<elf>.map`, or ESP-IDF's `<name>.map` beside it."""
+ idf = os.path.splitext(elf)[0] + '.map'
+ return idf if not os.path.isfile(elf + '.map') and os.path.isfile(idf) else elf + '.map'
+
+
def map_regions(map_path):
"""[(name, origin, length)] of a GNU ld map's Memory Configuration."""
if not os.path.isfile(map_path):
@@ -199,6 +212,8 @@
run location; one running where it loads is flash unless writable, when the
map region holding it decides. Raises RuntimeError rather than guess."""
sections, loads = elf_layout(elf)
+ if any(s[0] == '.flash.appdesc' for s in sections):
+ return _esp_idf_buckets(sections)
regions = map_regions(map_path)
buckets = {}
for name, sh_type, flags, addr, size in sections:
@@ -230,6 +245,24 @@
return buckets
+def _esp_idf_buckets(sections):
+ """section_buckets() of an ESP-IDF app (it has `.flash.appdesc`), whose ELF runs every
+ section where it loads. IDF's sections.ld names each output section in a flash-mapped region
+ `.flash*`; esptool's elf2image stores the rest in the image too, for the bootloader to load
+ into RAM (bin_image.py is_flash_addr())."""
+ buckets = {}
+ for name, sh_type, flags, _addr, _size in sections:
+ if not flags & SHF_ALLOC or (sh_type == SHT_NOBITS and name.startswith('.flash')):
+ buckets[name] = _NOT_COUNTED # reserved flash-window address space, not storage
+ elif sh_type == SHT_NOBITS:
+ buckets[name] = frozenset({'ram'})
+ elif name.startswith('.flash'):
+ buckets[name] = frozenset({'flash'})
+ else:
+ buckets[name] = frozenset({'flash', 'ram'})
+ return buckets
+
+
class _Sizes:
"""Accumulates one elf's filtered per-file flash/RAM, section and symbol sizes,
and its total flash/RAM."""
@@ -254,10 +287,47 @@
'symbols': self.symbols}
+@functools.lru_cache(maxsize=None)
+def _dwarf_sources(elf):
+ """{basename: {full source path}} of the elf's DWARF compile units."""
+ try:
+ from elftools.elf.elffile import ELFFile # pyelftools, a membrowse dependency
+ except ImportError as e:
+ raise RuntimeError(f'pyelftools is needed to resolve {elf}\'s source paths ({e})')
+ sources = {}
+ with open(elf, 'rb') as f:
+ elffile = ELFFile(f)
+ if not elffile.has_dwarf_info():
+ return sources
+ for cu in elffile.get_dwarf_info().iter_CUs():
+ top = cu.get_top_DIE().attributes
+ if 'DW_AT_name' in top:
+ comp_dir = top['DW_AT_comp_dir'].value.decode() if 'DW_AT_comp_dir' in top else ''
+ path = os.path.normpath(os.path.join(comp_dir, top['DW_AT_name'].value.decode()))
+ sources.setdefault(os.path.basename(path), set()).add(path)
+ return sources
+
+
+def _source_path(sym, elf, filters):
+ """A symbol's source path. An archive member has no object path and membrowse 1.2.9
+ keeps only the basename of its source, so the DWARF compile unit of that name gives it
+ back (ESP-IDF links TinyUSB from an archive)."""
+ path = sym.get('object_file') or sym.get('source_file')
+ if not path or '/' in path:
+ return path
+ candidates = _dwarf_sources(elf).get(path, set())
+ if len(candidates) == 1:
+ return next(iter(candidates))
+ if any(_relative_key(c, filters) for c in candidates):
+ raise RuntimeError(f'{elf}: {path} names several compile units ({", ".join(sorted(candidates))}), '
+ f'one of them filtered; cannot tell which holds {sym.get("name")}')
+ return path
+
+
def membrowse_sizes(elf, filters):
"""Symbols of `membrowse report`; linker-defined symbols without a section
(`__StackLimit`) are not counted."""
- buckets = section_buckets(elf, elf + '.map')
+ buckets = section_buckets(elf, _map_path(elf))
sizes = _Sizes(filters)
for sym in report_for_elf(elf).get('symbols', []):
section = sym.get('section')
@@ -265,8 +335,7 @@
continue
if section not in buckets:
raise RuntimeError(f'membrowse symbol {sym.get("name")} is in section {section}, not in {elf}')
- sizes.add(sym.get('object_file') or sym.get('source_file'), section, buckets[section], sym['size'],
- sym.get('name'))
+ sizes.add(_source_path(sym, elf, filters), section, buckets[section], sym['size'], sym.get('name'))
return sizes.result()
@@ -282,11 +351,11 @@
def linkermap_sizes(elf, filters):
- """Input sections of `<elf>.map` by full object path; analyze_map() is not used,
+ """Input sections of the elf's map by full object path; analyze_map() is not used,
it keeps only four sections. An archive member
(`lib.a(x.o)`) has no source dir, so it counts in 'all' only. The symbols are the
input sections (`.text.foo`), not the labels inside one."""
- map_path = elf + '.map'
+ map_path = _map_path(elf)
buckets = section_buckets(elf, map_path)
with open(map_path, encoding='utf-8', errors='replace') as f:
sections = _linkermap().parseSections(f)
@@ -313,7 +382,7 @@
rows = csv.DictReader(io.StringIO(r.stdout))
if not {'compileunits', 'sections', 'symbols', 'vmsize'} <= set(rows.fieldnames or ()):
raise RuntimeError(f'unexpected bloaty csv columns for {elf}: {rows.fieldnames}')
- buckets = section_buckets(elf, elf + '.map')
+ buckets = section_buckets(elf, _map_path(elf))
sizes = _Sizes(filters)
for row in rows:
section = row['sections']
@@ -333,8 +402,9 @@
# all_label: what the engine's 'all' total sums (totals of different engines are
# not comparable); install: how to get its tool when `sizes` raises FileNotFoundError
Engine = collections.namedtuple('Engine', 'sizes all_label install')
+MEMBROWSE_CI_VERSION = '1.2.12' # every CI install pins this; attribution can differ across versions
ENGINES = {
- 'membrowse': Engine(membrowse_sizes, 'all symbols', '`pip install membrowse`'),
+ 'membrowse': Engine(membrowse_sizes, 'all symbols', f'`pip install membrowse=={MEMBROWSE_CI_VERSION}`'),
'linkermap': Engine(linkermap_sizes, 'all input sections',
'`python3 tools/get_deps.py tools/linkermap`'),
'bloaty': Engine(bloaty_sizes, 'all accounted sections',
@@ -362,10 +432,22 @@
return f'+{delta}' if delta > 0 else str(delta)
+_MD_CELL = str.maketrans({**{c: f'&#{ord(c)};' for c in '&<>|`\\[]\r\n'}, '@': '@\u200b'})
+
+
+def _md_cell(text):
+ """A table cell safe from names that would open HTML, a link or code span, end or split
+ the row or @mention someone (a zero-width space breaks the mention)."""
+ return text.translate(_MD_CELL)
+
+
def md_table(header, rows, total=None):
"""Markdown table padded so its columns also line up as plain text: the first
column left-aligned, the others right-aligned. Cells are strings; a `total` row
follows the rows under a plain rule."""
+ header = [_md_cell(c) for c in header]
+ rows = [[_md_cell(c) for c in r] for r in rows]
+ total = [_md_cell(c) for c in total] if total else total
body = rows + [total] if total else rows
widths = [max(len(r[i]) for r in [header] + body) for i in range(len(header))]
@@ -459,10 +541,11 @@
return md_table([_row_label(symbols, engine)] + cols + ['size', '%'], lines, total_row) + '\n'
-def render_report(sizes, engine, failures=(), boards=(), symbols=False):
+def render_report(sizes, engine, failures=(), boards=(), symbols=False, files_label='TinyUSB'):
"""Markdown report of one tree's elfs keyed by (board, elf path), None for a
failed one; `failures` are (elf id, stage, message). One elf is inline, several
- get a summary table and each its own table in <details>."""
+ get a summary table and each its own table in <details>; `files_label` names the
+ filtered files' totals."""
sized = {i: s for i, s in sizes.items() if s is not None}
status = 'INCOMPLETE' if failures or not sized else 'complete'
all_label = ENGINES[engine].all_label
@@ -477,13 +560,13 @@
def totals(s):
src = _files_total(s['files'])
- return (f'filtered Flash {src["flash"]}, RAM {src["ram"]}; '
+ return (f'{files_label} Flash {src["flash"]}, RAM {src["ram"]}; '
f'{all_label} Flash {s["all"]["flash"]}, RAM {s["all"]["ram"]}')
if len(sized) == 1:
(elf_id, s), = sized.items()
return '\n'.join(lines + [f'`{_label(elf_id)}` {totals(s)}', '', size_table(s, symbols, engine)])
- lines.append(md_table(['Elf', 'filtered Flash', 'filtered RAM', f'{all_label} Flash', f'{all_label} RAM'],
+ lines.append(md_table(['Elf', f'{files_label} Flash', f'{files_label} RAM', f'{all_label} Flash', f'{all_label} RAM'],
[[_label(i)] + [str(t[k]) for t in (_files_total(s['files']), s['all']) for k in ('flash', 'ram')]
for i, s in sized.items()]))
for elf_id, s in sized.items():
@@ -581,8 +664,60 @@
or bool(_changed_files(b, c, symbols)))
+def _range(lo, hi):
+ return _fmt(lo) if lo == hi else f'{_fmt(lo)} → {_fmt(hi)}'
+
+
+COMMENT_TERMS = {'base-only': 'missing in this PR', 'current-only': 'new in this PR (no master size yet)',
+ 'base': 'master', 'current': 'PR'}
+
+
+def _problems(base_only, cur_only, failures, terms=None):
+ """Report lines for one-sided elfs and failures, each id named; `terms` rewords the report's words."""
+ word = lambda w: (terms or {}).get(w, w) # noqa: E731
+ lines = [f'- {word(label)}: ' + ', '.join(f'`{_label(i)}`' for i in ids)
+ for label, ids in (('base-only', base_only), ('current-only', cur_only)) if ids]
+ return lines + [f'- FAILED `{_label(elf_id)}` {word(side)} {stage}: {_md_escape(message)}'
+ for elf_id, side, stage, message in failures]
+
+
+def _changed_paths(stats):
+ """The changed files of _file_stats(), largest |Δ| first."""
+ return sorted((p for p, s in stats.items() if s['changed']),
+ key=lambda p: (-max(abs(v[0]) for k in ('flash', 'ram') for v in stats[p][k]), p))
+
+
+def render_comment(pairs, engine, base_only=(), cur_only=(), failures=(), symbols=False, files_label='TinyUSB'):
+ """The PR comment's view as (body, footnote): a row per changed file with its min to max
+ Δ over the builds containing it, so each changed driver shows however heavy the others
+ are. The footnote explains the whole-firmware totals, when the body has them."""
+ file_deltas = {i: _file_deltas(b, c) for i, (b, c) in pairs.items()}
+ changed = [i for i, (b, c) in pairs.items() if _pair_changed(b, c, symbols)]
+ head = (f'{len(pairs)} builds on {len({board for board, _elf in pairs})} boards compared, '
+ f'{len(changed)} changed')
+ footnote = ''
+ if changed:
+ flash, ram = zip(*(_delta(pairs[i][0]['all'], pairs[i][1]['all']) for i in changed))
+ head += f'. Whole firmware¹ Flash Δ {_range(min(flash), max(flash))}, RAM Δ {_range(min(ram), max(ram))}'
+ footnote = f'\n¹ Sum of {ENGINES[engine].all_label} measured by {engine}, not the exact image size.\n'
+ problems = _problems(base_only, cur_only, failures, COMMENT_TERMS)
+ lines = [head, ''] + (['Not compared:', '', *problems, ''] if problems else [])
+ if not changed:
+ return '\n'.join(lines + ['_no changes_' if pairs else '_no comparable pairs_', '']), footnote
+ stats = _file_stats(file_deltas)
+ rows = [[p] + [_range(*(v[0] for v in stats[p][k])) for k in ('flash', 'ram')] for p in _changed_paths(stats)]
+ if rows:
+ lines += [f'{files_label} file size: min to max change across builds containing the file.', '',
+ md_table(['File', 'Flash Δ', 'RAM Δ'], rows)]
+ other = sum(not any(any(d) for d in file_deltas[i].values()) for i in changed)
+ if other:
+ lines += ['', f'{other} other build{"" if other == 1 else "s"} changed without a '
+ f'{files_label} file-size change.']
+ return '\n'.join(lines) + '\n', footnote
+
+
def render_pairs(pairs, matched, engine, base_only=(), cur_only=(), failures=(), boards=(), symbols=False,
- labels=SIDE_LABELS):
+ labels=SIDE_LABELS, files_label='TinyUSB'):
"""Markdown report over paired elfs keyed by (board, elf path).
Every statistic is over per-pair deltas, sized by `engine`. `matched` counts
@@ -591,7 +726,7 @@
board-level failure. Failures or unmatched elfs mark the report INCOMPLETE.
`boards` lists the requested boards, so an unchanged or failed one is named.
`symbols` adds each file's changed symbols to each pair's section table; `labels`
- name the base and current sides.
+ name the base and current sides; `files_label` names the filtered files' totals.
"""
file_deltas = {i: _file_deltas(b, c) for i, (b, c) in pairs.items()}
changed = [i for i, (b, c) in pairs.items() if _pair_changed(b, c, symbols)]
@@ -601,12 +736,7 @@
f'compared, {len(changed)} changed']
if boards:
lines.append('- boards: ' + ', '.join(f'`{b}`' for b in boards))
- for label, ids in (('base-only', base_only), ('current-only', cur_only)):
- if ids:
- lines.append(f'- {label}: ' + ', '.join(f'`{_label(i)}`' for i in ids))
- for elf_id, side, stage, message in failures:
- lines.append(f'- FAILED `{_label(elf_id)}` {side} {stage}: {_md_escape(message)}')
- lines.append('')
+ lines += _problems(base_only, cur_only, failures) + ['']
if len(pairs) == 1:
(elf_id, (b, c)), = pairs.items()
@@ -619,18 +749,14 @@
if not changed:
return '\n'.join(lines + ['_no changes_', ''])
- pair_rows = []
- for elf_id in changed:
- b, c = pairs[elf_id]
- src = _delta(_files_total(b['files']), _files_total(c['files']))
- syms = _delta(b['all'], c['all'])
- pair_rows.append([_label(elf_id)] + [_fmt(d) for d in src + syms])
- lines.append(md_table(['Pair', 'filtered Flash Δ', 'filtered RAM Δ', f'{all_label} Flash Δ',
+ src = {i: _delta(_files_total(pairs[i][0]['files']), _files_total(pairs[i][1]['files'])) for i in changed}
+ pair_rows = [[_label(i)] + [_fmt(d) for d in src[i] + _delta(pairs[i][0]['all'], pairs[i][1]['all'])]
+ for i in changed]
+ lines.append(md_table(['Pair', f'{files_label} Flash Δ', f'{files_label} RAM Δ', f'{all_label} Flash Δ',
f'{all_label} RAM Δ'], pair_rows))
stats = _file_stats(file_deltas)
- rows = sorted((p for p, s in stats.items() if s['changed']),
- key=lambda p: (-max(abs(v[0]) for k in ('flash', 'ram') for v in stats[p][k]), p))
+ rows = _changed_paths(stats)
lines += ['', '_Changed / present: pairs where the file\'s Flash or RAM total changed / compared pairs '
'that contain it._', '',
md_table(['File', 'Changed / present', 'Flash Δ min', 'Flash Δ max', 'RAM Δ min', 'RAM Δ max'],
@@ -663,7 +789,8 @@
f'{engine} output broke its parsing (try another --engine to isolate)')
-def compare_sides(base, cur, engine, failures=(), boards=(), scope=None, symbols=False, labels=SIDE_LABELS):
+def compare_sides(base, cur, engine, failures=(), boards=(), scope=None, symbols=False, labels=SIDE_LABELS,
+ files_label='TinyUSB'):
"""Pair two sides and render them. Returns (md, failures, ok, data).
`failures` comes back with a filter failure for `scope` added when no
@@ -671,13 +798,14 @@
broke its parsing); pass `scope=None` when each scope was already checked.
`ok` is False on any failure or when nothing was compared. `data` is the
report's raw paired sizes, for JSON, symbols included only when `symbols` is set.
+ `files_label` is render_pairs()'.
"""
pairs, base_only, cur_only = pair_elfs(base, cur)
failures = list(failures)
if scope and pairs and not any(b['files'] or c['files'] for b, c in pairs.values()):
failures.append((scope, 'both', 'filter', _filter_failure(engine)))
md = render_pairs(pairs, len(base.keys() & cur.keys()), engine, base_only, cur_only, failures, boards,
- symbols, labels)
+ symbols, labels, files_label)
data = {
'engine': engine,
'boards': list(boards),
@@ -817,24 +945,117 @@
return error.replace(src_dir.rstrip(os.sep) + os.sep, '')
+ESP_IDF_IMAGE = 'espressif/idf:tinyusb' # .github/actions/setup_toolchain/espressif's tag
+ESP_IDF_MISSING = (f'ESP-IDF: source $IDF_PATH/export.sh, or docker with {ESP_IDF_IMAGE} '
+ f'(docker tag espressif/idf:v5.5.3 {ESP_IDF_IMAGE}, as CI does)')
+
+
+def _esp_examples(src_dir, board, example):
+ """The examples tools/build.py builds for espressif `board` (`example` alone when given):
+ those get_examples('espressif') lists that `src_dir` has, less skip_example's."""
+ import build # tools/build.py; its import has no side effects
+ with contextlib.chdir(src_dir): # build.py and build_utils read examples/ and hw/bsp from the cwd
+ return [e for e in build.get_examples('espressif')
+ if example in (None, e) and os.path.isdir(os.path.join('examples', e))
+ and not build_utils.skip_example(e, board)]
+
+
+def _link_hops(path):
+ """The paths a symlink resolves through, its real path last."""
+ hops = []
+ while os.path.islink(path):
+ path = os.path.normpath(os.path.join(os.path.dirname(path), os.readlink(path)))
+ hops.append(path)
+ return hops
+
+
+def is_espressif(board, src_dir=TINYUSB_ROOT):
+ return os.path.isdir(os.path.join(src_dir, 'hw', 'bsp', 'espressif', 'boards', board))
+
+
+def _idf_image():
+ return bool(shutil.which('docker')) and run(['docker', 'image', 'inspect', ESP_IDF_IMAGE]).returncode == 0
+
+
+def esp_without_idf(boards):
+ """The error for espressif `boards` no exported ESP-IDF or CI's image can build, else None."""
+ esp = [b for b in boards if is_espressif(b)]
+ if esp and not (shutil.which('idf.py') or _idf_image()):
+ return f'{", ".join(esp)} need {ESP_IDF_MISSING}'
+ return None
+
+
+def _idf_command(src_dir, build_dir, name):
+ """The argv that runs idf.py on `src_dir` into `build_dir`: an exported ESP-IDF, else CI's
+ image as this user in a container `name`, with both mounted at their own paths so the
+ elfs' DWARF matches the filters, and each symlinked dependency's real dir at every path
+ its link goes through (a base worktree's links point at the checkout's, which may point
+ at the main checkout's). None when neither is available."""
+ if shutil.which('idf.py'):
+ return ['idf.py']
+ if not _idf_image():
+ return None
+ # the image enables ccache; kept here it serves the next run (its default hash_dir keeps
+ # one tree's DWARF paths out of the other's objects)
+ cache = os.path.join(CODE_SIZE_DIR, '_ccache')
+ os.makedirs(cache, exist_ok=True)
+ mounts = {p: os.path.realpath(p) for p in map(os.path.abspath, (src_dir, build_dir, cache))}
+ for dep in runpy.run_path(os.path.join(src_dir, 'tools', 'get_deps.py'))['deps_all']:
+ path = os.path.join(src_dir, dep)
+ mounts.update((hop, os.path.realpath(path)) for hop in _link_hops(path))
+ cmd = ['docker', 'run', '--rm', '--name', name, '--user', f'{os.getuid()}:{os.getgid()}',
+ '-e', 'HOME=/tmp', '-e', f'CCACHE_DIR={os.path.abspath(cache)}']
+ for target, source in sorted(mounts.items()):
+ cmd += ['-v', f'{source}:{target}']
+ return cmd + [ESP_IDF_IMAGE, 'idf.py']
+
+
+def _build_idf(src_dir, build_dir, board, example):
+ """Build each ESP-IDF example project's app, as tools/build.py does (its bootloader is
+ not sized); the first failure stops."""
+ examples = _esp_examples(src_dir, board, example)
+ if not examples:
+ return subprocess.CompletedProcess([], 1, '', f'{board} builds no {example or "example"}')
+ container = f'tinyusb-code-size-{os.getpid()}'
+ idf = _idf_command(src_dir, build_dir, container)
+ if idf is None:
+ return subprocess.CompletedProcess([], 1, '', f'{board} needs {ESP_IDF_MISSING}')
+ for ex in examples:
+ ret = None
+ try:
+ ret = run(idf + ['-C', os.path.join(src_dir, 'examples', ex), '-B', os.path.join(build_dir, ex),
+ '-GNinja', f'-DBOARD={board}', 'app'], timeout=600)
+ finally:
+ # stopping the docker CLI leaves its container building; --rm only removes an exited one
+ if idf[0] == 'docker' and (ret is None or ret.returncode == 124):
+ run(['docker', 'rm', '-f', container])
+ if ret.returncode != 0:
+ break
+ return ret
+
+
def build_board(src_dir, build_dir, board, example, label):
"""Configure and build examples for a board as a `label` progress phase, printing
an excerpt of the output on failure. Returns None on success, else build_error().
When `example` is given, only that target is built (`ninja -C DIR NAME`),
- keeping single-example workflows fast.
+ keeping single-example workflows fast. An espressif board builds each example as
+ its own ESP-IDF project.
"""
phase = Phase(label)
os.makedirs(build_dir, exist_ok=True)
- ret = run(['cmake', '-B', build_dir, '-G', 'Ninja',
- f'-DBOARD={board}', '-DCMAKE_BUILD_TYPE=MinSizeRel',
- os.path.join(src_dir, 'examples')])
- if ret.returncode == 0:
- # ninja itself, not `cmake --build`: cmake does not pass a timeout's SIGTERM on
- cmd = ['ninja', '-C', build_dir]
- if example:
- cmd.append(os.path.basename(example))
- ret = run(cmd, timeout=600)
+ if is_espressif(board, src_dir):
+ ret = _build_idf(src_dir, build_dir, board, example)
+ else:
+ ret = run(['cmake', '-B', build_dir, '-G', 'Ninja',
+ f'-DBOARD={board}', '-DCMAKE_BUILD_TYPE=MinSizeRel',
+ os.path.join(src_dir, 'examples')])
+ if ret.returncode == 0:
+ # ninja itself, not `cmake --build`: cmake does not pass a timeout's SIGTERM on
+ cmd = ['ninja', '-C', build_dir]
+ if example:
+ cmd.append(os.path.basename(example))
+ ret = run(cmd, timeout=600)
failed = ret.returncode != 0
phase.done(failed=failed)
if not failed:
@@ -872,8 +1093,10 @@
"""
# escape the dir, not the wildcards: a checkout path is a path, not a pattern
root = glob.escape(build_dir)
- # <role>/<example>/*.elf: deeper elfs are helpers, e.g. pico-sdk's bs2_default.elf
- elfs = sorted(glob.glob(f'{root}/{example or "*/*"}/*.elf'))
+ # <role>/<example>/*.elf: deeper elfs are helpers, e.g. pico-sdk's bs2_default.elf, and
+ # cmake's _deps/ holds fetched tools, e.g. picotool's enc_bootloader.elf
+ elfs = sorted(e for e in glob.glob(f'{root}/{example or "*/*"}/*.elf')
+ if not os.path.relpath(e, build_dir).startswith('_deps' + os.sep))
if not elfs:
return {}, [(None, f'no .elf files in {build_dir}')]
@@ -901,7 +1124,7 @@
return sizes, errors
-def diff_summary(data, symbols):
+def diff_summary(data, symbols, files_label='TinyUSB'):
"""Console lines of one diff from its JSON-shaped `data`: coverage, the single
pair's filtered Δ, unmatched elfs and failures."""
pairs = {(p['board'], p['elf']): (p['base'], p['current']) for p in data['pairs']}
@@ -910,7 +1133,7 @@
if len(pairs) == 1:
(b, c), = pairs.values()
df, dr = _delta(_files_total(b['files']), _files_total(c['files']))
- line += f'; filtered Flash Δ {_fmt(df)}, RAM Δ {_fmt(dr)}'
+ line += f'; {files_label} Flash Δ {_fmt(df)}, RAM Δ {_fmt(dr)}'
lines = [line if data['status'] == 'complete' else f'INCOMPLETE: {line}']
for key in ('base_only', 'current_only'):
if data[key]:
@@ -920,13 +1143,13 @@
return lines
-def report_summary(sizes, failures, ok):
+def report_summary(sizes, failures, ok, files_label='TinyUSB'):
"""Console lines of one report: coverage, the single elf's filtered total, failures."""
sized = [s for s in sizes.values() if s is not None]
line = f'{len(sized)} of {len(sizes)} elfs sized'
if len(sized) == 1:
src = _files_total(sized[0]['files'])
- line += f'; filtered Flash {src["flash"]}, RAM {src["ram"]}'
+ line += f'; {files_label} Flash {src["flash"]}, RAM {src["ram"]}'
lines = [line if ok else f'INCOMPLETE: {line}']
return lines + [f'FAILED {_label(i)} {stage}: {message}' for i, stage, message in failures]
@@ -991,6 +1214,7 @@
def run_report(args):
"""`report`: build and size the working tree, one report per board and example."""
filters = args.filter or [tinyusb_src_filter(TINYUSB_ROOT)]
+ files_label = 'filtered' if args.filter else 'TinyUSB'
examples = args.example or [None]
drop_stale_reports(args.board, examples, 'report')
focused = _focused(args.board, examples)
@@ -1017,7 +1241,7 @@
if sized and not any(s['files'] for s in sized):
failures.append(((board, None), 'filter', _filter_failure(args.engine)))
phase.done(failed=bool(failures))
- md = render_report(sizes, args.engine, failures, [board], args.symbols)
+ md = render_report(sizes, args.engine, failures, [board], args.symbols, files_label)
ok = not failures and any(s is not None for s in sizes.values())
failed |= not ok
data = None
@@ -1030,11 +1254,409 @@
for i, stage, message in failures]}
tables = _labelled({i: [size_table(s, args.symbols, args.engine)] for i, s in sizes.items() if s},
len(sizes)) if focused else ()
- print_result(report_summary(sizes, failures, ok), f'{example}: ' if scope else '', tables)
+ print_result(report_summary(sizes, failures, ok, files_label), f'{example}: ' if scope else '',
+ tables)
write_report(report_path(board, example, 'report'), md, data)
return 1 if failed else 0
+SNAPSHOT_SCHEMA = 1
+_SHA_RE = re.compile(r'[0-9a-f]{40}')
+
+
+def _cmake_compiler(build_dir):
+ """{'id', 'version', 'name', 'build_type'} of a configured build dir, from CMake's own
+ compiler detection (not the host gcc), '' where CMake recorded none."""
+ info = {'id': '', 'version': '', 'name': '', 'build_type': ''}
+ found = glob.glob(os.path.join(glob.escape(build_dir), 'CMakeFiles', '*', 'CMakeCCompiler.cmake'))
+ if not found: # ESP-IDF: one project per <role>/<example>, all on the board's one toolchain
+ projects = sorted(glob.glob(os.path.join(glob.escape(build_dir), '*', '*', 'CMakeFiles')))
+ return _cmake_compiler(os.path.dirname(projects[0])) if projects else info
+ for path in found:
+ with open(path) as f:
+ text = f.read()
+ for key, var in (('id', 'CMAKE_C_COMPILER_ID'), ('version', 'CMAKE_C_COMPILER_VERSION'),
+ ('name', 'CMAKE_C_COMPILER')):
+ m = re.search(rf'^set\({var} "([^"]*)"\)', text, re.M)
+ if m:
+ info[key] = os.path.basename(m.group(1)) if key == 'name' else m.group(1)
+ cache = os.path.join(build_dir, 'CMakeCache.txt')
+ if os.path.isfile(cache):
+ with open(cache) as f:
+ m = re.search(r'^CMAKE_BUILD_TYPE:\w+=(.*)$', f.read(), re.M)
+ info['build_type'] = m.group(1) if m else ''
+ return info
+
+
+def _git_shas(event):
+ """(sha, base_sha, head_sha) of the checkout: a pull_request build checks out GitHub's
+ merge commit, whose first parent is the base branch tip it was built against and
+ second parent the PR head; any other build is its own base and head."""
+ ret = run(['git', '-C', TINYUSB_ROOT, 'rev-parse', 'HEAD', 'HEAD^1', 'HEAD^2']
+ if event == 'pull_request' else ['git', '-C', TINYUSB_ROOT, 'rev-parse', 'HEAD'])
+ shas = ret.stdout.split() if ret.returncode == 0 else []
+ if not shas or not all(_SHA_RE.fullmatch(s) for s in shas):
+ raise RuntimeError(f'cannot resolve the checkout commits for a {event} build: {ret.stderr.strip()}')
+ return (shas[0], shas[1], shas[2]) if len(shas) == 3 else (shas[0], shas[0], shas[0])
+
+
+def snapshot_boards(families, boards, examples, defines=()):
+ """The CI-pinned boards a build leg produced, as the Membrowse upload step picks them:
+ each family's pinned boards that build one of `examples` (build.py's
+ resolve_ci_boards, boards-only), plus each `-b` board that is pinned. `defines` are
+ the leg's -D tokens, which can enable examples (MAX3421_HOST=1)."""
+ import build # tools/build.py; its import has no side effects
+ pinned = ci_pinned_boards()
+ # build.py skips a -b board that builds none of the examples, before configuring it
+ picked = [b for b in boards if b in pinned and build.builds_any(b, examples, defines)]
+ for family in families:
+ picked += build.resolve_ci_boards(CI_PINNED_BOARDS, family, True, examples, extra_defines=defines)
+ return list(dict.fromkeys(picked))
+
+
+def board_snapshot(board, build_dir, examples, filters, defines=()):
+ """(elfs, failures) of one board's build dir: {elf path: sizes} of every sized elf,
+ and [{'elf', 'stage', 'message'}]. A missing build dir, an example of the scope the
+ board builds but has no elf for, or an elf that failed to size is a failure, never a
+ zero; a board whose elfs all match no TinyUSB file fails the filter check."""
+ if not os.path.isdir(build_dir):
+ return {}, [{'elf': None, 'stage': 'build', 'message': f'no build dir {_shown(build_dir)}'}]
+ elfs, failures = {}, []
+ scopes = [e for e in dict.fromkeys(examples)
+ if not build_utils.skip_example(e, board, defines)] if examples else [None]
+ for example in scopes:
+ sizes, errors = generate_sizes(build_dir, filters, example)
+ for rel, s in sizes.items():
+ if s is not None and not _valid_sizes(s):
+ errors.append((rel, 'sizes are not integer byte counts of files, sections and symbols'))
+ elif s is not None:
+ elfs[rel] = s
+ failures += [{'elf': rel, 'stage': 'build' if rel is None else 'report',
+ 'message': f'{example}: {msg}' if example and rel is None else msg}
+ for rel, msg in errors]
+ if elfs and not any(s['files'] for s in elfs.values()):
+ failures.append({'elf': None, 'stage': 'filter', 'message': _filter_failure('membrowse')})
+ return elfs, failures
+
+
+def run_snapshot(args):
+ """`snapshot`: size the CI-pinned boards a build leg already built, one
+ code-size-<board>.json each, for `compare` against another run's snapshots."""
+ os.chdir(TINYUSB_ROOT) # build_utils and build.py resolve hw/bsp and examples from the cwd
+ sha, base_sha, head_sha = _git_shas(args.event)
+ try:
+ from importlib.metadata import version
+ membrowse_version = version('membrowse')
+ except Exception:
+ membrowse_version = ''
+ filters = args.filter or [tinyusb_src_filter(TINYUSB_ROOT)]
+ defines = tuple(args.define_symbol)
+ boards = snapshot_boards(args.families, args.board, args.example, defines)
+ # a variant leg (--build-name) reports its one board under the build name
+ names = {args.build_name or b: b for b in boards}
+ os.makedirs(args.output, exist_ok=True)
+ # written even with no board: `compare` tells a leg that ran from one that never did
+ with open(os.path.join(args.output, 'leg.json'), 'w') as f:
+ json.dump({'schema': SNAPSHOT_SCHEMA, 'examples': args.example, 'boards': list(names),
+ 'build_outcome': args.build_outcome, 'sha': sha, 'base_sha': base_sha, 'head_sha': head_sha},
+ f, sort_keys=True)
+ if not boards:
+ print('snapshot: no CI-pinned board in this leg')
+ return 0
+ failed = False
+ for board, base_board in names.items():
+ build_dir = os.path.join(args.build_root, f'cmake-build-{board}')
+ elfs, failures = board_snapshot(base_board, build_dir, args.example, filters, defines)
+ # the leg's commits, examples and build outcome are in its leg.json
+ data = {'schema': SNAPSHOT_SCHEMA, 'board': board, 'engine': 'membrowse',
+ 'membrowse_version': membrowse_version, 'compiler': _cmake_compiler(build_dir),
+ 'elfs': {rel: _json_sizes(s, args.symbols) for rel, s in sorted(elfs.items())},
+ 'failures': failures}
+ path = os.path.join(args.output, f'code-size-{board}.json')
+ with open(path, 'w') as f:
+ json.dump(data, f, sort_keys=True, separators=(',', ':'))
+ failed |= bool(failures)
+ print(f' {board}: {len(elfs)} elfs sized, {len(failures)} failures -> {_shown(path)}')
+ for fail in failures:
+ print(f' FAILED {fail["elf"] or board} {fail["stage"]}: {fail["message"]}')
+ return 1 if failed else 0
+
+
+_NAME_RE = re.compile(r'[A-Za-z0-9_.+/-]+')
+COMMENT_LIMIT = 60000 # GitHub caps a comment at 65536 characters
+
+
+def _clean(text, limit=300):
+ """Snapshot text for a report: a snapshot is PR-built data, so nothing in it may open
+ Markdown, HTML or an @mention."""
+ return re.sub(r'[^A-Za-z0-9_.,:;=()/+ -]', '?', str(text))[:limit]
+
+
+def _is_count(value):
+ return isinstance(value, int) and not isinstance(value, bool)
+
+
+def _region(value):
+ return isinstance(value, dict) and set(value) == {'flash', 'ram'} and all(map(_is_count, value.values()))
+
+
+def _int_map(value, depth):
+ """Whether `value` is `depth` levels of str-keyed dicts over byte counts."""
+ return isinstance(value, dict) and all(
+ isinstance(k, str) and (_int_map(v, depth - 1) if depth > 1 else _is_count(v)) for k, v in value.items())
+
+
+def _valid_sizes(s):
+ return (isinstance(s, dict) and {'files', 'all', 'sections'} <= set(s) <= {'files', 'all', 'sections', 'symbols'}
+ and _region(s['all']) and isinstance(s['files'], dict)
+ and all(isinstance(k, str) and _region(v) for k, v in s['files'].items())
+ and _int_map(s['sections'], 2) and ('symbols' not in s or _int_map(s['symbols'], 3)))
+
+
+def _is_name(value):
+ return isinstance(value, str) and bool(_NAME_RE.fullmatch(value)) and '..' not in value.split('/')
+
+
+def _names(value, none_ok=False):
+ return (none_ok and value is None) or (isinstance(value, list) and all(map(_is_name, value)))
+
+
+def _commits(data):
+ """(sha, base_sha, head_sha), or None unless all are 40-hex."""
+ shas = tuple(data.get(k) for k in ('sha', 'base_sha', 'head_sha'))
+ return shas if all(isinstance(s, str) and _SHA_RE.fullmatch(s) for s in shas) else None
+
+
+def _scope_error(data):
+ legs, fam = data.get('legs'), data.get('family_examples', {})
+ if not isinstance(data.get('code_changed'), bool):
+ return 'code_changed is not a boolean'
+ if not (isinstance(legs, list) and all(isinstance(leg, dict) and isinstance(leg.get('toolchain'), str)
+ and isinstance(leg.get('arg'), str) for leg in legs)):
+ return 'legs are not {toolchain, arg} strings'
+ if not (isinstance(fam, dict) and all(map(_names, fam.values()))):
+ return 'family_examples is not a map of example lists'
+ return None
+
+
+def _leg_error(data):
+ if not isinstance(data.get('build_outcome'), str):
+ return 'build_outcome is not a string'
+ if not _names(data.get('boards')):
+ return 'boards is not a list of board names'
+ if 'examples' not in data or not _names(data['examples'], none_ok=True):
+ return 'examples is not a list of example names'
+ return None if _commits(data) else 'commit hashes are not 40-hex'
+
+
+def _shard_error(data, filename):
+ board = data.get('board')
+ if not (_is_name(board) and filename == f'code-size-{board}.json'):
+ return 'board name does not match the file'
+ if data.get('engine') != 'membrowse':
+ return 'unsupported engine'
+ elfs, failures = data.get('elfs'), data.get('failures')
+ if not isinstance(elfs, dict) or not isinstance(failures, list):
+ return 'no elfs/failures'
+ for elf, sizes in elfs.items():
+ if not _is_name(elf):
+ return 'invalid elf path'
+ if not _valid_sizes(sizes):
+ return f'{elf}: sizes are not integer byte counts'
+ if not all(isinstance(f, dict) and 'elf' in f and (f['elf'] is None or _is_name(f['elf']))
+ and isinstance(f.get('stage'), str) and isinstance(f.get('message'), str) for f in failures):
+ return 'failures are not {elf, stage, message} records'
+ return None
+
+
+def load_snapshots(root):
+ """Every snapshot file under `root` (one directory per downloaded artifact):
+ {'scope': scope.json or None, 'legs': {artifact dir: leg.json}, 'shards': {board: shard},
+ 'errors': [(file, why)]}. A shard records its artifact dir as '_leg'. An unusable file
+ is an error, never guessed at."""
+ run = {'scope': None, 'legs': {}, 'shards': {}, 'errors': []}
+ for dirpath, _dirs, files in sorted(os.walk(root)):
+ artifact = os.path.basename(dirpath)
+ for name in sorted(f for f in files if f.endswith('.json')):
+ rel = os.path.relpath(os.path.join(dirpath, name), root)
+ try:
+ with open(os.path.join(dirpath, name)) as f:
+ data = json.load(f)
+ except (OSError, ValueError):
+ run['errors'].append((rel, 'unreadable JSON'))
+ continue
+ if not isinstance(data, dict) or data.get('schema') != SNAPSHOT_SCHEMA:
+ why = 'unsupported snapshot schema'
+ elif name == 'scope.json':
+ why = _scope_error(data)
+ run['scope'] = None if why else data
+ elif name == 'leg.json':
+ why = _leg_error(data)
+ if not why:
+ run['legs'][artifact] = data
+ elif name.startswith('code-size-'):
+ why = _shard_error(data, name)
+ if not why and data['board'] in run['shards']:
+ why = f'second snapshot of {data["board"]}'
+ if not why:
+ run['shards'][data['board']] = {**data, '_leg': artifact}
+ else:
+ why = 'not a snapshot file'
+ if why:
+ run['errors'].append((rel, why))
+ return run
+
+
+def _leg_examples(scope, arg):
+ """The examples build_util.yml builds for a scope leg: its arg's own -e plus the scope's
+ family map entry for that arg, None for all."""
+ return sorted(set(re.findall(r' -e ([^ ]+)', arg) + scope.get('family_examples', {}).get(arg, []))) or None
+
+
+def _in_scope(elf, examples):
+ """Whether an elf path (<role>/<example>/<name>.elf) is one of `examples`, all when None."""
+ return examples is None or '/'.join(elf.split('/')[:2]) in examples
+
+
+def _usable_shards(run, side, failures, boards=None):
+ """`run`'s shards listed in the boards of their own leg, whose build succeeded, each given
+ its leg's examples as '_examples'. Returns ({board: shard}, {leg commits}); every
+ rejection is added to `failures`, only of `boards` when given."""
+ shards, commits = {}, {_commits(leg) for leg in run['legs'].values()}
+ for rel, why in run['errors']:
+ board = re.fullmatch(r'code-size-(.+)\.json', os.path.basename(rel))
+ if boards is None or not board or board.group(1) in boards:
+ failures.append(((_clean(rel), None), side, 'snapshot', _clean(why)))
+ for board, shard in sorted(run['shards'].items()):
+ if boards is not None and board not in boards:
+ continue
+ leg = run['legs'].get(shard['_leg'])
+ if leg is None or board not in leg['boards']:
+ failures.append(((board, None), side, 'snapshot', 'not a board of its leg\'s build'))
+ elif leg['build_outcome'] != 'success':
+ failures.append(((board, None), side, 'build', f'the build step ended {_clean(leg["build_outcome"])}'))
+ else:
+ shards[board] = {**shard, '_examples': leg['examples']}
+ for _artifact, leg in sorted(run['legs'].items()):
+ failures += [((board, None), side, 'snapshot', 'the leg left no snapshot of it') for board in leg['boards']
+ if board not in run['shards'] and (boards is None or board in boards)]
+ return shards, commits
+
+
+def _scope_failures(scope, legs):
+ """Legs of the scope that left no leg.json, or measured other examples than it selected,
+ and legs that are not of it."""
+ failures, expected = [], set()
+ for leg in scope['legs']:
+ tag = re.sub(r' -e [^ ]+', '', leg['arg']) # build_util.yml's ARTIFACT_TAG
+ artifact = f'code-size-{leg["toolchain"]}-{tag}'
+ expected.add(artifact)
+ if artifact not in legs:
+ failures.append(((_clean(artifact), None), 'current', 'snapshot', 'the build leg left no snapshot'))
+ elif (legs[artifact]['examples'] is not None # all examples covers any selection
+ and sorted(set(legs[artifact]['examples'])) != _leg_examples(scope, leg['arg'])):
+ failures.append(((_clean(artifact), None), 'current', 'scope', 'measured other examples than selected'))
+ failures += [((_clean(a), None), 'current', 'scope', 'not a leg of this run') for a in sorted(set(legs) - expected)]
+ return failures
+
+
+def compare_runs(base, cur, baseline=None, symbols=False):
+ """Compare two load_snapshots() runs, `cur` defining the scope. Returns
+ (full Markdown, comment Markdown, JSON data). `baseline` is code_size_ci.py's
+ description of the base run ({'sha', 'url', 'exact', 'note'}), or None. Anything that
+ leaves coverage short is a failure; a comparable difference is a note."""
+ notes, failures = [], []
+ scope = cur['scope']
+ if scope is None:
+ failures.append((('scope', None), 'current', 'snapshot', 'no usable scope manifest'))
+ elif not (scope['legs'] or cur['legs'] or cur['shards'] or cur['errors']):
+ md = 'Nothing to measure: no code change, or no CI-pinned build leg selected.\n'
+ return md, f'## Code size\n\n{md}', {'status': 'nothing measured'}
+ else:
+ failures += _scope_failures(scope, cur['legs'])
+ cur_shards, cur_commits = _usable_shards(cur, 'current', failures)
+ base_shards, base_commits = _usable_shards(base, 'base', failures, set(cur_shards))
+ labels = SIDE_LABELS
+ if len(cur_commits) > 1:
+ failures.append((('commits', None), 'current', 'snapshot', 'the snapshots are of different commits'))
+ cur_shards = {}
+ elif cur_commits:
+ (_sha, base_sha, head_sha), = cur_commits
+ labels = (base_sha[:10], head_sha[:10])
+ base_shas = {c[0] for c in base_commits}
+ if len(base_shas) > 1 or (baseline and base_shas and base_shas != {baseline.get('sha')}):
+ failures.append((('commits', None), 'base', 'snapshot', 'the snapshots are not of the selected baseline commit'))
+ base_shards = {}
+
+ sides, no_baseline, outside = {'base': {}, 'current': {}}, [], 0
+ for board, shard in sorted(cur_shards.items()):
+ failures += [((board, f['elf']), 'current', _clean(f['stage']), _clean(f['message'])) for f in shard['failures']]
+ b = base_shards.get(board)
+ if b is None:
+ no_baseline.append(board)
+ failures.append(((board, None), 'base', 'snapshot', 'no baseline snapshot'))
+ continue
+ for key in ('membrowse_version', 'compiler'):
+ if b.get(key) != shard.get(key):
+ notes.append(f'{board}: {key} differs from the baseline ({_clean(b.get(key))} -> '
+ f'{_clean(shard.get(key))}): comparable, not exact')
+ examples = shard['_examples']
+ # a baseline failure is the baseline's, never this PR's: its elf is not new here
+ base_failed = set()
+ for f in b['failures']:
+ if f['elf'] is None or _in_scope(f['elf'], examples):
+ failures.append(((board, f['elf']), 'base', _clean(f['stage']), _clean(f['message'])))
+ base_failed.add(f['elf'])
+ for elf, sizes in b['elfs'].items():
+ if _in_scope(elf, examples):
+ sides['base'][(board, elf)] = sizes
+ else:
+ outside += 1
+ sides['current'].update({(board, elf): s for elf, s in shard['elfs'].items() if elf not in base_failed})
+ if outside:
+ notes.append(f'{outside} baseline elfs are of examples this PR did not build (outside coverage)')
+
+ if symbols and not all('symbols' in s for side in sides.values() for s in side.values()):
+ symbols = False
+ notes.append('symbols omitted: some snapshots were taken without them')
+ head = []
+ if baseline:
+ kind = 'exact' if baseline.get('exact') else 'approximate'
+ head.append((f'Baseline: {_clean(baseline["sha"][:10])} ({kind}) {_clean(baseline.get("url"), 200)}'
+ if baseline.get('sha') else 'Baseline: unavailable')
+ + (f' - {_clean(baseline["note"])}' if baseline.get('note') else ''))
+ head += [f'- {n}' for n in notes]
+ boards = sorted(cur_shards)
+ full, _fails, _ok, data = compare_sides(sides['base'], sides['current'], 'membrowse', failures, boards,
+ symbols=symbols, labels=labels)
+ pairs, base_only, cur_only = pair_elfs(sides['base'], sides['current'])
+ body, footnote = render_comment(pairs, 'membrowse', base_only, cur_only, failures, symbols)
+ preface = '\n'.join(head) + ('\n\n' if head else '')
+ comment = f'## Code size\n\n{preface}{body}'
+ truncated = '\n\n_Truncated: see the full report._\n'
+ if len(comment) + len(footnote) > COMMENT_LIMIT:
+ comment = comment[:COMMENT_LIMIT - len(footnote) - len(truncated)].rsplit('\n', 1)[0] + truncated
+ comment += footnote
+ data.update({'notes': notes, 'baseline': baseline, 'no_baseline': no_baseline, 'outside': outside})
+ return preface + full, comment, data
+
+
+def run_compare(args):
+ """`compare`: compare a run's snapshots against a baseline run's, no build."""
+ baseline = None
+ if args.baseline_info:
+ with open(args.baseline_info) as f:
+ baseline = json.load(f)
+ base = load_snapshots(args.base) # os.walk of a missing directory yields nothing
+ full, comment, data = compare_runs(base, load_snapshots(args.current), baseline, args.symbols)
+ os.makedirs(args.output, exist_ok=True)
+ write_report(os.path.join(args.output, 'code-size'), full, data)
+ with open(os.path.join(args.output, 'comment.md'), 'w') as f:
+ f.write(comment)
+ print(f' comment: {_shown(os.path.join(args.output, "comment.md"))}')
+ return 0
+
+
def example_arg(value):
example = value.rstrip('/')
if not example:
@@ -1086,8 +1708,54 @@
parser.add_argument('--combined', action='store_true',
help='Also write one comparison over every board '
'(cmake-code-size/_combined/diff.md), in addition to per-board.')
+ snap = sub.add_parser('snapshot', help='size the CI-pinned boards a CI build leg already built, '
+ 'one code-size-<board>.json each (no build)')
+ snap.add_argument('families', nargs='*', default=[], help='Families the leg built (as tools/build.py)')
+ snap.add_argument('-b', '--board', action='append', default=[], help='Board the leg built (repeatable)')
+ snap.add_argument('-e', '--example', action='append', default=None, type=example_arg,
+ help='Examples the leg built (repeatable); omit when it built all')
+ snap.add_argument('--build-name', help='A variant leg\'s build name (as tools/build.py): its build dir, and '
+ 'the board it reports as')
+ snap.add_argument('-D', '--define-symbol', action='append', default=[],
+ help='A variant leg\'s build-system define (as tools/build.py); it needs --build-name')
+ snap.add_argument('--cflag', action='append', default=[],
+ help='A variant leg\'s compiler flag (as tools/build.py); it needs --build-name')
+ snap.add_argument('-o', '--output', required=True, help='Directory for the code-size-<board>.json files')
+ snap.add_argument('--build-root', default=os.path.join(TINYUSB_ROOT, 'cmake-build'),
+ help='Where tools/build.py put cmake-build-<board> (default: cmake-build)')
+ snap.add_argument('--event', default=os.environ.get('GITHUB_EVENT_NAME', 'push'),
+ help='CI event; pull_request reads base/head from the merge commit '
+ '(default: $GITHUB_EVENT_NAME, else push)')
+ snap.add_argument('--build-outcome', default='success', help='The Build step\'s outcome, recorded as is')
+ snap.add_argument('-f', '--filter', action='append', default=None,
+ help='Path-substring filter (repeatable); default: this checkout\'s <checkout>/src/')
+ snap.add_argument('--symbols', action='store_true', help='Include each file\'s symbols')
+ snap.add_argument('-v', '--verbose', action='store_true', help='Print commands')
+ cmp = sub.add_parser('compare', help='compare a CI run\'s snapshots against a baseline run\'s '
+ '(code-size.md, code-size.json, comment.md; no build)')
+ cmp.add_argument('current', help='The run\'s downloaded artifacts, one directory each')
+ cmp.add_argument('base', help='The baseline run\'s downloaded artifacts; a missing directory is no baseline')
+ cmp.add_argument('-o', '--output', required=True, help='Directory for the reports')
+ cmp.add_argument('--baseline-info', help='JSON describing the baseline run: {sha, url, exact, note}')
+ cmp.add_argument('--symbols', action='store_true', help='Show each changed file\'s symbols in the full report')
args = top.parse_args()
- verbose = args.verbose
+ verbose = getattr(args, 'verbose', False)
+
+ if args.command == 'compare':
+ return run_compare(args)
+
+ if args.command == 'snapshot':
+ # a leg whose build did not succeed may have skipped the membrowse install too;
+ # its snapshot still records that outcome, and any elf it left fails to size
+ if engine_missing('membrowse') and args.build_outcome == 'success':
+ snap.error(f'membrowse not found - install {ENGINES["membrowse"].install}')
+ if invalid := invalid_boards(args.board + ([args.build_name] if args.build_name else [])):
+ snap.error(f'invalid board name: {", ".join(invalid)}')
+ if args.build_name and (args.families or len(args.board) != 1):
+ snap.error('--build-name names exactly one -b board and no families')
+ if (args.define_symbol or args.cflag) and not args.build_name:
+ snap.error('-D or --cflag makes a variant, which needs --build-name to report under')
+ return run_snapshot(args)
if engine_missing(args.engine):
sub.choices[args.command].error(f'{args.engine} not found - install {ENGINES[args.engine].install}, '
@@ -1098,6 +1766,8 @@
report_parser.error('at least one -b BOARD is required')
if invalid := invalid_boards(args.board):
report_parser.error(f'invalid board name: {", ".join(invalid)}')
+ if error := esp_without_idf(args.board):
+ report_parser.error(error)
return run_report(args)
if args.bloaty and not args.example:
@@ -1113,8 +1783,12 @@
parser.error('at least one -b BOARD is required (or pass --ci)')
if invalid := invalid_boards(args.board):
parser.error(f'invalid board name: {", ".join(invalid)}')
+ # before any board builds: a --ci run would otherwise fail its espressif boards last
+ if error := esp_without_idf(args.board):
+ parser.error(error)
worktree_dir = os.path.join(CODE_SIZE_DIR, '_worktree')
+ files_label = 'filtered' if args.filter else 'TinyUSB'
# Per-side filters: when no override is given, each build uses its own
# absolute <checkout>/src/ path so we only match TinyUSB stack code from that
@@ -1202,7 +1876,7 @@
md, failures, ok, data = compare_sides(
sides['base'], sides['current'], args.engine, failures, [board], scope=(board, None),
- symbols=args.symbols, labels=labels)
+ symbols=args.symbols, labels=labels, files_label=files_label)
if not build_failure:
phase.done(failed=not ok)
failed |= not ok
@@ -1214,7 +1888,8 @@
args.engine, labels)
for p in data['pairs'] if _pair_changed(p['base'], p['current'], args.symbols)},
len(data['pairs'])) if focused else ()
- print_result(diff_summary(data, args.symbols), f'{example}: ' if scope else '', tables)
+ print_result(diff_summary(data, args.symbols, files_label), f'{example}: ' if scope else '',
+ tables)
write_report(report_path(board, example), md, report_data(data))
if args.bloaty and example and not build_failure:
@@ -1247,9 +1922,9 @@
# every scope was filter-checked above, and its failures carried over
md, _failures, ok, data = compare_sides(
combined_sides['base'], combined_sides['current'], args.engine,
- combined_failures, args.board, symbols=args.symbols, labels=labels)
+ combined_failures, args.board, symbols=args.symbols, labels=labels, files_label=files_label)
failed |= not ok
- print_result(diff_summary(data, args.symbols))
+ print_result(diff_summary(data, args.symbols, files_label))
write_report(os.path.join(combined_dir, 'diff'), md, report_data(data))
finally:
# an add stopped by a signal leaves it too, locked `initializing`
diff --git a/tools/membrowse_cli.py b/tools/membrowse_cli.py
index fe147d2..12b06f7 100644
--- a/tools/membrowse_cli.py
+++ b/tools/membrowse_cli.py
@@ -13,6 +13,8 @@
# `--defsym=SYM=VAL` or `--defsym,SYM=VAL` from a link line
DEFSYM_RE = re.compile(r'--defsym[=,](\S+)')
+# ESP-IDF's generated scripts, relative to the build dir (family_support.cmake's MEMBROWSE_LD_OVERRIDE)
+IDF_LD_SCRIPTS = ('esp-idf/esp_system/ld/memory.ld', 'esp-idf/esp_system/ld/sections.ld')
def link_command(ninja, build_dir, elf):
@@ -129,8 +131,7 @@
build = (f'idf.py -C {shlex.quote(f"examples/{example}")} -B {quoted_build_dir} '
f'-GNinja {shlex.quote(f"-DBOARD={board}")} build')
elf_path = f'{build_dir}/{basename}.elf'
- shim_scripts = ('esp-idf/esp_system/ld/memory.ld',
- 'esp-idf/esp_system/ld/sections.ld')
+ shim_scripts = IDF_LD_SCRIPTS
else:
configure = (f'cmake -S examples -B {quoted_build_dir} '
f'{shlex.quote(f"-DBOARD={board}")} -G Ninja '
