agents: add merge-queue and soft-failure CI lessons (#4393) Shepherding #4262 surfaced two agent mistakes: assuming merge-queue builds skip Buildkite's CI config approval step, and reporting soft-failed jobs as passed. Raw `bk job log` output also defeated simple greps. The CI skills now cover finding and unblocking merge-queue builds, which changed files trigger the approval step, checking soft-failed jobs against the latest main build, and cleaning job logs before grepping.
diff --git a/.agents/skills/buildkite-get-results/SKILL.md b/.agents/skills/buildkite-get-results/SKILL.md index 801e51c..f8cb2c8 100644 --- a/.agents/skills/buildkite-get-results/SKILL.md +++ b/.agents/skills/buildkite-get-results/SKILL.md
@@ -8,3 +8,9 @@ The `--jobs` flag can do glob-style filtering of jobs. The `--download` flag will download job logs. + +For quick checks with the `bk` CLI, `bk job log <job_id> --no-timestamps` +prints a job's log (it doesn't take `--build`). The output keeps a stray +`ESC _` prefix, cursor and color codes, and carriage returns; strip them +before grepping: +`sed 's/\x1b\[[0-9;]*[A-Za-z]//g; s/\x1b_//g; s/\r//g'`.
diff --git a/.agents/skills/merge-pr/SKILL.md b/.agents/skills/merge-pr/SKILL.md index 1939d71..056e51c 100644 --- a/.agents/skills/merge-pr/SKILL.md +++ b/.agents/skills/merge-pr/SKILL.md
@@ -31,9 +31,9 @@ transient network flakes (e.g., HTTP 504 gateway timeouts, downloader errors, `setup-bazel` `Unable to find Bazelisk version` errors) and unblock manual approval steps. - - **Blocked Builds (`.bazelci/presubmit.yml` Approval)**: When - `.bazelci/presubmit.yml` is modified, Buildkite blocks the build until - manually approved. Approve and unblock it immediately via + - **Blocked Builds (CI Config Approval)**: When the PR changes a file under + `.bazelci/` or a `.yml` file outside `.github/`, Buildkite blocks the + build until manually approved. Approve and unblock it immediately via `./.agents/skills/retry-ci-job/scripts/retry_ci_jobs.py --pr <pr_number> --unblock` (or `bk job unblock <job_id> --no-input`) so CI does not sit blocked indefinitely. @@ -44,6 +44,15 @@ `'.[].name | select(test("gh-readonly-queue/.*/pr-<pr_number>-"))'` and monitor commit statuses/Buildkite builds running on that temporary branch. + - **Blocked Merge Queue Builds**: If the PR changes a file under + `.bazelci/` or a `.yml` file outside `.github/`, the merge queue build + is blocked again, even with the `presubmit-auto-run` label, and + `monitor_remote_ci.py` doesn't watch it. Find it with `bk build list + --pipeline bazel/rules-python-python --branch <queue_branch> --json | + jq '.[] | {number, state, blocked}'`, then unblock it with + `./.agents/skills/retry-ci-job/scripts/retry_ci_jobs.py + --buildkite-build bazel/rules-python-python/<build_number> --unblock`. + `--pr <pr_number>` resolves to the PR's build, not the queue build. 4. **Queue Shepherding**: Periodically check `gh pr view <pr_number> --repo bazel-contrib/rules_python --json state,autoMergeRequest,mergeStateStatus,mergeable`. While `state` is
diff --git a/.agents/skills/monitor-ci-results/SKILL.md b/.agents/skills/monitor-ci-results/SKILL.md index c73295e..f0b6922 100644 --- a/.agents/skills/monitor-ci-results/SKILL.md +++ b/.agents/skills/monitor-ci-results/SKILL.md
@@ -43,7 +43,12 @@ 3. **Failure Reporting**: When any GitHub check or Buildkite job completes with errors, `monitor_remote_ci.py` dispatches a high-priority notification message reporting the failed check back to your conversation. - - Soft failures (e.g. `*rolling*` Bazel) are non-blocking warnings; ignore. + - Soft failures (e.g. `*rolling*` Bazel) don't block merging, but the + monitor counts them as passed. Before reporting that everything + passed, list them with `bk build view <build_number> --pipeline + bazel/rules-python-python --json | jq '.jobs[] | select(.soft_failed) + | {id, name}'` and compare their failing tests with the latest `main` + build's. - If `blocks-do-not-merge` fails while the PR intentionally carries the `"do not merge"` label, treat it as an expected merge gate rather than a code failure.