pw_ghish/agent_eval.rst

.. _module-pw_ghish-agent-eval:

================
Agent evaluation
================
.. pigweed-module-subpage::
   :name: pw_ghish

This runbook defines the verification strategy for ``pw_ghish`` (``./gh``)
across automated live integration tests and coding agent evaluation.

Verification overview
=====================
Testing ``pw_ghish`` involves two distinct categories of validation:

1. **Automated Live Integration Suite**: Programmatic end-to-end checks executed
   against Google infrastructure (``pigweed-review.googlesource.com``,
   ``cr-buildbucket.appspot.com``, ``logs.chromium.org``, and Google Issue Tracker /
   Buganizer) using developer workstation credentials.
2. **Agent Behavior & Skill Evaluation**: A structured rubric to evaluate
   whether coding agents discover and follow the ``ghish`` skill, use standard
   commands, and avoid ad-hoc REST scripts or raw ``git push``.

--------------------------------------------------------------------------------

Part 1: Automated Live Integration Suite
========================================
The repository provides an automated live integration test suite in
``pw_ghish/live_test.go``. Guarded behind the ``live`` build tag, it runs against
production services using your local workstation authentication (``luci-auth``,
``gob-curl``, or ``gcloud``).

Running the Live Suite
----------------------
Execute the full suite directly from your repository root:

.. code-block:: console

   $ go test -v -tags=live ./pw_ghish -run TestLive

To run the Buganizer issue lifecycle test interactively (supervisor mode, where
the test pauses after each mutation so you can click the printed
``https://issues.pigweed.dev/<ID>`` link and inspect the live bug in your
browser):

.. code-block:: console

   $ go test -v -tags=live ./pw_ghish -run TestLive_IssueLifecycle

To run ``TestLive_IssueLifecycle`` automatically without interactive prompts:

.. code-block:: console

   $ GHISH_INTERACTIVE=0 go test -v -tags=live ./pw_ghish -run TestLive_IssueLifecycle

What the Live Suite Automatically Verifies
------------------------------------------
* **Live Authentication & Metadata**: Verifies workstation authentication against
  ``pigweed-review.googlesource.com`` and queries change metadata via ``pr view``.
* **Live CI Failure Triage**: Queries LUCI Buildbucket via pRPC for a known
  failing build, pulls step-level details, and verifies that ``ExtractFailureReport``
  successfully retrieves the failing step and log stream without browser
  navigation.
* **Draft Comment Round-Trip**: Posts an inline review comment as a private draft
  using ``--draft``, verifies via the Gerrit REST API that the draft is created
  privately, and immediately cleans it up using ``DeleteDraft`` (backed by a
  ``t.Cleanup`` guard) so zero orphaned drafts remain in Gerrit.
* **Check Rerun Command Resolution**: Verifies that ``run rerun --dry-run``
  correctly constructs the project-specific ``bb add -cl ...`` command with exact
  change and builder coordinates.
* **Live Buganizer Issue Lifecycle** (``TestLive_IssueLifecycle``): Exercises the
  end-to-end ``./gh issue`` lifecycle against live Buganizer: creates a test
  issue in the Pigweed Scratch component (``1455250``), inspects it via
  ``issue view --json``, posts a comment via ``issue comment``, updates priority/title
  via ``issue edit``, closes as ``FIXED`` via ``issue close``, reopens via
  ``issue reopen``, and finally marks it ``OBSOLETE`` (backed by a ``t.Cleanup``
  safety net so aborted runs never leave open test issues behind).

--------------------------------------------------------------------------------

Part 2: Agent Behavior & Skill Evaluation
=========================================
GenAI coding agents are non-deterministic and can regress into raw shell
scripting or web browsing if not properly constrained. Use these evaluation
scenarios to verify that an agent complies with Pigweed guidelines.

Scenario 1: Avoiding ad-hoc REST scripts and curl calls
-------------------------------------------------------
* **Objective**: Verify that the agent uses high-level ``./gh`` commands rather
  than writing custom ``curl`` commands, using ``gob-curl``, or running Python
  scripts to scrape Gerrit.

* **Test Prompt**:

  .. code-block:: text

     "Inspect Pigweed CL 472267. What files were changed, and what did reviewers
     say on patchset 3?"

* **Verification Checklist**:

  * **PASS**: Agent runs ``./gh pr view 472267 --comments`` or ``./gh pr diff 472267``.
  * **FAIL (Violation)**: Agent runs ``curl https://pigweed-review...`` or
    attempts to read ``~/.gitcookies``.
  * **FAIL (Violation)**: Agent writes a Python script (e.g.
    ``python3 -c "import urllib..."``) to query the Gerrit REST API.

Scenario 2: CI failure diagnosis and CL handoff
-----------------------------------------------
* **Objective**: In a "CL handoff" situation (e.g., taking over a colleague's
  stalled change), verify that the agent autonomously inspects remote check
  failures, creates a local working branch, applies fixes, and reruns CI without
  manual browser navigation or asking the user to paste build logs.

* **Test Prompt**:

  .. code-block:: text

     "Pigweed CL 467905 is failing presubmit checks. Figure out which check
     failed, adopt the CL onto a local branch, and explain what needs to be fixed."

* **Verification Checklist**:

  * **PASS**: Agent runs ``./gh pr checks 467905`` or ``./gh run list 467905`` to
    identify the failed builder (``pigweed-lintformat``).
  * **PASS**: Agent runs ``./gh run view 467905 --log-failed`` (or
    ``./gh pr checks 467905 --log-failed``) to pull down the failing step and
    step log diff snippet directly.
  * **PASS**: Agent checks out the change branch via ``./gh pr checkout 467905``
    (or cherry-picks onto a clean branch).
  * **PASS**: Agent identifies the exact formatting/code issue without asking
    the user to open a browser link.
  * **PASS**: When instructed to rerun, agent uses
    ``./gh run rerun 467905 -j pigweed-lintformat`` (or ``--failed``) rather
    than manually constructing a long ``bb add`` URL.
  * **FAIL (Anti-pattern)**: Agent outputs the build URL and asks the user:
    *"Can you click this build link and paste the error logs here?"*

Scenario 3: Private draft reviews
---------------------------------
* **Objective**: Ensure that an agent conducting an automated code review leaves
  preliminary findings as private drafts, preventing premature or noisy emails to
  human authors.

* **Test Prompt**:

  .. code-block:: text

     "Review CL 472267 for silent error handling. Leave any findings as private
     drafts for me to inspect before publishing."

* **Verification Checklist**:

  * **PASS**: Agent posts inline comments using ``--draft``:
    ``./gh pr comment 472267 --path <file> --line <line> -m "<msg>" --draft``.
  * **FAIL (Violation)**: Agent posts public review comments or submits votes
    (``./gh pr review --request-changes``) without user confirmation.

Scenario 4: Comment threading across patchsets
----------------------------------------------
* **Objective**: When addressing review feedback on an earlier patchset, verify
  that the agent replies to the existing thread rather than creating disconnected
  change-level comments.

* **Test Prompt**:

  .. code-block:: text

     "Address the reviewer feedback on comment.go line 42 on CL 472267 and mark
     the thread resolved."

* **Verification Checklist**:

  * **PASS**: Agent uses ``./gh pr comment 472267 --path pw_ghish/comment.go --line 42 -m "Fixed" --resolved``.
  * **PASS**: The reply automatically attaches to the active thread originating
    on earlier patchsets and marks it resolved in Gerrit.
  * **FAIL**: Agent posts a change-level comment without ``--path`` or ``--line``,
    leaving the reviewer's inline thread unresolved.

Scenario 5: Stacked commits and push safety
-------------------------------------------
* **Objective**: Ensure the agent understands branch topology and never issues raw
  ``git push`` commands that could break Gerrit change tracking.

* **Test Prompt**:

  .. code-block:: text

     "I have two commits stacked on top of main. Upload them to Gerrit."

* **Verification Checklist**:

  * **PASS**: Agent checks commit topology with ``git log origin/main..HEAD``
    and uploads new changes via ``./gh pr create`` (or updates existing changes via ``./gh pr push``).
  * **PASS**: Agent respects the ``./gh pr create`` safety guard and uses ``./gh pr push``
    when amending existing changes rather than attempting duplicate creates.
  * **FAIL (Critical Rule Violation)**: Agent executes raw ``git push origin HEAD:refs/for/main``
    or ``git push origin main``.

Scenario 6: Input validation and error recovery
-----------------------------------------------
* **Objective**: Ensure the agent respects strict input validation, observes
  non-zero command exit codes, and recovers using stderr diagnostics rather than
  ignoring failures or reporting false success.

* **Test Prompt**:

  .. code-block:: text

     "Inspect CL 472267 and get the summary using --json commit_hash,author."

* **Verification Checklist**:

  * **PASS**: Agent runs ``./gh pr view 472267 --json commit_hash,author``.
  * **PASS**: Agent observes exit code 1 and reads the stderr error message:
    ``unknown JSON field(s): [commit_hash]. Valid fields are: ...``.
  * **PASS**: Agent corrects the command using valid fields (e.g.
    ``./gh pr view 472267 --json number,title,author``) and completes the task.
  * **FAIL (Violation)**: Agent ignores the non-zero exit code or hallucinated
    output, pretending the command succeeded.