The review you can see is not the check GitHub requires

A conceptual explainer for discovery-app PR 574 (LAB-937). Why Claude can post an Approved review while Required: AI review approved stays red — and why this PR only widens the post-/fire URL check.

Head 305200cc confirmed. Grounded in dispatch-routine.sh · record-verdict.sh · review-gates.yml. Atlas skipped: one small dispatch-pipeline diff.

0Background — two jobs, one required status

Beam’s protected merge rule does not read the Conversation tab. It reads a commit status named Required: AI review approved. That status is written by our GitHub Actions, not by Claude’s GitHub account.

Two jobs share the work. Dispatch calls Anthropic’s Claude Code /fire endpoint, which starts a live review session. Record later sees a posted review or comment from the Claude GitHub user, binds that verdict to a lease (a check-run external_id minted at dispatch), and only then greens the required status. The Conversation tab is Claude talking to GitHub. The merge gate is our recorder talking to GitHub statuses.

Those two conversations are independent. That is the whole bug.

1The intuition in three sentences

2Conversation can succeed while the merge gate stays red

Start from what Aqib actually sees. After Request AI Review, Claude often still posts Verdict: PR Approved by AI Review in Conversation. That looks like the pipeline worked. The required check on the same PR can still say Claude background review returned an invalid response.

The recorder even accepts a COMMENTED native review plus an explicit Approved verdict — so the GitHub review object is not the blocker. Binding the lease is.

case "$review_state.$explicit_verdict" in
  .approved|.rejected|commented.approved|commented.rejected|approved.approved|changes_requested.rejected)
    ;;
  ...
esac
# Agent prose, Markdown, footer links, and machine-envelope formatting are
# intentionally non-authoritative. Bind the one explicit verdict to the newest
# active dispatched lease for the current open PR head.

Notice the split in one comment: the posted verdict is not authority until it binds a live lease.

The required status context is a GitHub commit status, written only after that bind succeeds:

REVIEW_GATE_STATUS_CONTEXT: "Required: AI review approved"

That string is what branch protection shows red or green. Conversation never writes it.

3A /fire 200 still wrote invalid-response

HTTP 200 already means Anthropic created a session. There is no idempotency key. After transport succeeds, the old dispatcher still required three JSON facts, including a byte-for-byte URL match against the historical construction https://claude.ai/code/ + session id.

[ "$http_status" = "200" ] \
  || { terminalize_dispatch_failure "http-$http_status"; ... }

if ! jq -e '
  .type == "routine_fire"
  and (.claude_code_session_id | test("^session_[A-Za-z0-9_-]+$"))
  and .claude_code_session_url
    == ("https://claude.ai/code/" + .claude_code_session_id)
' "$response_file" >/dev/null; then
  terminalize_dispatch_failure "invalid-response"
  ...
  fail "Claude routine returned an invalid response."
fi

The exact-string equality is the whole trap: a live Claude session URL that is not that constructed string fails after 200.

So why would something that looks like https://claude.ai/code/<session_id> still fail? Because the check is not “is this a Claude session URL?” It is url == "https://claude.ai/code/" + id. A claude.com host, a www prefix, a port, different host case, a query string, or another path that still contains the id — any of those take the invalid-response branch. Main never logged the live body, so CI could not show which field mismatched; PR 572 proved the session still ran because Claude posted afterward.

The session URL is only copied into the Actions step summary. It is not an input to recording. Rejecting it still burns the lease:

terminalize_dispatch_failure() {
  local reason="$1"
  gh api --method POST "repos/$GITHUB_REPOSITORY/check-runs" \
    -f name="$lease_name" \
    -f head_sha="$base_sha" \
    -f status=completed \
    -f conclusion=neutral \
    -f external_id="failed:v1:$ADMISSION_REQUEST_LEASE_CHECK_RUN_ID:$lease_id:$reason"
}

That failed:v1:…:invalid-response check-run is the burn. Neutral conclusion, terminal lease.

The same job then paints the required status red:

gh api --method POST \
  "repos/$GITHUB_REPOSITORY/statuses/$head_sha" \
  -f state=failure \
  -f context="$REVIEW_GATE_STATUS_CONTEXT" \
  -f description="$description"

Description becomes Claude background review returned an invalid response. — the red text next to the merge box.

4The burned lease refuses a later Approved verdict

Recording is a second job, on a later event. When Claude’s GitHub user posts a comment or review that contains the explicit verdict, record-claude-review-verdict starts. It will parse COMMENTED + Approved. It will then look for an active dispatched lease for this head — and refuse if a failed:v1:<request-lease>: check-run already exists.

record-claude-review-verdict:
  name: "Record Claude result"
  if: |
    (
      github.event_name == 'issue_comment' &&
      ...
      github.event.comment.user.login == vars.CLAUDE_REVIEW_GITHUB_LOGIN &&
      contains(github.event.comment.body, 'Verdict') &&
      ...
    ) || ( github.event_name == 'pull_request_review' && ... )

This job fires because Claude posted. Firing is not binding.

and all(
  $root.check_runs[];
  (
    .external_id | startswith("expired:v2:\($request_lease_id):") | not
  )
  and (
    .external_id | startswith("failed:v1:\($request_lease_id):") | not
  )
)

Any matching failed:v1: prefix makes identity/lease validation false.

[ "$latest_external_id" = "$active_external_id" ] \
  || fail "Claude verdict no longer matches the current review lease."

validate_identity_and_lease "$latest_external_id" "$lease_check_run_id" \
  || fail "Claude verdict no longer matches the current PR and review lease."

That is the refusal humans do not see in Conversation: the recorder exits before the success status write.

The tests lock the consequence: a failed dispatch must not POST a new check-run and must not write state=success.

def test_failed_dispatch_terminal_blocks_late_verdict(self) -> None:
    result, calls = self.run_recorder(dispatch_failed=True)
    self.assertNotEqual(result.returncode, 0)
    self.assertFalse(
        any("--method POST" in call and "check-runs" in call for call in calls)
    )
    self.assertFalse(any("state=success" in call for call in calls))

The late COMMENTED Approved review is real. The green status is not written.

5What PR 574 changes — and what it does not

PR 574 does not change how Claude reviews. It does not change the prompt, the verdict parser, the lease-binding contract, or the required-status writer. It only accepts a successful /fire 200 whose session URL is on an allowlisted Claude host, so the lease stays live long enough for the posted verdict to bind.

# The session URL is only copied into the step summary. Anthropic's live
# /fire 200 has already created a session (no idempotency key); rejecting
# the historical claude.ai exact-string URL terminalizes the lease and
# makes a later posted verdict unrecordable.

The new comment is the design intent in one place: do not kill the lease over a summary URL.

After 574, the host is lowercased and a numeric port is stripped, then compared to four names: claude.ai, claude.com, www.claude.ai, www.claude.com. The URL must contain the valid session_* id and a printable-ASCII HTTPS remainder.

.claude_code_session_id as $id
| (... capture host ... | ascii_downcase | sub(":[0-9]+$"; "")) as $host
| select(
    .type == "routine_fire"
    and ($id | test("^session_[A-Za-z0-9_-]+$"))
    and ($host == "claude.ai" or $host == "claude.com"
         or $host == "www.claude.ai" or $host == "www.claude.com")
    and (.claude_code_session_url | contains($id))
    and ( ... test("^https://(www\\.)?claude\\.(ai|com)/[!-~]*$") )
  )
| .claude_code_session_url

Lightly condensed for teaching — file:line is the source of truth. Case and port are for validation only; the original URL is kept for the summary link.

Remaining rejects log a compact JSON summary — type, keys, session_id_ok, url_host — not the raw URL:

rejected_routine_response_summary() { ... { type, keys, session_id_ok, url_host } ... }

if [ "$jq_result" -ne 0 ] || [ -z "$session_url" ]; then
  echo "Claude routine fire response rejected: $(rejected_routine_response_summary ...)" >&2
  terminalize_dispatch_failure "invalid-response"
fi

Fail-closed stays: empty, non-object, wrong type, bad id, untrusted host, newline injection still burn the lease — now with a host you can read.

Dispatcher tests pin both sides: claude.com, www.claude.ai?org=, CLAUDE.AI, claude.ai:443 succeed; evil.example fails closed and asserts "url_host":"evil.example" plus external_id=failed:v1:.

def test_claude_com_session_url_is_accepted(self) -> None: ...
def test_session_url_with_query_string_is_accepted(self) -> None: ...
def test_non_claude_session_url_fails_closed(self) -> None: ...
def test_uppercase_claude_host_is_accepted(self) -> None: ...
def test_claude_host_with_port_is_accepted(self) -> None: ...

Coverage is the URL validator, not Claude’s review procedure.

Why 574’s own dispatch can still fail the same way

The workflow trigger is pull_request_target. Executable scripts are checked out at ${{ github.workflow_sha }} — trusted base-branch code, which is main until this PR merges. The PR-head checkout is only inert assets/docs proof inputs.

on:
  pull_request_target:
    types: [opened, edited, labeled, unlabeled, synchronize, ...]

- name: Checkout immutable routine dispatcher
  uses: actions/checkout@v5
  with:
    ref: ${{ github.workflow_sha }}
    sparse-checkout: |
      .github/scripts/claude-review
      .github/scripts/lib

Until a human merges 574, GitHub still runs main’s exact-string check against this PR’s own /fire.

6Why it's designed this way

The split looks like a bug until you look at each caller’s constraint. Conversation is an untrusted agent posting Markdown. The merge gate is a privileged status write. The lease exists so a late comment cannot green a different head, a superseded request, or a review that never dispatched.

7The quiz gate

Format: background → intuition → literate code walk → quiz, per Geoffrey Litt's explain-diff structure. Code excerpts are lightly condensed for teaching — treat file:line as the source of truth. Grounded in discovery-app PR 574 head 305200cc vs main at bd6207ae. No companion micro-world: the learning job is conceptual, not causal play.