Arrange Act Assert

Jag Reehal's thoughts on things, mostly product development

Shepherd fixes the PR, its CI review judges it

09 Oct 2026

Shepherd takes a pull request from opened to ready to merge.

Its review loop runs in your coding agent, reviews the change, fixes what it can and repairs the build.

Its review in CI reads each push, posts a verdict and sets a shepherd status your branch rule can require.

A Lego shepherd with a wrench fights red bugs on a PR platform beside a board of passing unit tests, lint and security checks, then sends a green PR brick across a bridge to a CI platform, where a second figure marks it APPROVED in red

The verdict comes from a separate reviewer that holds no token. The approval comes from you.

PostHog’s list of AI coding mistakes argues that lint, types and tests matter more once an agent writes the code. Shepherd runs those checks inside the pull request loop.

I built Shepherd in two parts. The loop prepares the change and can push a fix. The review in CI gives the verdict and sets the status that lets the PR merge. Shepherd never approves a pull request on GitHub: a person does.

---
alt: "Shepherd runs review rounds inside a coding agent: swarm reviews, triage fixes or defers threads, and simplify checks the result. After a round with no changes or new findings, Shepherd repairs CI and reads the verdict from its review in CI, which sets the shepherd status. The loop and the review record their work on GitHub, where a person approves."
---
flowchart LR
  subgraph inner["Shepherd: in your coding agent"]
    S["Swarm<br/>reviews the diff"] --> T["Triage<br/>fixes or defers threads"]
    T --> SI["Simplify"]
    SI --> S
  end
  inner --> C["CI repair"]
  C --> ST["Shepherd's review in CI<br/>policy gates + model review<br/>verdict and status"]
  ST --> GH[("GitHub<br/>threads, commits,<br/>statuses, verdicts")]
  inner --> GH
  GH --> H["A person approves"]
Figure 1: Shepherd's loop gets the PR ready, its review in CI gives the verdict, and a person approves.

How Shepherd works

Shepherd is a set of skills that runs inside a coding agent such as Claude Code or OpenCode. You install it with bunx @jagreehal/shepherd install, or with npx skills add jagreehal/shepherd for any agent the skills CLI supports, and run /shepherd 42 on a PR. One iteration works like this:

  1. Swarm runs the repository’s own lint, typecheck and nearby tests first, plus any scanners you have installed, such as gitleaks, semgrep and osv-scanner. A cheap model then reads the whole diff and hands the risky hunks to specialist lenses (correctness, security, simplicity, maintainability, slop, and any your team adds), which run in parallel. For a PR that closes an issue, a spec lens checks the diff against what the issue asked for. A second model checks each finding above a nit against the code before it posts, with the input that triggers it, the expected result and the actual one. It drops a finding it refutes.
  2. Triage works through the open review threads. It fixes the concrete ones, resolves the ones it can show are mistaken, and leaves the decisions that belong to the author. Threads another person has joined stay with that person.
  3. Simplify checks the result for code the change no longer needs. A fix gets another review round.
  4. After a round with no changes or new findings, CI repair brings the branch up to date and fixes what the PR broke. Shepherd then reads the verdict from its review in CI. The loop allows up to four review rounds before handing unresolved work back to you.

After the first round, swarm reads only what moved since the last head it reviewed. Shepherd compares the PR’s own change at both heads, so a base merge or a force-push doesn’t make the reviewers read the whole PR again. Reviewers read a diff with each line’s old and new numbers written out, so a finding cites its line directly.

On a PR stacked on another open PR, swarm reviews only this PR’s change over the parent’s head. On a PR opened against the wrong base, the loop reviews nothing and names the gh pr edit command that retargets it.

Wrap it in /loop 5m /shepherd 42 and you can leave it running. The loop never replies to people and never approves. Every comment it posts opens with Shepherd’s icon and Automated comment by **Shepherd swarm**, because it posts through your account and readers deserve to know a machine wrote it. A PR that /shepherd opens says in its first line that an agent wrote it and you haven’t reviewed it yet; only you change that line.

Set the checks and the fix mode

Your team can set the checks, reviewers and fix permissions in .shepherd/config.yml. A fresh shepherd init picks the model from your key, writes that file and one workflow, .github/workflows/shepherd.yml, sets the key as an Actions secret, and starts in review-only mode. Choose init --preset local-fix to leave fixes in your working tree, or init --preset push to let Shepherd commit and push them. The review in CI reads its own policy from the review: section of the same file.

You can extend an organisation’s policy pack and override settings for one repository. A pack from another repository needs a full commit SHA. Shepherd reads team policy from the default branch, so a PR author cannot use their change to remove a reviewer or relax the fix rules, and the review sends any edit to .shepherd/config.yml to a person.

Before merging a policy edit, run:

bunx @jagreehal/shepherd config validate
bunx @jagreehal/shepherd preview --working-tree --recipe pr

The preview lists the checks and models, the steps that can write code or touch GitHub, and the decisions left to a person. Then add CODEOWNERS entries for /.github/ and /.shepherd/ with Require review from Code Owners turned on, and run shepherd doctor. It reports a missing workflow, a model key that isn’t a secret, and a CODEOWNERS gap.

How the review in CI decides

The review runs in your own GitHub Actions on each push, on a PR marked ready, on a retargeted base, and on a /review comment from someone who can push. It skips drafts. shepherd gate <pr> --dry-run runs the same review on your machine and posts nothing.

It checks deterministic gates first: sensitive paths, size limits, credentials on added lines, test deletions and skip markers, and Shepherd’s own policy. A model then reads the diff, the checkout and your house rules, and reads the issues the PR closes to check the diff does what they ask. The model can refuse a change the gates passed, and only the gates can make a change eligible.

An approval has to name the fact that makes the change safe. Shepherd escalates a high-risk approval that rests on a fact nobody checked. Configure a second backend and a medium- or high-risk approval also needs the second model to agree; a disagreement goes to a person.

Each review ends with one of three verdicts:

Shepherd posts one review per push, always as a COMMENT, with the verdict in its heading, the merge risk, and an inline comment per finding that opens with ❌ Blocks approval or 🙋 Needs a person and a short title. Merge risk says what merging the head as it stands could break: high for data loss, a security hole, a wrong charge or broken callers. A refusal or an escalation reads at least medium.

The model review runs through the AI SDK, so you choose the provider with one setting. shepherd init --model writes it to the policy as models.loop, and a SHEPHERD_MODEL repository variable overrides it when you want to try a model without a PR: anthropic/claude-opus-5, openai/gpt-5.5, opencode-go/deepseek-v4.1-flash, openrouter/moonshotai/kimi-k3 or a Bedrock model.

The model reads the checkout through three tools, read_file, grep and glob, confined to the repository, and ends the review by calling submit_verdict, whose input schema is the verdict. Every provider I tested supports tool calling, so one verdict format works across them. A model that answers in prose gets one follow-up call that must be submit_verdict.

The model holds no token

The model runs no shell, never builds or runs the PR’s code, and holds no token. Shepherd’s own process makes each GitHub write after the model returns, and it splits the work across two jobs:

Job Permissions What it does
review contents: read, plus writes to PR comments and statuses Holds the model keys. Posts the review and sets the status to pending
settle contents: write Holds no keys and no checkout, and installs nothing. Reads each thread again, resolves the threads the review settled, then sets the final status

GitHub requires contents: write to resolve a review thread, so only the job without model keys gets it. The workflow uses the job’s own github.token: no GitHub App, no personal access token. Keep Allow GitHub Actions to create and approve pull requests turned off, so that token can’t approve.

The review job can still set the status, so you have to trust what it runs. Shepherd publishes as one bundled file with no runtime dependencies, and bunx fetches it and nothing else, at the version the workflow pins.

Teams that prefer Claude Code or Codex can run those as the reviewer instead, with backends: under review:. The review runs them in a pinned container with a read-only copy of the checkout and only their own credential, on a private network whose one route out leads to their own model provider. The egress proxy opens a tunnel only when the TLS handshake inside it names that provider’s host, so a reviewer can’t reach another site that shares the provider’s CDN address.

What a review costs

The review records each decision at the point it makes it, as I argued in You Can’t Support What You Didn’t Record, so every review leaves a gate-evidence-<pr> artifact on the run. A Kimi K3 review of a one-line change made five tool calls, read 22,656 of its 33,580 input tokens from cache, and took 57 seconds.

Limits from autotel’s guard bound each review: $2 of spend, three million tokens, a spin-loop rule, 80 tool calls, and a 15-minute timeout that cuts off a model call in flight. A review that hits a limit sets the status to error, and someone comments /review to run it again. Shepherd prices each call from the pricing: entry under review:, and from nothing else. shepherd init writes a dated list price for the model you choose, from models.dev, and you can edit it to your contract’s rate. A model with no price runs under the token ceiling alone, and the log says so. Set an OTLP endpoint and each review becomes a trace, with one span per model call and tool call, each cost tagged with its price date.

Read what the tests checked

Repositories that use executable-stories give Shepherd more than a test exit code. Shepherd reads each failed scenario’s steps, expected and actual values, and covers paths. It reruns the affected scenarios after a fix. For a skipped scenario, triage restores the test and fixes the code if that test fails.

Shepherd compares runs of the same scope. A focused run leaves out tests it did not target, so you can’t read a test missing from it as one the PR removed. For an incident or a linked issue, Shepherd runs executable-stories goal with the issue’s ticket against full current and base runs. The command checks that the required scenarios pass and flags scenarios the change dropped, skipped or shortened.

The review in CI can read the report CI uploads:

# .shepherd/config.yml
review:
  stories:
    workflow: ci.yml
    artifact: executable-stories
    file: index.review.json

The report needs executable-stories-formatters 1.27 or later to record the commit where the tests ran. Shepherd checks that commit against the PR head, or a merge whose parents match the reviewed base and head. It labels filtered runs, and reports stale or missing evidence. A comparison label such as headRef cannot establish where tests ran.

The PR author wrote those tests. Shepherd keeps the report inside its untrusted input and uses it as review context. A passing scenario, a declared covers path or a trace grants no exemption from a gate and supplies no independent approval. The report can only hold an approval back: a scenario that checks less than it did on the base branch goes to a person.

See where each PR stands

Every review sets the shepherd commit status on the head it reviewed. While it runs, the status reads pending and Shepherd posts a “shepherd is reviewing” comment with a link to the run, which it deletes when the run ends. Then:

An approval alone doesn’t turn the status green. Any open thread, Shepherd’s or a person’s, holds it at failure until someone resolves it. A person settles an escalation by approving the head on GitHub as someone who can push; the settle job reads the PR again and turns the status success. Each person’s latest review counts, so dismissing that approval sets failure again. An approval can’t settle a refusal: the author changes the code.

The status lives on GitHub, so it outlasts the session that set it. You can come back to a PR hours later and read what it needs from you without opening a transcript, or run shepherd status <pr> to see the review, the status, CI and open threads together. Make shepherd a required status check and a PR can’t merge while its review is running, refused, waiting on a person, or behind a push Shepherd hasn’t reviewed.

Shepherd reviews each head from scratch and keeps threads steady across pushes. Once a push changes the code under or near one of its threads, it resolves that thread and posts anything it still finds as a new thread on the new code. On code nobody touched, it posts nothing new beside an open thread of the same grade, so the thread keeps its history. It never resolves a thread a person wrote in. After a push that leaves the PR’s own diff byte-identical, such as a clean merge of the base, Shepherd keeps the standing verdict and calls no model.

You can trace a Shepherd iteration through autotel. With an OTLP endpoint configured, Shepherd exports a span for the iteration and one for each recorded stage, including the wait for the verdict. Repository, PR and commit fields let you correlate that work with the CI review. Separate processes can link their spans through traceparent values. The CLI records stage timing and outcomes; your coding-agent harness must supply model usage.

The gardener can read those traces from OTLP JSON exports, Grafana Tempo or Datadog. It flags failed stages, repeated retries, long approval waits and high review cost for investigation. It matches each sample to a known PR commit and reports gaps in collection or pricing. You can follow a proposal back to its trace ID and GitHub evidence; the trace alone cannot count a fix as accepted.

Run the loop before you open a PR

You can run the same review on your machine before GitHub sees the change. /shepherd --local captures your unpushed commits, staged and unstaged edits and untracked files, then runs the same lenses, verification and fix rules as the PR loop. Fixes land in your working tree. Shepherd never commits, pushes, comments or sets a status in this mode, so you review the fixes and commit them yourself.

A clean report needs shepherd local finish to say complete. That check confirms every lens finished, every changed line range reached a reviewer, and the files didn’t change mid-review. It proves each line went in front of a reviewer, which falls short of proving the reviewer understood it. A complete review writes a receipt keyed to the exact content, and shepherd local status tells you whether the files on disk match it. Put that in a pre-push hook if you want one.

Local mode ends at “local checks passed”. Merge readiness stays with the PR loop and the CI review. To see it work, bunx @jagreehal/shepherd local demo builds a repository with seven planted bugs, one for each way a change reaches the review, including a deleted check.

Rules your team already wrote

You don’t need to configure anything for the review to read your conventions. It reads AGENTS.md, CLAUDE.md, REVIEW.md, CONTRIBUTING.md and CODING_STANDARDS.md from the default branch, at the root and in each directory above a changed file, and the nearer file wins where they disagree. It also reads the Markdown files they link to.

A house rule never changes the verdict: a missing doc comment never refuses a merge. A breach of a REVIEW.md rule posts as a line comment quoting the rule, on any verdict, so triage can fix it, and its open thread holds the status until someone does. A breach of the other files gets a note. A PR that relaxes a house rule is a different matter, because later reviews trust that text: the review refuses it, and it reviews the PR against the rules as they stood.

One checklist for review and approval

A service team cares about observability and API compatibility. A frontend team cares about accessibility and its design system. When a rule must block a merge, write it as a lens on top of the built-in ones:

bunx @jagreehal/shepherd lens new observability --from observability --applies 'src/**'

That command copies Shepherd’s bundled observability pack into .shepherd/lenses/observability/SKILL.md and registers it in .shepherd/config.yml. Drop --from and you get a blank skill to fill in; other templates cover React, Next.js, Node APIs, Go, Rust, Java, .NET and Python. The ## Review section lists rules with IDs, so a finding names the rule it applies: [observability/error-context] means a caught error lost its cause or the IDs you need to find the request. The pack’s rules check for the telemetry Instrument Before You Know the Question says you need before production breaks. They cover outbound calls with no span or log, unstructured logs, secrets in telemetry, silent data drops and unbounded metric labels. The ## Fix section tells triage to use the logger and tracer the repository has, and to hand you any change that adds a telemetry dependency.

Shepherd reads the lens file from the default branch, so your team decides who reviews a PR and the PR itself has no say. You can try a lens on a real PR with /swarm --preview, which posts nothing.

The review in CI reads every lens in .shepherd/config.yml as a rule pack. The lens’s gate setting says what a broken rule does: refuse, escalate (the default) or note:

# .shepherd/config.yml
lenses:
  observability:
    skill: .shepherd/lenses/observability
    applies_to: ['src/**']
    gate: refuse

Your team edits one checklist for both the fixer and the CI reviewer. A lens can only add checks.

Catch an agent silencing a check

An agent chasing a green build can silence a rule and leave the code as it was. Swarm lists each suppression comment a PR adds, such as oxlint-disable, @ts-ignore or nosec, as a finding and reads the code it covers. The CI review flags the same comments for its model. The reviewer refuses a suppression that hides a finding the change introduced and approves a narrow one that gives its reason.

A PR that loosens lint, test or scanner config gets the same treatment. A linter can’t see its own config change, so the reviewer reads that diff and refuses a disabled rule, a widened ignore or a lowered coverage threshold. It approves tightening.

The tests gate holds a PR that deletes a test file, renames it out of the runner’s collection pattern or adds a recognised skip or focus marker. Conditional skips and rewritten assertions reach the reviewer. With that split, a model can’t explain away a disabled test the gate can identify.

After a failure reaches production

Run /incident <run-url|issue|error.txt> with a red default-branch build or a production failure. Shepherd tries to reproduce the symptom in a test, bisects with that test to find the PR that introduced it, and records whether Shepherd approved that PR. It opens a fix PR after verifying the fix, and the CI review refuses an incident fix with no reproducing test. If it cannot reproduce the failure, it stops without claiming a fix. On a public repository it stops on a security failure before it opens any issue or writes any test, because a reproducing test for a security failure is a working exploit.

The gardener reads those incidents alongside review history. For a regression Shepherd approved, it can propose a check that flags the culprit change and allows the fix, and it adds the incident to the held-out cases shepherd eval scores the reviewer against. You review the proposed rule before it affects later PRs.

Work alongside Claude Code Review

Claude Code Review posts findings on a PR and leaves the approval to you. Shepherd reads REVIEW.md the way Code Review does, with its @ includes, its skills and its size limits, so one file steers both. Their commands don’t collide: Code Review answers @claude review, and Shepherd answers /review. Name its check run in reviewer_checks under review: and the CI review waits up to five minutes while Code Review is still running, so those findings reach its model.

Triage fixes 🔴 Important and 🟡 Nit findings like any bot finding and leaves 🟣 Pre-existing ones to you as follow-ups. Shepherd posts no @claude review comment, because each one starts a billed review; its summary tells you when a review failed or ran out of budget. A PR that only adds paths to skip, caps findings or lowers a severity in REVIEW.md goes to a person, and one that relaxes it any other way is refused.

Spend money where judgement matters

Your team configuration sets the models. Name each model once as a tier, list tiers from cheapest to strongest in a ladder, and pin role groups or single lenses to a tier, from any provider your coding agent can reach:

tiers:
  fast: opencode-go/deepseek-v4.1-flash
  smart: opencode-go/kimi-k3
ladder: [fast, smart]
models:
  reviewers: fast    # the router and every lens
  fixers: smart      # triage, simplify, ci-repair, incident
  security: smart    # one lens beats its group
  loop: smart        # the review model in CI

Switching provider then changes only the tiers. In one swarm review through OpenCode, the router on DeepSeek V4 Flash cost $0.013, two specialist lenses on GLM-5.3 cost about $0.16 each, and the orchestrating session on Kimi K3 cost $0.71. The orchestrator ran up most of that bill, so I would tune its model first. Measure before you move a role down a rung: shepherd eval runs each configuration over the same frozen bugs and names the cheapest one whose held-out recall and precision sit within noise of the best.

aiagentsengineering