Suppose an agent opens a pull request that adds a summarization endpoint. CI is green. The diff also edits the shared config loader and one auth middleware file. That is the normal case, not the dramatic one. Agent-written PRs rarely fail r...
Suppose an agent opens a pull request that adds a summarization endpoint. CI is green. The diff also edits the shared config loader and one auth middleware file.
That is the normal case, not the dramatic one. Agent-written PRs rarely fail review on syntax. They fail on scope.
This article gives a three-bucket triage and a reproducible audit script. The goal is a review verdict a second engineer can re-run.
Green CI proves compilation, not intent
A passing pipeline answers one question. Does the code build and satisfy existing tests?
It does not answer whether the change belongs in this PR. It does not answer whether a shared default drifted. Treat the badge as a floor, never as evidence.
An agent optimizes for the stated task. Anything adjacent to the task looks like fair game to it. Config loaders, CI workflows, and middleware sit adjacent to almost every feature.
Step 1 — Sort the diff into three buckets
Before reading logic, classify every hunk. The bucket decides how much attention the hunk earns.
Bucket
Typical contents
Reviewer action
Trust
Renames, formatting, moved imports, generated lockfile churn
Skim, confirm tests still pass
Revert
Shared defaults, auth/middleware, CI workflows, migrations, dependency bumps, global exception handlers
Remove from the PR, open a separate one
Test
New endpoint logic, input validation, error mapping, outbound calls
Require a contract test and a failure-injection test
The asymmetry matters. A wrong trust hunk costs a revert. A wrong revert hunk costs an incident.
Step 2 — Measure blast radius with a script
The first artifact is a scope audit. It reads the diff, not the description.
#!/usr/bin/env python3
"""diff_scope.py - flag PR changes that fall outside the stated scope."""
import fnmatch
import subprocess
import sys
SENSITIVE = [
"config/*", "**/settings.py", "**/defaults.py",
"**/auth/*", "**/middleware/*",
".github/workflows/*",
"migrations/*", "**/migrations/*",
"requirements*.txt", "poetry.lock", "package-lock.json",
]
def changed_files(base):
out = subprocess.run(
["git", "diff", "--name-only", f"{base}...HEAD"],
capture_output=True, text=True, check=True,
).stdout
return [line for line in out.splitlines() if line.strip()]
def classify(files):
flagged = []
for path in files:
for pattern in SENSITIVE:
if fnmatch.fnmatch(path, pattern):
flagged.append((path, pattern))
break
return flagged
if __name__ == "__main__":
base = sys.argv[1] if len(sys.argv) > 1 else "origin/main"
files = changed_files(base)
print(f"files changed: {len(files)}")
for path, pattern in classify(files):
print(f"SENSITIVE {path} (matched {pattern})")
Run it against the PR branch:
python diff_scope.py origin/main
One honest caveat: fnmatch treats * as greedy across slashes. The list over-matches on purpose. A review aid should produce false positives, not silent misses.
Step 3 — Grep for the three hunk shapes that hide bugs
Path filters miss logic edits inside in-scope files. Three shapes deserve a targeted grep.
git diff -U0 origin/main...HEAD -- '*.py' \
| grep -nE '^\+.*(except Exception|except:|= None|os\.environ|default=)'
Read every hit. Swallowed exceptions turn a 500 into a 200 with an empty body. Changed defaults move behavior for every caller, not just the new one. A new os.environ read adds a deployment requirement the PR description probably omits.
Step 4 — Revert scope creep before reading logic
Revert first, review second. Reading logic inside a contaminated diff wastes attention.
Use file-level restore for files the PR never should have touched:
git checkout origin/main -- app/config/defaults.py .github/workflows/ci.yml
git commit -m "revert: out-of-scope changes, split into follow-up PRs"
Then re-run CI on the reduced diff. If the feature still passes, the reverted files were genuinely unrelated. If it fails, the agent had a hidden dependency you now know about.
That failure is the useful outcome. It converts an invisible coupling into a reviewable decision.
Step 5 — Test the contract, then the failure path
The test bucket needs two tests minimum. One for the happy contract, one for the failure the agent usually forgets.
# tests/test_summarize_contract.py - sketch; adapt imports to your app.
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
def test_empty_text_is_rejected():
r = client.post("/summarize", json={"text": ""})
assert r.status_code == 422
def test_upstream_timeout_is_not_a_200(monkeypatch):
def boom(_text):
raise TimeoutError("model timeout")
monkeypatch.setattr("app.summarize.call_model", boom)
r = client.post("/summarize", json={"text": "hello"})
assert r.status_code >= 500
assert "timeout" not in r.text.lower() # no internal detail leaked
The second assertion catches a specific failure. An agent that adds a global error handler often leaks exception text to the client. That is a quiet information disclosure, and it ships green.
Run only the new tests first:
pytest tests/test_summarize_contract.py -q
If they pass against the unreverted diff, re-run after the revert. Order matters.
Step 6 — Record the verdict in the PR
A verdict that lives in a chat thread is not reviewable. Write it where the next reader looks.
## Review verdict
- Files outside stated scope: app/config/defaults.py, .github/workflows/ci.yml
- Reverted: both, moving to follow-up PRs
- Kept: app/summarize.py, tests/test_summarize_contract.py
- Tests added: empty-input 422, upstream-timeout 5xx, no detail leak
- Not verified: live model latency, per-request cost, load behavior
The final line is the most important. Review states what was not checked. Silence reads as coverage.
Where free model access fits
A first pass over a large diff is mechanical work. Ask a model to list suspicious hunks, then verify each one by hand.
MonkeyCode offers free model access and a free server option, per the operator. That combination suits this narrow job: a stateless prompt that returns a candidate list. The reviewer keeps every decision, and the script remains the source of truth.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Keep the prompt small. Paste the diff, the stated task, and the sensitive-path list. Ask for hunks that match the list, with file and line. Ignore any answer you cannot confirm in git diff.
Limitations and who should skip this
This workflow is file-path driven. It cannot see a subtle logic change inside an in-scope file, and it never replaces running the suite. A single-file agent PR will pass the scope audit and can still be wrong.
The audit also assumes a stable origin/main. Rebase-heavy branches make ...HEAD comparisons noisy.
Skip this approach in three cases. Teams that require pinned, audited model versions should not paste diffs into any hosted endpoint. Teams with strict data-residency rules should keep the diff local. Teams under a hard deadline should revert aggressively rather than attempt a full model-assisted pass.
Free-tier availability and quotas can change without notice. Confirm current limits before depending on them in a pipeline.
The takeaway
Trust the mechanical, revert the adjacent, test the failure. That ordering survives model upgrades and tool churn.
The script is thirty lines. The verdict template is five. Both are cheaper than one incident caused by a config default nobody read.
Readers who want to try the model-assisted first pass can start with the free model access and free server option, then verify every flag against the raw diff.