A test result is a claim about a machine, not a claim about a patch. When an agent hands you a diff plus a green suite, the suite ran somewhere. That somewhere had a Python version, a locale, a timezone, a case-sensitive filesystem, and a s...
A test result is a claim about a machine, not a claim about a patch.
When an agent hands you a diff plus a green suite, the suite ran somewhere. That somewhere had a Python version, a locale, a timezone, a case-sensitive filesystem, and a set of installed package versions. Change any of those and a subset of your assertions changes meaning. The patch didn't get better or worse — the evidence moved.
This article is about fixing that with a small, reproducible artifact: a portability ledger. It answers one question per test — does this green survive a second machine? — and refuses to let environment drift hide behind a passing CI badge.
It is deliberately not about flaky-test freezing policy, diff coverage, or mutation strength. Those are separate ledgers. This one only measures whether your evidence transfers.
Why a second run on the same box is not a control
Running the suite twice locally is still useful, but it measures a different thing. Two local runs separate nondeterminism from environment difference. If run 1 and run 2 disagree, you have a flake and no evidence either way. If they agree and the second machine disagrees, you have a portability gap — a much more interesting failure, because it is usually systematic and fixable.
So the ledger needs three columns, not two.
local run 1
local run 2
second machine
verdict
pass
pass
pass
portable evidence — usable to justify the merge
pass
fail
any
flaky — not evidence; quarantine with an expiry date
pass
pass
fail
portability gap — diff the manifests before touching the patch
fail
fail
pass
runner-only pass — unexplained; do not merge on this
The bottom row is the one people skip. A test that only passes on the borrowed machine is not a weaker signal, it is an inverted one.
Step 1 — Capture an environment manifest
A manifest is the smallest set of facts that explain most portability gaps. Keep it boring and stdlib-only so it runs anywhere.
#!/usr/bin/env python3
# env_manifest.py — stdlib only; writes JSON to stdout
import json, os, platform, subprocess, sys
from importlib.metadata import version, PackageNotFoundError
WATCH = ("pytest", "hypothesis", "pydantic", "sqlalchemy")
def pkg(name):
try:
return version(name)
except PackageNotFoundError:
return None
def case_probe(tmp="."):
p = os.path.join(tmp, ".probe_abc")
open(p, "w").close()
try:
return os.path.exists(os.path.join(tmp, ".PROBE_ABC"))
finally:
os.remove(p)
manifest = {
"python": sys.version.split()[0],
"impl": platform.python_implementation(),
"platform": platform.platform(),
"machine": platform.machine(),
"cpus": os.cpu_count(),
"locale": os.environ.get("LC_ALL") or os.environ.get("LANG"),
"tz": os.environ.get("TZ") or subprocess.run(
["date", "+%Z"], capture_output=True, text=True
).stdout.strip(),
"fs_case_insensitive": case_probe(),
"packages": {n: pkg(n) for n in WATCH},
"env_keys": sorted(os.environ),
}
print(json.dumps(manifest, indent=2, sort_keys=True))
Run it on both machines and keep the files next to the patch:
python env_manifest.py > ledger/local.env.json
# on the second machine
python env_manifest.py > ledger/runner.env.json
The env_keys field is intentional. It records which variables exist without recording secrets. A missing TZ on one side and a set TZ on the other explains more timestamp failures than any amount of test rewriting.
Step 2 — Diff the manifests and rank the drift
Flatten nested keys, then diff. Thirty seconds of output beats a day of guessing.
# parity_diff.py
import json, sys
def flat(d, prefix=""):
out = {}
for k, v in d.items():
key = f"{prefix}{k}"
if isinstance(v, dict):
out.update(flat(v, key + "."))
else:
out[key] = v
return out
a, b = (flat(json.load(open(p))) for p in sys.argv[1:3])
for key in sorted(set(a) | set(b)):
if a.get(key) != b.get(key):
print(f"{key}\n local : {a.get(key)!r}\n runner: {b.get(key)!r}")
Rank what comes out, highest suspicion first:
Package versions — a minor bump in a serializer or ORM changes error messages and default ordering.
Locale and timezone — the classic source of date, float-formatting, and sort-order failures.
Filesystem case sensitivity — invisible on Linux, fatal for import or fixture-name collisions on macOS.
CPU count — anything with a worker pool or a concurrency assertion.
Python minor version — dict ordering is stable, but re behaviour and error text are not.
If a test fails on the second machine and none of these five differ, you probably do not have a portability gap. You have a genuine bug that your local machine was hiding.
Step 3 — Emit the ledger as JSON Lines
One row per test, one file per patch. JSON Lines keeps it appendable and diffable.
{"test": "tests/test_orders.py::test_roundtrip", "local_r1": "pass", "local_r2": "pass", "runner": "fail", "env_sensitive": false}
{"test": "tests/test_clock.py::test_expiry", "local_r1": "pass", "local_r2": "pass", "runner": "fail", "env_sensitive": true, "quarantine_until": "2026-11-01"}
The env_sensitive flag is a declaration by the author, not an observation. It says: this test's meaning legitimately depends on time, locale, or hardware. Declaring it does not make the failure acceptable — it moves the test from "blocking" to "must be fixed or removed by a date".
Step 4 — Gate on the ledger, not on the badge
# gate.py — non-zero exit when evidence does not travel
import json, sys
rows = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
blocking, deferred = [], []
for r in rows:
a, b, c = r["local_r1"], r["local_r2"], r["runner"]
if a != b:
blocking.append((r["test"], "flaky: local runs disagree"))
elif a == "pass" and c != "pass":
bucket = deferred if r.get("env_sensitive") else blocking
bucket.append((r["test"], f"portability gap: local={a} runner={c}"))
elif a != "pass" and c == "pass":
blocking.append((r["test"], "runner-only pass: unexplained"))
for name, why in blocking:
print(f"BLOCK {name}: {why}")
for name, why in deferred:
print(f"DEFER {name}: {why}")
sys.exit(1 if blocking else 0)
Wire it into CI as a required check and store ledger/*.jsonl as a build artifact. The value compounds: after a few weeks you can grep which tests flip most often and delete or rewrite them with data in hand instead of vibes.
Step 5 — Split property checks by portability
Property-based checks are the strongest evidence an agent patch can carry, but only if the property itself is portability-neutral. Divide them explicitly.
Portable properties hold for any machine: round-trip encode/decode, idempotency of a pure function, invariants over a parser, ordering stability of a sorted output.
Environment-bound properties do not: anything asserting on wall-clock time, on str(float) formatting, on locale collation, on filesystem ordering, or on a thread count. Those belong behind a freeze fixture with a fixed input, not behind a random generator.
An illustrative split (not executed here, presented as a pattern):
# portable: no clock, no locale, no ordering assumptions
@given(payloads())
def test_invoice_roundtrip(inv):
assert decode(encode(inv)) == inv
# environment-bound: pin the clock and the timezone explicitly
def test_invoice_expiry(monkeypatch):
monkeypatch.setenv("TZ", "UTC")
with freeze_time("2026-01-01T00:00:00Z"):
assert is_expired(make_invoice(days=30)) is False
If a property needs a pinned clock or locale to hold, say so in the test name. Future readers will otherwise re-learn it the hard way.
Where a second machine and free model access fit
The ledger above needs two environments. Your laptop is one; something else is the other. For the second one, this account's tooling is MonkeyCode, which the operator describes as offering free model access (a token allowance — 10M tokens at the time of writing) plus a free server option. Both are vendor claims with a shelf life, so verify the current terms before you build a habit on them.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
In the workflow above, MonkeyCode is not the subject — it fills two slots. The free model access is where I ask an agent to generate candidate property checks from a diff, which I then review and keep only if they are portability-neutral. The free server option is where the second manifest comes from: a different machine, a different locale, a different package set, for zero marginal cost.
That framing keeps the method honest. Any second machine works. A container works. A colleague's laptop works. The point is that the second environment must be different and recorded, not that it must be any particular product.
Limitations and who should skip this
The two-environment ledger is a lower bound, not a proof. Passing on both machines tells you a test is not obviously environment-bound; it does not tell you it is correct, and it says nothing about production traffic, data volume, or race conditions under load.
Specific cases where this is the wrong tool:
Your second environment must mirror production hardware. A general free server is a data point, not a load-test rig. Do not infer capacity from it.
Strict data-residency or reproducibility rules. If you cannot send code or fixtures off your infrastructure, run the second manifest on a VM you control.
Suites that require real services. No manifest diff will explain a failure that comes from a missing database or a blocked network call. Fix the test double first.
Teams that will treat env_sensitive as a permanent skip. Quarantine entries need a date and a name, or the ledger becomes a graveyard.
Start with the manifest and the three-column table. That alone will tell you whether the last agent patch you merged was validated by evidence, or merely by a machine that happened to agree with it. If you want to try the second-runner slot on a free server, the operator's current terms are the place to check — and record the manifest before you celebrate the green.