One `rm -f` Silently Disabled My Crash Detection

The daemon crashed, came back up, and recorded nothing. No gap, no warning, no log. The code was correct; the test cleanup had deleted the file the recovery logic keys on. This is from agentwatch, part of agentsec-ecosystem — a suite of open-source, harness-agnostic security tools for AI agents. The pattern generalizes to any logic that keys off a file's existence: migration markers, health files, lock files. Cleanup that doesn't understand what it's deleting can silently turn the feature off. The design On restart after a crash, the daemon synthesizes a recording-gap record — evidence that recording was interrupted, and for roughly how long. This is F1 in the fault taxonomy: a first-class fault the system must detect and record. The signal that separates "first ever start" from "back from a crash" is a file, daemon.pid: def _record_gap_if_needed(self): if not self.pid_path.exists(): return # first start, nothing to fill in records = self.store.records() if not records: return last = records[-1] gap_seconds = (datetime.now(timezone.utc) - last.started_at).total_seconds() if gap_seconds /tmp/daemon.log 2>&1 &) pgrep -f "[a]gentwatch-daemon" | head -1 > /data/agentwatch/daemon.pid The bracket around the a stops pgrep from matching itself — a small detail, but without it the pid file contains the pid of the pgrep process, which exits before the daemon can read it. Now the pid file is there before the daemon does anything, exactly like production, and after a crash-restart the gap record appears with a duration that matches the down time. After the fix: recording-gap anomaly present, duration_ms matching the crash-to-restart interval. FT-14 passes. Why the bug was invisible The failure had no error signal. The function returned normally — no exception, no error code, no log line. The gap anomaly was absent, but absence is not an error; it's just "nothing happened." The daemon didn't know it should have synthesized a gap. It didn't know the pid file had been deleted. It just saw "no pid file → first start → no gap." The logic was correct for the input it received. The input was wrong. This is what makes silent skips the worst failure mode. Three properties combine: No error signal. The function returns normally — no exception, no error code, no log. The daemon's _record_gap_if_needed returned None just as it would on a first start: the same return value, the same code path, the same "nothing to do" outcome. No observable effect. The gap anomaly is absent, but absence reads as "nothing happened," not "something failed." A missing row in the store doesn't look like an error — it looks like the daemon didn't have a gap to report. The only way to tell "should have reported but didn't" from "had nothing to report" is to know the pid file was deleted, and the daemon doesn't know that. No feedback loop. The daemon has no way to know the pid file was deleted. It just sees "no pid file" and does the right thing for that input. The logic is correct for the input it received. The input was wrong. There's no signal back saying "by the way, I didn't find a pid file, so I assumed this was a first start." The only way to catch a silent skip is to assert the expected behavior explicitly. FT-14 does this: after crash-restart, it asserts the recording-gap anomaly exists. Without that assertion, the silent skip would have gone to production, and real crash gaps would have been invisible. The bit that generalizes It's not really about pid files. Any file that acts as a signal can be broken by cleanup that doesn't understand what it's holding: Signal What it means What cleanup does What happens daemon.pid "daemon was running before" deleted as "stale" gap detection silently disabled .migration_complete "migration ran" deleted as "old marker" migration re-runs, data corrupted health_check.json "last health status" overwritten with empty health history lost .env "config was set" deleted as "temp" config reset to defaults In every case, cleanup sees "a file" and deletes it. The reader silently skips — no error, no warning. The system appears healthy. The signal is gone. tip: Not every file in a directory is stale. Some are signals, markers, or evidence. If your recovery logic checks whether a file exists, your cleanup isn't allowed to delete it. The cleanup must distinguish signals from stale state, or it will silently turn features off. Two different assertions "The daemon is running" and "the gap was written" are different assertions. The first is necessary but not sufficient — a daemon that starts but doesn't synthesize the gap has still lost the record of the interruption. The test has to assert recovery, not just startup, or it's testing the easy half and ignoring the half that matters. This shows up in a lot of recovery code. "Did the process restart" is the question most tests ask. "Did the process restore the state it was supposed to restore" is the question that actually matters, and it's the one that requires the recovery path to execute — which means the preconditions for recovery (the pid file, the marker, the lock) have to be present when the test runs. Where I landed If recovery keys off a file, that file has to survive the test, or the test isn't testing recovery. Reproducing the launcher's pid-file behavior in the harness is the difference between exercising the recovery path and skipping it. And the assertion has to be on the recovery behavior, not the startup — "the gap was written," not "the daemon is running." It's fixed in agentwatch v0.1.0, with FT-14 as the test that keeps the pid file — and the recovery path — honest. The question worth asking of any crash-recovery or migration code: what file or state does it key on, and does the test cleanup know the difference between stale state and evidence? If the cleanup deletes the signal, the recovery path never runs, and the test passes anyway. References Release: agentwatch v0.1.0 on GitHub · on PyPI Suite: agentsec-ecosystem — the full set of agent-security tools Field test report (FT-14, issue I-6) Daemon: daemon.py Test step: FT-14.sh