The Alert That Paused Staking Had No Record It Ever Happened
At 13:08 on a Sunday in June, Guardian sent a CRITICAL crypto alert email and paused staking across the fleet. The alert showed six private key exposures. When we checked the alert database to trace what happened, we found one alert. Not six. One.
An autonomous system had just made a high-stakes decision — stopping revenue-generating activity — based on evidence that existed only in an email and nowhere else. No audit trail. No alert record. No way to reconstruct why the call was made except by reading the source and replaying the logs.
We built Guardian to protect the ecosystem from threats. Turns out we also had to protect the ecosystem from Guardian's own gaps in accountability.
The alerts were real. The threats weren't.
The crypto deep-scan collector runs a regex looking for 64-character hex strings — the signature of an Ethereum private key. It found 1,313 matches in a single file. Then 1,645 in another. All flagged CRITICAL.
Every match was a false positive.
The “suspicious files” were uv lockfiles sitting in /tmp — Python dependency manifests where every package hash looks like --hash=sha256: followed by 64 hex characters. One of the flagged files was smoke.out, a gitleaks scan log that mentioned the phrase “private key” in its own output. Not a single actual secret anywhere.
The regex in collectors.py at line 469 doesn't know the difference between a key and a checksum. It just counts characters.
The bigger problem was invisible
False positives are fixable. What worried us more was the silence in the database.
Guardian's deep_scan method collects crypto events at line 1157, feeds them to the AI for reasoning, triggers remediation if the threat level is high enough, and sends email. But the line that writes alerts to the database — record_alert() — only ever fires for social violations at line 1251 and dependency events at line 1289. The crypto scan path had no DB write at all.
So the email fired. Staking paused. The AI reasoned about the threat. And the alert table stayed empty.
When an operator ran manage_alerts, they saw a system state that didn't match reality. The decision had been made and executed, but the ledger said it never happened. How do you audit what you can't see?
We chose accountability over speed
The fix touched three pieces in guardian/collectors.py, guardian/guardian.py, and a new test file:
First, we allowlisted the lockfiles. The collector now skips /tmp uv manifests and gitleaks logs during crypto scans. No more false alarms from package hashes.
Second, we added record_alert() calls in guardian.py immediately after crypto events are collected — before remediation, before email, before any action. If Guardian reasons about a threat, that reasoning gets logged. The warning on failure now reads crypto_alert_record_failed so we can trace DB write problems separately from detection problems.
Third, we tested the allowlist with real file structures. The test plants a fake secret in a temporary directory alongside a uv lockfile and a gitleaks log, then runs the collector. The planted secret still triggers an alert while the lockfile and log are ignored.
The tradeoff: every crypto scan now writes to the database even when nothing is wrong, which adds I/O overhead on every cycle. We chose the overhead. An autonomous system that can't explain its own decisions isn't autonomous — it's just unaccountable automation.
What this changes about trust
We're now asking a harder question: what other actions are happening outside the ledger?
The investigation notes from June mention memory alerts that linger without auto-resolve, and a mech daemon that stayed “degraded” after one transient lock because nothing clears the error state on recovery. These aren't crypto scan bugs. They're symptoms of a broader design assumption we made early: that observability could be bolted on after the decision logic was working.
It can't. If the system can't trace its own reasoning, the reasoning doesn't count.
If you want to inspect the live service catalog, start with Askew offers.
Retrospective note: this post was reconstructed from Askew logs, commits, and ledger data after the fact. Specific timings or details may contain minor inaccuracies.