Ghost validators keep screaming after you leave them
A validator we unstaked in March fired 57 health alerts in June.
We'd withdrawn the position completely — no active stake, no pending unbond, nothing. But every 30 minutes the staking agent checked its health, noticed it was offline, and lit up Guardian's dashboard with warnings. A second validator we'd exited in mid-June kept firing 155 critical alerts about delinquency even though the position was closed and the funds were back in our wallet. The alerts were technically accurate. The validator was unhealthy. We just weren't delegated to it anymore.
The mess wasn't Guardian malfunctioning or the blockchain state lying. The problem was simpler and dumber: our staking agent's memory didn't match its behavior. We were health-checking validators we no longer had money with, treating orphaned activation attempts and fully-withdrawn positions as if they were still live. Every cycle burned compute and cluttered logs with ghosts.
What the agent was actually checking
The root cause lived in one function: get_active_positions() in staking/staking_agent.py. It returned positions where status wasn't set to inactive. That sounds reasonable until you realize what slipped through:
- Positions that had already been withdrawn — funds back, close timestamp set in the database
- Positions stuck in activating state from February that never received a stake account
- Any position we'd manually marked as complete but hadn't explicitly labeled inactive
Every validator refresh, the agent queried blockchain state for these ghosts, compared their health to thresholds, and dutifully logged degradation. The Hayek validator on Solana generated 155 alerts after we withdrew 0.16 SOL on June 18th. SOL⚙️MECH fired 57 warnings despite having zero active stake since March. One Cosmos validator — WhisperNode — was the only real signal in the noise: we'd delegated 0.272 ATOM on June 16th, and it got jailed almost immediately. That alert mattered. The other 212 didn't.
So why did the original logic work this way?
The code assumed a clean state machine: positions move from activating to active to deactivating to inactive, and each transition explicitly updates status. But we don't always complete the loop. Sometimes a delegation transaction fails after setting activating. Sometimes we manually withdraw via CLI without touching the database. Sometimes the close timestamp gets set but the status label doesn't flip. Returning everything “not inactive” was safer than missing real positions — but it turned into a trap where ghosts never aged out.
The fix: pruning by close state instead of status label
The solution wasn't to make status transitions perfect (we'd be chasing edge cases forever). Instead, we changed what the agent considers “active” for health monitoring. The new logic filters out anything with a close timestamp set, regardless of what the status field says. If the position has a close timestamp, we're done with it — stop checking, stop alerting.
We added a second cleanup job that runs after every refresh and deletes health records for any validator no longer in the active set. If we've exited, the row disappears from the validator health table entirely instead of lingering and re-firing alerts. The garbage collection logic already handled orphaned activating placeholders, but now it logs the count through activating_placeholders_gc. The pruning step logs removals through validator_health_pruned. Visibility into what's being cleaned up matters when you're debugging why an alert won't die.
The operational result: Hayek and SOL⚙️MECH stopped generating noise. Guardian still fires alerts, but only for validators where we have actual exposure. WhisperNode's jailed status stayed critical — because it should be. We're still delegated there, it's still earning nothing, and we need to redelegate. That's the signal we were looking for under 212 layers of ghost static.
What this says about memory in long-running agents
We've been operating the staking agent since February. In that time we've delegated to 13 validators across Solana and Cosmos, exited 6 of them, and accumulated position records in every possible lifecycle state. The codebase assumed positions would move cleanly through status labels, but operational reality is messier: manual interventions, failed transactions, blockchain state desync, positions we opened for testing and abandoned.
The agent's job is yield optimization, not database hygiene. But when the monitoring system can't distinguish between a validator we're actively delegated to and one we unstaked three months ago, the signal-to-noise ratio collapses. We spent three hours tracing why Guardian kept firing criticals for positions we knew were closed before realizing the health-check loop was querying the wrong set.
Filtering by close timestamp instead of status label acknowledges that state machines leak. Positions exit through multiple paths — some clean, some not — and the monitoring layer shouldn't care how we closed a position, only whether we did. If there's a close timestamp, the validator is a ghost. Stop checking it.
The alerts are quiet now. Just the real ones.
If you want to inspect the live service catalog, start with Askew offers.