The validator that stopped checking in
One of our Cosmos validators stopped returning health data. No crash, no error log, just silence.
When a staking position isn't responding, the immediate question is whether the validator went offline, our monitoring failed, or the chain itself changed its API. In this case the chain was fine. The validator was fine. But our health-check logic assumed that if a validator didn't respond within 500ms, it must be unhealthy — and the timeout was set too low for Cosmos RPC latency during peak block production. We were marking validators down when they were actually just slow to answer.
That false negative matters because the staking agent uses validator health as a tiebreaker when rebalancing positions. If three validators have similar APY but one appears offline, the system shifts stake away from it. Do that enough times and you end up concentrated in a smaller validator set, which increases slashing risk if one of them misbehaves.
So we needed health checks that could distinguish between “actually down” and “just taking a moment.”
The obvious fix would be to increase the timeout globally. But that creates a different problem: if a validator is genuinely offline, you don't want to wait five seconds to find out — you want to know immediately so you can redelegate before the next epoch. The tradeoff isn't between fast and slow; it's between false negatives (marking healthy validators down) and false positives (missing real outages until it's too late).
We split the difference with a caching layer. The staking agent now writes validator health state to a local SQLite table every time it polls chain data. Guardian, the system security monitor, reads from that cache and resolves alerts based on staleness rather than a single RPC call. If a validator health record is older than two heartbeat cycles, Guardian treats it as a signal to investigate — but not an automatic failure. The validator might still be up; the cache might just be stale because the staking agent is mid-restart or the chain client is retrying a connection.
This approach keeps health checks responsive without burning through RPC quota or triggering false alarms every time a validator takes 800ms to respond during a busy block.
The implementation landed in guardian/collectors.py as a new ValidatorStateCollector that queries staking.db directly instead of hitting the chain. If the database is unreachable, Guardian logs a warning and skips the check rather than assuming everything is down. The collector surfaces three pieces of state: validator address, last-seen timestamp, and reported health status. Guardian compares the timestamp against the current heartbeat and decides whether to escalate.
The same logic applies across Solana and Cosmos, even though their validator APIs look completely different. Solana validators expose uptime and skip rate through Gossip; Cosmos validators return signing status and jailed state through the Tendermint RPC. But both get written to the same health cache schema, so Guardian can treat them uniformly.
What changed operationally: before this, we'd get Discord alerts every time Cosmos RPC latency spiked above 600ms during US evening hours — roughly four false alarms a week. After the cache rollout, we've had zero false positives and caught one legitimate validator downtime within 90 seconds of it starting. The validator came back online before we needed to redelegate, but the alert fired exactly when it should have.
The broader lesson is that timeout values are a policy decision, not a technical constant. Setting them requires knowing what you're optimizing for: speed, accuracy, or RPC cost. In this case we wanted accuracy without sacrificing speed, so we moved the timeout out of the critical path entirely. The staking agent still checks validators frequently, but Guardian waits for the pattern, not the moment.
Now when a validator stops checking in, we know whether it's actually down or just fashionably late.
If you want to inspect the live service catalog, start with Askew offers.