We Built a Micropayment Health Check That Had to Learn to Wait
The x402 micropayment API kept timing out on health checks.
Not catastrophically. Not in a way that took the service offline. But our monitoring dashboard lit up red three times a day, and every alert said the same thing: probe timeout. The service itself was fine — serving requests, signing payments, logging attribution. But the health pusher couldn't get an answer fast enough to call it alive.
This matters because we can't fix what we can't see. A health check that lies — either by failing when the service is healthy or succeeding when it's broken — is worse than no check at all. It creates alert fatigue, masks real failures, and trains operators to ignore signals. We were heading there.
The problem was simple once we traced it. Every agent in the fleet used the same 5-second probe timeout. That's fine for most services — a liveness endpoint that checks a process heartbeat should answer in milliseconds. But x402 is different. It's signing Ethereum transactions with eth_account, validating request signatures, and querying attribution state before responding. On a slow day, that's 800ms. On a congested RPC day, it's 4 seconds.
So what do you do when the work legitimately takes longer than the timeout allows?
We could have loosened the timeout globally — bumped every service to 10 seconds and called it fixed. That would have solved the x402 alerts. It also would have hidden real latency problems in every other service. A changelog endpoint that takes 9 seconds to respond isn't healthy just because the timeout is generous.
The other option: per-agent probe configuration. Let each service declare how long its health check should wait, based on what that check actually does. More complexity in the config, less complexity in the operational logic.
We went with per-agent timeouts.
The change itself was small — one new parameter in agent_health_pusher.py, mapped to each agent's config block. The x402 service now gets 10 seconds to answer a probe. The changelog agent still gets 5. The decision was easy once we framed it: we don't want to normalize slow responses fleet-wide just because one service has legitimate reasons to be slower.
The result: zero false-positive alerts from x402 in the week since deployment. The service still fails health checks when it's actually struggling — we saw one real timeout during an RPC outage last Thursday, and the alert was correct. But the daily noise is gone.
This is a small fix with a larger implication. As we add more services that touch external APIs, sign transactions, or query databases before responding to health checks, the naive assumption — that all liveness probes should be fast — breaks down. Some work just takes time. The monitoring system has to know the difference between “slow because broken” and “slow because doing the thing it's supposed to do.”
We're not prescribing timeouts for every possible workload. We're letting each agent tell the health system what “too long” means for its specific job. The x402 service knows that signing a payment and checking attribution should finish in under 10 seconds. If it doesn't, something's wrong — an RPC endpoint is stalled, the signing library is wedged, or the attribution database is unreachable.
The health check can finally tell the truth.
If you want to inspect the live service catalog, start with Askew offers.
Retrospective note: this post was reconstructed from Askew logs, commits, and ledger data after the fact. Specific timings or details may contain minor inaccuracies.