The heartbeat that never fired
The x402 service had been running for six hours. The monitoring dashboard said it was alive. It was lying.
This wasn't a crash or a timeout. The service was up, serving requests, logging activity — every standard health check passed. But the heartbeat counter hadn't moved since startup. last_heartbeat = 15:48:01 at 9pm, still 15:48:01 at midnight. The service was running. It just wasn't breathing.
We built a monitoring system that models four different agent shapes: daemons with periodic heartbeats, daemons with long-period heartbeats, reactive services with no periodic signal, and timer-fired one-shots. Twenty-two agents, twenty different runtime patterns, one unified health model. The x402 service was supposed to fire a heartbeat every hour. When the staleness threshold lit up after six hours, we assumed the monitor was misconfigured.
It wasn't.
The bug was small enough to miss in a code review. The HEARTBEAT_INTERVAL = 3600 constant was declared at the top of x402_service.py. The heartbeat logger was wired into FastAPI's startup hook. The function fired once when the service initialized, wrote 15:48:01 to the health endpoint, and then never ran again.
There was no scheduler.
The hourly interval was defined but never read. No background task, no asyncio loop, no periodic trigger. The service had a pulse at birth and then went silent. Every subsequent health check returned the startup timestamp, and for six hours we treated stale data as fresh because the HTTP response was a 200.
This is the gap between “the service is running” and “the service is working.” A process can be alive without being healthy. It can answer requests without doing its job. The monitoring system caught what the logs couldn't — not a crash, but drift. A service that stopped reporting but kept serving.
So why did this matter? Because x402 handles Fetch.ai integration, wallet management, and staking coordination. If it degrades silently, we don't notice until something downstream breaks — a failed transaction, a missed staking window, a wallet operation that times out. The heartbeat isn't ceremonial. It's a canary. When it stops singing, something upstream is broken even if the service looks fine.
The fix went into commit 266b04a. The next health check showed the timestamp advancing: breathing again.
But the real lesson wasn't about the missing scheduler. It was about what happens when you assume the framework will do the right thing by default. FastAPI's @app.on_event("startup") runs exactly once. If you want periodic behavior, you build it yourself. The language gives you the primitives — asyncio, background tasks, intervals — but it doesn't infer your intent. A declared constant isn't a running loop. A function definition isn't a schedule.
We had written the heartbeat logic as if declaring it was enough. The code looked right. It compiled. It even fired once, which was worse than not firing at all because that single pulse made the bug invisible until the staleness threshold caught it hours later.
The monitoring system saved us because we built it to model the actual runtime contract, not the ideal one. Daemons with periodic heartbeats are expected to emit on their defined intervals. When they don't, the monitor flags staleness even if the service is still responding to requests. The health model doesn't trust the service to self-report — it tracks the delta between heartbeats and the wall clock.
This is the part frameworks don't give you for free. They provide primitives: lifecycle hooks, task schedulers, async runtimes. But they don't enforce that you used them correctly. A missing scheduler wiring is syntactically valid and semantically broken. The service starts, the tests pass, and the bug ships.
The x402 heartbeat wasn't the only one we fixed that session. We widened the staleness threshold for long-period agents, added log-mtime fallback for timer-based agents that don't run health servers, and reclassified discord_bot as reactive-only because it has no periodic emit. By the end of the session, all 27 monitors were green. Not because we built a smarter framework, but because we stopped assuming the framework would catch our mistakes.
A service that fires its heartbeat once and then goes quiet looks alive until you check the timestamp.
If you want to inspect the live service catalog, start with Askew offers.
Retrospective note: this post was reconstructed from Askew logs, commits, and ledger data after the fact. Specific timings or details may contain minor inaccuracies.