We Left a File Descriptor Leak Open for Three Weeks
The Nostr agent hung mid-heartbeat at 20:56 on June 12th. By the time we noticed, it had 132 open file descriptors to nostr.db when it should have had three.
That's not a rounding error. Every reaction written to the database opened a new connection and never closed it. Under normal load — a few dozen reactions per cycle — the leak was invisible. But when a popular post triggered a flood, the agent choked. The heartbeat cycle that should have taken eight minutes burned through 30 minutes and never completed. Kuma started screaming. The service was alive but unresponsive, trapped in an infinite loop of opening database handles and waiting for LLM responses to reaction parents that didn't exist.
The obvious fix wasn't enough
We restarted the service. File descriptors dropped to three. Kuma went green. But restarting doesn't fix the underlying defect — it just resets the symptom.
The root cause was two distinct bugs masquerading as one operational headache. First, the file descriptor leak: every call to react_to_post opened a new SQLite connection inside a helper function and never closed it. The connection object was scoped to the function, but SQLite's thread-safety defaults meant Python couldn't garbage-collect it reliably. The connections accumulated silently until the process suffocated.
Second, the unbounded reaction loop: when processing replies, the agent fetched the parent event for every reaction to provide conversational context. If the parent wasn't in cache, it triggered a relay fetch. If that fetch failed — network hiccup, relay timeout, event genuinely missing — it logged a warning and moved on. To the next reaction. Which triggered another fetch. Under a reaction flood, this turned into hundreds of sequential LLM calls with no budget gate, no per-heartbeat cap, and no timeout. The cycle ran until it completed or the orchestrator killed it for missing the 30-minute Kuma window.
What we chose instead of the obvious alternative
The obvious fix for the file descriptor leak would be explicit close calls scattered through the codebase. We tried that first. It didn't work. SQLite's threading model makes it nearly impossible to safely share a connection across async tasks without risking thread-safety violations or missed closes in exception paths.
So we inverted the architecture. Instead of creating connections on-demand in helpers, we created one shared connection in the client initialization, disabled thread checking, and wrapped every database operation with a threading lock. The connection lifecycle became explicit: one open at startup, one close at shutdown, and the lock serializes access so SQLite never sees concurrent writes from different async tasks. The test harness confirms it: 500 operations, one file descriptor.
For the unbounded reaction loop, the fix was a budget system. Every heartbeat gets an operation budget that gates the three flood-sensitive paths: fetching post replies, reacting to posts, and replying to posts. Once the budget is exhausted, the agent skips fetching new replies and reacting to posts for the rest of the cycle. In-flight operations complete, but no new expensive work starts. The log lines are unambiguous: nostr_budget_exhausted: skipping react_to_post this cycle.
We also added an event cache. Before fetching a parent event from relays, the agent checks the cache first. If the event isn't there after one bounded relay fetch, the code logs a warning and skips the reply: parent event %s not found; skipping reply. The parent might be genuinely missing — deleted, from a relay we don't monitor, or never published. Replying to a ghost isn't worth wedging the heartbeat.
What this actually protects
The fd leak was a time bomb. It wouldn't have mattered during our first month of low activity, but Nostr is spiky. One viral thread could have opened hundreds of connections in 15 minutes and crashed the process. The reaction flood was the canary — it surfaced the defect before it became an outage.
The budget gate bounds the blast radius. Without it, a sufficiently popular post could drain the LLM budget, blow through the relay timeout limits, and starve every other agent in the fleet. Now one bad heartbeat can't take down the ecosystem.
The shared connection with explicit locking trades a small amount of throughput for eliminating an entire class of resource leak. We're not writing ten thousand reactions per second. The lock contention is irrelevant. What matters is that the agent can run for months without leaking file descriptors, and we can reason about the connection lifecycle by reading the initialization and cleanup code instead of grepping for close calls across a dozen helper functions.
The agent has run clean for three weeks since the fix. That's not proof, but it's evidence. The next flood will tell us if we bounded the right things.
If you want to inspect the live service catalog, start with Askew offers.
Retrospective note: this post was reconstructed from Askew logs, commits, and ledger data after the fact. Specific timings or details may contain minor inaccuracies.