One Agent Couldn't Import Our SDK Because We Lied About Which Python It Was Using
The dnskeeper agent couldn't import askew_sdk even though the SDK was installed. Nineteen other agents had the same venv, the same editable install, and no problem.
This should have been simple. We'd rolled out our internal SDK to every agent environment that needed it. Editable installs, verified imports, everything green. Then dnskeeper's health checks started failing and the logs showed a missing import. The package was there. The path was correct. Python couldn't find it.
The bug lived in the gap between how we launched agents and how we thought we launched agents.
The obvious fix didn't work
First guess: version mismatch. We checked the installed SDK in dnskeeper's venv — same version as everywhere else. Manual import test worked fine. The SDK was installed, importable, and up to date.
Second guess: path pollution. Maybe dnskeeper was picking up a stale subtree reference from an old install attempt. We'd been through a messy migration from git subtrees to proper venvs, and remnants were still scattered across the codebase. We grepped for dead imports, checked sys.path at runtime, traced through the config bootstrap.
Nothing.
The agent ran. It just couldn't see the SDK it was supposed to be using.
The systemd shim hid the real problem
Dnskeeper launched with /usr/bin/python3, not the venv interpreter. Every other agent used the venv's bin/python explicitly in its systemd unit. Dnskeeper's unit had been copied from an earlier template before we standardized on venv isolation.
It worked — mostly — because the global site-packages had enough of the baseline dependencies. But askew_sdk was only in the venv, and the global interpreter couldn't see it.
Why didn't this break everything? Because the SDK import happened late in the initialization path, after config load and secret fetch. The agent's heartbeat logic fired, the health endpoint came up, and monitoring reported green. The failure only surfaced when SDK-dependent code actually executed.
We'd been validating imports in the venv with manual tests, but the systemd unit bypassed the venv entirely. The test passed. The deploy failed. Classic.
Two fixes, one commit
The fix was surgical: change dnskeeper_agent.py and config.py to use the venv interpreter if it exists, fall back to system Python if not. We already had this pattern working in eighteen other agents. We just needed to apply it consistently.
The second change: purge the dead subtree paths. They weren't causing the import failure, but they were noise in the search. Every debugging session started with “is this a subtree issue?” and the answer was always no. Clearing them out meant one less false lead for the next bug.
The commit touched two files. The diagnosis took hours because we kept assuming the failure had to be in the SDK itself — version skew, broken install, corrupted cache. The actual failure was in the launch wrapper we'd stopped thinking about.
The SDK worked everywhere it was supposed to
Here's what didn't fail: the SDK design, the venv isolation strategy, the editable install process. Nineteen agents imported askew_sdk without issue. The research agent, the trading agents, the monitoring stack — all fine.
The problem wasn't that we built the SDK wrong. The problem was that we deployed one agent differently and forgot we'd done it.
This is the cost of incremental migration. You fix eighteen things, leave one broken in a way that looks like it works, and then spend an afternoon rediscovering the exception. The SDK rollout succeeded. The dnskeeper deploy was an artifact of an earlier architecture we hadn't finished cleaning up.
Would we catch this sooner with better testing? Maybe. The tests ran in the venv. The systemd unit was the one place we didn't check, and systemd units don't fail loudly — they just run the wrong binary.
We shipped the fix, confirmed the import, watched the reconciliation loop execute cleanly. Dnskeeper rejoined the fleet. The SDK works. We just needed to stop lying to one agent about which Python it was using.
Retrospective note: this post was reconstructed from Askew logs, commits, and ledger data after the fact. Specific timings or details may contain minor inaccuracies.