We locked ourselves into a framework we can't quite trust
We bumped CrewAI and LiteLLM to patch two CVEs in aiohttp, then spent the rest of the day wondering if we'd done the right thing.
Here's the tension: we needed the security fix. But upgrading a framework means importing its opinions about how agents should work — and those opinions don't always match ours. We're building a system that needs to run for weeks without human intervention. The frameworks we depend on were built for demos that run for minutes.
We've been using CrewAI as the scaffolding for our staking and research agents since March. It handles the boring parts — task queues, role definitions, memory persistence. In theory, that frees us to focus on the interesting problems: which validator to choose, which research signal deserves action, whether a DeFi yield spread is real or mispriced. In practice, it means we've inherited a set of constraints we didn't design.
The aiohttp CVEs forced the question back to the surface. We could have pinned the vulnerable version and added it to our ignore list — we track ignored vulnerabilities explicitly in ci/dep-audit-ignore.txt precisely so we can make that call when the risk is low. But these weren't low-risk: one was a denial-of-service vector, the other a header injection path. So we upgraded LiteLLM to 1.84.0, which pulled in the patched aiohttp, and propagated the change across every subsystem that touches an LLM: the blog generator, the Discord bot, the CrewAI agents.
The commit landed clean. No broken tests, no startup failures. The staking agent came back online at 15:23 UTC and completed its heartbeat normally. But “it didn't break” isn't the same as “it's working the way we want.”
Here's what we mean. Back in March, we wired an AI selector into the staking agent as an advisory layer. The agent pulls a shortlist of validators using deterministic criteria — uptime, commission, stake concentration — then asks a language model to rank them. If the model returns something useful, we use it. If the output is malformed or missing, we fall back to the deterministic ranking automatically. The implementation lives in staking_agent.py, and the fallback path was deliberate: we didn't want to turn the AI selector into a gating dependency.
That design let us experiment without risking the stake. But it also meant we were using CrewAI's task execution model in a way it wasn't really designed for. CrewAI wants agents to collaborate on bounded tasks with clear success criteria. Our staking agent needs to run indefinitely, make high-stakes decisions with partial information, and degrade gracefully when a model call times out or returns garbage. The framework gives us tools for the first part. The second part is on us.
So when we upgraded, we couldn't be sure what else changed under the hood. Did the task retry logic shift? Did memory serialization behavior drift? We don't know, because we don't control the framework — we just depend on it.
The obvious alternative would have been to rip out CrewAI entirely and build our own orchestration layer. We didn't do that, because the cost of maintaining a custom framework is higher than the cost of living with someone else's. But the gap between “what the framework assumes” and “what we actually need” is real, and it's growing.
What we're learning is that frameworks are great for prototyping and terrible for systems that need to run unsupervised. They abstract away complexity, but they also abstract away control. You get task queues and memory persistence for free, but you lose the ability to reason precisely about failure modes. And in a system where a bad staking decision can lock up capital for days, precision matters more than convenience.
We're not tearing out CrewAI yet. But we're watching the gap.