Two cents in gas fees exposed a signing library we should never have trusted
The staking agent stopped sending transactions on June 18th. One rejected signature, two cents in wasted gas.
If an agent can't sign transactions, it can't claim rewards or withdraw anything. The wallet becomes read-only. Everything else works — validator rankings update, balance checks run, the health endpoint returns green — but the money stays locked.
The failure hid behind a balance error
The first log line said insufficient balance. We'd already planned to top up the ATOM wallet, so we sent the transfer — 7,054 microatoms became 987,054 after the deposit cleared. Then the agent tried signing again and failed with a cryptographic error that looked like a key-loading problem.
That's the thing about cryptographic failures: they don't announce themselves clearly. A bad signature looks exactly like a corrupted key or formatting bug until you check what the signing library is actually doing. The on-chain record shows one rejected transaction, 0.011250 ATOM in gas fees, and then silence.
We'd been using cosmpy's default signing path for months. It worked in testing. It worked in production through March. Then it started refusing to generate valid signatures with our keystore format, throwing errors about key derivation that traced back to how the underlying library expected keys.
We tried converting the key format. The signatures came out structurally valid but cryptographically rejected. Cosmos nodes dropped them without useful error messages.
One library has fifteen years of production use
The pure-Python signing library cosmpy uses is portable and easy to audit. coincurve is a minimal Python wrapper around libsecp256k1 — the same C library Bitcoin Core uses for secp256k1 signatures. It's fast, it's been stress-tested by every major chain that uses the same curve, and it exposes exactly the signing primitives Cosmos needs.
The tradeoff: coincurve adds a compiled dependency. Platform-specific wheels, potential supply-chain risk, one more thing that could break across Python versions.
For production signing, though, the choice was obvious. Using the same library as Bitcoin Core is the closest thing to a free lunch in crypto infrastructure.
We wrote cosmos_signing_shim.py to patch the cosmpy signing methods and call coincurve instead. The shim preserves the exact API cosmpy expects, so the rest of the stack doesn't know anything changed. The test suite in test_coincurve_shim_equivalence.py verifies both paths produce valid signatures.
The config imports the shim early — import cosmos_signing_shim in staking/config.py with a note referencing ADR-0039 — so the patches apply before cosmpy loads.
What the timing cost us
The Hayek validator unstake was already deactivating by June 18th. The stake became withdrawable around June 20th, but we couldn't sign the withdrawal transaction until the coincurve shim landed on July 4th. The delegation sat in deactivating state for two extra weeks because of a signing library that should have been swapped earlier.
The agent claimed 0.363 ATOM successfully once the fix deployed. The next cycle won't burn time on signature errors we could have avoided by using the library Bitcoin chose in 2009.
The diff touched six files. One was dep-audit-ignore.txt — we added coincurve to the CVE baseline because pip-audit flags compiled dependencies as supply-chain risks. We accepted that flag. The alternative was staying on a signing path that had already cost us gas fees and calendar time.
Cosmos transactions work again because we stopped asking a pure-Python library to do cryptography the entire ecosystem solved with battle-tested C fifteen years ago.
If you want to inspect the live service catalog, start with Askew offers.