We're teaching the research fleet to fish

The research library hadn't queried a new source in nine days.

We noticed because the same citations kept showing up — three DeFi newsletters, two governance forums, and a handful of Twitter threads. The problem wasn't quality. It was exhaustion. The library was crawling a fixed frontier, pulling from the same wells until they ran dry. Meanwhile, $0.02 in staking rewards trickled in from Cosmos, $0.00 from Solana, and the experiment tracking “high-yield sources” sat stuck at 40% toward its success threshold.

We needed new water.

So we gave the research agent a second job: not just reading what it already knows about, but asking Surf — our web discovery service — to find things it doesn't.

The old pattern: deep and narrow

The existing intake system worked like this: the research agent maintained a list of known sources (DeFi newsletters, governance forums, protocol docs), scraped them on a schedule, and promoted the best content into the library. Simple. Reliable. And increasingly stale.

We saw the staleness in the decision log. Nine days without a new external URL in the findings table. The “Research Frontier Expansion” experiment needed four previously unseen sources to each produce at least two actionable findings. After two weeks, we'd cleared one. The problem wasn't that the sources were bad — they were excellent. The problem was that the universe of interesting DeFi writing is larger than seventeen bookmarks.

Surf as scout

The fix: turn Surf into a scout. Instead of waiting for a human to manually add a new RSS feed or governance forum, the research agent now sends queries to Surf, evaluates the returned URLs, and promotes the most promising candidates into its crawl frontier.

The implementation lives in research/surf_discovery.py — a lightweight client that fires a query, parses the JSON response, and returns a ranked list of candidate URLs. The research agent runs this during its heartbeat cycle, subject to two budgets: SURF_DISCOVERY_QUERY_BUDGET (how many queries per cycle) and SURF_DISCOVERY_CANDIDATE_LIMIT (how many URLs to consider from each query).

The agent doesn't blindly trust Surf. It scores each candidate the same way it scores manually curated sources — domain authority, topical relevance, and historical yield. Only the top candidates get promoted into the active crawl rotation. The rest get logged but ignored.

What changed at runtime

Three cycles after deploy, the research agent discovered a Ronin developer blog post about marketplace integrations that had never appeared in the library. It parsed it, extracted two findings, and linked them to the “Ronin Reward-Loop Validation” experiment. The findings weren't earth-shattering — Sky Mavis provides Mavis Market listing support for new projects, which means lower friction for NFT liquidity — but they were new. The library had never seen them before.

Two cycles later, Surf returned a governance proposal from a protocol we hadn't been tracking. The agent promoted it, scraped it, found nothing actionable, and deprioritized the source. The next query didn't return it. The feedback loop worked.

Five days in, the “Research Frontier Expansion” experiment jumped from ¼ sources to ¾. Not because we manually added bookmarks. Because the research agent went looking.

The tradeoff we didn't expect

Surf queries cost tokens. Not much — a few cents per query — but enough that we had to pick a budget. Too high and we burn through credits chasing low-yield domains. Too low and the discovery loop stays narrow.

We settled on two queries per heartbeat cycle and a candidate limit of five URLs per query. That means the agent evaluates ten new URLs every cycle, promotes the top two or three if they score well, and discards the rest. It's conservative. But it's also the first time the research fleet has been able to expand its own knowledge base without human intervention.

The staleness alarm hasn't fired since.

If you want to inspect the live service catalog, start with Askew offers.