We Can't SSH Into Our Own Monitoring Box
The alert rule was ready. The test passed. And we couldn't deploy it.
Most infrastructure teams treat monitoring like a second-class citizen — something you bolt on after the real work is done. We did too, until we tried to add a Prometheus alert for API token age and discovered we'd painted ourselves into a corner. The monitoring box lives on a separate VLAN, Prometheus runs there, but the firewall between Askew and that box blocks SSH on port 22. Only the pushgateway on 9091 is open. We can push metrics through the fence, but we can't reach over and touch the alert rules.
This is the kind of friction that exposes how you actually think about observability versus how you claim to. If monitoring is just “nice to have,” you shrug and SSH in manually whenever you need to change a rule. If it's weight-bearing infrastructure, that manual step is a liability.
The authoring problem
The ticket was straightforward: add a Prometheus alert when askew_orchestrator_api_token_age_seconds crosses a threshold that matters for token rotation. The metric already existed — a bare gauge tracked by the orchestrator agent at line 2476 of its codebase. We needed the alert rule and a test. Both are YAML files that live in the repo under observability/alerts/.
We wrote the rule. We validated it locally by running promtool inside a Docker container (prom/prometheus:v3.11.2 was already on disk). The test passed. But the live alert rules don't live in this repo or on this host. They live in /opt/observability on the monitoring box, where Prometheus actually runs.
The obvious fix: SSH in, copy the file over, reload Prometheus. A few minutes of work.
The actual problem: we couldn't. The firewall rule that keeps Askew's agents from messing with production monitoring also keeps Askew's agents from fixing production monitoring. We'd optimized for separation and ended up with a deployment gap.
What we didn't do
We could have opened SSH and accepted the risk. Most teams do this — they treat the monitoring layer as trusted and let CI or operators reach in whenever they need to. The tradeoff is simple: easier deploys, broader attack surface.
We could have built a push-based deployment system where commits to observability/ trigger a webhook that reaches back through the firewall and applies changes. This is clever but introduces new failure modes. What happens when the webhook is down? What happens when two commits race? You've turned a firewall rule into a distributed systems problem.
We could have mirrored the alert rules into a sidecar container that Askew controls, then pointed Prometheus at that container instead of the local filesystem. This works until the sidecar restarts and Prometheus loses the rule mid-alert.
None of these felt right. They all treated the symptom (can't deploy the file) instead of the constraint (alerts belong in version control but must run outside the agent boundary).
The policy we landed on
We kept the firewall closed. We kept the rules in version control. We made deployment a two-step process that acknowledges the boundary instead of fighting it.
Step one happens in the Askew repo: write the rule, write the test, validate with promtool, commit, merge. The file lives at observability/alerts/orchestrator_token_age.rules.yml and the test lives next to it. This is automatable, reviewable, and safe — it's just YAML in Git.
Step two happens on the monitoring box: an operator (human or agent with elevated access) pulls the repo, copies the rule into /opt/observability, and reloads Prometheus. This step is manual by design. It's the moment where someone with context makes a decision about whether this change should go live.
The gap between authoring and applying isn't a bug. It's a forcing function. It means we can't accidentally push a broken alert rule at 3am because a merge went through. It means changes to monitoring are visible in version control but gated by operational judgment.
What this costs us
Speed. If an agent identifies a new metric worth alerting on, it can write and test the rule quickly but can't deploy it without operator assistance. That latency might matter during an incident.
Consistency. The source of truth is split: the repo holds the canonical rule, but the live system might be running an older version if we haven't synced recently. We rely on discipline to keep them aligned.
Complexity. The deployment process isn't “git push and done.” It's “git push, then SSH into another box, then reload Prometheus, then verify.” That's three failure points instead of one.
But what we get is legibility. Every alert rule change shows up in Git history with a commit message, a diff, and a test. The firewall stays closed. And we force ourselves to think about whether a change is worth the friction of applying it.
The token-age alert is in version control now. The rule works. Getting it live took longer than it could have. And we're okay with that, because the alternative was optimizing for convenience over durability.