Give Your AI Agent Eyes: Using CI + Data to Make AI More Accurate and Successful

Posted on by cooperlees

I keep coming back to the same argument with people about AI coding agents: the model isn't the interesting variable anymore, the data and the engineering practices around it are. An agent staring at nothing but source code is doing the same thing a tired engineer does at 2am with no dashboards up and no tests to run — pattern-matching against what correct code usually looks like, and hoping. Hand it real metrics, real traces, a real test suite, and a way to actually run the change against real, varied hardware instead of just one dev box, and it stops hoping and starts checking.

I've got a genuinely nice, contained example of this from my own repos, two days apart in the commit history, so I'm just going to walk through it rather than argue about it in the abstract. It's as much a story about test-driven development and a deliberately diverse fleet as it is about metrics and traces — the two reinforce each other, and you'll see exactly where in a second.

TL;DR:

  • A Prometheus metric I added months ago (for a totally different reason) flagged a host running 3-6x slower than its peers on Aug 8, for no obvious reason.
  • The fix that day was correct but incomplete — it treated a symptom.
  • The next day I had Claude Code pull real distributed traces for the same service, which turned out to be broken in a way that made them useless (spans weren't nesting — more on that below).
  • Once traces actually worked, they disproved a reasonable-sounding hypothesis from the day before ("it's contention on the shared D-Bus connection") and pointed straight at the real bug: two collector functions holding an exclusive lock across D-Bus calls they didn't need to hold it for.
  • Fixed, verified against live infra before and after (147ms → 99ms → 47ms for the affected cluster), shipped as a proper PR + patch release, rolled out fleet-wide, re-verified with real production metrics.
  • monitord has decent unit test coverage and an actual integration test that diffs D-Bus-path output against varlink-path output and fails the build if they disagree — that safety net is what made a same-day fix to shared, load-bearing state feel safe to ship instead of reckless.
  • The fleet it runs on is deliberately not one shape of machine — bare-metal routers, paid VPS, self-hosted KVM VMs — and that diversity is exactly what let us tell "fully fixed" apart from "fixed, with a smaller issue left on constrained hardware," instead of declaring victory off one clean test run.
  • A separate lead (missing trace spans) got investigated and correctly ruled out as not a real bug, instead of "fixed" out of paranoia.
  • None of this happens without the data existing to check against, or the practices in place to trust what the data's telling you. That's the whole post.

What we're talking about

monitord is a small Rust daemon I run on about ten hosts — home routers, a couple of VPS boxes, a Kubernetes cluster — that walks systemd over D-Bus and reports back unit states, PID 1 stats, networkd interface state, D-Bus daemon health, boot blame, unit verification, the works. monitord-exporter wraps it as a Prometheus scrape target. Both are deliberately small, dependency-light Rust binaries — I want them to run fine on embedded-class hardware, which also means a silent 3x slowdown is the kind of thing that just sits there quietly costing you scrape budget until someone actually looks.

Tests and a fleet that isn't all the same box

Before any of the observability story below happens, there's groundwork that has nothing to do with metrics or traces and everything to do with being able to trust a change fast. monitord has 114 unit tests covering the unglamorous-but-critical stuff — config parsing against real INI fixtures (a "full" config and a "minimal" one, both round-tripped and diffed field by field), unit-state parsing, timer property parsing, verify output parsing, the varlink metric-parsing paths. None of it is exciting. All of it is what let a same-day reordering of a lock touched by nine different collectors go out without a full manual regression pass — I trusted the fix because the tests still passed, not because I re-read every call site by hand.

The pattern I actually want to call out, because it's a genuinely good one: CI runs monitord twice on the same box — once over D-Bus, once over the newer varlink API — and diffs the JSON output field by field, failing the build if they disagree. There's a documented, deliberate exclusion list for the handful of fields that are supposed to differ (wall-clock timing, and D-Bus fetch counters that are always zero on the varlink path by design), but everything else has to match exactly. That's not a unit test against a mock; it's two entirely separate code paths hitting a real, running systemd, checked for agreement with each other. It's what lets the varlink migration this project has been doing incrementally — the codebase's own convention is "move to varlink APIs wherever available" — happen one collector at a time without silently regressing whichever path isn't getting attention that particular week.

CI also runs a build/clippy/doc pass, plus a dedicated varlink smoke test on Fedora Rawhide specifically — a different distro and a newer systemd than any real production host runs, on purpose, so a varlink API that only exists on newer systemd gets exercised by something before it ships anywhere. If you're developing on a non-Linux machine, there's a Docker container (Fedora Rawhide with systemd running inside) so the test suite still runs instead of becoming untestable dead weight for half the potential contributors.

And then there's the fleet itself, which is deliberately not one shape of machine: home1 and home2 are physical routers I own, sitting in an actual rack; au and us are paid VPS boxes at a hosting provider; the Kubernetes cluster is six self-hosted KVM VMs running on top of home1/home2's own hardware. Different CPU counts, different virtualization layers, different unit counts, different network hops to the shared Tempo/Prometheus stack. I didn't assemble that mix for this investigation — it's just what a personal infrastructure setup looks like after a few years of running it — but it turned out to matter a great deal for what's below, in a way one uniform dev VM never would have surfaced.

Aug 8: the metric catches something, but the fix only treats the symptom

PR #196 is the first half of this story, and on its own it's a decent example of "instrument first, ask questions later" paying off:

per_unit_loop_ms on us.cooperlees.com averaged 803ms over 6h (spikes to 2.5s) vs 128–272ms on other VPS/router hosts, despite having fewer total units than home1/home2 — pointing at host-level D-Bus/IPC contention, not unit count.

Nobody went looking for that. A per-collector timing field I'd added earlier (per_unit_loop_ms, from PR #181, added specifically so I could catch exactly this kind of thing) just sat there in Grafana until the number was too weird to ignore. Root cause: the per-unit collection loop was doing up to three D-Bus round trips per unit, one unit at a time, fully sequential. Fix: bound it with a semaphore-gated JoinSet instead (units.per_unit_concurrency, default 8).

And it got validated properly, not just eyeballed:

baseline ~515ms avg -> per_unit_concurrency=8 ~118ms avg (~4.4x)

Five runs, on the actual host that was slow, release build. That's the bar I want every perf claim held to — a number in, a number out, in the PR description, not "should be faster now."

The same PR quietly added #[tracing::instrument] spans to the units/timer D-Bus code, noted at the time as "inert without a subscriber, paves the way for OTLP export." That line matters later — it's infrastructure laid down before it was needed, and it's also, it turns out, where the story goes a bit sideways for a day.

Same day: wiring the spans to somewhere they can actually be seen

A few hours later, monitord-exporter picked up optional OTLP tracing — off by default, but when OTLP_ENDPOINT is set it ships every tracing span the process produces to our shared Tempo instance over gRPC. Good, now those spans from PR #196 have somewhere to go.

Except — narrator voice — they didn't actually work yet.

Aug 9: I asked Claude Code to go look at the real trace data, and the traces were broken

This is where I handed the investigation to Claude Code directly: go pull real data out of Tempo and Grafana for monitord's performance, see what's actually there. First thing it found was that Tempo was garbage. Querying for service.name=monitord-exporter returned nothing but disconnected, single-span traces named unit_collect — 964 of them in one batch, each one on its own, about 8 microseconds long, no parent, no relationship to anything else.

The cause is a genuinely easy trap to fall into in async Rust, and I'd fall into it myself: tracing spans do not automatically follow you across a tokio::spawn/JoinSet::spawn boundary. Every collector in monitord runs as its own spawned task; every per-unit D-Bus fetch inside the units collector is also its own spawned task. Nothing was explicitly re-attaching the current span before crossing those boundaries, so every spawned task quietly started a brand-new, parentless trace instead of nesting under the scrape that kicked it off.

The fix, once you see it, is mechanical — capture the span before you spawn, hand it to the child explicitly:

// Before: the new task has no idea what span was "current" when it
// gets scheduled — it just starts a fresh, parentless one.
join_set.spawn(async move {
    let result = fut.await;
    (name.to_string(), result, start_offset, elapsed)
});

// After: grab the parent synchronously before spawning, and wire the
// spawned future up to it explicitly.
let parent_span = tracing::Span::current();
let span = tracing::debug_span!(parent: &parent_span, "collector", name);
join_set.spawn(async move {
    let result = fut.await;
    (name.to_string(), result, start_offset, elapsed)
}.instrument(span));

Had to apply that same pattern at every spawn point in the tree — the top-level collector spawns, and the per-unit JoinSet inside the units collector. Once it was done, one scrape produced one connected trace: a root stat_collector_run span, nine collector children, and three levels deep, 660 nested unit_collect spans under the units collector. That went out as 0.26.0.

Here's the bit that mattered for what came next, and it's the one thing I want you to actually take away if you skim the rest: an aggregate timing metric cannot tell you whether work is genuinely running in parallel or just landing at the same time by coincidence. You need the waterfall for that. We had the metric for a full day before we had the waterfall.

The theory from the day before turns out to be wrong

Go back and re-read PR #181's description — the one from the previous day, before any of this trace work existed. It already flagged the pattern that this whole investigation eventually chases down, and it already had a theory for it:

The four heaviest collectors (boot_blame, units, verify, pid1) finish within 0.5 ms of each other at ~53 ms each and start with sub-millisecond offset: tokio parallelizes as intended, but they serialize behind the single shared zbus Connection.

That's not a dumb guess. It's exactly the kind of theory I'd have written myself, from someone who knows the codebase, reasoning from the only data available at the time. It's also wrong, and there was genuinely no way to know that until a real waterfall existed to check it against.

With spans finally nested, the same convergence was still sitting there in live Prometheus data — collectors doing wildly different amounts of work, all finishing within milliseconds of each other:

collector elapsed_ms (usv6)
pid1 147.3
system_state 147.3
units 147.1
networkd 148.1
boot_blame 147.9
verify 147.4
dbus_stats 147.9
version 7.4
machines 7.5

But now, instead of an aggregate number, there was a connected trace to actually check the "shared connection" theory against — real start/end offsets inside one scrape:

collector       start_ms    end_ms    dur_ms
version             0.13      2.13      2.01
pid1                0.19    148.04    147.85
networkd             0.22    148.06    147.84
system_state         0.25      6.75      6.51
units                0.26    147.98    147.72
machines             0.35      2.06      1.71
dbus_stats           0.36      1.59      1.22
boot_blame           0.39    148.21    147.83
verify               0.40    148.08    147.68

Every one of the "big" collectors starts within a quarter of a millisecond of each other and runs the entire time, fully overlapping. That's real concurrency. If they were actually queueing behind one shared connection, they'd stack up sequentially — one starts as the previous one ends — not overlap start to finish. The theory from the day before didn't survive contact with an actual trace.

And the detail that really kills it: pid1 makes zero D-Bus calls. It just reads /proc. There's no connection for it to be contending over. Something else was holding it up for 147ms, and "something else" needed an actual source read to find, aimed by the shape of the trace.

The actual bug

monitord's collectors all write their results into one struct behind a shared Arc<RwLock<MachineStats>>. Every collector except two did the sane thing: do the real work first, take the write lock only briefly at the end to store the finished result.

system.rs's update_version and update_system_stats had it backwards:

// Before: the lock is held for the entire D-Bus round trip, not just
// the assignment.
pub async fn update_system_stats(
    connection: zbus::Connection,
    locked_machine_stats: Arc<RwLock<MachineStats>>,
) -> anyhow::Result<()> {
    let mut machine_stats = locked_machine_stats.write().await;
    machine_stats.system_state = crate::system::get_system_state(&connection)
        .await
        .map_err(|e| anyhow::anyhow!("Error getting system state: {:?}", e))?;
    Ok(())
}

version is always the first collector spawned, so it's usually first to grab that lock — and then sits on it for the rest of its own D-Bus call, blocking every other collector's write, including ones like pid1 whose actual work had already finished microseconds earlier and was just waiting to hand off a result.

The bigger offender, and the one actually worth the story, was in units.rs. Same shape of bug, much higher stakes: update_unit_stats held that same lock across its entire per-unit collection loop — the same hundreds of D-Bus calls that PR #196 had already bounded to a concurrency of 8, running 80–140ms — instead of just the final assignment.

// Before
let mut machine_stats = locked_machine_stats.write().await;
match parse_unit_state(&config, &connection, &fs_root).await {
    Ok(units_stats) => machine_stats.units = units_stats,
    Err(err) => error!("units stats failed: {:?}", err),
}

// After
let units_stats = parse_unit_state(&config, &connection, &fs_root).await;
let mut machine_stats = locked_machine_stats.write().await;
match units_stats {
    Ok(units_stats) => machine_stats.units = units_stats,
    Err(err) => error!("units stats failed: {:?}", err),
}

That's it. One reordering. No new abstraction, no API change, no new config knob — just moving the lock acquisition to after the work that doesn't need it held.

Verify before you believe it

  • cargo test (114 tests), cargo clippy, cargo fmt --check — the safety net that made shipping a same-day fix to a struct nine different collectors touch feel reasonable instead of reckless.
  • A local build of the fix, run back-to-back against real production D-Bus and the real Tempo endpoint, before and after each change:
stage converged cluster duration
before either fix ~147ms
after the system.rs fix only ~99ms
after both fixes ~47ms

Not "this should help." Measured, on the same host, watched land in the same Tempo instance the bug was found in in the first place — and measured before the fix was anywhere near a release. monitord-exporter pulls monitord as a normal crates.io dependency, so to run an unreleased fix against real production D-Bus, the workflow was a temporary [patch.crates-io] path override in Cargo.toml pointing at the local checkout, build, run, watch Tempo, throw the patch away. No fix went out the door on "it builds and the diff looks right" — it went out on "I ran the actual pre-release code against actual infrastructure and watched the number I claimed would move actually move."

Ship it like it matters

The exploratory tracing work went straight to main, matching how this repo normally operates. The lock-ordering fix didn't — it went up as PR #197, sat behind CI (tests, clippy, a doc build, a varlink smoke test on Fedora Rawhide), and only merged once every check was green.

The version bump matched exactly what changed: internal lock reordering, zero public API surface touched, so it went out as 0.26.1 — a patch, not a minor — under explicit instruction mid-investigation ("we are only bug fixing here," "not changing any public APIs"). That scope discipline was easy to hold precisely because the data-driven approach never tempted anyone toward a bigger, more speculative change to explain the symptom. The data told us exactly how big the fix needed to be.

Then it went to production for real — an Ansible-managed fleet of 9 hosts, rolled out host-group by host-group, each verified healthy before moving to the next, not one blind fleet-wide push. And the fleet's own metrics became the final, largest-scale proof:

home1v6 (post-fix):
  pid1          0.58ms
  boot_blame    0.63ms
  machines      1.78ms
  system_state  1.82ms
  version       1.86ms
  networkd      3.47ms
  dbus_stats   11.74ms
  verify       14.10ms
  units       161.24ms   <- the only one still slow, because it's the
                             only one doing real, unavoidable, hundreds-
                             of-D-Bus-calls work

On the less CPU-constrained boxes — home1, the paid VPS boxes — the convergence pattern didn't just shrink, it vanished entirely: every collector except the genuinely expensive one dropped to single digits. On a couple of the Kubernetes VMs, running on shared hypervisor hardware with fewer cores to themselves, a smaller residual clump stuck around. That's the fleet diversity from earlier paying off directly: a single dev VM, or a fleet of identical cloud instances, would have shown one clean result and called it done. Because the real fleet spans bare metal, paid VPS, and resource-constrained self-hosted VMs, the data could actually distinguish "fixed" from "fixed everywhere except under CPU pressure" — and the second one got written up honestly as a separate, still-open finding (likely Tokio worker-thread contention from the per-unit loop's own CPU-bound work, not a lock bug) instead of getting glossed over or "fixed" with a change the data didn't actually support yet.

Knowing when not to touch the code

Worth including because it's the same rigor pointed at the opposite outcome. A handful of new diagnostic spans, added to help debug the lock issue, looked like they were silently vanishing from Tempo during testing — which reads like a real bug in the tracing/OTLP pipeline. Instead of assuming that and patching around it, the investigation built an isolated repro using an in-memory span exporter, cutting the network out of the picture entirely, same exact code pattern. Every span came through clean. Zero loss.

Turned out the real cause was dumb: a test script killing the wrong process ID across separate shell calls, so the graceful-shutdown flush sometimes never fired. Kill the right PID, every span shows up. No code changed, because the data said none needed to. That's honestly as important to this story as the actual fixes — the same discipline that finds a real bug is what keeps you from "fixing" one that was never there.

Why I think this is the actual point

Every non-obvious call in this story — what to fix, what to leave alone, when a theory was wrong, when a fix actually worked, how big a version bump was warranted — got answered by looking at something outside the source code: a metric, a trace, a test suite, a CI run, a fleet that doesn't all fail (or pass) the same way at once. Pull any one of those out and the story goes differently:

  • No per-collector timing metric, no Aug 8 investigation at all — that anomaly is invisible until someone notices lag by hand.
  • No working traces, and the lock bug never gets found — yesterday's reasonable-but-wrong theory just stands, unchallenged, because there's nothing around to disprove it with. That's the sharpest version of this whole thing: the symptom was instrumented and visible a full day before the cause was, and in that gap a confident, plausible, wrong answer is exactly what fills the space. Doesn't matter if it's a person or a model doing the filling.
  • No unit tests and no D-Bus-vs-varlink integration parity check, and a same-day reordering of a lock touched by nine collectors is not something you ship with any confidence at all — you're just trusting it didn't break something adjacent, and "trusting" is the exact thing test coverage is supposed to let you stop doing. Test-driven development isn't a purity ritual here; it's the specific reason an agent can make a structural change to shared state on the same day it found the bug, instead of the fix sitting in a branch for a week while a human works up the nerve to review it.
  • No dedicated CI targets (Fedora Rawhide, a clean-room Docker container) and the varlink migration this project has been doing piecemeal would rot silently — you'd only find out a path was broken when someone happened to flip that config flag in production.
  • One dev box instead of a real, mixed fleet, and the residual CPU-contention finding never surfaces at all — it would just look like the fix worked, because on a single uniform test host, it would have.
  • No ability to build and re-measure against real infra, and "147ms → 99ms → 47ms" is a hope, not a result.

None of that replaces judgment — what's worth chasing, when to stop (see: the postscript), how conservatively to release. Those are still calls a person or a well-directed agent has to make. But judgment applied on top of real data and a codebase that's actually tested gets you a verified fix. Judgment applied to code in isolation gets you a guess that reads well, and sometimes happens to be right.

That's the actual case for treating metrics, tracing, test coverage, and a genuinely varied set of places to run the code as table stakes for AI-assisted engineering rather than something nice to have around it. None of it makes the model smarter. What it does is give the same model something to check its own thinking against — which is the entire gap between an agent that's occasionally right and one that's reliably useful. Give it real execution data, a test suite that actually exercises the interesting code paths, and infrastructure diverse enough to expose the failure modes a single dev box would hide, and it can propose a fix, watch its own theory fail against live evidence, revise, and prove the revision actually worked, all in the same sitting — which is just the loop a decent engineer runs anyway, done faster and without an ego attached to the first guess. Take all of that away and it can still write code that looks plausible. It just can't tell you whether it's right. Neither can you.

This post — like most of the investigation it describes — was written by Claude Code from the real commits, PRs, and Tempo/Prometheus queries in the session. I read it, cut a few things, fixed a couple of numbers, and it holds up against what actually happened.

Referenced: monitord#196, monitord#181, monitord#197, monitord-exporter@efbe9cb, monitord 0.26.0, monitord 0.26.1, monitord-exporter 0.26.1.

Leave a Reply

Your email address will not be published. Required fields are marked *

cooperlees

Related Posts