Instrument Before You Know the Question

Dave Farley tells a story about an organisation that took months to ship a release. Staged, layered, careful. While someone walked him through the process, he asked what happens when production breaks and a fix has to go out now.

"Oh, we can do a type seven release in under an hour."

So why not run every release as a type seven?

"We couldn't possibly do that. The risk is too high."

Type seven was their name for skipping every check the normal process existed to run, shipping the diff, and hoping. That organisation kept two paths to production: one so slow nobody could learn from it, and one so dangerous they saved it for emergencies.

Your observability has a type seven too.

You know the shape of it. Production is misbehaving, your dashboards answer none of the questions you have, so someone adds a log line and ships it. Waits for the deploy. Reads the output. Adds another log line. Or flips debug logging on across a service for twenty minutes and flips it off before the bill arrives. Or attaches a debugger to a live process and holds their breath.

Nobody files that under "the normal process failed". It goes under incident response, where uncomfortable things go to be forgiven.

The talk this story comes from is worth watching in full:

Organisations that get this right never need a type seven. Releasing is already cheap and fast enough that even a real emergency uses the normal path, with the safety still in it. Fast and safe stop being a trade-off. Two paths exist when you plan your way through a system that only yields to learning.

The 88 percent

Bain and Company put the number of business transformation efforts that fall short of their aims at 88 percent. Farley quotes it from Dave Clack's book, and his reading of it is the one I find convincing: at that rate you are not looking at variance. You are looking at a method that does not work.

His diagnosis comes from Cynefin. An organisation that builds software is a complex adaptive system. Two properties define that kind of system: you can see cause and effect only in hindsight, and the system shifts while you interact with it. A fixed multi-step plan is worse than useless in a system like that, because the first step changes the ground the other steps were standing on.

Give the same feature to two developers and you get two different systems, and you cannot say in advance which will be easier to change next month. Someone hands in their notice and the project speeds up, slows down, or falls over, and nobody can tell you which beforehand.

Guessing without feedback is how you leave complex and enter chaotic. A type seven debug session is that slide: you add a log line because the data is not there, you wait, you guess again. Always-on instrumentation is what keeps the work in probe, sense, and respond. Run a small change, watch what it does, keep it or step back. Farley lists four conditions that make this possible:

1. Safety to be wrong when you were trying to be right
2. Changes small enough and reversible enough that being wrong is cheap
3. A guess, a way to check the guess, and a short gap between them
4. Language that lands with the people you need on board

Condition one is a leadership behaviour. Condition four is a conversation. Condition two is version control, automated tests, and a deployment pipeline, which most teams have spent a decade getting good at.

He named a practice, not a product. What that practice looks like in the code is a tooling gap, and it is the one that gets skipped.

A change with no check is a gamble

Farley is blunt about it:

"A change that has no expected outcome and no way to measure the result isn't an experiment at all. It's just a change, and that's a gamble that you can't learn from."

Count the changes your team merged last month. Now count the ones where somebody wrote down what they expected to see, and could go and look. If your answer is anything like the teams I work with, the second number is small and clusters around the changes that scared people.

Everything else went out as a gamble the team called continuous improvement.

The reason is timing, and it is boring. Instrumentation gets treated as something you add when you need it, which means you add it once you needed it, which is after the moment you could have learned anything from it. The type seven debug session is what that policy looks like at 2am.

autotel exists to move the check to the same moment as the change. Every example in this post is exercised in the repo, and the assertions exit non-zero when a claim stops holding.

The loop is guess, instrument, change, observe, compare, learn.

Most teams can already guess and change. They can usually observe something. The gaps are instrumenting before you know the question, and comparing what happened against what you expected.

You cannot know which dimension will matter

If cause and effect are visible only in hindsight, then hindsight needs the data to have been there at the time. Picking three fields to log in advance is a bet that you know which three will explain a failure you have not seen yet. In a complex adaptive system, that is a bet you should expect to lose.

So write one event per unit of work carrying the dimensions that describe the work, not the person: plan, region, flag values, payment provider, which cache the read came from, how many items were in the cart. Not because you will look at all of them. Because you cannot know which one you will need. That is the wide event argument I made in Logging Sucks, arriving from the other side: there it was about the questions logs cannot answer, here it is about the questions an experiment cannot answer.

You guess the outcome, which is the thing your change is about: p95, conversion, error rate. You instrument the context, which is the thing that will explain the outcome: plan, region, flag, provider, cart size. Knowing what you are aiming at has never told anyone which variable will turn out to matter.

Redact anything that identifies a person at the boundary. Every field still costs storage and query time, which is what sampling is for, and what bucket() below is for. Neither is a reason to pick three fields up front and hope.

The guess ships with the code

A span is the start. The dimensions and the guess have to go in the same file.

import { trace } from 'autotel';

export const checkout = trace(async (cart: Cart) => {
  const quote = await priceCart(cart);
  const payment = await charge(quote);
  return fulfil(payment);
});

That function is now a span: timing, failures, and a parent for everything it calls. Anything instrumented that runs inside it attaches itself without being handed anything.

Timing and errors tell you a change did not break. They do not tell you whether your guess held, or why. For that you name the dimensions the request ran under, while you are writing the code:

import { withTracing, getRequestLogger } from 'autotel';

export const postCheckout = withTracing({})(
  (ctx) => async (req: Request, res: Response) => {
    const log = getRequestLogger(ctx);

    const user = await getAuth(req);
    const body = await readBody(req);
    log.set({
      user: { id: user.id },
      plan: user.plan,
      cart: { items: body.items?.length },
      pricing: 'v2',
    });

    const result = await processCheckout(user.id, body);
    log.set({ result: { orderId: result.id } });
    log.emitNow();

    return res.json(result);
  },
);

One event per request, carrying the state that request ran under. This costs about thirty seconds while you are already in the file. It costs an incident to add later.

That records what happened. It does not record what you expected, which is the other half of Farley's condition. So autotel has a primitive for the guess itself:

import { trace, experiment } from 'autotel';

export const checkout = trace(async (cart: Cart) => {
  experiment({
    name: 'checkout-pricing',
    variant: 'v2',
    expect: 'p95 drops, conversion holds',
  });

  const quote = await priceCart(cart);
  return fulfil(await charge(quote));
});

Every span in that request now carries experiment.name, experiment.variant and experiment.expectation. The two cohorts you want to compare are selectable from the telemetry rather than reconstructed a week later from deploy timestamps, and the expectation travels with them, so whoever opens the trace in a month reads the claim alongside the result.

That string is not itself a check, and it would be dishonest to call it one. It is the claim, written down at the moment you are least tempted to flatter it, and carried to the place the result will show up. That is the honest measurement he asked for. The checking happens later, in the comparison. Writing it here is what makes the comparison possible at all, because a claim nobody recorded cannot be checked, only remembered generously.

Numeric fields need one more thing before they can explain anything:

import { bucket } from 'autotel/analysis';

log.set({ 'cart.size': bucket(cart.items.length, [1, 5, 20, 100]) });

A raw count takes a near-unique value per request, and a value that never repeats cannot describe a group. Bucketing at instrumentation time is what turns it into a field that can.

Mark, change, compare

Once every request carries its dimensions, probe-sense-respond stops being a metaphor and becomes something you run.

import { compareCohorts } from 'autotel/analysis';

const differences = compareCohorts({
  outlier: slowCheckouts,
  baseline: normalCheckouts,
});

compareCohorts ranks the field and value pairs that separate the group you are investigating from a normal population. It answers the question you have during an incident: what is different about the ones that broke.

Two details in it matter more than the function does. It skips fields whose values never repeat, which is why bucket() exists at the instrumentation end. And its output is labelled a hypothesis, to be confirmed against individual traces. That is incident exploration, not Farley's check. His check is whether the outcome you named showed up. The person in Compare, reading the claim that travelled with the spans, is how you decide you have seen it.

The devtools UI is closer to the loop he describes. Mark a moment. Change something. Compare after against before. The mark is the probe, the comparison is the sense, and what you do next is the respond.

Because experiment() writes the name and the arm onto every span, Compare reads them back: pick checkout-pricing and it offers you its arms, instead of asking you to type two queries and remember what you called the variants.

The speed of the loop is the speed you learn

That short gap is the rest of condition three. Farley puts it in one line: "the speed of that loop is the speed at which your organization can learn."

This is where most observability stacks lose. Your telemetry lives in a vendor's backend, so to see the result of a change you deploy it, wait for the pipeline, wait for ingestion, open a dashboard, and by then it is tomorrow and you are on a different problem. The loop is a day long, so the team runs a handful of real experiments a year and calls the rest "delivery".

autotel-devtools puts an OTLP receiver on your machine. Spans arrive as you exercise the code, in a UI with a query language over them:

service = api AND duration > 100
name CONTAINS checkout
http.status_code = 500

The loop is seconds. Every field a service emits is queryable without being declared anywhere, so a question you did not anticipate does not need a schema change and a deploy before you can ask it.

A team that can answer a question in seconds asks more questions. That is the whole mechanism.

A production experiment still waits on a deploy and on whatever backend you ship to, and no receiver running on your laptop changes that. What it changes is what you ship. You find out on your machine that the field you needed is missing, while adding it costs thirty seconds, instead of finding out from a dashboard a week after the change went out and the window closed. Production is not where you rehearse the question. It is where the answer matters.

You cannot experiment on a part of the system you cannot see

Some of your code emits nothing. Not the parts you would guess, either. npx autotel map records every entry point and scores what it emits. Coverage lists the handlers that have emitted nothing: the places where you can change them, ship them, and learn nothing. A missing map 404s rather than looking clean, because a false bill of health is worse than no report.

The trace you cannot afford to lose

Sampling is what makes always-on instrumentation affordable, and it is how you lose the declined payment that looked like the other ten thousand. A request carrying baggage: autotel.debug=1 keeps its trace with no deploy and no debug logging across the service. That is a better type seven, not the end of them. Farley's version is that the emergency path disappears because the normal one is already fast and safe enough: if the request that broke already emitted its plan, its provider and its cart size, you are reading, not reproducing.

Survival is not mandatory

Deming said it, and Farley quotes it: it is not necessary to change, because survival is not mandatory.

The companies that come through a decade of this are not the ones that predicted it. Nobody predicts a complex adaptive system. They are the ones that got cheap at finding things out, and stayed cheap at it while everyone else was building a business case for a dashboard.

I did not write autotel to produce traces. Traces are a means. I wrote it so that the gap between changing something and knowing what you changed collapses to something short enough that your team runs experiments without scheduling them, and so that the answer is waiting when the question arrives rather than being bolted on at 2am by someone who is guessing.

Farley ends with a swap: stop asking how do we get this right, start asking how do we learn what to do next.

That second question has a precondition nobody says out loud. It assumes your system can answer. Most cannot, which is why so many teams ask the first question instead. It is the only one their tooling can hear.

Every API in this post ships in autotel 7.2.0 and later.

The loop runs in apps/example-experiment: 800 checkouts across both arms of a pricing experiment, one planted cause for the slow ones, and an assertion that compareCohorts finds it without being told. Sampling, coverage, --baseline and --min-score have their own checks, in that app and in apps/example-hono.

They exit non-zero when a claim stops being true, which is the bargain this post is making: a claim with no way to check it is a gamble.

autotel is open source and works with any OTLP backend: github.com/jagreehal/autotel.

If your team still equates "we adopted OTel" with the three-pillars stack, read OpenTelemetry Is Not the Three Pillars (draft) and run apps/example-pillars-vs-unified. Put an LLM in the call stack and the same rule holds: You Can't Ship Agents Like Software (draft).

The type seven story, the four conditions, and the Cynefin framing are Dave Farley's, from How to Build a Team of Devs That Improves Itself on the Modern Software Engineering channel. The 88% figure is Bain and Company's, quoted in Dave Clack's The Case for Connection.