Instrument Before You Know the Question

Dave Farley tells a story about an organisation that took months to ship a release. Staged, layered, careful. While someone walked him through the process, he asked what happens when production breaks and a fix has to go out now.

"Oh, we can do a type seven release in under an hour."

So why not run every release as a type seven?

"We couldn't possibly do that. The risk is too high."

Type seven was their name for skipping every check the normal process existed to run, shipping the diff, and hoping. That organisation kept two paths to production: one so slow nobody could learn from it, and one so dangerous they saved it for emergencies.

Your observability has a type seven too.

You know the shape of it. Production is misbehaving, your dashboards answer none of the questions you have, so someone adds a log line and ships it. Waits for the deploy. Reads the output. Adds another log line. Or flips debug logging on across a service for twenty minutes and flips it off before the bill arrives. Or attaches a debugger to a live process and holds their breath.

Nobody files that under "the normal process failed". It goes under incident response, where uncomfortable things go to be forgiven.

The talk this story comes from is worth watching in full:

Organisations that get this right never need a type seven. Releasing is already cheap and fast enough that even a real emergency uses the normal path, with the safety still in it. Fast and safe stop being a trade-off. Two paths exist when you plan your way through a system that only yields to learning.

Instrumenting Message Queues? Autotel Handles the Ceremony

The OneUptime team is spot on in their Instrument Message Queues with OpenTelemetry post.

Inject trace context on the producer, extract on the consumer; use PRODUCER and CONSUMER span kinds; set semantic conventions (messaging.system, messaging.destination.name, messaging.operation, Kafka partition/offset/consumer group).

They show the raw OpenTelemetry code. It's comprehensive. It's also verbose. Every team ends up re-implementing the same patterns: inject, extract, span kinds, semantic attributes, error handling.

We've all been there: copying "best practice" code from blog posts and adapting it for our broker.

Their key insight:

For batch processing, use a batch span with links or child spans to contributing traces.

But there's still a gap...

Message Isolation? Autotel Makes Tenant Context Flow

The Signadot team is spot on in their Testing Event-Driven Architectures with OpenTelemetry post.

Message isolation using a shared queue: propagate tenant ID in Kafka message headers; consumers use tenant ID for selective message consumption.

They make the case that infrastructure duplication is expensive. Instead of separate Kafka clusters per environment, use tenant ID filtering on a shared queue. Instrument producers and consumers for context propagation.

We've all been there: maintaining four "identical" Kafka setups that slowly drift apart.

Their key insight:

Requires modifying consumers and using OpenTelemetry for context propagation.

But there's still a gap...

Request-Level Isolation? Autotel Propagates Context Automatically

The CNCF team is spot on in their Testing Asynchronous Workflows using OpenTelemetry and Istio post.

Request-level isolation is the most cost-effective approach.

They make the case against duplicating infrastructure for testing. Instead of spinning up separate Kafka clusters per tenant, use OpenTelemetry Baggage to propagate tenant ID through async flows. Consumers filter by tenant ID. Istio handles routing.

We've all been there: every team has their own "staging Kafka" and costs balloon.

Their key insight:

Use OpenTelemetry Baggage to propagate tenant ID through sync and async. When publishing to Kafka, producers inject trace context (including baggage) into message headers; consumers extract and make routing decisions.

But there's still a gap...

End-to-End Tracing? Autotel Makes It Automatic

The OSO team is spot on in their End-to-End Tracing in Event Driven Architectures post.

Traces break at queues unless you extract context from message headers and put it in the appropriate context.

They walk through the real pain: stateful processing loses trace context in caches, Kafka Connect can only do batch-level tracing, and every team ends up writing custom interceptors and state store wrappers.

We've all been there.

Their key insight:

In Kafka Streams and Kafka Connect this often means manual work: interceptors, state stores, batch spans, or extending tracing logic to extract from headers.

But there's still a gap...

Logging Sucks. Autotel Makes Wide Events the Default

Boris is spot on in his Logging Sucks post

logs are optimised for writing, not querying

He explains why debugging in production feels like archaeology.

You grep for user-123, find it logged 47 different ways, then spend an hour correlating timestamps across services.

We've all been there.

His wide event example nails it:

{
  "user": {"id": "user_456", "subscription": "premium", "lifetime_value_cents": 284700},
  "cart": {"item_count": 3, "total_cents": 15999, "coupon_applied": "SAVE20"},
  "payment": {"method": "card", "provider": "stripe", "latency_ms": 1089},
  "error": {"type": "PaymentError", "code": "card_declined", "stripe_decline_code": "insufficient_funds"}
}

One event. High-cardinality keys (user.id, traceId). High dimensionality. Queryable.

But there’s still a gap…