Arrange Act Assert

The Best Consultancies Make Themselves Less Necessary

Thu, 17 Sep 2026 00:00:00 GMT

I’m preparing for a talk with Equal Experts, and it’s got me thinking about consultancy.

I’ve worked for consultancies, alongside them, and inside companies while consultancies have come in to assess them.

Some experiences have been brilliant.

Others have made me uncomfortable.

And the more I’ve seen, the more I think the difference between a good consultancy and a bad one often has little to do with technical ability.

It comes down to culture, honesty, empathy and transparency.

But above all, it comes down to incentives.

Every consultancy has to make money. Of course it does.

But how it makes that money matters.

The business model does the pulling

If more people, more billable days and longer engagements mean more revenue, there’s an obvious tension.

An assessment can become a sales pipeline.

A problem can become another engagement.

A gap in a team can become an opportunity to place somebody.

You don’t need bad people for this to happen.

Sometimes the business model does the pulling for you.

I wouldn’t have done it like that

Years ago, I was part of a consultancy engagement that I still think about today.

I can’t name the consultancy or the company involved, but our mandate was clear: go in, assess the organisation, find the problems, make recommendations and create openings to bring in more of our own people.

I watched us do real damage.

We found fault with the code, the testing, the processes, the architecture and then the people.

But we had something they didn’t.

Hindsight.

We hadn’t lived through their deadlines, changing requirements, budget constraints, people leaving or the compromises that had got them there.

It’s easy to arrive afterwards and say:

“I wouldn’t have done it like that.”

Of course you wouldn’t.

You already know how the story ends.

That’s why Norm Kerth’s Prime Directive has stuck with me: start from the belief that people did the best they could with the information, skills, resources and circumstances they had at the time.

You can still call out bad decisions.

You start with curiosity rather than judgement.

He wasn’t full stack

One example from that engagement has always stayed with me.

There was a backend engineer who was good at what he did. He knew the system, cared about the product and delivered value.

We invented a problem: he wasn’t “full stack”.

There was no problem to solve.

The company already had people doing the frontend work. He was a backend engineer, enjoyed being one and was good at it.

He left.

That left a gap.

A gap a consultancy could fill.

I’ve written about that engineer before, from the team’s side, in good products die when people become resources.

That bothered me then.

It bothers me even more now.

Sometimes the best solution is boring

I’ve seen the same thing with technology.

A straightforward problem needs microservices, CQRS, event sourcing or whatever happens to be interesting at the time.

It looks like CV-driven development.

I love new technology. Our job as engineers is to solve problems, though, and a clever system is not the same as a solved problem.

Sometimes the best solution is boring.

The same instinct hires a scrum master for a team that’s already working, or adopts a framework nobody asked for. I’ve seen what that did to one team, and I’ve written about why you should understand the problem before you change the process.

A consultancy that finds problems which don’t exist is doing the same thing, with a bill attached.

They don’t land and expand

I’ve also seen consultancy done properly.

Consultants who listen before telling, who understand why something exists before trying to replace it, and who share what they know so the existing team gets stronger.

They don’t create dependency.

That is what I like about Equal Experts.

One of their values is to put customer success ahead of their own short-term interests. They also explain what that means in practice.

They say they don’t “land and expand”.

If they’re no longer the right solution, they should leave. When they run out of ways to add value, they aim to leave the customer in a better place than they found them.

That is a different incentive.

Their engineering principles struck a chord with me too: understand the context, choose the right tool for the job and avoid over-engineering.

That is good consultancy.

Come in. Understand. Help. Share what you know. Leave people stronger than you found them.

And know when to leave.

Same idea, different subject

It’s also why I’m pleased to be doing this talk with Equal Experts.

It’s called: Magic vs Science: Making AI Coding Agents Prove Their Work

The talk is about moving past AI coding tricks and witchcraft, and using engineering practices to make generated code prove itself: testing, feedback, maintainability and observability.

Equal Experts’ own engineering principles talk about fast feedback, well-tested code and proactive monitoring.

Different subject, same idea.

Good engineering shouldn’t rely on magic.

It should demonstrate its value.

Test WebMCP tools like an API (with Playwright)

Mon, 07 Sep 2026 00:00:00 GMT

The Chrome team’s best practices and use cases guides end on evaluation tests.

Evals cover the agent’s judgement. Did it pick the right tool, did it fill the form the way the user meant, did it stop when it should have asked. You can’t hard-code those outcomes, which is why the guides send you to evaluation-driven development.

Underneath that sits a layer that is deterministic and dull. Once your page calls registerTool, it advertises callable functions with typed schemas to whatever agent is driving. That’s an API. Get an enum wrong and an agent walks a path your buttons never allow, and the eval that catches it will report that the model got confused. Your UI tests will never see it.

I measured what Chrome 152 does when an agent calls these methods.

The API you didn’t know you shipped

This is the imperative API: registerTool in script. The same contract test applies when Chrome synthesises the schema from toolname on a form.

The cart page registers two, add_to_cart and get_cart. This is the one with a schema worth testing:

await document.modelContext.registerTool({
  name: 'add_to_cart',
  description: 'Add a quantity of one catalogue item to the cart',
  inputSchema: {
    type: 'object',
    properties: {
      sku: { type: 'string', enum: ['espresso', 'cold-brew'] },
      qty: { type: 'number' },
    },
    required: ['sku'],
  },
  execute: (input) => JSON.stringify(addToCart(input.sku, input.qty ?? 1)),
});

Read that as a developer and it’s a function. Read it as an agent and it’s a contract: these are the arguments, this one is mandatory, and the only two SKUs in the world are espresso and cold brew. The agent builds its call from the schema before it ever touches your handler.

A typo in that enum is a production bug your UI tests will never see. Your buttons don’t read the schema. The agent reads nothing else.

Two problems before you write a single test

Bundled Chromium has no WebMCP. Playwright ships its own Chromium, and document.modelContext isn’t in it. Your everyday suite can’t see the API at all.

Chrome ships it behind flags, on a moving draft. The surface has already moved once, from navigator to document. A suite pinned to today’s behaviour goes stale without telling you.

Solve the first by faking the browser and you inherit the second: your fake drifts away from Chrome and your tests keep passing. So card 42 runs two lanes.

Lane	Browser	Runs	Proves
default	bundled Chromium, `document.modelContext` double	every `pnpm test`	your page’s tools: schemas, handlers, and the state they share with the UI
native	real Chrome with the flags on	on demand	the double still matches the browser

You’ll spend your time in the default lane. It runs on any machine, needs no flagged browser, and tests the code you wrote. The native lane exists to stop the double turning into fiction.

The everyday lane

The double goes in through addInitScript, which runs before any page script, so the page finds document.modelContext exactly where it expects it. Reverse that order and the page registers nothing.

Then you test the tools the way an agent would use them.

The schema is the contract, so assert on it.

test('a tool schema names the arguments an agent must send', async ({ page }) => {
  const [tool] = await page.evaluate(() => document.modelContext!.getTools());

  const schema = JSON.parse(tool!.inputSchema);

  expect(schema.required).toEqual(['sku']);
  expect(schema.properties.sku?.enum).toEqual(['espresso', 'cold-brew']);
});

Note the JSON.parse. The spec stringifies inputSchema on the descriptor for the imperative API, so tool.inputSchema.properties is undefined and a test that forgets this passes for the wrong reason.

One cart, two doors. An agent calling a tool and a user clicking a button have to land in the same state. This is the assertion I’d fight to keep if I could only have one:

test('a tool call and a click reach the same cart', async ({ page }) => {
  await page.getByTestId('add-cold-brew').click();

  await page.evaluate(async () => {
    const tools = await document.modelContext!.getTools();
    const add = tools.find((tool) => tool.name === 'add_to_cart')!;
    await document.modelContext!.executeTool!(add, JSON.stringify({ sku: 'espresso' }));
  });

  await expect(page.getByTestId('cart-count')).toHaveText('2');
  await expect(page.getByTestId('cart')).toContainText('1 x cold-brew');
  await expect(page.getByTestId('cart')).toContainText('1 x espresso');
});

A tool that reports success while the cart stays empty has lied to the agent, and only the UI assertion catches it. The best practices guide asks you to update the interface once a function completes, because agents read the interface to plan their next step. This is that bullet, written as a test.

The browser that never shipped the API. Most of your users are on one. Drop the double and check the page still sells coffee:

await page.addInitScript(() => {
  Object.defineProperty(document, 'modelContext', { configurable: true, value: undefined });
});

The best practices guide also asks you to register tools when a page state makes them useful and unregister them after. An AbortSignal owns that lifetime, which is how a component drops its tools on unmount. Assert both halves, because a tool that outlives the screen it belongs to is an agent calling into a page state you’ve already torn down:

test('an AbortSignal drops the registration it owns', async ({ page }) => {
  const names = await page.evaluate(async () => {
    const controller = new AbortController();
    await document.modelContext!.registerTool(
      {
        name: 'temporary_probe',
        description: 'Registered for the length of one controller',
        execute: () => 'ok',
      },
      { signal: controller.signal },
    );
    const before = (await document.modelContext!.getTools()).map((t) => t.name);
    controller.abort();
    const after = (await document.modelContext!.getTools()).map((t) => t.name);
    return { after, before };
  });

  expect(names.before).toContain('temporary_probe');
  expect(names.after).not.toContain('temporary_probe');
});

That one runs in the native lane. The everyday lane could run it against the double, but I am the one who taught the double to drop a tool on abort, so the test would only ever agree with itself. A claim about the browser has to be checked against the browser.

What Chrome actually does

Measured against Chrome 152.0.7977.65, not read off the explainer:

Behaviour	What Chrome does
`getTools()` order	sorted by name, ignoring registration order
`inputSchema` on a descriptor	a JSON string, so parse before asserting
`executeTool(tool, input)`	the draft takes an object; Chrome 152 wants a JSON string, and an object rejects with `UnknownError`
handler returns an object	serialised, so `{a:1}` arrives as `'{"a":1}'`
handler returns `undefined`	arrives as the string `"undefined"`
handler throws	the call rejects with `UnknownError`, and your message is replaced with a generic one
duplicate tool name	`InvalidStateError`
`AbortSignal` on registration	aborting withdraws the tool

The throwing case is the one that will cost you an afternoon. Your handler raises Unknown SKU: tea, and the agent receives none of that. It gets a rejection and a sentence about a script function that failed.

The guides tell you to validate strictly in code and loosely in schema, and to write errors the model can self-correct from. That advice only works if you return the error rather than throw it:

execute: (input) => {
  try {
    return JSON.stringify(addToCart(input.sku, input.qty ?? 1));
  } catch (error) {
    return `Could not add that item: ${error.message}`;
  }
}

The double is a claim

A test double says “this is what the browser does”. Nothing keeps that claim true except reading the original again, which is the native lane’s only job. It drives real Chrome with the flags on, and the files end in .contract.ts so the everyday run skips them.

It earned its place while I was building the card. My double resolved a throwing handler with the error message as text, which is what the WebMCP libraries I’d read do: they catch, and hand the agent something readable. I credited the browser for library behaviour. The first native run rejected instead, and both lanes now assert what Chrome does.

When one of these fails, suspect the double before the browser.

test('the browser owns the context, not a page script', async ({ page }) => {
  const shape = await page.evaluate(() => ({
    registerTool: document.modelContext!.registerTool.toString(),
    onPrototype: Object.getOwnPropertyNames(
      Object.getPrototypeOf(document.modelContext!),
    ).sort(),
  }));

  expect(shape.registerTool).toContain('[native code]');
  expect(shape.onPrototype).toContain('registerTool');
});

That test fails if the double leaks into the native lane, which is the failure mode that would quietly invalidate everything else.

Two traps

A data: URL has no WebMCP. Its origin is opaque, and document.modelContext is absent there in a browser that exposes it on every real page. Probe on http://localhost. I lost an hour concluding the feature was missing from a Chrome that had it.

Version checks lie. >= 152 passes on a build with the flags off and fails on a future build that renames nothing. Ask the browser, on the page you’re about to test:

const available = await page.evaluate(
  () => typeof document.modelContext?.registerTool === 'function',
);
test.skip(!available, 'This Chrome exposes no document.modelContext');

Then feature-detect executeTool on its own. The draft puts it on ModelContext, but Chrome’s testing surface is behind WebMCPTesting. A browser can expose registration without the hook, or the hook can disagree with the IDL, which is why the native lane exists.

Where the evals start

None of this tells you whether an agent picks add_to_cart over search_products, whether your description reads as an execution or an initiation, or whether an agent can complete the shopping list on the use-cases page without adding the same item to two carts. Those are the questions the Chrome guides point at, and they need an agent in the loop.

They also assume the layer underneath holds. Every eval you run against a wrong enum measures your model’s patience with your schema.

pnpm test:webmcp            # default lane, no special browser
pnpm test:webmcp:native     # native lane, needs a Chrome with WebMCP

The native lane wants a Chrome that ships WebMCP, launched with --enable-experimental-web-platform-features and --enable-features=WebMCPTesting,DevToolsWebMCPSupport. Point CHROME_BIN at it if the default search misses yours.

The pattern is also an agent skill, playwright-webmcp. It wires the two lanes into a repo and carries the Chrome behaviours as a reference file the agent has to read before it writes the double. That’s the part I’d otherwise expect an agent to get wrong, because the libraries it has seen catch handler errors and return the message as text, and it will write that down as browser behaviour. Mine did.

WebMCP is a draft under active discussion, so some of the table above will age. The native lane is how I’ll find out when it does.

The card: src/42-webmcp-tools
Live card docs: Card 42
Live harness: /webmcp
The skill: skills/playwright-webmcp
The explainer: webmachinelearning/webmcp
Chrome’s guides: best practices, use cases, build tools, secure tools

Instrument Before You Know the Question

Tue, 01 Sep 2026 00:00:00 GMT

Dave Farley tells a story about an organisation that took months to ship a release. Staged, layered, careful. While someone walked him through the process, he asked what happens when production breaks and a fix has to go out now.

“Oh, we can do a type seven release in under an hour.”

So why not run every release as a type seven?

“We couldn’t possibly do that. The risk is too high.”

Type seven was their name for skipping every check the normal process existed to run, shipping the diff, and hoping. That organisation kept two paths to production: one so slow nobody could learn from it, and one so dangerous they saved it for emergencies.

Your observability has a type seven too.

You know the shape of it. Production is misbehaving, your dashboards answer none of the questions you have, so someone adds a log line and ships it. Waits for the deploy. Reads the output. Adds another log line. Or flips debug logging on across a service for twenty minutes and flips it off before the bill arrives. Or attaches a debugger to a live process and holds their breath.

Nobody files that under “the normal process failed”. It goes under incident response, where uncomfortable things go to be forgiven.

The talk this story comes from is worth watching in full:

Organisations that get this right never need a type seven. Releasing is already cheap and fast enough that even a real emergency uses the normal path, with the safety still in it. Fast and safe stop being a trade-off. Two paths exist when you plan your way through a system that only yields to learning.

The 88 percent

Bain and Company put the number of business transformation efforts that fall short of their aims at 88 percent. Farley quotes it from Dave Clack’s book, and his reading of it is the one I find convincing: at that rate you are not looking at variance. You are looking at a method that does not work.

His diagnosis comes from Cynefin. An organisation that builds software is a complex adaptive system. Two properties define that kind of system: you can see cause and effect only in hindsight, and the system shifts while you interact with it. A fixed multi-step plan is worse than useless in a system like that, because the first step changes the ground the other steps were standing on.

Give the same feature to two developers and you get two different systems, and you cannot say in advance which will be easier to change next month. Someone hands in their notice and the project speeds up, slows down, or falls over, and nobody can tell you which beforehand.

Guessing without feedback is how you leave complex and enter chaotic. A type seven debug session is that slide: you add a log line because the data is not there, you wait, you guess again. Always-on instrumentation is what keeps the work in probe, sense, and respond. Run a small change, watch what it does, keep it or step back. Farley lists four conditions that make this possible:

Safety to be wrong when you were trying to be right
Changes small enough and reversible enough that being wrong is cheap
A guess, a way to check the guess, and a short gap between them
Language that lands with the people you need on board

Condition one is a leadership behaviour. Condition four is a conversation. Condition two is version control, automated tests, and a deployment pipeline, which most teams have spent a decade getting good at.

He named a practice, not a product. What that practice looks like in the code is a tooling gap, and it is the one that gets skipped.

A change with no check is a gamble

Farley is blunt about it:

“A change that has no expected outcome and no way to measure the result isn’t an experiment at all. It’s just a change, and that’s a gamble that you can’t learn from.”

Count the changes your team merged last month. Now count the ones where somebody wrote down what they expected to see, and could go and look. If your answer is anything like the teams I work with, the second number is small and clusters around the changes that scared people.

Everything else went out as a gamble the team called continuous improvement.

The reason is timing, and it is boring. Instrumentation gets treated as something you add when you need it, which means you add it once you needed it, which is after the moment you could have learned anything from it. The type seven debug session is what that policy looks like at 2am.

autotel exists to move the check to the same moment as the change. Every example in this post is exercised in the repo, and the assertions exit non-zero when a claim stops holding.

The loop is guess, instrument, change, observe, compare, learn.

Most teams can already guess and change. They can usually observe something. The gaps are instrumenting before you know the question, and comparing what happened against what you expected.

You cannot know which dimension will matter

If cause and effect are visible only in hindsight, then hindsight needs the data to have been there at the time. Picking three fields to log in advance is a bet that you know which three will explain a failure you have not seen yet. In a complex adaptive system, that is a bet you should expect to lose.

So write one event per unit of work carrying the dimensions that describe the work, not the person: plan, region, flag values, payment provider, which cache the read came from, how many items were in the cart. Not because you will look at all of them. Because you cannot know which one you will need. That is the wide event argument I made in Logging Sucks, arriving from the other side: there it was about the questions logs cannot answer, here it is about the questions an experiment cannot answer.

You guess the outcome, which is the thing your change is about: p95, conversion, error rate. You instrument the context, which is the thing that will explain the outcome: plan, region, flag, provider, cart size. Knowing what you are aiming at has never told anyone which variable will turn out to matter.

Redact anything that identifies a person at the boundary. Every field still costs storage and query time, which is what sampling is for, and what bucket() below is for. Neither is a reason to pick three fields up front and hope.

The guess ships with the code

A span is the start. The dimensions and the guess have to go in the same file.

import { trace } from 'autotel';

export const checkout = trace(async (cart: Cart) => {
  const quote = await priceCart(cart);
  const payment = await charge(quote);
  return fulfil(payment);
});

That function is now a span: timing, failures, and a parent for everything it calls. Anything instrumented that runs inside it attaches itself without being handed anything.

Timing and errors tell you a change did not break. They do not tell you whether your guess held, or why. For that you name the dimensions the request ran under, while you are writing the code:

import { withTracing, getRequestLogger } from 'autotel';

export const postCheckout = withTracing({})(
  (ctx) => async (req: Request, res: Response) => {
    const log = getRequestLogger(ctx);

    const user = await getAuth(req);
    const body = await readBody(req);
    log.set({
      user: { id: user.id },
      plan: user.plan,
      cart: { items: body.items?.length },
      pricing: 'v2',
    });

    const result = await processCheckout(user.id, body);
    log.set({ result: { orderId: result.id } });
    log.emitNow();

    return res.json(result);
  },
);

One event per request, carrying the state that request ran under. This costs about thirty seconds while you are already in the file. It costs an incident to add later.

That records what happened. It does not record what you expected, which is the other half of Farley’s condition. So autotel has a primitive for the guess itself:

import { trace, experiment } from 'autotel';

export const checkout = trace(async (cart: Cart) => {
  experiment({
    name: 'checkout-pricing',
    variant: 'v2',
    expect: 'p95 drops, conversion holds',
  });

  const quote = await priceCart(cart);
  return fulfil(await charge(quote));
});

Every span in that request now carries experiment.name, experiment.variant and experiment.expectation. The two cohorts you want to compare are selectable from the telemetry rather than reconstructed a week later from deploy timestamps, and the expectation travels with them, so whoever opens the trace in a month reads the claim alongside the result.

That string is not itself a check, and it would be dishonest to call it one. It is the claim, written down at the moment you are least tempted to flatter it, and carried to the place the result will show up. That is the honest measurement he asked for. The checking happens later, in the comparison. Writing it here is what makes the comparison possible at all, because a claim nobody recorded cannot be checked, only remembered generously.

Numeric fields need one more thing before they can explain anything:

import { bucket } from 'autotel/analysis';

log.set({ 'cart.size': bucket(cart.items.length, [1, 5, 20, 100]) });

A raw count takes a near-unique value per request, and a value that never repeats cannot describe a group. Bucketing at instrumentation time is what turns it into a field that can.

Mark, change, compare

Once every request carries its dimensions, probe-sense-respond stops being a metaphor and becomes something you run.

import { compareCohorts } from 'autotel/analysis';

const differences = compareCohorts({
  outlier: slowCheckouts,
  baseline: normalCheckouts,
});

compareCohorts ranks the field and value pairs that separate the group you are investigating from a normal population. It answers the question you have during an incident: what is different about the ones that broke.

Two details in it matter more than the function does. It skips fields whose values never repeat, which is why bucket() exists at the instrumentation end. And its output is labelled a hypothesis, to be confirmed against individual traces. That is incident exploration, not Farley’s check. His check is whether the outcome you named showed up. The person in Compare, reading the claim that travelled with the spans, is how you decide you have seen it.

The devtools UI is closer to the loop he describes. Mark a moment. Change something. Compare after against before. The mark is the probe, the comparison is the sense, and what you do next is the respond.

Because experiment() writes the name and the arm onto every span, Compare reads them back: pick checkout-pricing and it offers you its arms, instead of asking you to type two queries and remember what you called the variants.

The speed of the loop is the speed you learn

That short gap is the rest of condition three. Farley puts it in one line: “the speed of that loop is the speed at which your organization can learn.”

This is where most observability stacks lose. Your telemetry lives in a vendor’s backend, so to see the result of a change you deploy it, wait for the pipeline, wait for ingestion, open a dashboard, and by then it is tomorrow and you are on a different problem. The loop is a day long, so the team runs a handful of real experiments a year and calls the rest “delivery”.

autotel-devtools puts an OTLP receiver on your machine. Spans arrive as you exercise the code, in a UI with a query language over them:

service = api AND duration > 100
name CONTAINS checkout
http.status_code = 500

The loop is seconds. Every field a service emits is queryable without being declared anywhere, so a question you did not anticipate does not need a schema change and a deploy before you can ask it.

A team that can answer a question in seconds asks more questions. That is the whole mechanism.

A production experiment still waits on a deploy and on whatever backend you ship to, and no receiver running on your laptop changes that. What it changes is what you ship. You find out on your machine that the field you needed is missing, while adding it costs thirty seconds, instead of finding out from a dashboard a week after the change went out and the window closed. Production is not where you rehearse the question. It is where the answer matters.

You cannot experiment on a part of the system you cannot see

Some of your code emits nothing. Not the parts you would guess, either. npx autotel map records every entry point and scores what it emits. Coverage lists the handlers that have emitted nothing: the places where you can change them, ship them, and learn nothing. A missing map 404s rather than looking clean, because a false bill of health is worse than no report.

The trace you cannot afford to lose

Sampling is what makes always-on instrumentation affordable, and it is how you lose the declined payment that looked like the other ten thousand. A request carrying baggage: autotel.debug=1 keeps its trace with no deploy and no debug logging across the service. That is a better type seven, not the end of them. Farley’s version is that the emergency path disappears because the normal one is already fast and safe enough: if the request that broke already emitted its plan, its provider and its cart size, you are reading, not reproducing.

Survival is not mandatory

Deming said it, and Farley quotes it: it is not necessary to change, because survival is not mandatory.

The companies that come through a decade of this are not the ones that predicted it. Nobody predicts a complex adaptive system. They are the ones that got cheap at finding things out, and stayed cheap at it while everyone else was building a business case for a dashboard.

I did not write autotel to produce traces. Traces are a means. I wrote it so that the gap between changing something and knowing what you changed collapses to something short enough that your team runs experiments without scheduling them, and so that the answer is waiting when the question arrives rather than being bolted on at 2am by someone who is guessing.

Farley ends with a swap: stop asking how do we get this right, start asking how do we learn what to do next.

That second question has a precondition nobody says out loud. It assumes your system can answer. Most cannot, which is why so many teams ask the first question instead. It is the only one their tooling can hear.

Every API in this post ships in autotel 7.2.0 and later.

The loop runs in apps/example-experiment: 800 checkouts across both arms of a pricing experiment, one planted cause for the slow ones, and an assertion that compareCohorts finds it without being told. Sampling, coverage, --baseline and --min-score have their own checks, in that app and in apps/example-hono.

They exit non-zero when a claim stops being true, which is the bargain this post is making: a claim with no way to check it is a gamble.

autotel is open source and works with any OTLP backend: github.com/jagreehal/autotel.

If your team still equates “we adopted OTel” with the three-pillars stack, read OpenTelemetry Is Not the Three Pillars (draft) and run apps/example-pillars-vs-unified. Put an LLM in the call stack and the same rule holds: You Can’t Ship Agents Like Software (draft).

The type seven story, the four conditions, and the Cynefin framing are Dave Farley’s, from How to Build a Team of Devs That Improves Itself on the Modern Software Engineering channel. The 88% figure is Bain and Company’s, quoted in Dave Clack’s The Case for Connection.

The $1 House Still Needs a Site Survey

Mon, 24 Aug 2026 00:00:00 GMT

Matt Pocock got Uncle Bob on a livestream, and the bit I keep coming back to is the house.

Bob asks you to imagine that every change to a house costs a dollar. The foundation, the roof, moving the kitchen, moving the stairs. All of it, a dollar.

Would you pay an architect thousands to get the plan perfect before the contractor starts? Or would you tell them to put the foundation there, walk through it, decide the traffic pattern is wrong, and move the stairs?

He is right about the plan. He leaves out the other half of what an architect does.

What Bob gave up, and what he kept

Bob is blunt about heavy up front specification. He tried it, he was trying it again the week of the interview, and he calls the outcome the same every time: you write the plan, the agents run off half cocked on some nonsense the plan never anticipated, and you stop, rewrite, and start them over.

The agents themselves make this worse, because they love writing plans. The plans come back gorgeous and full of things nobody can build. So he went back to the agile loop. Do a story or two, look at what came out, reorganise, do a bit more.

What he did not give up is constraint.

Later in the same conversation he describes a file that pins which module is allowed to depend on which, and how the dependencies should flow. A checker runs at the end of the loop.

If the agents violated the rules they have to fix it, usually by inverting a dependency or splitting a module in half. He runs crap scores over the code and mutation testing over the tests, and the agents sit in that loop until the tools go green.

Ask him about specifications and he says they are ephemeral and he throws them away. The dependency rules and the thresholds are the things he keeps.

The distinction worth pulling out is prescription versus context.

Matt raises this in the conversation and lets it go. He asks how much planning happens before the gauntlet starts, because putting the wrong work through it is wasteful, and he says out loud that building the wrong thing is still possible. Bob answers with the history of waterfall and the agile correction, which is a good answer to how much planning and not an answer to how you know it is the right work. The gauntlet is a delivery machine. Crap scores, mutation testing and a dependency checker will tell you the thing was built well. None of them can tell you whether it was worth building.

The architect’s other job

An architect does not exist because moving a wall is expensive. Before anyone decides where the walls go, someone has to know whether the plot floods, which way the prevailing wind comes from, where the sun rises, what the planning authority will allow, where the utilities run, whether anyone living there uses a wheelchair, and whether the family will still fit in the house in twenty years.

None of those answers get cheaper because the contractor works for a dollar a change. You can iterate at enormous speed and end up with a beautifully built house facing the wrong way on a floodplain. You are placing a bet either way. The survey is what tells you the odds.

The interview nobody passes by typing

There is a version of this you have probably sat through. You walk into a principal engineer system design interview and the panel says: design a system for X. If your first answer is “let’s start coding and discover the architecture as we go”, you have not shown them any system design yet.

What you do instead is ask. How many users? What is the read to write ratio? What consistency guarantees do we need? What happens when a region fails? What data are we storing and where is it allowed to live? What is the growth curve? Which parts of this cannot go down?

They are the forces acting on the architecture, and they decide which implementations are even in the running.

They get to this themselves. Matt reaches for John Ousterhout’s split between tactical and strategic programming, and both of them stop to pull the book off the shelf. Bob takes it as given: agents are strong at the tactical work and weak at the strategic. That list of questions is the strategic work. The agents did not make any of it cheaper.

Context that compiles

Bob’s reason for preferring tools over prompts has a name: lost in the middle. Anything you write at the top of a long context loses its grip as the window fills. A checker that fails the build does not, which is why I keep putting standards in linters instead of prompt files. Sort your context into two piles.

The first pile compiles. Module dependency rules in a file with a checker behind them, the way Bob does it. Complexity and coverage thresholds, picked so a green gate means something. A performance budget expressed as a test that fails at 200ms. Data residency as a lint rule that rejects an import of the wrong region’s client. A public contract pinned by contract tests. Agents obey these because they run into them, not because they read them.

The second pile does not compile. What are we optimising for. Which constraints are fixed and which are negotiable. What this system has to become when the thing the company announced last quarter actually ships. Who this is for and what happens to them when it breaks. No checker exists for any of that, and you cannot leave it in a prompt and expect it to survive. It also decides whether the foundations hold: what you know is coming changes how you build now.

Bob’s answer about juniors sits in that second pile. Asked how someone starting out learns the strategic half, he suggested they spend a few months being treated like an agent, given agent sized tasks and put through the same deterministic gauntlet, “horribly unproductive but learning a hell of a lot”.

His own judgement came from somewhere else. He can spot an agent thrashing because he has watched systems thrash for fifty years. A principal engineer asks about regional failure because they were on call the night a region went down, and about data residency because they have watched a regulatory constraint arrive long after the architecture was set. The tactical work was the apprenticeship that produced the context.

There is a second question underneath the one about juniors learning to code. Where does the next generation get its context from, if the ten years of implementation that used to supply it are the part we are handing to the agents? I do not have a good answer yet.

Direction is not a plan

The vision does not need to be two hundred pages. It can be a paragraph.

This system will serve millions of users across several regions, hold regulated financial data, and keep taking payments while a region is down.

That paragraph says nothing about the implementation. It rules out a large class of the implementations an agent would otherwise produce. You can still throw code away six times an hour inside it, which is the part Bob and Matt are right about. I still would not put customers in that house, for the same reason I would not put them in a vibe-coded prototype.

That paragraph is a fence. It says what the system must never do. It does not say what anyone is trying to achieve, and an agent held inside a fence will build something nobody wanted, and build it well, at speed.

When a customer’s card fails, they keep their service and understand what happened, without having to call us.

Matt has shown us something important about working with agents: we need to watch their tendency to go heads down too quickly. Give an agent a prompt and it starts writing code. Put it in plan mode and it produces a plan that often reads like the code in prose. In both cases, it has gone straight to cutting.

What’s missing is what Brooks called the design concept: the underlying picture of the solution that sharpens through conversation before implementation begins.

Matt’s grill-me is that conversation. The agent interviews until every branch is visited and nothing is left silently assumed. You work collaboratively with the agent to understand why the thing should exist, what outcome it should produce and which decisions follow from that. It does not build until you confirm that you share the same understanding.

Then it executes while that conversation is still in context. It’s measure twice, cut once, even when cutting costs almost nothing.

Back to the house

The contractor moves any wall for a dollar, so use that. Do not spend six months deciding whether the kitchen island sits 20cm to the left. Build it, walk around it, move it.

Before the first dollar, though, somebody has to say: we are on a floodplain, the wind comes off the west, one of the people living here uses a wheelchair, and we would like the house standing in thirty years.

This is the site survey. It covers the ground, and it covers who is going to live on it: a family who cook together every night and have never once eaten in the dining room.

One half tells you what the plot will allow, the other what the house is for.

Neither half is a floor plan. A contractor who moves any wall for a dollar can satisfy both while you are still arguing about the island.

As the cost of producing code falls towards nothing, working out the forces and where the system is heading is a larger share of what engineers are paid for.

You Can't Support What You Didn't Record

Mon, 27 Jul 2026 00:00:00 GMT

Support asks a year after the payment: “Why did the customer receive €117? And which card did we charge?”

A supportable system answers that with one query. If answering means trawling archived logs, guessing what the exchange rate was that day, and hoping nobody edited the payment method since, you don’t have a supportable system.

Record the decision at the point you make it, and never destroy the data it points to.

Two rules

Rule 1: never overwrite reference data. A payer’s stored payment method is reference data. When the payer changes it, insert a new row. When they remove it, set deletedAt. No UPDATE, no DELETE, ever.

Rule 2: snapshot the decision inputs on every transaction. At the moment a payment executes, write down everything the decision depended on: which payment method, what rate, what amounts. Write the values as they were at that moment, on the payment record itself.

I covered how the payment flow gathers those inputs into a context object in Tracking Every Decision in Payments. This post is about writing that context onto the durable record so support can still read it a year later.

A payment record is a claim about the past, and if the data underneath it can change, the claim stops being true.

Both rules protect that claim.

Internal data: pointer plus snapshot

For data you own, keep both. The transfer record stores a pointer into the reference table and a snapshot of what the pointer resolved to at execution time:

type PaymentMethod = {
  id: string;
  type: 'card' | 'bank_account';
  last4: string;
  deletedAt: string | null; // soft delete only
};

type TransferRecord = {
  transferId: string;
  // ...
  paymentMethodId: string; // pointer, resolvable forever
  paymentMethod: { type: string; last4: string }; // what it was at execution time
};

The pointer works because of Rule 1. The snapshot means support skips the join: the record itself says “card ending 4242”, and keeps saying it. When a dispute or audit needs the full reference row, soft-deleted or not, the pointer still resolves.

External data: capture the value

The exchange rate came from someone else’s API. You can’t foreign-key into a third party’s history, so the value you captured is the only record you will ever have:

type TransferRecord = {
  // ...
  rate: number; // the rate we converted at
  rateCapturedAt: string; // when we fetched it
  convertedAmount: number; // what it produced
};

The workflow stamps the rate the moment it fetches it:

const rate = await fetchRate({ from: 'GBP', to: 'EUR' });
// External data: capture it now, we can't point at the provider's history later
const rateCapturedAt = new Date().toISOString();

A month or a year later, “why €117?” has an answer: rate 1.17, captured at 14:53:12 UTC, applied to £100. Call the rate API at support time instead and you get today’s rate, a different number answering a different question.

Model output: the worst external data of all

An exchange rate at least has a provider you could argue with. A model call has nothing. Same prompt, same model, different answer, and no history to refetch. If you didn’t record it, it never happened.

So record the whole step: the model and version, the prompt, the tool calls it asked for, the arguments it sent, what each tool returned, and the final text.

type ModelStep = {
  model: string;
  prompt: string;
  toolCalls: Array<{
    toolName: string;
    input: unknown; // exactly as the model sent it
    output?: unknown; // absent if the tool never ran
    error?: string;
  }>;
  text: string;
  usage: { inputTokens: number; outputTokens: number };
};

The interesting field is the one that isn’t there. Here is a real run against a local llama3.2 with a calculator tool, arguments typed as z.number():

{
  "type": "tool-call",
  "toolName": "calculator",
  "input": { "a": "23", "b": "7", "op": "add" },
  "invalid": true,
  "error": "AI_InvalidToolInputError: expected number, received string at path a"
}

The model sent strings. Validation rejected them, so execute never ran. Then the model answered:

Using the calculator tool, I get: 23 + 7 = 30

It did not use the calculator tool. It guessed, and it happened to guess right. finishReason was stop. Nothing threw. Every dashboard stayed green.

That is the whole argument for recording the step. The final text is the model’s account of what it did, and the model is not a reliable narrator. The tool call record is evidence. It shows the arguments the model actually produced, and its missing output shows the tool never executed. The next question is only answerable because you kept it: was 30 arithmetic or a coin flip?

Record the absences too, then. A tool that was offered and never called, a step that ended early, a validation failure that resolved itself. null is a finding.

What goes wrong without this

Someone updates or hard-deletes a payment method row. Historic payments claim the wrong card, or point at nothing. Nothing errors. A chargeback investigation dead-ends eighteen months later.
Someone answers a rate query by calling the rate API at support time. They report today’s rate to the customer, and it’s wrong.
Someone logs only the model’s final text. A month later a customer disputes the number in it, and there is no way to tell whether a tool computed it or the model invented it. Re-running the prompt produces a different answer, which settles nothing.

Wide events, same idea

I’ve written about how wide events beat scattered logs: one canonical event per request, carrying all the context that request accumulated.

You apply the same discipline to storage. The wide event snapshots a request; the transfer record snapshots a transaction. The workflow writes the same values to both:

span.setAttributes({
  'transfer.id': transfer.transferId,
  'transfer.rate': rate,
  'transfer.rate.captured_at': rateCapturedAt,
  'payment_method.id': paymentMethod.id,
  'payment_method.last4': paymentMethod.last4,
});

Model steps go on the same event: gen_ai.request.model, gen_ai.tool.name, gen_ai.usage.input_tokens. The span shows you this week’s latency spike; the record answers next year’s “did the tool run?”

Keep both. Telemetry retention runs out in days or weeks, so the wide event serves this week’s incident. The transfer record outlives it and serves next year’s dispute.

The payoff

Support needs one endpoint:

app.get('/api/transfer/:id', (c) => {
  const record = getTransferRecord(c.req.param('id'));
  return record ? c.json(record) : c.json({ error: 'NotFound' }, 404);
});

{
  "transferId": "TXN-001",
  "requestedAmount": 100,
  "fromCurrency": "GBP",
  "toCurrency": "EUR",
  "rate": 1.17,
  "rateCapturedAt": "2026-07-22T14:53:12.000Z",
  "convertedAmount": 117,
  "paymentMethodId": "pm_001",
  "paymentMethod": { "type": "card", "last4": "4242" },
  "status": "completed"
}

In production, put the endpoint behind auth and scope the lookup to the record’s owner; the record answers questions for support, not for whoever guesses an ID.

Every decision sits next to the inputs you wrote when you made it. A year later, “why €117, and which card?” is one authenticated GET.

Good products die when people become resources

Mon, 13 Jul 2026 00:00:00 GMT

Throughout my career the one thing I’ve watched kill good products is a single word: resource.

Once a company calls its people resources, it starts treating them like resources. It adds them, rotates them, splits them across teams.

The product is the first thing to suffer.

Yes, you want a team. The dynamic of that team decides far more than the org chart admits.

Chemistry you cannot buy

People talk about forming, storming, norming and performing, and then forget what it means. A team is not a list of job titles. It is a living system, and when the team changes the system changes with it. One common reading of Tuckman’s model is that a new member can push a team back into storming.

The chemistry between people either works or it does not, and you cannot buy it after the fact. It reminds me of arranged marriages. Back in the eighties, when I was growing up, they were commonplace, and I assumed one was the path for me too. People would grow to love each other, we were told. But I saw enough unhappiness to know chemistry mattered. In my own family and the world around me, that has changed a lot. People are far more likely to choose their own partners now, and families have had to change their culture and thinking to accept it.

Good product teams work the same way. There is a line from Simon Sinek I like: “A team is not a group of people who work together. A team is a group of people who trust each other.” You can assemble one backend engineer, one frontend engineer, one tester and one product person on a spreadsheet and call it a team. You have capacity on paper.

Trust is a separate thing, and it takes time.

The person I would choose again

I have worked with someone I would choose again tomorrow. At one company they were seen as too quiet. At another they did not work out. For me they were one of the best people I have worked with.

Not for agreeing with me. For how they challenged me. The way they set out an argument worked. The way they shared what they knew worked. The way they pushed back made the product better, without ego getting in the way. We put the business first, and we put the product first, and the work clicked. You need conflict, but it has to be the right conflict.

Marty Cagan makes the case for durable, empowered product teams, and SVPG has long argued against outsourcing core competencies when they are central to the business. His phrase for the teams that work is missionaries, not mercenaries. His measure is outcomes over output. You lose both when you move people around like pieces on a board.

The cost lands on a person

I have been on teams where the people making the staffing calls sat nowhere near the day-to-day work, and their bad choices did not stop at the team. They landed on a person.

A brilliant backend engineer was told to become full-stack. He did not want to be. It was not what he enjoyed, not the books he was reading, not what he spent his own time on. He could not bring himself to care about the front end. But he loved the product, so he accepted it.

It drove him into a downward spiral. He lost motivation, and we all do our best work on the things we love, so the work showed it. He felt trapped. Every task was uphill, work he had never chosen and had no pull towards. He had written React before, and he still got no dopamine from solving a problem in that part of the stack.

He had been added as a resource to fill a full-stack gap, and no one asked what he actually wanted to do. The team suffered. People started to read the mismatch as a performance problem, when it was a role-design problem. He had been placed into a slot, not chosen for the work he was best at. So how was he ever going to do good work on something he did not want to do?

There is a place for disagree and commit. This was not it. Adding him that way, and bending the team around a role no one needed to fill like that, was bad for the product and worse for him. It was bad for the company too. He walked away not long after.

Product thinking or resource thinking

That story is the difference between product thinking and resource thinking. The two ask different questions. Product thinking asks whether a change improves the outcome, protects context, and helps the team understand the customer. Resource thinking asks whether you can add a body, rotate a body, or fill a gap. Those are not the same question, and I now sort companies by which one they reach for first. The best product culture I have been part of came from people, not process, which I wrote about in creating a product culture at Cambridge University Press.

A lot of the confusion comes from ideas companies have read but never lived. The Spotify model is the obvious one, a way of working that even Spotify does not use, which I wrote about in before you change the process, understand the problem. These hyped ways of organising teams can look right on paper. Before you adopt one, ask who actually benefits from it.

AI raises the stakes. It is already part of everyday development, and the more raw production work it absorbs, the more the human work shifts towards judgement, taste, context and attention to detail. Dotting the i’s and crossing the t’s stops being the small part of the job and becomes the job. So the question sharpens: what is the resource for, and what outcome do you expect from it?

How the decline starts

I have worked at companies I would call fantastic product companies, and I have watched some of them decline. The decline starts quietly.

Leadership decides to bring in offshore teams, or rotate people, or split the work across locations, without thinking about the dynamic. The problem is not geography. The problem is breaking context and chemistry while pretending nothing has changed. In a planning deck it reads as efficient. In the room, the product suffers. I once saw the dynamic of a team die inside a day.

Before that, testers, engineers, product people and stakeholders solved problems together. Testers had a real voice, treated as first-class members of the team rather than a service on the side. I had spent a long time breaking down the silos the Jira lanes enforced, which is the work I wrote about in the team I built before I fixed anything. Then the resource thinking arrived. People got added without care. The work fell back into silos. Context leaked away. The people who cared started to care less. The harmony broke. People who had respected each other found it hard to work together.

That is the part leaders miss. When you treat the team casually, the team sees it, and that carelessness spreads into the decisions and into the product. Fred Brooks named the mechanism decades ago in The Mythical Man-Month: “Adding manpower to a late software project makes it later.” Nine women cannot produce a baby in one month.

Some work does not get faster because more people have been assigned to it. Coordination, onboarding and communication all take time, and they can make a late project later. We still believe people are interchangeable units of delivery, and they are not.

Consulting runs on the same incentive

This is not only a product-company problem. Consulting has its own version of the same trap.

I have spent years as a consultant across many consultancies, and the model there is plain. The more consultants on a project, the more the consultancy earns. That does not automatically make it right for the company paying the bill. They care about the product, not the staffing model.

I have since written about what that incentive looks like from inside a consultancy in the best consultancies make themselves less necessary.

The line often attributed to Charlie Munger applies here: “Show me the incentive and I will show you the outcome.” The consultancy’s incentive and the client’s incentive are not the same, and everyone involved should be honest about that.

Knowledge without churn

The same holds for the bus factor. You do not want all the knowledge trapped in one head, so pairing and documentation matter. Knowledge sharing works when it comes from a stable team, not from constant churn. Spreading knowledge and disturbing the team are two different acts.

AI can help with the bus factor, but only if the team uses it deliberately. It can keep documentation fresh, capture decisions, explain workflows, and remove some of the mundane work. Done well, it helps knowledge stay in the system. But it does not replace the judgement, taste and context of the people building the product.

If the value sits in judgement, shared context and knowing why past decisions were made, you cannot rotate it in every three months. I hold six months as the minimum contract for serious product work. You need that time to understand the product, its history, the decisions taken, and the decisions the team chose against. Some companies understand this instinctively. Your opinion in week one is worth a dollar. Three weeks in it is worth three. Two months in, five. The context that makes it worth anything never shows up on a resourcing plan.

Autonomy and fit

Not everyone fits everywhere, and that is fine. Two people click, another two are chalk and cheese. Neither is a bad person. The chemistry is wrong for that context, and the product carries the cost when every conversation turns into work and every decision turns into conflict. I have watched that curdle into office politics and a bitterness that spreads.

The product shows it. Instead of something clear and coherent, you get a run of compromises made by people who are tired, frustrated, and ready to give up so they can move on. People stop pushing. The product turns into a mishmash of clashing patterns and styles and half-finished initiatives, a platform that hardens into its own kind of minefield over time.

The alternative is worth the discomfort. Resource thinking optimises for the average, and average is where products go to die. I would take a team with real chemistry, even if it does not last forever, over a safe, interchangeable one that never risks anything. Better to have built something you were proud of and watched it end than to have never built it at all.

Amy Edmondson defined team psychological safety as “a shared belief that the team is safe for interpersonal risk taking.” That is not about being nice. It is about people speaking honestly, challenging well, and disagreeing without it turning personal, so the loudest voice does not run the room.

The product has to come first, and for it to come first the team has to work. Communication has to flow, the way I described in teams should communicate like a team fielding in cricket.

Get the chemistry right and the safety sits there, invisible, so everyone keeps their eyes on the product instead of on the friction.

If you hold any power over how teams are formed, be careful with the word resource, and with the arbitrary rotation that tends to follow it. You would not choose a partner from a spec sheet. Do not choose a team from one either.

Chemistry is not the soft part of the job. It is often what decides whether a good product survives.

Good questions protect the room

Mon, 06 Jul 2026 00:00:00 GMT

Our software craftsmanship group encourages new speakers to come and practise their talks.

One is coming up in August. At the start of every meetup we read aloud the code of conduct.

It can feel formal, but the repetition is the point. It sets the culture and reminds us who we want to be.

A question can be good and still land badly. The timing and the room decide.

Something happens when someone gives a talk for the first time, or steps out of their shell because public speaking can be terrifying. A lot of it comes down to the questions people ask.

The question that helps gives something back to the speaker and never leaves them feeling small or put off speaking again. A question that does the opposite is the anti-pattern, because it teaches the whole community to stay quiet, and the discussions and the respect we have for each other are the reason the group works at all.

When good questions land badly

A couple of months ago I spoke at an AI meetup. The speaker before me had built an app with AI, and the app had security flaws. Someone in the audience spotted them and kept asking, question after question, until the speaker looked uncomfortable.

The questions were good. I’m sure the speaker would have welcomed the feedback in a code review, or over coffee. Asked in that room, at that moment, nobody else put their hand up. The room went quiet. The speaker felt awful.

The Recurse Center, a programming retreat, has a useful rule for one smaller version of this: no “well-actuallys”, which they define as a correction with “no bearing on the crux of the conversation”. Correctness matters. The harm is that some corrections move a room from learning towards status.

The security questions at the AI meetup were more serious than a minor correction, but the same lesson holds. A valid point can land in a way that shuts the rest of the room down.

From where I was sitting, it seemed unlikely the speaker could answer those questions well in that setting, so I won’t judge what the audience member should or shouldn’t have said. I care about the effect it had, on the speaker, and on every other person in the room who might have been weighing up their own first talk and decided against it.

Questions that are assumptions

Some questions are assumptions in disguise. The person has reached a conclusion and wraps it in a question mark to make it land.

Edgar Schein called the opposite of this humble inquiry, “the fine art of drawing someone out, of asking questions to which you do not know the answer.” He subtitled the book The Gentle Art of Asking Instead of Telling, and that’s the line.

A question you know the answer to is an instruction. Even a genuine question carries a risk the asker doesn’t see. It can derail the talk. The question can be valid on its own terms and pull the room away from the point the speaker came to make. Ten minutes later you’re deep in an edge case that mattered to one person, and the thread the speaker spent weeks preparing has gone.

What makes a question good

A good question helps everyone in the room. It’s the one where someone leans over afterwards and says, “I’m glad you asked, I was wondering the same thing.” It doesn’t put the speaker on the spot, it furthers their learning, and yours.

Sometimes it’s the single clarification the talk needs to land, because one undefined term changes the whole meaning.

Zenger and Folkman spent years studying what great listeners do. They found that “people perceive the best listeners to be those who periodically ask questions that promote discovery and insight,” and that “these questions gently challenge old assumptions, but do so in a constructive way.”

A good question leaves the speaker ready to move forward.

Amy Edmondson gave this a name in 1999: psychological safety, “a shared belief held by members of a team that the team is safe for interpersonal risk taking,” a confidence that the team “will not embarrass, reject, or punish someone for speaking up.”

Edmondson’s talk on the same idea, The Fearless Organization, is worth watching:

Standing up to talk is an interpersonal risk. So is asking a question, admitting you don’t know something, or floating a half-formed idea. Protect that risk and people take it again. Punish it once in public and they stop.

Candor has to be built in

As Ed Catmull wrote in his book “Creativity, Inc.”:

“A hallmark of a healthy, creative culture is that its people feel free to share ideas, opinions and criticisms… without the critical ingredient that is candor, there can be no trust. And without trust, creative collaboration is not possible.”

Pixar built a structure for that candor, the Brain Trust. It is a regular meeting where a director gets honest feedback on a film in progress. The comments are suggestions, not orders, and they are aimed at the work, not the person. It is the same thing a good question does at a meetup, made into a habit.

Draw quieter voices in

After a meetup once, a good friend pulled me aside. He said it was good that I had the confidence to contribute and ask questions, but my real aim should be to bring the people who weren’t talking into the conversation. That would be the real win. It stuck with me.

Asking a good question is one skill. Drawing in someone who hasn’t spoken is the harder one, and it does more for the room than any point I could make myself. It’s the same move as a fielding team calling each other in, which I compared to a cricket side.

Before you ask

At the next meetup, I run through these in my head:

Can the speaker answer this in the room, right now?
Does this help the room, or only prove you noticed something?
Are you asking because you don’t know, or because you’ve decided?
Who in the room hasn’t spoken yet, and could your question invite them in?

Those keep me from asking the wrong thing. For asking the right thing, I borrow Edmondson’s own set, in two kinds. Some broaden the discussion:

What do others think?
What are we missing?
What other options could we consider?
Who has a different perspective?

Some deepen it:

What leads you to think so?
What is the concern you have about that?
Can you give us an example?
Can you explain that further?
What do you think might happen if we did X?

Notice “who has a different perspective?” rather than “does anyone have a different perspective?”. The first assumes another view is out there and worth hearing. The second sets a hurdle almost no one in a full room wants to clear.

The same forces play out inside companies, where one confident voice can take over the room and everyone pays for it. I wrote about that in The loudest voice is not the smartest group.

The loudest voice is not the smartest group

Mon, 06 Jul 2026 00:00:00 GMT

A confident voice can ask a sharp question at the wrong moment and close a room down. In a meetup, the cost is one nervous speaker and a room that goes quiet. In a company, the same habit compounds, meeting after meeting, until the same few people decide what the team is allowed to say out loud.

You see the pattern at conferences first. The people who are confident keep speaking. The people finding their voice hold back. Over time the room narrows. The same voices return, the same ideas circulate, and the different takes never get a hearing.

When one voice dominates

Anita Woolley and her colleagues measured what this does to a group. In their 2010 study in Science, a group’s collective intelligence had little to do with the smartest person in it. It tracked social sensitivity and whether people took turns or one voice dominated.

Groups where a few people dominated the conversation showed less collective intelligence than groups where more people took turns. Let the loudest voices dominate and the group gets duller.

The room with the cleverest individual is not the room that thinks best. You want the room where more people get to speak.

Meetings that go quiet

I’ve sat in too many meetings where the same few people talk, the leader says “any questions?”, and the silence pushes everyone to the anonymous form.

Sometimes anonymity is right. If someone’s job is on the line, or the company is about to change under people’s feet, of course. For the day to day, you have to give people room to speak and build the culture that makes them want to.

A quick yes is a warning

A room can go quiet in another way. Everyone nods, the plan passes, and it looks like agreement. Amy Edmondson uses a cartoon of a management team waving a decision through, every one of them saying aye, while their thought bubbles say the opposite. In a real meeting you cannot see the thought bubbles, but they are there.

Alfred Sloan ran General Motors on the opposite instinct. When his senior team agreed a big decision on the spot, he told them he took it they were all in agreement, then proposed they postpone it to the next meeting to give themselves time to disagree. The quick yes was the part that worried him.

Drive out fear

W. Edwards Deming put driving out fear at the centre of how he thought about management, not only in high-stakes moments but in the everyday. He made it his eighth point: “Drive out fear, so that everyone may work effectively for the company.”

I sit on the Deming side of this. You hire smart people for their ideas. No single person, however senior, has thought of every angle of a problem, and in a busy week the viewpoints are the reason you hired them. The sharing shouldn’t depend on the leader organising it either.

A team should be able to give each other the space and the time to speak. That’s the same habit I compared to a cricket side fielding together, sharing information out loud and backing each other up.

I’ve been on both sides

Just before Covid I gave a talk on how I build apps at a local UX meetup. The room was mostly designers. They asked how I worked with their mockups and handoffs, I answered, and we both came away with something.

A month later I ran the same deck at a company I’d just joined. Six colleagues watched the practice session and liked it. At the company-wide session the organiser told the room this was meant to be a discussion, not a presentation, then interrupted me within the first few minutes. She talked over my answers for the rest of it. At the end she told the room I’d wasted everyone’s time.

Afterwards several attendees messaged me to apologise for her. What I didn’t understand at the time was that nobody there had done these sessions before. I was the first.

Show and tell beats abstract talks

That was alien to me, because I’d come from a place where we ran a regular show and tell and everyone was encouraged to present. I did one on TypeScript and Tailwind.

I learned more from watching colleagues talk through their own approaches than any conference could teach me, because it was about our domain and the code we shipped.

People quote Conway’s Law about systems mirroring the organisations that build them. This is the other half of it. Sharing ideas and arguing about how things work, out loud, breaks down the walls between teams.

Talks tied to what someone shipped on Tuesday land. Abstract ones don’t.

Count the voices

At your next meeting, count how many people speak and how many ask a question. Note the number. Your job is to make it bigger next time.

One way in is an icebreaker, if you’re leading, or you can suggest one to whoever is. Skip the cheesy and personal ones. Most people dread sharing a childhood fun fact before a sprint review, and it ignores their privacy. Keep it trivial. Ask the room whether it’s jam or cream on a scone first.

There’s no right answer and nothing at stake, so anyone can answer, and in half a minute the whole room has spoken once.

What matters is that each person has used their voice early, before anything is on the line. When someone speaks, the rest of the room notices they are there. On a video call that counts for more than it should. The software puts the current speaker front and centre and buries the rest, so in a big meeting you can go the whole hour without seeing half the people in it. One round of small answers pulls the quiet ones back onto the screen, and into the room.

The loudest voice can make a room feel decisive. It also makes the room duller. Build rooms where more people speak.

I wrote about the meetup side in Good questions protect the room.

Claude Code sandbox: close the gaps before you auto-allow

Tue, 30 Jun 2026 00:00:00 GMT

Most Claude Code sandbox guides sell convenience. Fewer prompts, smoother flow, less time approving commands.

But what about the blast radius?

When an AI agent runs a command, it does not only run the command you had in mind. It runs every child process, package script, setup hook and recovery step that command sets off.

A helpful agent reads a README, installs a dependency, retries a failed setup step, and turns “get this project running” into “execute whatever this project tells me to execute.”

The sandbox is a boundary, not a trust button. And the boundary only protects you to the extent you configure it.

Stop thinking of it as a productivity feature

Describing the sandbox as a productivity feature is tempting. Without it, Claude asks before running most commands. With it, sandboxed Bash commands run on their own. You get fewer interruptions and a smoother session.

True, and the least interesting part.

The question I want answered is the one a permission prompt never asks: if this command goes bad, what can it still read, where can it write, and who can it talk to?

That is the model I care about. Not “do I trust Claude,” not “did I approve this one command,” not even “does the repo look clean.” A coding agent does not run isolated commands. It explores, retries, follows instructions, reads error messages, installs packages, and tries to recover when a stranger’s script fails. That is the point of using an agent, and it is also where the danger lives.

Give an agent full access to your machine and every package script and setup step inherits that access. The agent may act in good faith. The process tree it starts will not.

The sandbox moves the unit of trust. Instead of trusting each command the moment it appears, you decide the boundary up front. Inside it, development moves fast. Outside it, the operating system says no.

A clean repo is not the same as safe execution

This is not hypothetical. In late June 2026 Mozilla’s 0DIN team published a proof of concept where a normal-looking GitHub repository walks an AI coding agent into opening a reverse shell, with nothing malicious sitting in the repo for a reviewer to catch.

The chain is what makes it work. A Python package is built to fail on first use and tells the agent to run an init command. That command resolves an attacker-controlled DNS TXT record, decodes the value it gets back, and pipes it to bash.

The payload, a reverse shell, never lived in the repository. The agent fetched and ran it at runtime, and the shell ran with the developer’s own access to environment variables, API keys and local config. Tom’s Hardware and BleepingComputer both covered it, and 0DIN found Cursor, Copilot and Gemini CLI vulnerable to versions of the same trick.

No single evil file appears anywhere. The damage comes from a chain: a boring-looking repo, normal setup instructions, a failed command, a helpful recovery step, a payload pulled in at runtime, and a shell with your privileges. Agents are good at completing exactly that chain.

A human reviewing a command judges it locally. You see npm install, composer install, or a tool-specific setup step, and you approve what is in front of you. The agent acts on the next step. The sandbox is where you cap what that whole execution tree can reach.

What the sandbox covers

The Claude Code sandbox applies to Bash commands and their child processes. That scope matters. When a sandboxed command runs, you can restrict what it writes, what it reads if you configure that, which network destinations it reaches, and whether a failed sandboxed command may retry outside the sandbox.

On macOS it uses Apple’s Seatbelt framework. On Linux and WSL2 it uses bubblewrap for filesystem isolation and a proxy for network control. Native Windows is not supported, so on Windows you run Claude Code inside a WSL2 distribution to get the Linux sandbox.

That is a useful boundary, not a magic one. It does not turn arbitrary code into trusted code, does not put every Claude Code tool in a container, does not inspect encrypted traffic, and does not know on its own which of your files are secrets. You configure the boundary you want.

The write boundary is a sensible default

By default, sandboxed commands write to the current working directory and the session temp directory. That covers most of what an agent needs: a build tool writing generated files, a test runner writing temporary output, a package manager updating project files, a framework scaffolding routes or migrations.

What it stops is a command casually rewriting your shell config, replacing a system binary, or dropping files somewhere unrelated to the project. Most of the agent’s real work lives in the repository. Let it work there, and do not hand it the rest of your machine by accident.

The read gap is the part people miss

Here is where I want you to slow down. The sandbox limits writes well by default. It does not limit reads the same way.

Do not assume a sandboxed command cannot read a sensitive file just because it cannot write outside the project. Without extra configuration, the default read policy still reaches files you would never want a package script or a hijacked setup step to see:

~/.ssh
~/.aws/credentials
cloud provider config
npm tokens
GitHub tokens in the environment
other credentials under your home directory

This is the gap between “the sandbox is enabled” and “the sandbox is useful for my threat model.” A process that can read your credentials and reach the network does not need write access to ruin your day. It reads what it finds and sends it out. So the first serious change is to close reads.

Start with credentials

Claude Code gives you a dedicated sandbox.credentials block (v2.1.187 and later). It denies reads of named credential files and unsets sensitive environment variables before each sandboxed command runs. This is the minimum I would start with:

{
  "sandbox": {
    "enabled": true,
    "autoAllowBashIfSandboxed": true,
    "credentials": {
      "files": [
        { "path": "~/.aws/credentials", "mode": "deny" },
        { "path": "~/.ssh", "mode": "deny" },
        { "path": "~/.config/gcloud", "mode": "deny" },
        { "path": "~/.npmrc", "mode": "deny" }
      ],
      "envVars": [
        { "name": "AWS_ACCESS_KEY_ID", "mode": "deny" },
        { "name": "AWS_SECRET_ACCESS_KEY", "mode": "deny" },
        { "name": "AWS_SESSION_TOKEN", "mode": "deny" },
        { "name": "GITHUB_TOKEN", "mode": "deny" },
        { "name": "NPM_TOKEN", "mode": "deny" }
      ]
    }
  }
}

Treat this as the baseline, not as caution. If an agent starts a process you did not expect, that process has no business inheriting your most useful secrets.

Then close the home-directory read path

For a stronger project setup, deny reads from the home directory and re-allow the project. Put this in the project’s .claude/settings.json, not your user-level ~/.claude/settings.json, because . only resolves to the project root when the config lives in project settings.

{
  "sandbox": {
    "enabled": true,
    "filesystem": {
      "denyRead": ["~/", "./.env", "./.env.*", "./secrets"],
      "allowRead": ["."]
    }
  }
}

There is a trap in allowRead: ["."]. Re-allowing the project pulls every project file back into reach, including .env and anything under secrets/, so deny those paths in the same block. This is where the layer distinction bites. A permissions rule like Read(./.env) blocks Claude’s own Read tool, and it does nothing to a sandboxed cat .env in a postinstall script. Only a filesystem.denyRead entry stops Bash, because the OS enforces it whatever the command does. Keep both: the permission rule for Claude’s tools, the sandbox rule for everything Bash spawns. That gives you the boundary most people assume the word “sandbox” already includes.

A config for daily development

For normal web work I want the agent to move quickly inside the repo: run tests, install packages, start a dev server, inspect project files. I do not want it reading secrets or publishing anything without me. Here is what I run for a typical Next.js, Vite or React project:

{
  "sandbox": {
    "enabled": true,
    "autoAllowBashIfSandboxed": true,
    "allowUnsandboxedCommands": true,
    "network": {
      "allowLocalBinding": true,
      "allowedDomains": ["registry.npmjs.org", "api.github.com"]
    },
    "filesystem": {
      "denyRead": ["~/", "./.env", "./.env.*", "./secrets"],
      "allowRead": ["."]
    },
    "credentials": {
      "files": [
        { "path": "~/.aws/credentials", "mode": "deny" },
        { "path": "~/.ssh", "mode": "deny" },
        { "path": "~/.npmrc", "mode": "deny" }
      ],
      "envVars": [
        { "name": "GITHUB_TOKEN", "mode": "deny" },
        { "name": "NPM_TOKEN", "mode": "deny" },
        { "name": "AWS_ACCESS_KEY_ID", "mode": "deny" },
        { "name": "AWS_SECRET_ACCESS_KEY", "mode": "deny" },
        { "name": "AWS_SESSION_TOKEN", "mode": "deny" }
      ]
    }
  },
  "permissions": {
    "deny": ["Read(./.env)", "Read(./.env.*)", "Read(./secrets/**)"],
    "ask": ["Bash(git push *)", "Bash(npm publish *)", "Bash(pnpm publish *)"]
  }
}

A few choices are deliberate. autoAllowBashIfSandboxed is on, because the point is to let safe work continue without constant prompts. allowLocalBinding is on, because frontend dev servers bind local ports. Reads of the home directory and the project’s own secrets are denied, the project is otherwise readable, credential files and token variables are blocked, and publishing and pushing still ask. Fast inside the boundary, deliberate at the edges.

The two-domain allowlist is a starting point, not a finished list. A real project usually needs more: registry.yarnpkg.com or a private registry, pypi.org and crates.io for polyglot repos, plus CDN, font and storage hosts your app pulls at build time. Do not try to predict them all. Run with the short list, and when an install or a dev server fails with a host-not-allowed error, add that exact host. The prompt tells you what to add, so you widen the boundary by evidence rather than by guessing a wildcard.

Broad network domains undo the point

Network control is one of the sandbox’s most useful features, and one of the easiest to weaken. The built-in proxy decides access by hostname, which stops a sandboxed command from calling random domains. What it does not do is open TLS and inspect the encrypted contents.

So a broad domain is a broad escape route. github.com is not one destination in practice. It is a platform of user-controlled content, redirects, raw files, releases, issues and gists. Allowing it wholesale gives a payload somewhere to phone home. That is not a reason to ban GitHub, it is a reason to be specific. Prefer this:

{
  "sandbox": {
    "network": {
      "allowedDomains": ["registry.npmjs.org", "api.github.com"]
    }
  }
}

over a wildcard list like ["*.github.com", "*"], which only looks like a boundary.

Specific cuts both ways, though. api.github.com does not cover raw.githubusercontent.com, codeload.github.com, or a git clone over HTTPS, which talks to github.com itself. A repo that installs git-based or private dependencies will hit the escape hatch on every one of those until you add the exact host. That is the trade you want: each host is a decision you made, not a door you left open. For higher-risk environments, route sandbox traffic through a corporate proxy that terminates TLS and applies real egress rules. For local development the rule is simpler: allow the few domains your tools need, and make everything else earn a prompt.

Know what the escape hatch does

Claude Code has a useful fallback. When a command fails because of sandbox restrictions, it can retry the command outside the sandbox through the normal permission flow. For a tool that cannot run sandboxed at all, that saves you turning the whole thing off.

For untrusted code, that fallback is the wrong default. When you are reviewing a random repository, auditing a suspicious dependency, or testing a branch from someone you do not know, switch the hatch off:

{
  "sandbox": {
    "enabled": true,
    "autoAllowBashIfSandboxed": false,
    "allowUnsandboxedCommands": false,
    "network": {
      "allowedDomains": []
    },
    "filesystem": {
      "denyRead": ["~/", "./.env", "./.env.*", "./secrets"],
      "allowRead": ["."]
    },
    "credentials": {
      "files": [
        { "path": "~/.ssh", "mode": "deny" },
        { "path": "~/.aws/credentials", "mode": "deny" },
        { "path": "~/.config/gcloud", "mode": "deny" },
        { "path": "~/.npmrc", "mode": "deny" }
      ],
      "envVars": [
        { "name": "GITHUB_TOKEN", "mode": "deny" },
        { "name": "NPM_TOKEN", "mode": "deny" },
        { "name": "AWS_ACCESS_KEY_ID", "mode": "deny" },
        { "name": "AWS_SECRET_ACCESS_KEY", "mode": "deny" },
        { "name": "AWS_SESSION_TOKEN", "mode": "deny" }
      ]
    }
  },
  "permissions": {
    "ask": ["Bash"],
    "deny": ["WebFetch", "WebSearch", "Read(./.env)", "Read(./.env.*)", "Read(./secrets/**)"]
  }
}

The credentials block matters more here than in daily development, not less. The 0DIN payload went straight for environment variables and local config, and untrusted code is exactly when you want those stripped before any command runs. This is slower by design. With untrusted code you do not want the agent deciding that a failed sandboxed command deserves a retry with more access. You want the failure in front of you so you can choose the next step.

Docker is host access

Docker does not sit neatly inside the sandbox. The Docker CLI talks to the Docker daemon over a Unix socket, and access to that socket is access to the host. Allow a sandboxed process to use the Docker socket and you may have handed back the power you were trying to remove.

So do not solve Docker by punching a hole through the sandbox. Exclude the Docker commands instead, so they run outside it and stay visible through the normal permission flow:

{
  "sandbox": {
    "enabled": true,
    "autoAllowBashIfSandboxed": true,
    "excludedCommands": ["docker *", "docker-compose *"],
    "network": {
      "allowedDomains": ["registry.npmjs.org", "ghcr.io"]
    },
    "filesystem": {
      "denyRead": ["~/", "./.env", "./.env.*", "./secrets"],
      "allowRead": ["."]
    },
    "credentials": {
      "files": [
        { "path": "~/.ssh", "mode": "deny" },
        { "path": "~/.aws/credentials", "mode": "deny" }
      ],
      "envVars": [
        { "name": "GITHUB_TOKEN", "mode": "deny" },
        { "name": "NPM_TOKEN", "mode": "deny" }
      ]
    }
  },
  "permissions": {
    "ask": ["Bash(docker *)", "Bash(docker-compose *)", "Bash(git push *)"],
    "deny": ["Read(./.env)", "Read(./.env.*)", "Read(./secrets/**)"]
  }
}

Docker stays a conscious action, and everything else stays boxed in.

When the agent genuinely needs Docker — building images, running docker compose, spinning up a multi-service stack — excluding those commands trades safety for incomplete work. The follow-on in this series, Docker Sandboxes: when Claude Code needs Docker, covers moving the boundary outward: run Claude Code inside a microVM with its own Docker daemon, instead of punching the host socket through the inner sandbox.

Sandboxing and permissions are different layers

This trips people up. The sandbox controls what a Bash command can access once it runs. Permissions control whether a Claude Code tool runs at all. They are not the same control.

The built-in file tools, Read, Edit and Write, go through the permission system. They do not become OS-sandboxed Bash processes when the sandbox is on. So you still need permission rules alongside the sandbox:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env.*)",
      "Read(./secrets/**)",
      "Read(./private/**)"
    ],
    "ask": ["Bash(git push *)", "Bash(npm publish *)", "Bash(pnpm publish *)"]
  }
}

Four layers do four jobs:

Layer	What it controls
Permissions	What Claude may attempt with any tool
Sandbox filesystem	What Bash can read and write once it starts
Credentials	The secret files and env vars Bash never inherits
Network	Where Bash and its child processes can connect

You want all four. Drop one and the others have a gap to cover for it.

What I would check into a team repo

For a team I would commit a conservative policy at .claude/settings.json so the security posture shows up in code review:

{
  "sandbox": {
    "enabled": true,
    "autoAllowBashIfSandboxed": true,
    "allowUnsandboxedCommands": true,
    "network": {
      "allowLocalBinding": true,
      "allowedDomains": ["registry.npmjs.org", "api.github.com"]
    },
    "filesystem": {
      "denyRead": ["~/", "./.env", "./.env.*", "./secrets"],
      "allowRead": ["."]
    },
    "credentials": {
      "files": [
        { "path": "~/.ssh", "mode": "deny" },
        { "path": "~/.aws/credentials", "mode": "deny" },
        { "path": "~/.npmrc", "mode": "deny" }
      ],
      "envVars": [
        { "name": "GITHUB_TOKEN", "mode": "deny" },
        { "name": "NPM_TOKEN", "mode": "deny" },
        { "name": "AWS_ACCESS_KEY_ID", "mode": "deny" },
        { "name": "AWS_SECRET_ACCESS_KEY", "mode": "deny" },
        { "name": "AWS_SESSION_TOKEN", "mode": "deny" }
      ]
    }
  },
  "permissions": {
    "deny": ["Read(./.env)", "Read(./.env.*)", "Read(./secrets/**)"],
    "ask": ["Bash(git push *)", "Bash(npm publish *)", "Bash(pnpm publish *)"]
  }
}

Then I would add a short note to the README so a new joiner knows the rules: the policy allows normal project work inside the repo, blocks reads from your home directory and the project’s own .env and secrets/, removes common token variables from sandboxed commands, and keeps publishing and pushing behind manual approval. Run /sandbox and select auto-allow for everyday development. For an untrusted branch or a dependency audit, drop to strict mode by setting autoAllowBashIfSandboxed and allowUnsandboxedCommands to false and emptying allowedDomains.

What the sandbox does not solve

The sandbox reduces risk without removing responsibility. It will not stop Claude making a bad edit inside your project, so use git. It will not make a broad network allowlist safe, so keep the list narrow. It will not make Docker harmless, so treat the socket as host access. It will not protect a secret you forgot to block, so add the credential rules that fit your environment. It will not sandbox every Claude Code tool, so the built-in file tools still rely on permission rules. And it will not make untrusted code safe to run. It makes the damage smaller when something goes wrong, which is still worth a lot.

What should the standard be?

It shouldn’t be “Claude asked, and I approved it.”

It is this: if this command misbehaves, it cannot read my home directory, cannot inherit my tokens, cannot write outside the project, cannot reach random hosts, and cannot silently retry outside the sandbox.

That is a better place to work from, and a more honest way to use agents. We are going to let these tools install packages, run tests, generate files, read errors and keep going, because that is where the speed comes from. Autonomy without a boundary is just delegation to whatever the agent meets next. The sandbox lets you draw that boundary and have the operating system enforce it.

So do not turn on auto-allow because you are tired of prompts. Turn it on once the boundary is real: reads closed, credentials removed, network narrowed, escape hatch understood, dangerous actions still behind a prompt. Then let the agent work.

When does a vibe-coded app become a real product?

Thu, 25 Jun 2026 00:00:00 GMT

I’ve seen non-developers build apps that would have taken engineers days, if not weeks.

Not just functional apps, but polished apps with rich functionality that bring joy to users. They meet real needs, and the attention to detail is unmatched.

Product people and domain experts can build good apps, and they build them fast.

So the question is: when does a vibe-coded app become a real product?

That is the question a lot of companies are about to face.

In many businesses, engineers are no longer the only people building the next prototype. Client managers, sales teams, product people and domain experts build it too, using tools like Lovable, Base44, Claude and other AI app builders.

And that is a good thing.

A client explains a problem. A domain expert knows the workflow. They sit together and, instead of writing a long requirements document, they build something. In days, sometimes hours, not the six months a normal roadmap would take.

That moves the conversation from “tell me what you need” to “is this what you mean?”

Every layer in between is a chance to lose the point. A requirements document, a spec, a planning meeting, a handoff to someone who has never spoken to the customer: each one is where the original need turns into something slightly different. The person with the domain knowledge skips those layers. They are close to the customer, in direct contact, and they build what the two of them want.

Andrej Karpathy named the feeling when he coined the term in February 2025: “it’s not really coding — I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.” That speed is intoxicating, and it closes deals that would have died in a six-month backlog.

Part of it is the hit of seeing the idea live. A working app you can click within minutes beats a slide or a spec, and that feeling separates the people who make things from the people who only talk about them. Give someone that hit once and they chase it again.

The instinct is not new. Tim Brown of IDEO has argued for years that you build a rough prototype to think with, and that it beats a thick stack of spec documents. Vibe coding makes that prototype cheaper to build than ever.

Make it work, make it right, make it fast

For years, engineers have said things like:

Make it work. Make it right. Make it fast. (I dug into how that maxim holds up once AI writes the first draft in Make It Work, Make It Right, Make It Fast in the Age of AI.)

Vibe-coded apps are brilliant at the first part. They turn a vague idea into something real that people can click, challenge, reject, improve and sell.

As an engineer, I value testing, linting, accessibility, observability, dependency management, secure deployments and maintainable code.

But I understand why a sales team does not want all of that on day one.

A prototype exists to move fast and to learn. It does not exist to look like a fully governed production system.

As Charity Majors puts it, “the true product of every (good) software engineering team is production.” An app that works in the demo says nothing about month six, when the dependencies are stale, the API key has rotated, and someone files a support ticket.

When the prototype crosses a trust boundary

A prototype is scaffolding. It helps people see the shape of the thing quickly, test the idea, move around it, and decide whether it is worth building properly.

But you do not ask customers to live in the scaffolding.

Agents will rebuild the walls for a dollar, and you will still have a house on a floodplain if nobody did the site survey.

Once the app carries the company logo, connects to real systems, stores customer data or has people depending on it, the business has moved from scaffolding to building, often without anyone deciding that shift has happened.

Someone has to own the foundations, the wiring, the fire exits and the maintenance. The same shift happens when the app starts costing money to run, when someone expects support, when the company’s reputation rides on how good it is.

The specifics bite fast. A prototype humming along on someone’s personal API key turns into a security and billing problem the moment it is business-critical. Exposed keys do real damage: one developer pushed AWS keys to GitHub, bots spun up servers to mine cryptocurrency within hours, and the bill climbed into four figures before he noticed. An app parked on a free Vercel Hobby plan breaks the rules as soon as it earns money: Vercel restricts that tier to “non-commercial personal use only” and pushes commercial projects onto a paid plan. The bill and the liability land on the company, not on the person who built it in an afternoon.

That is the moment the conversation has to change, because the app has crossed a trust boundary. Engineers are not trying to slow things down, and product people have not done anything wrong.

Before that point, it is experimentation. After it, it is software the business is responsible for.

This is where many companies will struggle.

Involve engineering too early and the process becomes the bottleneck vibe coding was meant to avoid. Involve engineering too late and the company inherits risk: security, accessibility, cost, operations, dependencies and support.

A graduation path

Banning vibe coding would be the wrong lesson. So would forcing every vibe-coded app through the same governance as a core banking system or a main customer platform, which kills the value that made it worth building.

What works is a graduation path: lightweight, visible, and clear to the people using it.

For example:

Level 0: Personal build — A live site, but no company data, no client access, no company brand and no production expectations. The API keys belong to the builder, not the business. It is hosted on Lovable or similar, not on company infrastructure.

Level 1: Company prototype — The company sees the value and wants real users to try it. Company branding may appear. Before it ships, someone reviews API key terms, hosting, data use and commercial restrictions. The app is still experimental, but it is no longer purely personal.

Level 2: Paid pilot — The app now needs paid usage above the free tiers, so it costs the company money. The business reviews and approves that spend. Costs are tracked, ownership is clear, and basic security, accessibility and dependency checks apply. Still no company APIs and no company login.

Level 3: Product — Uses real customer data, internal systems, company authentication or paid APIs. Now it enters the software development lifecycle.

A Level 3 app does not automatically need months of process. It means the business has accepted that this is no longer a demo but something the company owns.

The important part is not the exact labels. It is that someone can look at an app and quickly decide what level it is at, who owns it, and what has to happen before it moves up.

The handover problem

Moving an app up that path runs into a practical engineering problem.

If a team builds the app in Lovable and engineers then fix it directly in GitHub, where is the source of truth?

Is it the version in Lovable, or the version in the repository? Can the business team still iterate, or has the app moved into an engineering-owned lifecycle? The handover point decides who owns the next change.

Martin Fowler would call a fast prototype deliberate, prudent technical debt: you ship now and accept you will pay it back later. The trouble starts when that debt turns reckless and invisible, when the app drifts into production and nobody records that the handover happened.

Vibe coding does not remove the need for engineering judgement. Research by Advait Sarkar and Ian Drosos argues it redistributes that judgement toward managing context, evaluating output fast, and knowing when to drop the AI and edit by hand. The handover is where that judgement has to show up.

Do not kill the energy that built it

When the app graduates, remember why it exists. One person saw a problem, felt it, and built something. That vision and that drive are the reason you are here, not the engineering team and not the roadmap.

Most engineers, given the choice, want to build, not run product. Product thinking is its own craft. It takes discipline, customer context and a different set of instincts from engineering. The person who built the prototype may have those instincts because they were closest to the problem.

So the real risk at handover is human, not technical. Take the app away from the person who built it, tidy it up “properly”, and you can drain the energy that made it good. Kill that energy and you kill the product.

Security and operations teams are part of that handover, but they cannot be the whole of it. Their job is to protect the business, keep systems stable and reduce surprises. That matters. But it is not the same as owning the customer problem or carrying the product vision. If the app becomes only a thing to secure, host, monitor and control, the company may make it safer while draining the reason it was worth saving.

The business needs to see this coming. The aim is to keep the initiator driving while engineering makes the thing safe to own, not to hand the whole thing over.

The tools are not the enemy

It would be easy to write these tools off as toys. They are not.

Take Lovable. It syncs two ways with GitHub, so the code can be cloned, edited in your own IDE, or self-hosted. It runs security checks too, from dependency audits to keeping API keys server-side. These builders are moving toward the enterprise problem, not away from it.

So the answer cannot be “stop using them”. Platform guardrails are not the same as company ownership, and that is the line that matters.

A platform can generate the code, deploy it, scan it for obvious holes, and sync it to GitHub. It cannot decide whether the app now represents the company, whether a client treats it as an official product, who pays for the API usage, who gets paged when it breaks, or whether the business accepts the data and reputational risk. Those are company decisions.

This is not a new problem

Vibe coding is the AI-native version of something companies have wrestled with for years: citizen development. Low-code and no-code platforms made the same promise and hit the same wall. Microsoft warns that when security gets cumbersome, people bypass it and drift into “shadow IT”. Gartner makes the point that low-code and no-code speed development but need support and governance so citizen developers can build safely at scale.

The useful framing comes from OWASP, whose Citizen Development Top 10 treats the goal as safe adoption rather than a ban: guardrails that let innovation thrive while keeping security intact. A graduation path is how you bring that spirit to vibe coding.

Engineering as enabler

Engineering’s job here is to enable the people building these apps, not to guard a gate. I made the fuller case for that shift in From Gatekeeper to Enabler, where I argued that engineering adds more value handing citizen developers safe defaults than standing in their way.

There is a practical reason for this too. Every engineering team I have worked on is already at capacity. The roadmap is full, and no spare team is waiting to adopt a stack of vibe-coded apps. Much of what people are building is work that sat in the backlog for months, deferred behind something more urgent. Vibe coding gives those ideas another route to daylight. Funnelling every app through engineering is not feasible, so enablement is the only model that scales.

Give non-technical teams room to create: safe defaults, approved places to host, guidance on API keys, company accounts, data, branding and security, and a fast route to ask for help before the app becomes a liability. And give engineers the context to see why these apps matter.

There is a competitive point here too. If your company makes this too hard, the ideas will not wait. The people closest to the customer will find another way to build, or a competitor will get there first. One missed prototype does not matter much. Repeat the pattern, though, and you lose more than experiments. You lose innovation velocity.

In the AI era, the ability to turn customer insight into working software is becoming a first-class business utility. Security, operations and engineering should make that utility safe to use, not so hard to reach that people route around it. Make it hard and you do not stop the work. You push it into personal accounts, unmanaged tools, or the hands of a competitor who moves faster.

Vibe coding is already happening, so the question is not whether to allow it. The choice companies face is whether to treat it as shadow IT or to build a path where creativity, speed and engineering discipline work together.

Vibe-coded apps should not be blocked. They should be given a clear path, from a quick prototype to a product the company owns. The danger is not that people outside engineering are building software. The danger is that nobody knows when the business has become responsible for it.

Businesses still need to move fast. The trick is knowing when the thing built to learn has become something the company must own.

How sandbox runs risky installs in a throwaway container

Mon, 15 Jun 2026 00:00:00 GMT

@jagreehal/sandbox-node runs your npm install in a throwaway container that can see your project and the registry, and nothing else.

Install scripts still run. node-gyp still builds. Your SSH keys, npm token, cloud credentials, and .env are not in the box, so a malicious dependency has nothing to steal and nowhere to send it.

The one-picture version

Put sandbox in front of the command you already run:

sandbox npm install
sandbox pnpm add zod
sandbox npm run dev

Under the hood the risky code runs in a container that can reach your project files and the npm registry. It cannot reach your home directory, your credentials, or anywhere else on the internet. When the command finishes, the box is deleted and your installed dependencies stay behind.

This is install-time containment, not a general sandbox. It targets the one moment that the previous post is about: when untrusted dependency code executes with your access.

Why this is worth it

You do not have to stop using normal package managers. You do not have to disable install scripts and break native modules. You do not have to read every transitive dependency before you clone a repo.

You keep the workflow you already use. The difference is that install-time code runs somewhere with no secrets to steal, nowhere to phone home, and fewer places to persist. Same workflow, smaller blast radius.

That gives you a handful of practical benefits:

Safer installs without changing package managers
Install scripts still work
Better defaults for agents and CI
Less reliance on perfect dependency review
One tool and one config, not a toolchain to wire up

One attack path, end to end

You add a package. One dependency under it ships a postinstall script. During install, that script reads ~/.npmrc, checks ~/.aws/credentials, writes a hook into .git/hooks, and makes an HTTPS request to a host you have never heard of.

On your host, that works because the script runs as you. In sandbox, each step hits a different wall. The credentials are not mounted. The common persistence paths are read-only. The network request has no route unless you allowed that host.

Three layers: check it, contain it, catch it

A supply-chain worm needs three things to work: a version fresh enough to still be live, something worth stealing, and a way out. sandbox works in three layers that line up against those, so no single bypass defeats it: it checks a package before the box runs, contains the code while it runs, and catches anything that tries to get out.

Check it before the box runs

Before any dependency code executes, sandbox vets what you are about to pull. This is the part that reaches out to external sources to confirm there are no known issues:

Risk hints inspect each direct package before it installs. By default they use the metadata npm already returns, with no extra network calls: install scripts, versions published in the last hours or days, brand-new packages, newly added binaries, deprecations, a name within one or two edits of a popular package (a typosquat, checked against a corpus of around 2,500 names), a release that dropped the npm provenance attestations its earlier versions shipped, and a first-time or long-dormant publisher (a maintainer-takeover signal). The thorough level, which the strict preset turns on, reaches further out to the npm downloads API and DNS to flag missing repository or license metadata, suspiciously low download counts, and a maintainer email domain that no longer resolves (a known route to seizing an npm account). They warn by default; --fail-on-risk makes them block.
A release-age cooldown refuses any version published more recently than your threshold: three days on the balanced and agent presets, seven on strict. Worms detonate within hours, so the install never even resolves a version that fresh, falling back to the last one that has aged past the threshold. It is one of the strongest controls against publish-and-detonate worms, but a cooldown alone leaves gaps, which is why the layers below exist alongside it. Minimum release age is necessary, but not enough walks through where it falls short.
A known-malware check queries the OSV database for the exact version you would pull and refuses anything flagged as malware (MAL-...), even a version old enough to clear the cooldown.
Your own blocklist blocks immediately when you already know a package is bad, without waiting for OSV to catch up: a committed sandbox.advisories.json for your team, plus malware feeds you trust, such as Aikido’s public npm malware list.
sandbox preflight runs all of the above without installing anything, so you can review a package, or a whole pull request’s dependency changes, before it touches your machine.

Contain it while it runs

A check is a signal, not a guarantee, so the vetted install still runs inside the throwaway box. Inside, there is nothing to take and no way out:

No credentials. The container starts almost empty: SANDBOX=1, HOME=/root, and little else. No ~/.ssh, ~/.npmrc, ~/.aws, or home directory is mounted, so a script that greps process.env for an AWS key finds nothing.
Persistence is read-only. .git, .husky, .claude, .vscode, and package.json are mounted read-only, so an install cannot plant an auto-running hook.
No route off-box. The container sits on an internal network with no gateway. Its only exit is a proxy that forwards to allowlisted hosts and nothing else, so even malware that ignores HTTP_PROXY is stuck. UDP and ICMP have no route either, which closes DNS tunnelling and ICMP exfiltration, not just HTTP.
No cloud-metadata pivot. The metadata endpoints (IMDS) are blackholed, so a dependency cannot reach the magic address that would hand it your cloud role’s credentials.
Stripped capabilities. --cap-drop ALL, --security-opt no-new-privileges, and container-root is not host-root.

Catch it and prove the box held

The last layer is evidence: knowing something tried, and being able to show it did not get out.

An egress tripwire logs every blocked outbound attempt, the highest-signal event the tool produces, and --fail-on-egress turns one into a failed build.
Canary honeytokens plant fake-but-realistic AWS, Stripe, and Slack credentials in the install environment, exactly what a thief greps for. If one ever shows up leaving the box, that is unambiguous proof of theft and the run fails hard. On by default in the strict and agent presets.
A retroactive sweep (sandbox scan) re-checks your committed lockfile against OSV later, catching dependencies that turned malicious after you installed them.
A committed-secrets scan (sandbox secrets) checks the repo itself for credentials you may have committed, reporting where, with the values redacted.
Signed receipts and a tamper-evident audit log turn “the boundary held” into something a later CI stage or a third party can verify without re-running it.
sandbox demo runs four real supply-chain attacks (credential theft, a persistence hook, a cloud-metadata pivot, network exfiltration) against the live sandbox and shows each one contained. It exits non-zero if any attack gets through, so it doubles as a test that the boundary still holds.

It works across npm, pnpm, yarn, and bun, plus runners like npx, bunx, node, tsx, and vite. Anything that pulls new versions goes through the same checks, so update, upgrade, and audit fix are covered too, not just first install.

One tool instead of a toolchain

None of these checks is a new idea on its own. You could assemble most of them from separate tools: one to review packages before install, one to block known malware, one to enforce a minimum release age, and a devcontainer for isolation. Then you wire them together, keep four configurations in sync, and hope every developer set them up the same way.

sandbox puts review, age-gating, malware-blocking, and containment behind one prefix and one config file. You get the safety and the developer experience in one place, instead of orchestrating a toolchain and hoping it holds.

That single file is also what turns this from a personal habit into a team decision. Commit sandbox.config.json and the boundary is reviewed in a pull request, like any other code. sandbox verify enforces it in CI so nothing merges around it, a personal override that loosens it is flagged on every run, and a badge can advertise that installs go through it. Everyone installs against the same policy, whether or not they remember to.

Limits

You should know the edges before you adopt it.

Your source tree stays writable during a normal install, because package managers need a writable root and pnpm writes temp files there. A malicious dependency can still overwrite a file in src/ during install. You will see it in git diff, and --frozen makes the whole tree read-only on every package manager except pnpm. What sandbox blocks is credential theft, persistence, and exfiltration. It does not promise source immutability unless you ask for it.

Anything you grant is in scope on purpose. Forwarding a private-registry token with --env NPM_TOKEN, mounting a path, or turning network on are explicit opt-ins. Grant the minimum.

It needs Docker (or a Docker-compatible engine like OrbStack or Podman) and runs on macOS or Linux, or Windows under WSL2.

How it sits next to other tools

Devcontainers give you a full containerised workspace, but a devcontainer does not make npm install safe: inside it, the install still runs with whatever the container can reach. sandbox fills the install-time gap a devcontainer leaves open, and it can generate the devcontainer for you when you want a persistent environment. That is composition, not competition.

The same is true of Claude Code’s built-in OS sandbox: run that for the agent’s whole blast radius, and sandbox for the supply-chain surface on top.

Bottom line

sandbox does not try to read the dependency code in time, recognise the package, or out-guess the release timing. It runs the install in a box that has nothing worth stealing and no way to send it. Same workflow, smaller blast radius.

The next post covers setup: getting the sandbox command, the one-button onboarding, making it automatic so you never type the prefix, and wiring it into agents and CI.

How to put sandbox in front of npm, pnpm, yarn, and bun

Mon, 15 Jun 2026 00:00:00 GMT

Get the sandbox command, run one setup step, and put sandbox in front of the commands you already run.

New to Node, or not sure why npm install is risky in the first place? Start with why install-time code is a trust decision, then what running it in a throwaway container buys you. Otherwise, read on.

Requirements

One dependency: Docker. Docker Desktop, OrbStack, or any Docker-compatible engine works, and the CLI builds its own images on first run. You also need Node 20 or newer, on macOS or Linux (on Windows, run it inside WSL2). Make sure the engine is running before you start.

The fast path is short:

npx @jagreehal/sandbox-node@latest setup --vibe
npx @jagreehal/sandbox-node@latest npm install

That is enough to try it on a fresh clone without changing your shell or project setup first.

What you get

This is not a new workflow. It is the same install and run commands with the risky part moved behind a boundary.

Same workflow, smaller blast radius.

That means:

You keep install scripts working
You keep npm, pnpm, yarn, and bun
You can apply the same boundary in local dev, AI-agent sessions, and CI
You stop depending on memory and caution every time you clone a repo

Get the command

Use @latest so npx does not reuse a stale cached version. For day-to-day work, add it to the project:

npm install -D @jagreehal/sandbox-node

The CLI installs as both sandbox and sandbox-node.

One-button setup

sandbox setup writes a config if you do not have one, checks Docker, builds the images if needed, and prints the next commands:

sandbox setup --vibe         # safe defaults for exploring and cloning repos
sandbox npm install          # install deps, lifecycle scripts contained
sandbox pnpm add zod         # add a dependency
sandbox npm run dev          # run a dev server, tests, a build

--vibe is the relaxed preset for exploring: containment is on, but the release-age gate is warn-only so a fresh clone is not blocked. When you want the gate enforcing, pick a stricter preset:

sandbox init --preset balanced   # blocks versions younger than 3 days (the default)
sandbox init --preset strict     # blocks 7 days, plus the OSV malware check and canary tokens

Preview any command in plain English before it runs, without executing it:

sandbox --dry-run npm install

It prints what would be mounted, which hosts the install can reach, and that your host credentials stay out.

Make it automatic

Remembering to type sandbox on every install is a rule you will break under pressure. Set the boundary once instead.

sandbox path installs shell functions so a bare npm install routes through the sandbox out of habit:

sandbox path install      # auto-detects zsh, bash, or fish
sandbox path status
sandbox path uninstall

After you open a new terminal, the install vector (npm install, pnpm add, npm ci, npx, and friends) runs sandboxed, while npm run dev, npm test, and node app.js hit the real tool on the host untouched. Two escape hatches are always there:

command npm install        # bypass the wrapper for one call
export SANDBOX_OFF=1       # disable the wrappers for the whole shell

This is a convenience guardrail, not the boundary itself. The real protection is still sandbox running the command in a container.

Cloning a repo or letting an agent install

This is where containment earns its keep:

git clone https://github.com/some/app && cd app
sandbox setup --vibe
sandbox npm install
sandbox npm run dev          # common dev ports are auto-forwarded

For coding agents like Claude Code, the --agent preset stops relying on the model to remember the rule:

sandbox init --agent

It writes .sandbox/AGENT.md (guidance the agent reads), a PreToolUse hook that blocks a bare npm install or npx on the host and tells the agent to re-run it through sandbox, and permissions.deny rules so the agent cannot read .env or secrets/** into its own context. A model that ignores the markdown still hits the hook.

To contain the agent’s whole session rather than each operation, generate a hardened devcontainer from the same config:

sandbox devcontainer init
# then: open in VS Code, "Reopen in Container", run the agent inside

The per-operation form (default) and the per-session form (devcontainer) apply the same hardening at two lifecycles, both driven by one sandbox.config.json.

Give the agent the install skill

The repo also ships a Claude Code skill, sandbox-install, that turns the agent into the human-in-the-loop reviewer. Instead of installing blind, the agent runs sandbox preflight first (the gates, without installing anything), shows you each finding with a recommended action, and runs the real install only once you have cleared the risk, with the flags that match your choices. It will not proceed past a known-malware advisory on its own.

Install it with the skills CLI:

npx skills add jagreehal/sandbox-node

That drops the skill into your agent’s skills directory, and the agent picks it up when you ask it to install or vet a package. If you would rather not use the CLI, copy skills/sandbox-install from the repo into your project’s .claude/skills (or ~/.claude/skills for every project) by hand.

Wire it into CI

CI is where untrusted dependency code runs unattended, so it is where containment pays off most. The pattern is --frozen (reproducible, read-only install) plus --fail-on-egress (fail the build if install-time code tries to phone home):

# .github/workflows/install.yml
jobs:
  install:
    runs-on: ubuntu-latest   # the runner already has Docker
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm i -g @jagreehal/sandbox-node
      - run: sandbox --frozen --fail-on-egress npm install
      - run: sandbox npm test

The guarantee this gives you has two honest halves. Host secrets stay out: anything you do not explicitly grant, your home credentials and any environment variable you did not forward, is never in the container for dependency code to read. Nothing leaves without a route: if a host is not in egress.allow, install-time code cannot send data there, and --fail-on-egress turns the attempt into a failed build instead of a silent leak.

What the install can read is the project tree it runs against. A secret committed into the repo, or the checkout token below, sits in that tree and is readable inside the box. Default-deny egress is what contains it: readable, but with nowhere to send it.

--frozen needs a committed, in-sync lockfile. If an install legitimately needs a host (a private registry, nodejs.org for native modules), allow it once so it is committed in your config:

sandbox allow npm.pkg.github.com

One CI-specific gotcha worth knowing: actions/checkout writes the job’s GitHub token into .git/config by default. sandbox mounts .git read-only, so install code can read that token but not change it, and default-deny egress stops it leaving. To keep it out of the tree entirely, set persist-credentials: false on the install job:

- uses: actions/checkout@v4
  with:
    persist-credentials: false

To gate only what a pull request changes, sandbox delta diffs the lockfile against the merge target and runs the age, malware, and deprecation gates over just the added or bumped versions. Run sandbox scan on a nightly schedule to catch dependencies that turn malicious after merge, and gate merges with sandbox verify so the committed boundary cannot be quietly loosened.

Grant a secret when an install needs one

The container starts with an almost-empty environment, so nothing from your shell or CI runner is in the box by default. When an install genuinely needs one host secret, like a private-registry token, grant exactly that and nothing more:

sandbox --env NPM_TOKEN npm install        # forward one host var by name (its value only)
sandbox allow npm.pkg.github.com           # plus allow the private-registry host

To pull values from an env file, use --env-from, and narrow it to specific keys with a :KEY,KEY suffix so the rest of the file stays out:

sandbox --env-from .env npm install              # inject every key in .env
sandbox --env-from .env:NPM_TOKEN npm install    # inject only NPM_TOKEN; everything else stays out
sandbox --env-from .env tsx script.ts            # one-off script with its env injected

The file is never mounted; only the values you ask for become container env vars. To make a grant permanent for the project, put it in sandbox.config.json under grants.env (named vars) or grants.envFiles (files, with the same :KEY,KEY filtering), so it is reviewed in one place instead of retyped each run.

One thing to know: a .env that lives inside the project is not a grant. It is part of the tree sandbox mounts so the install can run, so dependency code can read it like any other project file. Default-deny egress is what keeps that contained: the code can read a project .env, but under the registry-only allowlist it has nowhere to send the contents. Keep egress.allow tight if your tree holds real secrets, and prefer ssh-agent over mounting key files.

Going further

The defaults cover everyday installs. A few more commands are worth knowing once it is running.

Review before you install. sandbox preflight runs the same gates as a real install (release-age, malware, deprecation, risk hints) without installing anything. It prints each finding and exits non-zero exactly when the install would be blocked, so it drops into a pre-merge check. Tighten it per run:

sandbox --min-release-age 7 --fail-on-advisory preflight npm install

Check your setup. sandbox doctor validates the config, package manager, and container backend, and flags a container-escape CVE in the runtime or an end-of-life Node line in the image.

Block packages you already know are bad. When you do not want to wait for OSV, drop a committed sandbox.advisories.json in the repo, or list malware feeds you trust and cache them locally:

// sandbox.config.json — Aikido's public npm malware list
{ "install": { "malwareFeeds": ["https://malware-list.aikido.dev/malware_predictions.json"] } }

sandbox feeds update          # fetch and cache the feeds; the install check then reads them offline

A match in either source always blocks, independent of the OSV lookup.

Catch theft in the act. --canaries plants fake AWS, Stripe, and Slack credentials in the install environment. If one ever leaves the box, the run fails with proof of exfiltration. It is on by default in the strict and agent presets.

Re-check later, and scan the repo. Dependencies can turn malicious after you install them, and secrets can slip into the repo:

sandbox scan                  # re-query OSV for every version in the committed lockfile
sandbox secrets               # offline scan for committed credentials (values redacted)

Neither needs a container, and both exit non-zero on a finding, so they fit a nightly cron or a CI step.

Prove the boundary held. For a chain of custody, sandbox verify --sign emits an Ed25519-signed receipt that the boundary is intact, which a later stage checks with sandbox verify-receipt. Set SANDBOX_AUDIT_LOG=<path> to append a tamper-evident, hash-chained log of every run.

Watch it work

You do not have to take the claims on faith:

sandbox demo

It runs real supply-chain attacks (credential theft, a persistence hook, an IMDS pivot, egress exfiltration) against the live sandbox in a throwaway project and shows each one contained. It exits non-zero if any attack is not contained.

Bottom line

Get Docker running, npm install -D @jagreehal/sandbox-node, run sandbox setup, and put sandbox in front of the commands you already type. Wire sandbox path into your shell so you stop having to remember, and add the --frozen --fail-on-egress step in CI.

Your installs run the same as before. Same workflow, smaller blast radius. The code they execute has nothing of yours to steal and nowhere to send it.

Minimum release age is necessary, but not enough

Mon, 15 Jun 2026 00:00:00 GMT

The most-recommended npm defence right now is a cooldown: refuse to install any version published more recently than a few days ago. pnpm shipped it as minimumReleaseAge, Aikido Safe Chain turns it on by default, and after the recent Shai-Hulud-style npm worms, it became the advice everyone repeats.

It is good advice. It closes the window these worms detonate in. But “set a cooldown” is being sold as if it were the whole answer, and it is not. A cooldown knows one thing about a package: how old it is. There are at least three reasons it is not enough on its own, and for those, age is the wrong question to ask.

What a cooldown does

A minimum release age refuses to resolve any version published more recently than your threshold. Set it to three days and a version published two hours ago does not get installed; the resolver falls back to the last version that has aged past the line.

It is one of the strongest controls against publish-and-detonate worms. Those worms compromise a maintainer account, publish a malicious version, and spread within hours. Some of the versions developers pulled were five minutes and around an hour old. A cooldown refuses to even resolve anything that fresh, so the bad version is never the one you install.

Keep the cooldown. Everything below is about what it leaves open, not a case against it.

Gap 1: the malicious version that aged in

A cooldown asks one question: is this version old enough? It does not ask: is this version bad?

Most of the time those line up, because worms detonate fast and get caught fast. Not all of the time. A version can sit past your threshold and still be malicious. The attacker who plants something and waits a week beats a three-day cooldown by definition. Once a version crosses your age line, the cooldown no longer filters it at all.

So you need a second axis: not “too new to trust” but “known to be bad.” That is a malware check. In sandbox-node’s case, the second gate is an OSV malware check: @jagreehal/sandbox-node queries the OSV database for the exact version an install would pull and refuses anything flagged as malware, even a version old enough to clear the cooldown:

sandbox --fail-on-advisory npm install   # block a known-malicious version
sandbox init --preset strict              # strict turns this on for you

The two gates are complementary. The cooldown catches too-new-to-trust; the malware check catches known-bad. A zero-day worm is too fresh for any advisory, so the cooldown stops it. A republished known-bad version has aged in, so the malware check stops it. Neither covers the other.

Gap 2: the version that turned bad after you installed it

This second gap defeats both gates at once. An advisory database only knows what has been reported so far. Most supply-chain compromises surface later: a version is published, looks clean, passes your cooldown, passes the malware check because nothing is filed yet, you install it, and days afterwards OSV files a MAL-... advisory for it.

At install time, every check you can run returns green. The package was old enough and not yet known-bad. The cooldown is irrelevant here because the version had already aged in. The install-time malware check is irrelevant because the advisory did not exist yet.

This is why sandbox-node also re-checks after the fact:

sandbox scan          # re-query OSV for every version in your committed lockfile

sandbox scan re-queries OSV for the versions in your committed lockfile and exits non-zero if any installed package is now flagged as malware. Run it nightly, or fold it into a CI gate, and a dependency that turned out bad after you installed it gets caught on the next sweep instead of never. A point-in-time gate cannot do this; only re-checking can.

Gap 3: the one that catches you

The cooldown is blunt in the other direction too. It does not only block other people’s fresh malware; it blocks your own fresh publishes and any legitimate newly released dependency you want. So teams tune it down, exempt their own scope, or turn it off on the repos where the friction bites, which are often the repos that pull the most.

That is not a bug in the cooldown, it is the nature of a time-based gate: the only knob is a number of days, and that one number trades safety against friction for every package at once. You end up either too loose to stop a patient attacker or too tight to ship. sandbox-node lets you exempt your own scope so the gate stays strict for everything else:

{ "install": { "minReleaseAgeDays": 7, "minReleaseAgeExclude": ["@myscope/*"] } }

But exempting a scope is still trusting it by name, which is the assumption a maintainer-account takeover breaks.

The layer that does not depend on timing or recognition

Each gate above depends on knowing something in advance. The cooldown depends on the bad version being fresh. The malware check depends on someone having filed an advisory. Your own blocklist depends on you already knowing. Every one of them comes down to recognising the package, reading the code, or timing the release, and a brand-new, unrecognised, plausibly-aged package slips past all three.

The control that does not depend on any of that is containment. If the install runs in a throwaway box that cannot see your SSH keys, cannot reach your npm token, and has no outbound route beyond the registry it needs to fetch packages, it does not matter whether the version was fresh, flagged, or recognised. The credentials are not in the box. Even if a script tries to exfiltrate, the default-deny egress allowlist gives it no general-purpose route out. The box is deleted afterwards and your installed dependencies stay.

That is the difference worth being precise about. A cooldown, a malware check, and a blocklist all try to answer “should this code run?” and they answer it from incomplete information, every time. Containment changes the question to “if it runs, what can it reach?” and answers that one in full, regardless of what the gates knew.

sandbox npm install   # the install runs with no credentials, default-deny egress, IMDS blackholed

Be precise about the claim. sandbox-node does not have a magically better blocklist: its malware check has publish lag like any other, and it blocks known malware, not every vulnerability. What sets it apart is that it does not rely on the blocklist being complete. The cooldown and the OSV check are the cheap front line; containment is the backstop for everything they miss.

Use them together, not instead

This is not cooldown versus sandbox. sandbox-node runs the cooldown and the malware check and the retroactive sweep and the containment, behind one prefix and one config:

sandbox init --preset strict   # 7-day cooldown + OSV malware check + canaries + containment
sandbox npm install
sandbox scan                    # later, catch what turned bad after install

A worm too fresh to be flagged hits the cooldown. A known-bad version old enough to clear the cooldown hits the malware check. A version that turns bad after you install it gets caught by the next sandbox scan. And the one that beats all three, fresh enough or obscure enough that nothing flagged it, runs in a box with nothing of yours to steal and nowhere to send it.

Bottom line

A minimum release age is necessary. It closes the worm window better than anything else, and you should turn it on. It is not sufficient, because it asks only how old a package is, and a patient attacker, a late advisory, and your own release schedule all make age the wrong question.

The fix is not a longer cooldown or a better blocklist. It is to stop betting everything on the gate being right and run the install where a wrong answer cannot reach your credentials or your host environment. If you want the full picture, the safe-installs series walks through why install-time code is a trust decision, what running it in a throwaway container buys you, and how to put sandbox in front of the commands you already run.

npm install runs code you never read

Mon, 15 Jun 2026 00:00:00 GMT

You run npm install without thinking. It looks like a download step. It is a trust decision.

Every package that resolves into your tree can run code the moment it installs, with the same access you have: your SSH keys, the npm token in ~/.npmrc, your cloud credentials, your .env. Not the package you typed. Any of the hundreds underneath it. You approved none of them by name, and the package manager never asked.

The fix is not to read more or trust less. It is to run that install somewhere your secrets are not.

First, what `npm install` actually does

If you are new to Node, here is the part that makes the rest matter.

When you run npm install, npm reads the packages your project asked for, downloads them from the public npm registry (a shared store anyone can publish to), and drops them into a node_modules folder. Each of those packages depends on others, and those depend on more. You ask for a handful and you get hundreds. These are called transitive dependencies: the ones you never named, pulled in on behalf of the ones you did. Most are written by people you will never meet.

Downloading is not the whole story. A package can include scripts that npm runs automatically as part of installing it, called lifecycle scripts (preinstall, install, postinstall). They exist for good reasons, like compiling native code or fetching a binary. But it means “install” is not a passive copy. It executes code from those hundreds of packages, on your machine, as you.

That last fact is the one most people miss, and it is the whole reason this series exists.

The install is where the code runs

That install-time code runs with everything your shell can reach. It can read files, open network connections, and write to disk before a single line of your own code executes. node-gyp, which builds native modules, runs during install without even needing a lifecycle script, so “just turn scripts off” does not close the gap.

The package manager answers operational questions. Can I resolve this version? Can I fetch the artifact? It does not ask the trust questions: should this code run on this machine, with this access, right now?

The agent makes it routine

You told the agent to clone the repo and get it running. It ran npm install. You were not watching the output, because not watching is the point of an agent.

Somewhere three levels down the dependency tree, a postinstall script read your ~/.npmrc and your AWS credentials and sent them out. The install finished green. You found out later, or you never did.

Coding agents industrialise the thing that was already risky. They clone unfamiliar repos, they add dependencies on your behalf, and they run installs faster than you can review them. The trust decision you used to make a few times a day now happens unattended, on code neither of you has read.

CI runs it unattended, against bigger keys

The same install runs on every build, in CI, with no one watching. And the secrets there are worth more than any laptop’s: your registry tokens, your cloud deploy credentials, the token your pipeline uses to push releases.

A postinstall script on a build runner has the same access the build does. One compromised dependency does not steal a single developer’s keys. It steals the organisation’s, on a machine nobody is looking at.

The usual advice is partial

The standard supply-chain advice does not hold up on its own.

“Audit your dependencies” assumes you can read every transitive package, and you cannot. A mid-sized project pulls in hundreds of them, most written by people you will never meet.

“Use --ignore-scripts” misses node-gyp, which runs during native builds without a lifecycle script. It also breaks packages that genuinely need their install step.

“Pin your versions” does nothing the day a pinned version turns out to be the compromised one.

Each piece of advice closes one door. Install-time code still runs, and it still runs with everything you have.

“We would notice” fails as a control

Recent incidents follow one pattern. An attacker compromises a maintainer account, publishes a malicious release, and waits for developers to pull it before anyone reacts.

The June 4, 2026 Shai-Hulud worm showed the compressed version. It pushed malicious releases and spread within hours. Some of the versions developers pulled were five minutes and around an hour old. By the time a dashboard flags the release, it is already on laptops and CI runners.

“We would notice” is not a control, because you notice too late. The damage happens during the short window between publish and takedown, and an automatic install inside that window is all it takes.

Run install where there is nothing to steal

Review tools warn you before install. Blocking tools refuse packages already confirmed as malware. Age gates refuse versions too fresh to trust. Each one helps, and each one depends on either reading the code, recognising the package, or timing the release. A brand-new, unrecognised, plausibly-aged package slips past all three.

If the postinstall script runs in a throwaway box that cannot see your SSH keys, cannot reach your npm token, and cannot send anything out, it does not matter whether you recognised the package or read its code. The credentials are not in the box. There is no way out. The box is deleted afterwards, and your installed dependencies stay.

In plain terms, here is what that buys you. You can install anything, clone any repo, and let an agent add packages, without betting your SSH keys, your tokens, and your cloud credentials on every one of those packages being honest. You work the way you already do. A malicious package just finds an empty room.

That is the approach the rest of this series is about. I built @jagreehal/sandbox-node to do exactly this: put sandbox in front of the npm, pnpm, yarn, or bun command you already run, and the risky part happens in a container with your secrets left out.

Bottom line

npm install is the most common command in your day and one of the least examined. It runs code you never read, with access you never granted by name, at a moment an attacker can predict.

The fix is not to read more or trust less. It is to run install somewhere your secrets simply are not, so a bad package has nothing to take and no way to send it. You keep your normal workflow and stop betting it on every dependency. The next post covers what that looks like.

The GitHub and npm settings open source maintainers should turn on before they need them

Mon, 08 Jun 2026 00:00:00 GMT

In the middle of a supply-chain incident, the maintainer is not just fixing packages. They are locked out of their account, answering reports, trying to contact registries, trying to warn users, and trying to prove what happened.

That is the part we do not talk about enough.

Most open source maintainers are not companies. They do not have incident response teams. They do not have a security department. They have a GitHub account, an npm account, a laptop, and a lot of people depending on them.

Security advice often assumes the maintainer is the weak link. That is backwards. The maintainer is the last line of defence, usually unpaid, usually alone, and often locked out of the systems they need during the incident.

This is the checklist I wish every maintainer had before something goes wrong. No single setting saves you, so it works in layers: the account, the branch, the release path, the workflow, the tokens, the files, the tripwires, and the recovery plan.

1. Protect your GitHub account

Everything below assumes the attacker does not control your GitHub account. Start here, because once someone has push access, a lot of “secure CI” assumptions collapse.

Use phishing-resistant login where you can. GitHub supports passkeys and hardware security keys, and a passkey signs you in and satisfies 2FA in one step. An OTP app is better than SMS; a passkey or security key is better than both, because there is no code to phish.

The ten-minute audit:

Enable 2FA, then add a passkey or hardware security key
Remove SSH keys you no longer recognise or use (settings/keys)
Remove old personal access tokens; replace classic PATs with fine-grained ones scoped to specific repos
Review authorised OAuth apps and GitHub Apps, and revoke anything you stopped using
Check active sessions for devices you do not recognise

Calendar this twice a year. Credentials accumulate; nobody deletes them without an appointment.

If you maintain important packages, separate authority across accounts. Keep a rarely used admin identity for rulesets, environments, access changes, and recovery, and do day-to-day development from a non-admin maintainer account. If you can, keep approvals on a third account or teammate identity that does not share the same browser session or device posture. The goal is not ceremony. It is to stop one stolen day-to-day session from also being allowed to remove the protections that were meant to catch it.

Solo maintainer? At minimum, separate a rarely used admin account from your daily development account.

2. Protect your release branch

Every package repo should protect main. In Settings → Branches (or the newer rulesets):

Require pull requests before merging
Require at least one approval
Require status checks to pass
Block force pushes
Block branch deletion
Require conversation resolution
Restrict who can push directly, if your plan allows it

For solo maintainers there is an obvious objection: who reviews my PR? Be honest about what this layer does. Branch protection will not save you from yourself, and it will not save you from a fully compromised account that can change the rules. But it stops accidental pushes, basic automation abuse, and compromise of a lower-privilege collaborator. It also creates friction, and sometimes friction is the control. In the June 2026 campaign, some attackers had enough GitHub privilege to remove branch protection before pushing to main, which is exactly why you cannot treat branch rules as sufficient on their own. Section 3 closes the OIDC snapshot-branch path, which does not depend on main branch protection noticing the push.

The same June 2026 campaign also landed direct commits on main with stolen credentials, disguised as ordinary feature work. Required review and signed commits still raise the bar, but they help only while the attacker does not have enough privilege to disable or rewrite those protections first. The trusted publishing post covers the OIDC half.

3. Protect your release environment

This is the layer that stopped mattering being theoretical in June 2026, when a worm published malicious versions through projects’ own trusted publishing pipelines by pushing throwaway snapshot-* branches containing a copy of the release workflow. The publishes carried valid provenance. See the mechanics and the fix for details; the short version:

Create a GitHub environment called release (Settings → Environments)
Restrict it to main, or to the tags you actually release from
Add required reviewers if you can
In npm’s trusted publisher settings, set Environment name to release

Jobs that reference an environment must satisfy its protection rules before they run. A workflow on a throwaway branch does not get the environment, does not get the OIDC claim, and the publish is rejected.

The principle:

A workflow should not be allowed to publish just because it exists. It should have to come from the release path.

4. Lock down GitHub Actions permissions

Default workflow permissions should be boring. Release permissions should exist only on the release job.

# workflow level: read-only
permissions:
  contents: read

jobs:
  publish:
    environment: release
    permissions:
      contents: read
      id-token: write # only here, never workflow-wide

The rest of the Actions checklist:

In Settings → Actions → General, set default workflow permissions to read-only
Pin third-party actions to full commit SHAs, not tags. GitHub’s own guidance says a full-length SHA is the only way to use an action as an immutable release; tags can be moved to malicious commits.
Avoid pull_request_target unless you fully understand what it grants. It runs with secrets in the context of your repo against code from a fork.
Never run untrusted PR code in a job that has secrets

5. Remove publish tokens from CI

The best npm token is the one that does not exist.

If trusted publishing is available for your setup, use it and delete the automation tokens it replaces. A long-lived publish token in CI secrets is the single most valuable thing an attacker can exfiltrate from your repo, and stealers now harvest CI secrets as standard practice.

The npm side of the audit:

Use trusted publishing for CI releases; remove old automation publish tokens
Where you still need tokens (e.g. installing private packages), use read-only granular tokens with short expiry
In each package’s settings, require 2FA for publishing and settings changes, and disallow token-based publishing where you can
Audit the maintainers list on every package; remove people who no longer need access
Check the trusted publisher settings themselves, and limit them to publish only, not every action

6. Treat release files as sensitive

Some files in your repo are not ordinary code. If a file can change what gets published, treat it as release infrastructure.

A CODEOWNERS file makes that surface visible, and combined with branch protection’s “require review from code owners”, changes to it demand sign-off:

# .github/CODEOWNERS
/.github/workflows/   @your-username
package.json          @your-username
package-lock.json     @your-username
pnpm-lock.yaml        @your-username
yarn.lock             @your-username
npm-shrinkwrap.json   @your-username
rollup.config.*       @your-username
vite.config.*         @your-username
webpack.config.*      @your-username
tsup.config.*         @your-username
/scripts/             @your-username

Even in a small project where you are the only owner, the file documents which paths are dangerous. It says: these files are not ordinary code. Future you, future co-maintainers, and future security reviewers all benefit from that being written down.

7. Watch for snapshot branches

The June 2026 worm used throwaway branches with random names. Most maintainers do not have a SOC, and they do not need one. They need cheap tripwires:

New branches with random or generated names
Workflow files added or changed outside your normal branches
Changes to release.yml or whatever your publish workflow is called
package.json scripts changing, especially postinstall or preinstall appearing
npm versions of your package you did not publish
Actions runs with names you do not recognise (“Dependabot Updates” that you never configured)

Two free tools cover most of this. Watch your own repo (Watch → Custom → Releases and Discussions plus rule-based notifications), and subscribe to your own package on npm so new-version emails reach you. The cheapest alert that something is wrong is an email about a release you did not make, arriving while you still have account access.

8. Prepare for account lockout

This is the section maintainers need most, and the one nobody writes.

When an account is compromised, platforms suspend it. That is the right call for stopping the spread, and it has a brutal side effect: the maintainer loses access to the security log, the audit trail, the commit history, and the settings pages — the very evidence needed to prove which actions were not theirs, at exactly the moment everyone is asking.

So keep a recovery file outside GitHub: a password-manager secure note, an encrypted local file, even a printed sheet. For each package:

- npm package name
- GitHub repo URL
- npm owner account
- Trusted publisher settings (repo, workflow file, environment name)
- Normal release workflow path
- npm support: https://www.npmjs.com/support
- GitHub support: https://support.github.com
- Emergency contacts (co-maintainers, employer security team if relevant)
- Last known safe version
- The deprecate command:
    npm deprecate <pkg>@<bad-version> "compromised - do not install"
- The supersede command (publish a clean version ABOVE the bad one,
  because deprecation does not stop semver ranges resolving to it)

Add to it the moment anything changes. The file costs you twenty minutes on a quiet afternoon. Its absence costs you days in the middle of the worst week of your maintainer life.

Bottom line

Open source maintainers should not need to become incident response teams overnight.

Until the platforms build better defaults, a small set of controls protects you before the panic starts: phishing-resistant login, a protected release branch, a locked release environment, minimal Actions permissions, no long-lived publish tokens, release files treated as infrastructure, cheap tripwires, and a recovery plan that still works if your GitHub or npm access disappears.

None of this makes a maintainer invincible.

It just means one stolen credential does not immediately become dozens of compromised packages, hundreds of malicious versions, and a maintainer locked out of the audit logs mid-incident.

If you want the deeper dives: the trusted publishing setting that would have blocked the snapshot-branch attacks covers layer three in detail, and the package manager settings that would have blocked the Axios attack covers the consumer side. Publisher, consumer, and now the person in between.

The trusted publishing setting that would have blocked the npm snapshot-branch attacks

Mon, 08 Jun 2026 00:00:00 GMT

In June 2026, a self-propagating npm worm compromised maintainer accounts and republished their packages with malware inside. Maintainers woke to hundreds of malicious versions across dozens of packages, often published in a few minutes overnight.

Many of those versions went out through npm’s trusted publishing, the tokenless way to publish from CI, and they carried valid provenance. The green badge, the Sigstore attestation, the “built from this repo by this workflow” proof: all real.

The worm never needed an npm password for those publishes. It used the release pipeline itself.

Trusted publishing remains the right approach. Many configurations leave one field blank, and that blank field is the gap. Closing it takes about five minutes per package. This post shows the attack and the settings that would have stopped the snapshot-branch half of it.

How trusted publishing works (and what it actually checks)

Trusted publishing replaces long-lived npm tokens with a handshake between npm and your CI provider. You tell npm: “versions of this package may be published by this GitHub repository, running this workflow file.” When the workflow runs, GitHub mints a short-lived OIDC token (a signed statement describing the run), npm verifies the claims, and the publish goes through. No token sits in CI secrets waiting to be stolen.

The default binding checks:

the repository ✅
the workflow filename ✅ (e.g. .github/workflows/release.yml)
the branch ❌

The binding says which file may publish, but not which ref it may run from.

The attack: bring your own branch

The worm operated with stolen GitHub credentials that had push access. It avoided main, where branch protection might have noticed. Instead, for each repository it:

Pushed a throwaway branch with a random name (snapshot-a1b2c3d4)
Added its own .github/workflows/release.yml on that branch, the same filename as the project’s real release workflow, disguised with an innocent display name (“Dependabot Updates”)
Gave the job id-token: write and published

GitHub minted the OIDC token: correct repository ✅, correct workflow filename ✅. npm checked the claims: all match ✅. Publish accepted, provenance attached, badge green.

The filename check fails when the attacker controls the file. Workflow files live in the repository, so anyone who can push a branch can create a workflow with any filename they like. The OIDC token includes which ref the workflow ran on. Default npm trusted publisher config does not require it. The worm used side branches because that path was easier, not because branch data was missing from the token.

Every one of those malicious versions displayed valid provenance on npmjs.com. Provenance tells you a package was built where it says it was built. It says nothing about whether anyone authorised that build. If you treat the badge as a trust signal, this attack breaks that assumption.

The worm published with stolen credentials. GitHub and npm treated the runs as legitimate maintainer activity. The OIDC tokens and provenance were real. Authorisation was not. Platforms cannot distinguish your session from a stolen one without release rules you define.

The fix: make the branch part of the contract

npm’s trusted publisher config has an optional field many people leave blank: Environment name. It adds the missing branch dimension, provided you configure both halves.

Half one: the npm side

In your package’s trusted publisher settings, set Environment name to release. npm now requires the OIDC token to carry an environment: release claim. A workflow only gets that claim if its job declares:

jobs:
  publish:
    environment: release

Half two: the GitHub side (this is the half that does the work)

The attacker controls the workflow file. If the release environment has no rules attached, they add environment: release to their rogue workflow, GitHub grants it, and the publish goes through anyway. Configure the npm field and the GitHub environment together, or you gain nothing.

The protection lives in the environment’s rules. In your repo: Settings → Environments → New environment → release, then:

Deployment branches and tags → Selected branches and tags → add main (and v* tags if you publish from tags)
Required reviewers → add someone other than the account that pushes code, if you can (a co-maintainer or org teammate; solo maintainers need a second approver account or this layer does little, and it only helps while the attacker cannot disable the rule)

The chain now holds:

npm demands an environment: release claim
GitHub only grants that environment to jobs running from main
A workflow running from snapshot-a1b2c3d4 gets the environment refused, so no claim, so the publish is rejected

A throwaway branch does not satisfy a main-only environment.

With Required reviewers on top, a malicious commit on main should pause the publish job until another human approves it. That only works when the approver is a different account from the one the attacker stole and the attacker cannot first remove or weaken the protection.

The hardened release workflow

# .github/workflows/release.yml
name: Release

on:
  push:
    branches: [main]

permissions:
  contents: read # default-deny everywhere else

jobs:
  publish:
    runs-on: ubuntu-latest
    environment: release # grants the claim npm now demands
    # refuse to run in forks/unexpected contexts
    if: github.repository_owner == 'your-username'
    permissions:
      contents: read
      id-token: write # ONLY on this job, never workflow-wide
    steps:
      - uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5, pin to SHA
      - uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4, pin to SHA
        with:
          node-version: 24
          registry-url: https://registry.npmjs.org
      - run: npm ci --ignore-scripts
      - run: npm run build && npm test
      - run: npm publish # OIDC, no token, no NODE_AUTH_TOKEN

The details that matter:

environment: release on the publish job pairs with the npm field
id-token: write goes only on the publish job, never at workflow level
Pin third-party actions to commit SHAs, not tags. Tags can be moved to malicious commits.
The repository-owner guard is a cheap extra tripwire
In the npm trusted publisher settings, under allowed actions, tick only what you use (publish) and drop the rest

Why this combination works

What the attacker did	What blocks it
Pushed a rogue workflow on a throwaway branch	Environment restricted to `main` → no OIDC claim
Reused the trusted workflow’s filename	Filename no longer sufficient; the branch is checked
Could declare `environment: release` themselves	Declaring ≠ receiving; GitHub enforces the rules
Publish from `main` without a second approver	Required reviewers, if the rule still exists and someone else must approve
Relied on valid provenance to look legitimate	Provenance valid; the publish does not complete

What this closes

Environment binding blocks the snapshot-branch OIDC bypass: push snapshot-*, add a rogue release.yml, publish with valid provenance. Restrict the release environment to main and the publish fails before it reaches npm.

The same June 2026 campaign used other paths. Attackers with stolen credentials pushed commits directly to main, sometimes with forged commit metadata, and in some cases had enough GitHub privilege to remove branch protection first. Environment binding does not stop that. Pull request review and signed commits still raise the bar, but they help only while those protections still stand and the attacker cannot edit them. See the GitHub and npm settings checklist section 2 for that layer.

The practical way to make that harder is to split authority. Let the account that pushes code be a non-admin maintainer account. Keep rulesets, environments, and recovery on a separate admin identity that is not used for normal development. If you use required reviewers, make that a third identity or a teammate account, not the same session that authors the change. That does not make account compromise harmless. It does stop one stolen day-to-day account from automatically being allowed to dismantle the repo settings first.

If someone fully controls your GitHub account and can change environment rules or branch protection, no repo setting saves you on its own. Account hygiene (passkeys, credential separation, recovery notes outside GitHub) matters too. Entry in these campaigns is often credential harvest on a dev machine, not a misconfigured workflow.

One stolen credential should not become silent publishes and silent commits. Different gates for different paths.

Bottom line

Trusted publishing without an environment binding trusts a workflow filename. Workflow filenames are part of the repository, and an attacker who can push a branch can create a workflow file with the same name.

The missing control for OIDC publishes is the branch rule:

Set the Environment name in npm’s trusted publisher config
Create the matching GitHub environment
Restrict it to main, or to the tags you actually release from
Add a required reviewer if you have someone else who can approve

That tightens provenance from “this came from a workflow with the right name” to “this came from a release path you defined.” It would have blocked the snapshot-branch OIDC publishes from the June 2026 campaign, and it blocks the same branch trick next time someone tries it.

None of this makes a maintainer invincible. Pair it with branch protection, release-file controls, tripwires, and a recovery plan: the GitHub and npm settings checklist covers the rest.

While you harden publishing: the package manager settings that would have blocked the Axios attack protect you as a consumer of packages. This post protects you as a publisher. You need both.

Secure AI Apps Start With Functions, Not SQL

Wed, 13 May 2026 00:00:00 GMT

If you’re building AI features into your product, the temptation is to hand the model a database connection and let it write SQL.

It feels powerful. It feels like the modern way.

It’s the wrong place to start.

The same reasoning applies whether you’re building a CLI, a Slack bot, an MCP server, or an agent.

tl;dr: Build a typed, composable function layer first. Lock down the database, shape responses with DTOs, redact sensitive fields, and expose only those safe functions to CLIs, Slack bots, MCP servers, and agents. The AI layer is easy to swap. The function layer is what matters.

Functions, not prompts

In most product use cases, you don’t need text-to-SQL at all. You need a small set of predefined functions that do useful things.

If you’re building around payments, that might look like:

getLatestPaymentsForClient(clientId)
getFailedPaymentsForClient(clientId)
getClientPaymentSummary(clientId)

Each function does one thing, takes a fixed set of arguments, and returns predictable JSON. Good JSDoc, typed inputs, typed outputs, a clear purpose. That is the foundation.

Not prompts. Not agents. Not text-to-SQL. Functions.

The reason this matters is composability. If a function has a single responsibility and returns structured JSON, anything can sit on top of it: a CLI, a Slack bot, an MCP server, an HTTP API, an agent. The contract is fixed, so you always know what goes in and what comes out.

flowchart TD
    DB[("Approved data source")]
    Core["Core functions<br/>typed inputs, safe outputs"]
    CLI["CLI"]
    Slack["Slack bot"]
    MCP["MCP server"]
    API["HTTP API"]
    Agents["AI agents"]

    DB --> Core
    Core --> CLI
    Core --> Slack
    Core --> MCP
    Core --> API
    Core --> Agents

Read that diagram as: one safe layer in the middle, many ways for people and machines to use it.

Build the CLI or Slack bot first

Before you add any AI at all, I’d actually recommend wiring up a CLI or a Slack bot.

It sounds backwards, but it forces you to design the interaction model properly. Start with a simple menu of domains:

Clients
Payments
Invoices
Orders

Within each domain the user picks an action: latest payments for a client, failed payments, account summary, recent invoices. They’re calling the same predefined functions an agent would call, just through a menu instead of natural language.

This is a good thing, because the first job is not making it intelligent. The first job is making it safe.

Least privilege starts at the database

Do not expose more than you need to.

Start with a read-only database connection. Better still, create a database user that only has access to the specific tables or collections the tooling needs.

If your database contains API keys, secrets, admin data, audit logs, or sensitive customer records, your AI tooling should not be able to touch those tables at all. That’s the first security boundary.

The second boundary is the function itself. Even when the database user can read a table, the function should not return everything from it. This is where DTOs earn their keep.

A payment record might have twenty internal fields. The function returns six:

{
  "paymentId": "pay_123",
  "clientName": "Acme Ltd",
  "amount": 1200,
  "currency": "GBP",
  "status": "paid",
  "createdAt": "2026-05-01"
}

No raw metadata, no internal IDs, no private fields, no PII unless it’s genuinely required.

That JSON is what the engineer sees. The business user sees something like this in Slack:

Acme Ltd Last payment: £1,200 on 1 May 2026 Status: Paid Outstanding invoices: 2 (£3,400) Open support tickets: 1

Same data, same function call, just rendered for a human instead of piped to another tool. A salesperson can pull that up before a call, a support agent can quote it on a ticket, and an AI agent can read it as context. None of them need direct database access to do it.

Database permissions can restrict tables and sometimes columns, but they can’t shape responses per use case. DTOs can, and they give you a natural place to redact. If a value should never reach Slack, the CLI, the MCP server, or the model, redact it before it leaves the function. Your AI layer should never be the thing deciding what’s safe.

What you’ve actually built

By this point you have something stronger than “an AI app”:

a read-only database user with restricted table access
a set of approved functions
DTOs that only expose safe fields
redaction at the function boundary
JSON schemas for predictable outputs
typed inputs and outputs end to end

And because the output is structured JSON, the CLI gets a useful side effect for free. You can pipe through jq:

payments list --client acme --json | jq '.[] | select(.status == "failed")'

That’s the same composability the AI layer will need, just exposed to humans first. If jq can chain your tools cleanly, an agent can too.

Now you’re ready for AI

The key is not “give the model SQL access”. The key is deciding which use cases AI should actually support.

Maybe you need a payments agent, a client insights agent, a support agent, a sales enablement agent. Each one only gets the tools it needs. The payments agent gets payment tools. The support agent gets support tools. No agent gets broad access. No agent invents arbitrary queries.

You can put a routing agent at the front to send a request to the right specialist, and each specialist can have its own instructions and few-shot examples. But the agents are not the foundation. The tools are. The functions are. The schemas are. The safe data shapes are.

With TypeScript, Pydantic, Zod, or JSON Schema, the entire chain is type-safe end to end. You know what each tool takes, what it returns, how to validate it, and how to document it. That matters a lot once a function becomes an AI tool, because the description, the name, the schema, and the return shape all become part of the prompt the model reads.

This is why the core function layer deserves the most effort.

A note on the database itself

Your AI features and internal tools should ideally not hit production directly.

A read replica is a good first step: a bad query won’t take down the live application. A better setup is change data capture, like CDC or MongoDB change streams. You capture events from production and write them into a separate database that exists specifically for internal tooling, reporting, and AI.

That database is read-only from the AI app’s perspective. It only contains the fields these tools are allowed to consume. It can store redacted or transformed versions of records. If something ever goes wrong, you revoke one connection string without touching production.

flowchart LR
    Prod[("Production database")]
    Stream["Change events"]
    Tool[("Tooling database<br/>read-only, redacted")]
    Core["Core functions"]
    Users["CLI / Slack / MCP / Agents"]

    Prod -->|CDC or events| Stream
    Stream --> Tool
    Tool --> Core
    Core --> Users

The arrows only go one way. Nothing on the right can write back to production.

If you already have event-driven architecture, this is even cleaner. The tooling database is just another subscriber. You can choose between thin events (minimal payload, requires another lookup) and fat events (the full approved payload). For AI and internal tooling, fat events are often more useful, because the event itself can carry exactly what the tools are allowed to see.

If CDC or events aren’t on the table yet, dual-writing from the application to a tooling database is a fallback. It’s not perfect, but it’s still better than handing AI broad access to production.

Don’t forget observability

Whatever shape this takes, you need to know what’s happening: which tools are being called, by whom, with what arguments, what they returned, which agents called which functions, where errors are happening, and whether anyone is trying to use the tools in ways the design didn’t anticipate.

Security isn’t only about blocking access. It’s also about being able to see what’s going on.

What this unlocks

With this foundation, the same function layer powers a lot of useful things at once. Customer support can answer questions faster. Account managers can pull client activity without filing tickets. Sales can get context before a call. Operations can investigate issues without engineering help.

In the repo, the effort goes into one place: a core package holding the functions, DTOs, schemas, types, redaction, descriptions, OpenAPI specs, and MCP tool definitions. Everything else just consumes core:

/packages
  /core
  /cli
  /slackbot
  /mcp-server
  /api
  /agents

Every surface shares the same logic, the same DTOs, the same safety rules.

The mistake is thinking the AI layer is the product. It usually isn’t.

The real product is the secure, typed, composable function layer underneath.

AI is just another interface on top of it, and probably not the last one you’ll add.

node-env-resolver Makes Safe, Typed Node Config the Default

Thu, 23 Apr 2026 00:00:00 GMT

Liran Tal is spot on in his Environment variables and configuration anti-patterns in Node.js applications post

You may inadvertently expose sensitive information like database credentials and API keys as part of error messages, stack traces, and other forms of data returned to consuming clients.

He explains why process.env feels safe right up until it isn’t.

You add dotenv.config() on line one, scatter process.env.DB_PASSWORD across twelve files, then someone’s error reporter serialises a request object and your Stripe key ends up in a third-party log.

If you’ve shipped a Node app, you’ve probably seen some version of this happen.

His anti-pattern example nails it:

const port = process.env.PORT || 3000;
const dbUsername = process.env.DB_USERNAME;
const dbPassword = process.env.DB_PASSWORD;
const dbHost = process.env.DB_HOST;
const apiBaseUrl = process.env.API_BASE_URL;
const apiToken = process.env.API_TOKEN;

Untyped. Unvalidated. Globally readable. One step away from leaking through logs, traces, or error reporters.

But there’s still a gap…

Liran’s article names the anti-patterns. It stops short of offering a concrete implementation.

node-env-resolver is a strong answer to that gap. It resolves config into a typed object, limits reliance on global process.env, blocks .env files in production by default, and adds runtime redaction safeguards.

The Problem You’re Solving

A single Node app reads config from five places:

// server.ts
const port = process.env.PORT || 3000;

// database.ts
const dbUrl = process.env.DATABASE_URL;

// auth.ts
const jwtSecret = process.env.JWT_SECRET;

// routes.ts
const apiKey = process.env.API_KEY;

// stripe.ts
const stripeKey = process.env.STRIPE_SECRET_KEY;

Five files. Five untyped reads. Zero validation. Five modules that can be tested only by mutating process.env in beforeEach.

And every one of those values sits in global process state, where any module can reach it and where logs, traces, serialised errors, or child-process boundaries can expose it.

One Schema. Zero process.env Writes

With node-env-resolver, the same app gets one typed config object, assembled at startup, injected into the modules that need it:

import { resolve } from 'node-env-resolver';
import {
  postgres,
  string,
  number,
  secret,
} from 'node-env-resolver/validators';

export const config = resolve({
  PORT: number({ default: 3000 }),
  DATABASE_URL: postgres(),
  JWT_SECRET: secret(),
  API_KEY: string({ min: 32 }),
  STRIPE_SECRET_KEY: string(),
});

If DATABASE_URL is missing, the app fails at startup with a descriptive error. Not on the first request. Not in production at 3am. At startup.

config.PORT is number. config.DATABASE_URL is string and guaranteed to parse as a Postgres URL. process.env.STRIPE_SECRET_KEY stays undefined, because preventProcessEnvWrite is on by default.

One typed config object, validated once at boot.

Which values are enforced?

Every key in your schema. Required validators throw if missing. Bare literals act as typed defaults. Array literals become enums:

const config = resolve({
  PORT: 3000,                                          // number, default 3000
  HOST: 'localhost',                                   // string, default
  NODE_ENV: ['development', 'production', 'test'] as const, // enum, required
  DATABASE_URL: postgres(),                            // required, parsed
  API_KEY: string({ min: 32, pattern: '^[a-zA-Z0-9]+$' }), // required, validated
});

Safety rails

In production, node-env-resolver applies stricter policies automatically: .env files are blocked, audit logging is on, process.env writes are prevented, missing required keys fail fast. The application code stays the same, while production gets stricter defaults automatically.

Manual process.env vs node-env-resolver

The manual approach reads process.env directly, with a light dusting of defaults:

// Manual: scattered, untyped, unsafe
import 'dotenv/config';

export function createDatabase() {
  const url = process.env.DATABASE_URL;
  if (!url) throw new Error('DATABASE_URL required');
  return connect(url, {
    poolSize: Number(process.env.DB_POOL_SIZE) || 10,
  });
}

export function createAuth() {
  return jwt.sign({
    secret: process.env.JWT_SECRET,
    expiry: process.env.JWT_EXPIRY || '1h',
  });
}

With node-env-resolver, config is a typed value you pass in:

import { resolve } from 'node-env-resolver';
import {
  postgres,
  secret,
  number,
  duration,
} from 'node-env-resolver/validators';

export const config = resolve({
  DATABASE_URL: postgres(),
  DB_POOL_SIZE: number({ default: 10 }),
  JWT_SECRET: secret(),
  JWT_EXPIRY: duration({ default: '1h' }),
});

export function createDatabase(
  cfg: Pick<typeof config, 'DATABASE_URL' | 'DB_POOL_SIZE'>,
) {
  return connect(cfg.DATABASE_URL, { poolSize: cfg.DB_POOL_SIZE });
}

export function createAuth(
  cfg: Pick<typeof config, 'JWT_SECRET' | 'JWT_EXPIRY'>,
) {
  return jwt.sign({ secret: cfg.JWT_SECRET, expiry: cfg.JWT_EXPIRY });
}

No process.env reads inside modules. No coercion. No runtime undefined. Tests pass a literal config object, no vi.mock required.

Manual dotenv / process.env	node-env-resolver
`process.env.FOO` may be undefined	Inferred types: `number`, `string`, enum literals
Values written to `process.env`	`preventProcessEnvWrite` on by default
`.env` works everywhere, including prod	Blocked in production by default
No validation	Required validators throw at startup
No provenance	Every value records its source
Secrets may leak via logs/errors	`protect()` patches console and redacts
Test by mutating `process.env`	Test by passing a plain object

On “dotenv Increases Exposure”

He’s right to flag it.

dotenv.config() loads your .env into process.env, and process.env is global state. Any module, any dependency, any serialised error object can read it. That doesn’t guarantee a leak, but it widens the surface area considerably, and most leaks I’ve seen in the wild took this path.

node-env-resolver treats process.env as an input, not an output. Resolved values land in a typed object that only the modules you inject into can see. In production, .env files are refused by default:

// In production, dotenv is ignored by default.
// Config must come from processEnv, secrets managers, or mounted files.
const config = await resolveAsync({
  resolvers: [
    [processEnv(), { PORT: number({ default: 3000 }) }],
    [awsSecrets({ secretId: 'prod/app' }), {
      DATABASE_URL: postgres(),
      JWT_SECRET: secret(),
      STRIPE_SECRET_KEY: string(),
    }],
  ],
  options: {
    policies: {
      enforceAllowedSources: {
        DATABASE_URL: ['aws-secrets'],
        JWT_SECRET: ['aws-secrets'],
        STRIPE_SECRET_KEY: ['aws-secrets'],
      },
    },
  },
});

enforceAllowedSources means a deploy that accidentally drops STRIPE_SECRET_KEY into process.env fails startup instead of silently overriding the value from AWS.

Runtime Redaction Without the Boilerplate

Knowing the secrets never reach process.env is half the battle. The other half is keeping them out of logs and HTTP responses.

import { resolve } from 'node-env-resolver';
import {
  protect,
  createResponseMiddleware,
} from 'node-env-resolver/runtime';
import { string, secret } from 'node-env-resolver/validators';

const config = resolve({
  API_KEY: string(),
  DB_PASSWORD: secret(),
});

const unprotect = protect(config);

console.log(`Connecting with key: ${config.API_KEY}`);
// Connecting with key: [REDACTED]

app.use(createResponseMiddleware(config));
// Response bodies are scanned for resolved secret values and redacted before send.

protect() patches console.log, console.error, and console.warn to match the resolved secret values and replace them with [REDACTED]. The Express and Hono middleware do the same scan on response bodies.

It’s a safety net, not a silver bullet. If a secret arrives through a different channel, or gets transformed before logging, the redactor won’t catch it. Treat it as defence in depth on top of not leaking in the first place.

Provenance When You Need It

When an incident hits and someone asks “where did this value come from”, most apps have no answer. node-env-resolver records the source of every resolved value, auto-enabled in production:

import { getAuditLog, createDebugView } from 'node-env-resolver';

const events = getAuditLog();
// env_loaded  key=DATABASE_URL  source=processEnv
// env_loaded  key=API_KEY       source=aws-secrets  resolvedVia=aws-sm://prod/api-key

const debug = createDebugView(config);
// DATABASE_URL: [REDACTED]  source: processEnv
// API_KEY:      [REDACTED]  source: aws-secrets

getAuditLog() gives event history, createDebugView(config) gives a redacted current snapshot, and provenance tracks source metadata in both views. Raw values are never included, only names and sources.

Adopting it in an existing app

If you’re retrofitting this into a running service, the order matters more than the final shape. I’d do it in three passes:

Create a single config.ts and resolve everything there. Don’t change any callers yet. Just prove the schema validates at startup in every environment.
Move modules off direct process.env reads one at a time, passing Pick<typeof config, ...> into the functions that need them. Tests get easier with each one.
Turn on the stricter policies last: enforceAllowedSources, response middleware, audit logging. By this point the schema is the source of truth and you can tighten without hunting.

The mistake is doing all three at once. Each pass is a small, safe PR. Together they replace the ad-hoc config layer without a big-bang rewrite.

Try It Yourself

npm install node-env-resolver
# Optional integrations
npm install node-env-resolver-aws

Minimal usage, sync:

import { resolve } from 'node-env-resolver';
import { postgres, string, number } from 'node-env-resolver/validators';

export const config = resolve({
  PORT: number({ default: 3000 }),
  DATABASE_URL: postgres(),
  API_KEY: string({ min: 32 }),
});

app.listen(config.PORT);

With AWS Secrets Manager, async:

import { resolveAsync } from 'node-env-resolver';
import { processEnv } from 'node-env-resolver/resolvers';
import { awsSecrets } from 'node-env-resolver-aws';
import { postgres, string, number } from 'node-env-resolver/validators';

export const config = await resolveAsync({
  resolvers: [
    [processEnv(), { PORT: number({ default: 3000 }) }],
    [awsSecrets({ secretId: 'prod/app' }), {
      DATABASE_URL: postgres(),
      API_KEY: string(),
    }],
  ],
});

Scan a repo for hardcoded secrets before they ship:

npx node-env-resolver scan src/
npx node-env-resolver scan --staged

The Config You Want

One file, every environment:

// config.ts
import { resolve } from 'node-env-resolver';
import {
  postgres,
  redis,
  string,
  number,
  boolean,
  duration,
  email,
  secret,
} from 'node-env-resolver/validators';

export const config = resolve({
  PORT: number({ default: 3000 }),
  NODE_ENV: ['development', 'production', 'test'] as const,

  DATABASE_URL: postgres(),
  DB_POOL_SIZE: number({ default: 10 }),
  REDIS_URL: redis({ optional: true }),
  CACHE_TTL: duration({ default: '5m' }),

  JWT_SECRET: secret(),
  JWT_EXPIRY: duration({ default: '1h' }),

  ADMIN_EMAIL: email(),
  ENABLE_METRICS: boolean({ default: false }),
  LOG_LEVEL: ['debug', 'info', 'warn', 'error'] as const,
});

Modules take what they need:

// database.ts
export function createDatabase(
  cfg: Pick<typeof config, 'DATABASE_URL' | 'DB_POOL_SIZE'>,
) {
  return connect(cfg.DATABASE_URL, { poolSize: cfg.DB_POOL_SIZE });
}

// server.ts
import { config } from './config';
import { createDatabase } from './database';

const db = createDatabase(config);
app.listen(config.PORT);

Tests stop mocking process.env:

it('connects with the provided URL', () => {
  const db = createDatabase({
    DATABASE_URL: 'postgres://testuser:testpass@localhost:5432/testdb',
    DB_POOL_SIZE: 2,
  });
  // assertions
});

Zod and Valibot schemas are supported via resolveZod and resolveValibot if you’re already using them.

Repo: github.com/jagreehal/node-env-resolver

Log the Fingerprint, Not the Secret

Tue, 21 Apr 2026 00:00:00 GMT

When something goes wrong with your application’s configuration, you need to know what was loaded and where it came from.

Most tools for answering that question either tell you too much or nothing at all.

There’s a third option most people haven’t considered.

tl;dr: At startup, log SHA-256 fingerprints of secrets, not the secrets themselves. You can verify consistency and rotation without exposing credentials.

It’s 2 a.m. An alert fires… payment processing is down. Your first instinct is a credentials issue: something rotated, or didn’t rotate when it should have.

You need to know which secrets were loaded at startup, which resolver provided them, and whether the rotation propagated correctly.

Most config systems leave you with two choices:

Option A. Add debug logging. Print config values at startup so you can see what the application saw. The answer appears, and so do your database password and payment processor API key, now living in your log aggregator, possibly forwarded to a third-party service, accessible to anyone with log access. You’ve answered the debugging question by widening the blast radius.

Option B. Don’t log values. Fly blind. Redeploy, restart, check metrics, try to reason about what the application saw without being able to see it directly.

Neither is good.

The assumption underneath both options

To debug configuration, people assume they need to see the values. They usually don’t.

What you need to know in a debugging scenario:

Did this variable resolve at all?
Which resolver provided it (.env, process.env, AWS Secrets Manager)?
Did the rotation work? Is the value different from what it was yesterday?
Is every pod seeing the same value?

None of those questions require seeing the plaintext value. They require an identity signal: something that lets you verify whether a value is present, correct, and consistent without being the value itself.

A cryptographic fingerprint is that signal.

For very low-entropy values (e.g., a boolean flag, a small integer timeout, a known environment name like production), a SHA-256 fingerprint can be brute-forced by an attacker who guesses the few possible inputs.

In those cases, prefer length or none over fingerprint. For high-entropy secrets (API keys, passwords, tokens), fingerprinting is safe.

A common middle ground is redaction: logging [REDACTED] in place of the value. That tells you a value existed, but nothing else: not whether it changed, not whether pods are consistent.

Redaction hides the secret but gives you no verification signal. Fingerprints give you both.

What fingerprints give you

A short SHA-256 fingerprint usually gives you enough signal to debug an incident without exposing the secret itself.

If DATABASE_URL produces fingerprint sha256:4f91b0c2 across all your pods, the value is consistent. If one pod shows sha256:4f91b0c2 and another shows sha256:94e1b7aa, something is wrong and you’ve located it without printing either credential. If after rotating a secret the fingerprint changes from sha256:4f91b0c2 to sha256:7a3c9f11, the rotation propagated. If it didn’t change, it didn’t.

Truncating the hash to eight hex characters leaves about 4 billion possible values. For a handful of config keys per service, the chance of two different secrets producing the same prefix is negligible.

The prefix is for operational convenience (short log lines, easy to grep), not a security boundary, and the full hash is available if you ever need it.

The sha256: prefix is part of the fingerprint string by convention, which leaves room for a stronger algorithm later (say sha512: or blake3:) without breaking consumers that parse the prefix.

What safe debug output looks like

In any language, the pattern is the same: compute a SHA-256 fingerprint, log it with metadata like key and source, then discard the plaintext. Here’s how it looks with node-env-resolver:

import { resolveAsync } from 'node-env-resolver';
import { postgres, string } from 'node-env-resolver/validators';
import { awsSecretHandler } from 'node-env-resolver-aws';

const debugEntries = [];

const config = await resolveAsync({
  schema: {
    DATABASE_URL: postgres(),
    STRIPE_SECRET_KEY: string(),
  },
  references: {
    handlers: {
      'aws-sm': awsSecretHandler,
    },
  },
  options: {
    debug: {
      enabled: true,
      valueMode: 'fingerprint',
      includeSource: true,
      onDebugEntry: (entry) => debugEntries.push(entry),
    },
  },
});

logger.info({ event: 'config_loaded', entries: debugEntries }, 'Config loaded');

Each entry looks like this:

{
  "key": "DATABASE_URL",
  "resolved": true,
  "source": "process.env",
  "reference": "aws-sm://prod/app/database_url",
  "resolvedVia": "aws-sm",
  "sensitive": true,
  "length": 53,
  "fingerprint": "sha256:d0976d8b",
  "preview": null
}

No plaintext. No partial reveal. Source, provenance, fingerprint. These answer most of the questions you have during an incident.

The reference and resolvedVia fields show the full chain: the reference string loaded from process.env, the handler (aws-sm) that dereferenced it, and the resulting fingerprint of the final value.

Incident response in practice

Back in the 2 a.m. scenario. You pull the config snapshot from your log aggregator:

jq -r '.entries[] | "\(.key) \(.fingerprint // "none") \(.resolvedVia // .source)"' config-snapshot-*.json | sort

DATABASE_URL sha256:4f91b0c2 aws-sm
STRIPE_SECRET_KEY sha256:7a3c9f11 aws-sm

Across all pods, the fingerprints are consistent and the sources are correct.

Check whether the rotation propagated:

# yesterday
DATABASE_URL sha256:e2d9a1c3 aws-sm

# tonight
DATABASE_URL sha256:4f91b0c2 aws-sm

Fingerprint changed. Rotation worked. Confirmed in thirty seconds, from logs.

Post-incident rotations are this scenario at scale. After the Vercel April 2026 security incident, affected customers were asked to rotate any credentials stored as non-sensitive environment variables. That means dozens of secrets rotated under time pressure, each requiring confirmation that every pod picked up the new value. Fingerprint logs turn that confirmation into a diff; without them, you’re trusting that rotation worked.

The four visibility modes

The debug system supports four modes:

Mode	What it shows	When to use
`none`	Key, resolved state, source	Default production logging
`length`	Above + value length	Minimal leak risk, basic verification
`fingerprint`	Above + SHA-256 hash, length	Incident response, rotation verification
`masked`	Partial preview (first/last chars)	Never in production, only for local troubleshooting

For production incident diagnostics, fingerprint is often the most useful choice for high-entropy secrets. Stricter environments may prefer none by default and only turn on fingerprinting during an incident. length sits between them when you want the lowest possible leak surface and only need to confirm a value was loaded. masked exists for local development only: a partial reveal like sk_live_...4f91 still leaks information and should never appear in production logs.

For hard masking of values that slip into console logs or HTTP responses, libraries like node-env-resolver’s protect() replace sensitive values with [REDACTED] at the output boundary: a different tool, for a different job.

Log once, at startup

Log the snapshot once when config loads. The goal is a startup audit trail, not request-level logging.

// At startup, after resolve():
logger.info(
  {
    event: 'config_loaded',
    service: process.env.SERVICE_NAME,
    entries: debugEntries,
  },
  'Config loaded successfully',
);

Store these in a place your security team can access during an incident, such as a separate log stream or dedicated audit index that won’t get mixed in with noisy application logs and rolled off before anyone needs it.

The principle

The assumption that you need to see a value to know it’s correct is wrong. A fingerprint tells you whether a value is present, whether it is consistent across instances, and whether it has changed.

Observability and security don’t have to conflict. The tension comes from bad tooling. With identity signals, you can verify configuration state without exposing the values themselves.

If a secret ends up in your logs, your incident response just created a new incident.

Fingerprints let you answer the question without creating the problem.

Your .env File Is Not a Secret Store

Tue, 21 Apr 2026 00:00:00 GMT

Most teams make one of two mistakes with secrets.

The first is obvious.

The second is more common in teams that have already fixed the first one.

Both come from the same misunderstanding about what a .env file is supposed to do.

tl;dr: Don’t use committed or shared .env files as a secret store. Load secrets from a dedicated manager at startup. Merge multiple config sources in a single pass.

Everything can look right on the surface. Let’s take a team that has AWS Secrets Manager set up, credentials rotated on a schedule, production access locked down.

Application code fetched secrets from the manager at startup. They’d taken the security seriously.

Then I looked at their .env:

DATABASE_URL=postgres://user:pass@prod-db.internal/app

The application code fetched from AWS, but the .env held a copy of the same credential: the value someone had pasted in months ago. The secret had rotated twice since. The .env hadn’t.

Two sources of truth. No connection between them.

The two mistakes

Mistake one is obvious: putting plaintext credentials directly into .env files. Now everyone on the team effectively has the prod password. It appears in git blame. It gets pasted into Slack when someone shares their config for debugging. This mistake is well understood, and many teams have moved past it.

Mistake two is less obvious. Teams set up a secret manager (AWS Secrets Manager, Vault, Doppler, 1Password Secrets) and write code to fetch secrets at startup. But nothing links the config system to the secret manager. Someone rotates a secret; the application still reads the old value. The rotation happened; the propagation didn’t.

Both mistakes come from the same root cause: treating .env as a configuration database instead of a local convenience.

.env is not the environment

12-factor says config should come from environment variables injected at runtime. A .env file is just a development convenience for populating those variables locally.

It is not the environment itself. The execution environment (a container orchestrator, a cloud platform, a CI runner) can inject values from any number of backing stores.

The natural extension: secrets should come from secret managers, not from .env files. Non-secret config can come from .env, process environment, CLI arguments, or anywhere else.

Your application should know how to pull from multiple sources and merge them into a single, validated config object.

This is what it looks like with a resolver-based approach. The pattern is implementable with any config system. The resolver approach shown here is one example:

import { resolveAsync, processEnv } from 'node-env-resolver';
import { dotenv } from 'node-env-resolver/resolvers';
import { awsSecrets } from 'node-env-resolver-aws';
import { postgres, string } from 'node-env-resolver/validators';

const config = await resolveAsync(
  // Non-secret config from .env
  [dotenv({ path: '.env' }), {
    NODE_ENV: ['development', 'production'] as const,
    LOG_LEVEL: 'info',
    PORT: 3000,
  }],
  // Secrets from AWS, never touch .env
  [awsSecrets({ secretId: 'prod/app' }), {
    DATABASE_URL: postgres(),
    STRIPE_SECRET_KEY: string(),
  }],
);

Each resolver is a named source. Later resolvers override earlier ones by default. Validators run on the merged result. Your application gets one typed config object and never needs to know where each value came from.

flowchart LR
    A[".env"]:::config --> D["merge + validate"]:::decision
    B["process.env"]:::config --> D
    C["secrets manager"]:::config --> D
    D --> E["config object"]:::config

    classDef config fill:#e1f5ff
    classDef decision fill:#fff4e1

The diagram shows three common sources, but any number can participate: SSM Parameter Store, Docker secrets, CLI arguments, custom databases. Each plugs into the same pipeline.

Local development

Running locally without access to a secret manager is the most common objection. The resolver model handles it naturally: swap the source, keep the schema.

// Development: secrets from a gitignored .env.secrets
// Production: secrets from AWS
const isProd = process.env.NODE_ENV === 'production';

const config = await resolveAsync(
  [dotenv({ path: '.env' }), { NODE_ENV: ['development', 'production'] as const }],
  isProd
    ? [awsSecrets({ secretId: 'prod/app' }), { DATABASE_URL: postgres() }]
    : [dotenv({ path: '.env.secrets' }), { DATABASE_URL: postgres() }],
);

The .env.secrets file stays out of git. Developers populate it once with local test credentials. The schema and validation are identical. Only the source changes.

If you prefer a single config without conditionals, use priority control. Normally later resolvers override earlier ones; priority: 'first' reverses that:

const config = await resolveAsync(
  [dotenv({ path: '.env' }), { NODE_ENV: ['development', 'production'] as const }],
  [dotenv({ path: '.env.secrets' }), { DATABASE_URL: postgres() }],
  [awsSecrets({ secretId: 'prod/app' }), { DATABASE_URL: postgres() }],
  { priority: 'first' },  // Local .env.secrets wins if it exists
);

In production, .env.secrets doesn’t exist, so the resolver returns nothing and AWS provides the value.

The operational payoff

The practical improvement, no more manual copy-paste after secret rotation, is real. But the bigger gain is structural.

When config sources are explicit, the data flow is visible. You can see at a glance that DATABASE_URL comes from a secrets manager and PORT comes from the environment. You can audit which variables come from which providers. You can verify that production secrets don’t accidentally mix with staging config.

Provenance becomes part of the code, not tribal knowledge.

When something goes wrong, you know exactly where to look. Not “it might be the .env, or it might be the startup script, or it might be the Kubernetes secret.” One resolver, one chain.

For runtime secret rotation, resolvers fetch at startup by default. Most applications pick up rotated secrets on the next deploy or restart. For those that need live reload, wrapping a resolver in a short-TTL cache and calling resolve on each request works. It trades startup-time simplicity for per-request latency, so it is a deliberate choice rather than a free upgrade.

A concrete example

The Vercel April 2026 security incident is a public instance of this failure mode. An attacker compromised a third-party AI tool, pivoted through a Vercel employee’s Google Workspace account, and read environment variables from Vercel projects that were not flagged as “sensitive.” API keys, tokens, and database credentials stored as plain env vars were exposed. Vercel’s guidance after the incident is to mark secrets as “sensitive” going forward, so they can’t be read back from the dashboard.

That helps. A stronger mitigation is to not store the secret in the platform env layer at all. Keep a reference like aws-sm://prod/stripe-key in Vercel, and let the secret manager hold the value with its own access controls and audit trail. An attacker reading the dashboard sees a URI, not a credential. The resolver dereferences it at startup.

const config = await resolveAsync({
  resolvers: [[processEnv(), { STRIPE_KEY: string() }]],
  references: {
    handlers: { 'aws-sm': awsSecretHandler },
  },
});

This shifts the blast radius. Compromise of the platform control plane now exposes which secrets a project uses and where they live, not their values.

The principle

Most teams treat secret managers and configuration systems as separate concerns. One stores secrets; the other loads config. The problem is the gap between them: that’s where values get copied, drift, and go stale.

Explicit config sources close that gap. They turn config loading into a single, typed pass that pulls from the right places, validates everything, and gives your application a clean config object.

If your .env file isn’t safe to read on a shared screen, it’s doing too much.

Algebraic Thinking Without the Ceremony

Wed, 15 Apr 2026 00:00:00 GMT

Christian Ekrem’s post, Effect Without Effect-TS: Algebraic Thinking in Plain TypeScript, shows something useful: typed errors, explicit dependencies, and composition do not require a framework.

This post takes the same principles and shows where awaitly helps once the manual approach starts to repeat. The point is not “no ceremony.”

The point is where ceremony lives.

For most code, run() is the starting point. createWorkflow() is for orchestration features like retries, caching, observability, or managed dependency wiring.

Raw async/await       Manual Result pattern      awaitly
──────────────────    ──────────────────────      ──────────────────────
Least setup           Most boilerplate            Moderate setup
Least honesty         Most explicit control       Lower repeated boilerplate
Errors invisible      Errors visible (expected)   Stronger guarantees
                      Unexpected errors: 🤷        Unexpected errors: typed

The function that lies to you

async function signupUser(email: string, password: string): Promise<User> {
  if (!isValidEmail(email)) throw new Error('Invalid email');

  const existing = await db.findUserByEmail(email);
  if (existing) throw new Error('Email already registered');

  const user = await db.createUser({
    email,
    passwordHash: await hash(password),
  });

  await emailService.sendWelcome(user.email);
  await analytics.track('user_signed_up', { userId: user.id });

  return user;
}

Promise<User> looks honest, but it is not. The function can fail in several ways and depends on external systems. None of that is visible in the signature.

Idea 1: Honest errors

Manual Result style works:

type SignupError =
  | { _tag: 'InvalidEmail' }
  | { _tag: 'EmailTaken'; email: string }
  | { _tag: 'DbError'; cause: unknown }
  | { _tag: 'EmailServiceDown' };

type Result<T, E> = { ok: true; value: T } | { ok: false; error: E };

This is good engineering. You can share helpers and keep it clean. The maintenance cost appears when each module owns its own error union and every step change modifies that union.

Awaitly equivalent

import { ok, err, type AsyncResult } from 'awaitly';

const validateEmail = async (
  email: string,
): AsyncResult<string, 'INVALID_EMAIL'> =>
  isValidEmail(email) ? ok(email) : err('INVALID_EMAIL');

const checkDuplicate = async (
  email: string,
): AsyncResult<void, 'EMAIL_TAKEN'> => {
  const existing = await db.findUserByEmail(email);
  return existing ? err('EMAIL_TAKEN') : ok();
};

The difference here is small but practical: an AsyncResult alias, concise error codes, and ok() handling void naturally.

Idea 2: Honest dependencies

The dependency-in-signature pattern is excellent. awaitly supports that same shape in two levels.

Shape 1: `run` + `ErrorsOf`

Plain functions, with no framework registration:

import { ok, err, type AsyncResult, type ErrorsOf } from 'awaitly';
import { run } from 'awaitly/run';

const validateEmail = async (
  email: string,
): AsyncResult<string, 'INVALID_EMAIL'> =>
  isValidEmail(email) ? ok(email) : err('INVALID_EMAIL');

const findUser = async (email: string): AsyncResult<User | null, 'DB_ERROR'> =>
  ok(await db.findUserByEmail(email));

const checkNotTaken = async (
  user: User | null,
): AsyncResult<void, 'EMAIL_TAKEN'> => (user ? err('EMAIL_TAKEN') : ok());

const createUser = async (
  input: CreateUserInput,
): AsyncResult<User, 'DB_ERROR'> => ok(await db.createUser(input));

const sendWelcome = async (
  email: string,
): AsyncResult<void, 'EMAIL_SERVICE_DOWN'> =>
  ok(await emailService.sendWelcome(email));

const deps = {
  validateEmail,
  findUser,
  checkNotTaken,
  createUser,
  sendWelcome,
};
type SignupError = ErrorsOf<typeof deps>;

const result = await run<User, SignupError>(async ({ step }) => {
  const email = await step('validate', validateEmail(rawEmail));
  const existing = await step('find', findUser(email));
  await step('checkNotTaken', checkNotTaken(existing));
  const user = await step(
    'create',
    createUser({ email, passwordHash: await hash(password) }),
  );
  await step('welcome', sendWelcome(email));
  return user;
});

You can also keep fn(args, deps) exactly:

type SignupDeps = typeof deps;
type SignupErrors = ErrorsOf<SignupDeps>;

const signupUser = async (
  args: { email: string; password: string },
  deps: SignupDeps,
) => {
  return run<User, SignupErrors>(async ({ step }) => {
    const email = await step('validate', deps.validateEmail(args.email));
    const existing = await step('find', deps.findUser(email));
    await step('checkNotTaken', deps.checkNotTaken(existing));
    const user = await step(
      'create',
      deps.createUser({ email, passwordHash: await hash(args.password) }),
    );
    await step('welcome', deps.sendWelcome(email));
    return user;
  });
};

Testability stays the same as manual style: inject fake deps directly.

Shape 2: `createWorkflow` for when you need more

import { createWorkflow, ok, err, type AsyncResult } from 'awaitly';

const signup = createWorkflow('signup', {
  validateEmail,
  findUser,
  checkNotTaken,
  createUser,
  sendWelcome,
});

const result = await signup.run(async ({ step, deps }) => {
  const email = await step('validate', deps.validateEmail(rawEmail));
  const existing = await step('find', deps.findUser(email));
  await step('checkNotTaken', deps.checkNotTaken(existing));
  const user = await step(
    'create',
    deps.createUser({ email, passwordHash: await hash(password) }),
  );
  await step('welcome', deps.sendWelcome(email));
  return user;
});

The error union is inferred automatically from registered functions, with no separate ErrorsOf<...> type needed. You also get step caching, retry/timeout config, and observability events.

Workflows can also be statically analyzed with awaitly-analyze, which lets you generate diagrams like this. The result is living documentation that makes onboarding and code reviews easier.

flowchart LR
  VE["validateEmail"] -->|ok| FU["findUser"]
  FU["findUser"] -->|ok| CNT["checkNotTaken"]
  CNT["checkNotTaken"] -->|ok| CU["createUser"]
  CU["createUser"] -->|ok| SW["sendWelcome"]
  SW["sendWelcome"] -->|ok| Done((Success))
  VE -->|err| VEE["'INVALID_EMAIL'"]
  FU -->|err| FUE["'DB_ERROR'"]
  CNT -->|err| CNTE["'EMAIL_TAKEN'"]
  CU -->|err| CUE["'DB_ERROR'"]
  SW -->|err| SWE["'EMAIL_SERVICE_DOWN'"]

Idea 3: Composition

Manual andThen and early-return flows are valid and readable for small chains. As chains grow, composition noise grows too.

const result = await run(async ({ step }) => {
  const email = await step('validate', validateEmail(rawEmail));
  const existing = await step('find', findUser(email));
  await step('checkNotTaken', checkNotTaken(existing));
  const account = await step('create', createAccount(email));
  await step('welcome', sendWelcome(account.id));
  return account;
});

One gap manual Result often leaves open

The manual approach models expected errors well. A common gap is unexpected throws from dependencies.

const result = await run(async ({ step }) => {
  const email = await step('validate', validateEmail(rawEmail));
  const existing = await step('find', findUser(email));
  await step('checkNotTaken', checkNotTaken(existing));

  // Imagine createAccount throws internally
  const account = await step('create', createAccount(email));
  return account;
});

// result.error:
// "INVALID_EMAIL" | "EMAIL_TAKEN" | "DB_ERROR" | UnexpectedError

You can still build this boundary manually (runSafe wrapper). awaitly includes it by default.

`step.try()` for throwing APIs

const result = await run(async ({ step }) => {
  const data = await step.try('parse', () => JSON.parse(rawInput), {
    error: 'PARSE_FAILED' as const,
  });

  const response = await step.try('callSDK', () => thirdPartySDK.call(data), {
    onError: (e) => ({
      type: 'SDK_FAILED' as const,
      message: e instanceof Error ? e.message : String(e),
    }),
  });

  return response;
});

The trade-off is one more concept (step.try), in exchange for not re-writing try/catch-and-map code around every throwing dependency.

Scorecard

Dimension	Raw async/await	Manual Result	awaitly
Setup cost	None	`Result` type, `ok`/`err` helpers, `assertNever`	`npm install`, learn `run`/`step`/`ErrorsOf`
Per-module cost	Low	Error union + early returns per call	`step()` calls
Error visibility	None (catch unknown)	Expected errors typed	Expected + unexpected typed
Error union maintenance	N/A	Hand-maintained	Inferred (`ErrorsOf<typeof deps>` or `createWorkflow`)
Unexpected errors	Uncaught rejections	Usually uncaught unless you add safe boundary	`UnexpectedError` in union
Composition at scale	try/catch nesting	`andThen` nesting or early returns	`step()`
Dep injection	Module imports	`fn(deps, ...args)`	`fn(args, deps)` or `createWorkflow`
Test story	Often mocks/stubs	Pass fake deps	Pass fake deps

Takeaway

The original post teaches the right principles and is honest about trade-offs. awaitly is a lightweight extension on top: infer error unions with ErrorsOf, reduce repeated early-exit checks with step(), and include a default boundary for unexpected throws.

The ideas are similar. The difference is where the ceremony lives: hand-written per module, or centralized behind a small execution boundary.

The package manager settings that would have blocked the Axios attack

Tue, 07 Apr 2026 00:00:00 GMT

On the 31st of March 2026, attackers hijacked an npm maintainer account and published malicious versions of axios with a remote access trojan baked in. npm pulled the bad releases after about two or three hours, but that was enough. Anyone who ran npm install axios during that window could have installed the trojan. The article Post Mortem: axios npm supply chain compromise has all the details.

This kind of attack keeps happening and the playbook barely changes: compromise an account, push a malicious update, hope people install it before anyone notices, get removed a few hours later.

Every major package manager now lets you defend against this. In this post I’ll show you the setup for npm, pnpm, Bun and Yarn.

Why the Axios attack worked

The malicious versions were published as ordinary patch releases. If your project depended on "axios": "^1.14.0" and performed a fresh resolution without a lockfile pinning a safe version, install could resolve to the poisoned version automatically. And if the attacker included a postinstall script, npm ran it during installation without asking.

So one npm install could download the compromised package, run arbitrary code on your machine or CI runner, and exfiltrate credentials, tokens, SSH keys, source code - whatever it wanted.

The whole thing hinges on two assumptions: you install the bad version quickly, and npm runs the install script. The three settings break both.

1. Delay new releases

min-release-age=7

This tells npm to skip any version published in the last 7 days.

Seven days sounds like a lot. But most malicious releases get caught within hours. The Axios compromise was live for roughly 2-3 hours. With this setting, npm would have refused to resolve those versions until they were a week old, by which point they’d already been yanked.

Every major package manager has an equivalent, though the config key and units differ. npm calls this setting min-release-age (in days); pnpm and Bun call it minimumReleaseAge (minutes and seconds respectively); Yarn uses npmMinimalAgeGate (human-readable durations).

# pnpm-workspace.yaml (minutes, so 10080 = 7 days)
minimumReleaseAge: 10080

# bunfig.toml (seconds, so 604800 = 7 days)
[install]
minimumReleaseAge = 604800

# .yarnrc.yml
npmMinimalAgeGate: '7d'

2. Disable install scripts globally

ignore-scripts=true

npm packages can define lifecycle hooks (preinstall, install, postinstall) and by default npm runs them automatically. Attackers exploit this because their code executes during installation. You don’t even need to import the package.

With ignore-scripts=true, npm downloads the package but refuses to run any install-time code. The install-time RAT payload described in the Axios compromise would not have executed during install.

The tradeoff

Some legitimate packages need install scripts, especially native modules: sharp, bcrypt, sqlite3, esbuild, puppeteer. If something breaks, rebuild only the packages you trust:

npm ci --ignore-scripts
npm rebuild sharp esbuild

You can also override per project if you need to:

# project/.npmrc
ignore-scripts=false

3. Pin exact versions

save-exact=true

By default npm saves dependencies with a caret (^1.14.0), which means future patch and minor versions get pulled in automatically. That’s the convenience that made the Axios attack possible.

With save-exact=true, npm records "axios": "1.14.0". No range, no floating. You still update dependencies, but through a PR or lockfile update. Not because someone pushed a release five minutes ago.

Setup for your package manager

Pick yours, copy the config, and you are done.

npm

Requires npm v11.10.0 or later. To make sure your team is on a compatible version, add engines to your package.json and set engine-strict=true in .npmrc.

# ~/.npmrc (or project .npmrc)
ignore-scripts=true
min-release-age=7
save-exact=true
engine-strict=true

pnpm

minimumReleaseAge uses minutes. 10080 = 7 days.

# pnpm-workspace.yaml
minimumReleaseAge: 10080
onlyBuiltDependencies:
  - sharp
  - esbuild

pnpm can block arbitrary install scripts using onlyBuiltDependencies. Once configured, only the listed packages may run build/install scripts. Without it, lifecycle scripts still run normally.

Bun

minimumReleaseAge uses seconds. 604800 = 7 days. Bun is more restrictive by default: it does not run arbitrary dependency lifecycle scripts, uses a built-in allowlist for many common packages, and lets you allow additional ones with trustedDependencies in package.json. If you want to disable lifecycle scripts entirely, use bun install --ignore-scripts.

# bunfig.toml
[install]
minimumReleaseAge = 604800

// package.json
{
  "trustedDependencies": ["sharp", "esbuild"]
}

Yarn (current releases)

npmMinimalAgeGate accepts human-readable durations. Disable scripts globally and allow them per package with dependenciesMeta.

# .yarnrc.yml
npmMinimalAgeGate: '7d'
enableScripts: false
dependenciesMeta:
  sharp:
    built: true
  esbuild:
    built: true

Lockfiles and CI

Even with these settings, commit your lockfile. A committed package-lock.json (or pnpm-lock.yaml, bun.lock, yarn.lock) means CI gets the same dependency graph every time, fresh installs don’t resolve to something unexpected, and dependency updates show up in PRs where you can actually review them.

In CI, always use the lockfile-only install command for your package manager:

npm ci              # not npm install
pnpm install --frozen-lockfile
bun install --frozen-lockfile
yarn install --immutable

npm install in CI can silently update the lockfile and pull in new versions. npm ci refuses to do that. The other managers behave the same way with their respective flags.

Commit the lockfile. Review lockfile diffs. Update dependencies on purpose. Don’t let CI grab whatever happens to be newest.

Why this combination works

The Axios attack needed three things to go right: the victim installs quickly, the dependency range floats to the bad version, and install scripts run automatically.

What the attacker needs	What blocks it
Install before the package is noticed	`min-release-age=7`
Auto-upgrade to the newest release	`save-exact=true`
Execute malware during install	Disable install scripts / allowlist trusted packages

Most supply chain attacks follow the same pattern. These settings won’t stop everything, but they would have stopped this one, and they will probably stop the next few too.

Stay safe out there

Scroll up to the Setup for your package manager, pick your package manager, and paste the config.

Commit your lockfiles, use lockfile-only installs in CI, review dependency changes in PRs, and keep an allowlist for packages that genuinely need install scripts.

Engineering decisions are bets, not proofs

Mon, 06 Apr 2026 00:00:00 GMT

Most engineering decisions are not proofs.

They are bets.

You choose a direction with incomplete information, under time pressure, and with trade-offs you cannot fully test in advance.

That does not make the decision weak.

It makes it real.

In a previous post, I wrote about how design by committee leaves engineering change unfinished. The deeper reason is simple: many organisations treat technical decisions as though they should be certain before the work begins. But most meaningful engineering decisions are not certainties. They are bets.

Annie Duke puts this well in Thinking in Bets. Decision-making is more like poker than chess. In chess, everything is visible. In poker, you work with hidden information, incomplete knowledge, and luck. You can play the hand perfectly and still lose.

Engineering decisions work the same way. You choose a direction based on what you know now, not what you will know in six months. The quality of a decision is not whether it later turned out to be right. It is whether it was a good bet given the information available at the time.

The outcome trap

Teams often judge decisions by outcomes alone: good outcome, good decision; bad outcome, bad decision.

But that is too simple. A good decision can have a bad outcome. A bad decision can get lucky.

Norman Kerth put it well in his Prime Directive for retrospectives:

“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.”

If the team made a reasonable bet with the information they had, the fact that it did not work out does not mean the decision was wrong. It means the context changed, or something was unknowable.

I have seen teams adopt a framework that later proved the wrong fit, and conclude that the decision-making process itself was broken. But when you looked back at the information they had at the time, the decision was reasonable. What changed was the context. New requirements appeared. The team’s needs shifted. The framework’s maintainer abandoned it.

I have also seen teams make a questionable architectural choice that happened to work because the product never scaled past the point where the weakness mattered. That is not a good decision. That is luck.

If you only evaluate decisions by results, you learn the wrong lessons. You punish reasonable bets that did not pay off and reward reckless ones that happened to land. The right response is to update the bet, not to blame the people who made it.

What good bets look like in engineering

Why now?

Not every decision needs to be made upfront. We once delayed a data model decision on purpose because the first integration was still unclear. Two weeks later, the integration taught us what mattered and the choice became obvious.

When a decision does need to be made, there should be an honest answer to “why this, why now.”

What are the trade-offs?

The useful conversation is usually about what gets worse, not what gets better. You can feel it when someone says, “This speeds up deploys, but on-call gets harder for teams without strong Kubernetes skills.” That is a real trade-off.

How small can we make the bet?

Instead of migrating the whole platform, move one service first. You learn more from the first awkward deployment than from three meetings about what might happen.

How will we test it?

If you are betting that Kubernetes will simplify deployment, the first migrated service should tell you whether that is true. If you are betting on event-driven architecture, the first event contract usually exposes the integration pain you were hand-waving in diagrams.

What would make us change our mind?

This is where teams fool themselves. The useful question is, “What result would make us stop and rethink?” A short ADR helps when it captures what we believed and what would trigger a change, not when it tries to predict every edge case.

Is it reversible?

Not every decision can be reversed cheaply, but reversible bets should move with less ceremony. If a choice can be rolled back in a day, decide quickly and learn. If it locks you into years of tooling and contracts, slow down and pressure-test it harder.

Some teams formalise this by constraining time and scope. Basecamp’s Shape Up process, for example, treats projects as bounded bets: decide how much time something is worth, shape the work to fit, and if it does not ship within six weeks it does not automatically get more time. The principle is simple. Smaller bets create faster learning.

The learning loop

At University Press, when we migrated services onto a new platform, we treated each service as a bet. We expected some assumptions to be wrong. The point was not to be right the first time. It was to make being wrong early cheap.

One team discovered that their networking assumptions were off. Another found that the deployment configuration needed to be more flexible than anyone had anticipated. Those were not failures. That was the loop working.

That is the learning loop: make the decision, observe what happened, separate what was skill from what was luck, and update your beliefs.

Most teams skip the middle step. They see a result and jump to “we were right” or “we were wrong” without asking how much came from the decision itself versus circumstances outside their control.

The lesson should not be “we should have known.” It should be “we learned, and now we update.”

Two heads are better than one, but not twenty

Getting a second perspective helps you spot blind spots, challenge assumptions, and test whether your reasoning holds up. Duke calls this the “buddy system” and I think she is right about it.

Two or three people pressure-testing a decision is valuable. But there is a threshold where adding more people stops improving the decision and starts slowing it down. That threshold is lower than most organisations think.

The best engineering decisions I have been part of involved a small group who understood the problem, had the relevant context, and were willing to disagree with each other. The worst involved large groups where nobody had enough context to push back meaningfully, so the discussion became about politics and risk aversion rather than trade-offs and evidence.

This is the same dynamic that turns ADRs into bureaucracy. The instinct is to involve more people to reduce risk. But past a certain point, involvement does not reduce risk. It diffuses ownership.

Pre-mortem and backcasting

In her book there are two techniques Duke discusses.

A pre-mortem starts with a simple question: imagine this decision failed. Why? This forces the team to think about risks they might otherwise dismiss because they are optimistic about the outcome. It is not about pessimism. It is about preparation.

Backcasting asks the opposite: imagine this decision succeeded. What did we do to get there? This helps identify the steps that actually matter and separates them from the busywork that feels productive but does not move the needle.

Both work backwards from reality rather than forwards from assumptions. They are also quick. A fifteen-minute pre-mortem before a significant decision is worth more than a two-hour planning meeting that tries to account for every scenario.

ADRs as bet records

If you think of decisions as bets, ADRs become something different. They are not proof that the team thought carefully. They are a record of the bet.

What did we believe at the time? What information did we have? What were the alternatives? What trade-offs did we accept? What confidence level did we have? And what would cause us to revisit this?

That is a useful document. It helps future teams understand not just what was decided, but the state of knowledge at the time. It makes it easier to judge whether the decision should be revisited because the context has changed, or whether the original reasoning still holds.

The key is brevity. A bet record should be short because the bet itself should be bounded. If the ADR is long, the decision is probably too big.

Show, don’t tell

The best teams I have seen do not close a migration plan and declare certainty. They move one service, hit a rough edge, update the record, then move the second service with better knowledge.

You can watch the behaviour: an assumption breaks, the team rewrites the ADR while the details are fresh, guardrails get sharper, and the next team starts from a better place than the one before.

That is what good decision-making looks like in engineering. Not proving the future on paper, but learning faster than the uncertainty in front of you.

Why design by committee leaves engineering change unfinished

Sun, 05 Apr 2026 00:00:00 GMT

Engineering change often gets stuck for the same reason: design by committee. Not because anyone has bad intentions, but because the pursuit of alignment quietly replaces the pursuit of learning.

It usually starts with good intentions. People want alignment, consistency, and lower risk. So a proposed change gets pulled into more meetings, more reviews, more stakeholders. Before long, the goal is no longer to try something, learn from it, and improve it. The goal becomes finding one perfect solution for every team.

That is usually where momentum dies.

Because in real organisations, there is rarely one perfect solution. Teams carry different risks and constraints, and those differences show up the moment delivery starts.

I have watched plans look complete in review and come apart in week one because one service had awkward networking, another had brittle deployment scripts, and a third had compliance constraints nobody raised in the room.

That is not failure. That is normal.

The failure is building a process that treats learning as a problem instead of the whole point.

The biggest risk is not just slowness

When change is driven by committee, the biggest problem is often not speed. It is that it never really finishes.

Instead of delivering something small and useful, organisations create another half-complete initiative stranded in no man’s land. The old way has already been disrupted. The new way is not fully in place. The documentation no longer matches reality. Teams are unclear on what they are supposed to follow. And the organisation is carrying the cost of both the old world and the new one without getting the full benefit of either.

That is how things get worse, not better.

You end up with yet another initiative that started with a grand ambition and ended with confusion and drift. Nobody can clearly say it succeeded, but everyone has to live with the mess it left behind.

To me, that is one of the strongest arguments against trying to solve too much upfront.

If you make the scope too broad and the coordination too heavy, you massively increase the chance that nothing lands cleanly. You do not just delay progress. You create unfinished change.

And unfinished change is expensive.

The silver bullet mindset

There is a belief that enough meetings and smart people in a room can design the final answer upfront. It sounds responsible. It often produces the opposite of progress.

A new platform, architecture, or engineering practice cannot be discussed into existence. The assumptions that look sensible in a slide deck are not the same ones that survive contact with a real codebase, a real deployment pipeline, or a real team under pressure.

That is why I keep coming back to a simple pattern:

Make it work. Make it good. Make it fast.

In that order.

Too many organisations invert that. They try to make it perfect before they have made it real.

ADRs are not the problem

I am not against ADRs. Martin Fowler’s description of what they should be is close to what I actually want. Short. A page, maybe two. One decision per record. The context that led to it, the alternatives considered, the trade-offs, the consequences. Immutable once accepted. If the thinking changes, you write a new record that supersedes the old one.

Fowler also makes a point I think is underappreciated: the act of writing an ADR is often more valuable than the record itself. Writing a document of consequence forces you to say what you actually believe and why. In a group, that surfaces disagreements that might otherwise stay hidden until implementation. That clarifying effect is the real value.

If every team used ADRs the way Fowler describes, I would have nothing to write about. But they do not. My problem with ADRs has never been the format. It is what happens to them in practice.

Michael Nygard coined the term in 2011. His argument was simple: large documents are never kept up to date, nobody reads them, and without understanding the rationale behind a decision people either blindly accept it or blindly change it. Both are bad. His solution was a short text file, one or two pages, written as a conversation with a future developer. Context, decision, consequences. That is it.

Fifteen years on, teams still get this wrong in the same ways.

The failure modes are well documented by people like Zimmermann, Keeling, and others. Reviewing wording instead of decision quality. Skipping alternatives. Reopening accepted records instead of superseding them. ADRs becoming architecture bureaucracy. Records written long after the decision was forgotten, or written up after the fact to justify a choice already made in a hallway conversation.

I have seen most of these in person. ADRs that try to be mini design documents. ADRs so short they record the decision but not the reasoning. Records created after implementation rather than before. Documentation theatre, where the ADR exists to satisfy a process rather than support a decision.

ADRs absorb the habits of the organisation using them. When a company falls into design by committee, ADRs become one of the places that habit shows up. Instead of recording a decision, they become a vehicle for trying to eliminate uncertainty upfront. They stop being a note of what we believe now and start becoming an attempt to define the whole future before the work begins.

That is when they start to behave like waterfall.

A bloated ADR tries to solve the needs of many teams, in many contexts, all at once. It leaves little room to say, “This is our current bet, but what we learn in step one may change step three.” It starts sounding less like a decision record and more like a contract with a future nobody fully understands yet.

A good record should follow the inverted pyramid: the decision and rationale first, details later. It should say what we think, why we think it, what constraints matter, and what we expect to learn. If there is supporting material, link to it. And when reality gives us better information, we write a new record that supersedes the old one. That way there is a clear log of decisions and how long each one governed the work.

I am not arguing against ADRs, and I am not arguing for chaos. Written decisions can be useful. The problem starts when teams treat them as a way to discover the final answer before they have learned enough to know it. In practice, meaningful engineering change is iterative. You make a sensible bet, try it in a real context, learn from the result, and refine the decision. The real question is not whether you wrote an ADR. It is whether the process helped you move, learn, and adapt.

Two migrations, two approaches

I have seen both sides of this.

At a previous company, the leader announced to the entire organisation that everything would be on Kubernetes within three months. The decision had the confidence and theatre of certainty, but not the evidence. I remember sketching the iceberg model to show how much of the real implementation work was still hidden beneath the surface. The three-month target was not a plan. It was a guess dressed up as a commitment.

Unsurprisingly, we missed the target. A follow-up message had to explain that it had not happened. You cannot commit to a deadline for something you do not yet understand.

At University Press, we approached a similar migration very differently. We moved services onto Kubernetes across multiple teams. We did not rely on formal ADRs or a large upfront design exercise. Instead, teams stayed aligned through regular cross-team conversations, a shared direction of travel, and a leaner implementation approach. We did just enough to move forward, learned from the work, and adapted based on reality.

I have written before about the importance of understanding the problem before changing the process. That is what we did here.

We started with one service. That gave us real information very quickly. We learned what deployment actually needed, what configuration mattered, what the rough edges were, and what flexibility teams would need in practice. From there, the ops people could define sensible constraints without pretending they should dictate every implementation detail for every team.

That effort succeeded not because we had the perfect document, but because we created the conditions to learn. One was decree first, learning later. The other was vision first, learning through execution.

Large-scale engineering change always contains unknown unknowns. Some things can be reasoned about upfront. Others only reveal themselves in delivery. That is why the goal should not be to eliminate uncertainty on paper, but to reduce it through small moves, fast feedback, and real learning.

One pattern does not mean one identical solution

Success does not mean every team ends up using the exact same solution in the exact same way. In fact, that is often a sign the standard is too rigid. A service with complex networking needs is not the same as a simpler internal application. A mature team with strong operational habits is not in the same place as a team still dealing with legacy systems. If you force both into the same implementation, one of them will carry unnecessary pain.

So the goal should not be to impose one identical answer for everybody.

The goal should be to define the lines clearly enough that teams can colour within them, while still having the agency to choose what works best for their own problem.

That is not a failure of standardisation. It is a smarter form of it.

The platform team might define the shape of the contract: security expectations, deployment structure, required interfaces, observability standards, the mechanism by which a service is discovered and deployed. But within those boundaries, teams should have room to decide what makes sense for their service’s lifecycle, networking needs, runtime, and operational realities.

For example, if an organisation adopts event driven architecture, it may matter that events follow the same naming conventions, versioning rules, and schema standards. It may matter that teams agree what guarantees exist around ordering, retries, and idempotency. But it does not necessarily matter whether one team uses Kafka and another uses RabbitMQ, if both satisfy the same contract.

Good standards define the boundaries that let teams work together. Bad standards try to make every team solve the problem in exactly the same way.

You want consistency where consistency is valuable, and flexibility where flexibility is necessary.

How you drive change determines what kind of team you get

When change is imposed through a committee mandate and a rigid document, you can see disengagement quickly. Teams attend the meeting, stop challenging assumptions, wait for the approved pattern, and implement the minimum needed to satisfy the document. The awkward edge cases sit unowned because nobody feels accountable for improving the model itself.

When change is driven by people who go first and learn in public, behaviour changes. One team solves secrets management first. Another finds the networking issues. Another improves the deployment path. Those lessons feed back into shared guidance so the next team starts with fewer mistakes.

That is how organisations learn well. Not through one giant design produced centrally, but through smaller experiments feeding better knowledge back into the system.

Stop confusing alignment with progress

One reason design by committee is so seductive is that it looks productive.

There are meetings, diagrams, approvals, and a polished document. It feels like progress because everyone is busy.

Then the mess shows up. The old approach is half-deprecated. The new one is half-adopted. Documentation contradicts what teams are actually doing. Different teams believe they are compliant while making different trade-offs. The committee keeps meeting because nobody wants to say the grand plan never fully landed.

Activity is not the same as learning, and it is not the same as delivery.

A finished small success is worth more than a grand cross-team programme that drifts into another unfinished initiative. One leaves you with evidence and something to build on. The other leaves you with partial adoption, confused teams, and stale documentation.

My view

As usual, the answer is balance. Doing nothing is one extreme. Decisions live in people’s heads, get forgotten, and get relitigated six months later. That is how organisations lose institutional memory. Trying to design the final, absolute answer upfront is the other extreme. That is how organisations lose momentum.

Engineering change is not a finite problem you can solve on paper. It is an ongoing process of making bets, learning from them, and adjusting. Good organisations build that learning into the way they make decisions, not around it. The goal is not to be certain upfront. It is to make the best bet you can with the information you have, then update quickly as reality gives you better information. I wrote more about this in Engineering decisions are bets, not proofs.

ADRs are a tool. A good one, when used well. But no document can substitute for actually doing the work and learning from it.

I am not anti-ADR. I am anti-pretending that a document can replace learning. In engineering change, you usually understand more by getting something real into motion than by trying to agree the whole future in advance.

Start small. Make a bet. Try it in practice. Give teams guardrails, not handcuffs. Let them learn what works for their own context. Finish something real. Then spread what you learned.

That is how change becomes real.

Make it work. Make it good. Make it fast.

fn(args, deps) Is Not Anti-Mock

Mon, 23 Mar 2026 00:00:00 GMT

People hear “don’t use vi.mock” and assume the alternative is “don’t mock anything.” That is not the argument.

The real distinction is simpler: mock explicit collaborators, not implicit imports.

The earlier post argued that vi.mock tends to couple tests to module wiring, while dependency injection tends to couple tests to the contract your code actually depends on.

That remains true.

But it leaves a follow-up question:

If the function already follows fn(args, deps), what should the test use for deps?

The answer is simple:

for unit tests, pass a fake deps object
if you want the nicest ergonomics, generate that object with vitest-mock-extended
reserve vi.mock for the cases where you truly cannot inject the dependency cleanly

That is not a different philosophy. It is the same one, applied consistently. The fn(args, deps) pattern makes dependencies explicit, local, and easy to substitute in tests. Tests should pass deps as data, not intercept the module system with global shared state.

The misunderstanding

Here is the mistaken mental model:

vi.mock = mocking
dependency injection = no mocking

That is the wrong split.

The real split is:

module mocking: replacing what code imports
dependency mocking: passing in a fake implementation of what the function declares it needs

Those are not the same thing.

The first changes module resolution. The second passes a different argument.

That is why fn(args, deps) fits so naturally with test doubles. The pattern already separates:

what varies per call: args
what the function needs from the outside world: deps

That separation is the boundary. The test just supplies a different deps.

The default many teams reach for

Take a simple use case:

import { saveUser } from './db';
import { sendWelcomeEmail } from './mailer';

export async function createUser(name: string, email: string) {
  const user = await saveUser({ name, email });
  await sendWelcomeEmail(user);
  return user;
}

To unit test this, the usual move is:

vi.mock('./db', () => ({
  saveUser: vi.fn(),
}));

vi.mock('./mailer', () => ({
  sendWelcomeEmail: vi.fn(),
}));

That works.

It also couples the test to:

the file paths
the export names
Vitest’s hoisting rules
global mock state that has to be managed between tests

This is the architecture critique behind the anti-vi.mock argument: the test is asserting on module wiring, not just behavior. vi.mock relies on global shared state, while fn(args, deps) uses direct injection and each test owns its own dependencies.

The shape we actually want

Now look at the same use case with explicit dependencies:

export type CreateUserArgs = {
  name: string;
  email: string;
};

export type CreateUserDeps = {
  db: {
    save: (user: { name: string; email: string }) => Promise<User>;
  };
  mailer: {
    sendWelcome: (user: User) => Promise<void>;
  };
};

export async function createUser(args: CreateUserArgs, deps: CreateUserDeps) {
  const user = await deps.db.save(args);
  await deps.mailer.sendWelcome(user);
  return user;
}

Now the unit test has no reason to touch vi.mock at all:

import { describe, it, expect, vi } from 'vitest';
import { createUser } from './createUser';

describe('createUser', () => {
  it('creates a user and sends welcome email', async () => {
    const mockUser = { id: '1', name: 'Alice', email: 'alice@test.com' };

    const deps = {
      db: { save: vi.fn().mockResolvedValue(mockUser) },
      mailer: { sendWelcome: vi.fn().mockResolvedValue(undefined) },
    };

    const result = await createUser(
      { name: 'Alice', email: 'alice@test.com' },
      deps,
    );

    expect(result).toEqual(mockUser);
    expect(deps.mailer.sendWelcome).toHaveBeenCalledWith(mockUser);
  });
});

No hoisting. No path coupling. No module interception. The mock is just an object. The function signature makes the seam explicit, and the test uses it. That is exactly the testing story fn(args, deps) is designed to support.

Where vitest-mock-extended fits

The obvious objection is ergonomic.

Manually writing fake dependency objects is fine at first. But once the dependency surface gets slightly deeper, hand-writing every nested vi.fn() becomes repetitive.

That is where vitest-mock-extended earns its keep.

import { describe, it, expect } from 'vitest';
import { mock } from 'vitest-mock-extended';
import { createUser, type CreateUserDeps } from './createUser';

describe('createUser', () => {
  it('creates a user and sends welcome email', async () => {
    const mockUser = { id: '1', name: 'Alice', email: 'alice@test.com' };

    const deps = mock<CreateUserDeps>();
    deps.db.save.mockResolvedValue(mockUser);

    const result = await createUser(
      { name: 'Alice', email: 'alice@test.com' },
      deps,
    );

    expect(result).toEqual(mockUser);
    expect(deps.mailer.sendWelcome).toHaveBeenCalledWith(mockUser);
  });
});

That is not a compromise with the pattern. It is the pattern with better tooling. mock<CreateUserDeps>() produces an object with the right methods, backed by vi.fn(), while TypeScript enforces the shape. If a dep method gets renamed, the test fails to compile instead of drifting into runtime surprises.

Why this is different from vi.mock

Both approaches use the word “mock,” but the mechanics are very different.

With vi.mock, the test reaches out into the runtime and says:

when code imports this module, secretly replace it

With mock<CreateUserDeps>(), the test says:

here is the deps object this function asked for

One relies on indirection. The other relies on explicitness.

But fn(args, deps) is not primarily a testing pattern. It is an architectural boundary. The same shape that makes unit tests simple also:

keeps business logic independent from infrastructure
maps cleanly onto ports and adapters
makes composition roots straightforward
makes abstractions cheap to extract and cheap to undo

That is why the pattern shows up repeatedly across the other posts in this series. In the hexagonal architecture piece, the function signature itself becomes the boundary between the application core and external capabilities. In the composition piece, growing deps signal when a responsibility has stabilized into its own function. In the legacy code piece, fn(args, deps = defaultDeps) becomes the seam that lets you migrate safely without breaking callers.

Testing is just where the benefits become obvious first.

A rule of thumb

Here is the rule I would actually give a team:

Mock explicit collaborators, not implicit imports.

That means:

if your function takes deps, mock deps
if the deps object is annoying to hand-write, use vitest-mock-extended
if the code under test imports a third-party module directly and you cannot introduce a seam yet, use vi.mock pragmatically
if you own the business logic, prefer explicit injection over module interception

That rule preserves the core distinction.

It says “yes” to mocks as test doubles.

It says “no by default” to mocking the module system.

A worked example: repository + mailer

Here is the full progression.

Hidden imports

import { db } from './db';
import { mailer } from './mailer';

export async function inviteUser(email: string) {
  const user = await db.users.insert({ email });
  await mailer.sendInvite(user);
  return user;
}

Module-mocked test

vi.mock('./db', () => ({
  db: {
    users: {
      insert: vi.fn(),
    },
  },
}));

vi.mock('./mailer', () => ({
  mailer: {
    sendInvite: vi.fn(),
  },
}));

Explicit deps

export type InviteUserArgs = {
  email: string;
};

export type InviteUserDeps = {
  db: {
    users: {
      insert: (input: { email: string }) => Promise<User>;
    };
  };
  mailer: {
    sendInvite: (user: User) => Promise<void>;
  };
};

export async function inviteUser(args: InviteUserArgs, deps: InviteUserDeps) {
  const user = await deps.db.users.insert({ email: args.email });
  await deps.mailer.sendInvite(user);
  return user;
}

Typed test with vitest-mock-extended

import { describe, it, expect } from 'vitest';
import { mock } from 'vitest-mock-extended';
import { inviteUser, type InviteUserDeps } from './inviteUser';

describe('inviteUser', () => {
  it('creates a user and sends an invite', async () => {
    const user = { id: 'u1', email: 'alice@test.com' };

    const deps = mock<InviteUserDeps>();
    deps.db.users.insert.mockResolvedValue(user);

    const result = await inviteUser({ email: 'alice@test.com' }, deps);

    expect(result).toEqual(user);
    expect(deps.mailer.sendInvite).toHaveBeenCalledWith(user);
  });
});

That is the pattern at its cleanest:

app code wires real deps at the boundary
test code passes fake deps directly
no module interception needed

What about partial application?

One concern people have is call-site noise.

Do I really have to pass deps through every call?

No.

The core implementation stays:

fn(args, deps)

But at the boundary you can bind deps once and expose a clean function to the rest of the app:

export const bindDeps =
  <Args, Deps, Out>(fn: (args: Args, deps: Deps) => Out) =>
  (deps: Deps) =>
  (args: Args) =>
    fn(args, deps);

So application code can use:

const inviteUserWithDeps = bindDeps(inviteUser)(realDeps);
await inviteUserWithDeps({ email: 'alice@test.com' });

While tests still hit the core function directly:

await inviteUser({ email: 'alice@test.com' }, mock<InviteUserDeps>());

That preserves the explicit core while keeping wiring ergonomic at the composition boundary. Implement in the explicit fn(args, deps) form, then partially apply deps where it improves ergonomics.

When vi.mock is still the right tool

None of this means vi.mock is forbidden.

There are still legitimate uses:

third-party libraries that call globals internally
platform APIs in code you do not control
legacy modules where introducing a seam is not yet practical
migration steps where defaultDeps or wrappers are still being introduced

Defaults are acceptable as a strangler-fig migration seam, and vi.mock still has legitimate uses when the dependency is truly external or not yet injectable.

The problem is not mocking as such.

The problem is using module mocking as the default testing architecture for business code you control.

The pattern in one sentence

Here is the simplest way to say it:

fn(args, deps) is not anti-mock. It is anti-hidden dependency.

And that is why vitest-mock-extended fits so well.

It does not undermine the pattern. It reinforces it.

It says:

keep dependencies explicit
keep tests local
keep the type system honest
mock the collaborator, not the module system

That is the same architectural move from a different angle.

The takeaway

If your function already follows fn(args, deps), then vitest-mock-extended is not a compromise. It is one of the best ways to test that function.

Use vi.mock when you are forced to intercept a module boundary.

Use mock<Deps>() when the function has already declared its boundary explicitly.

That is the whole point:

explicit deps in production
typed fake deps in unit tests
real wiring at the boundary

No magic. No hidden state. No pretending the import graph is the contract.

Just functions, data, and capabilities.

Before You Change the Process, Understand the Problem

Fri, 20 Mar 2026 00:00:00 GMT

Strong opinions should be earned, not borrowed. If you want to propose a change, do the work first.

Understand the trade-offs before you walk into the room.

I see the same pattern again and again in software teams. Someone reads a blog post, watches a conference talk, picks up a new buzzword, or sees how a company like Spotify talks about working, and by the next meeting they are proposing that the whole team change how it works.

The idea often sounds compelling at first. But the moment you ask about the trade-offs, the case falls apart. There is no real research, no serious risk assessment, and no clear plan for what to do if the change creates new problems rather than solving the old ones.

That gap between enthusiasm and understanding is where teams get into trouble.

The real-world test

When I returned to University Press as principal architect on a project called Higher Education, my team lead, Clare, and I had to make a foundational decision on day one.

How should the team work?

The team had grown up inside a Scrum process that had become rigid and overly ceremonial.

All the artefacts were there, but the collaboration was not. In practice, the work moved in sequence. Design finished and handed over to development. Development finished and handed over to test. It was Scrum in structure, but not in spirit.

That was the real problem.

The answer was not to throw out Scrum and replace it with something fashionable. It was to understand why the team was struggling and improve the way we worked within a process people already knew.

Clare asked me a simple question: “You’re the principal. What do you think we should do?”

I told her I wanted to come back with a proper answer.

Clare and I were not best friends, but there was trust and respect between us. She had brought Agile into University Press and had given talks about it, which helped me enormously. She challenged my thinking, gave me honest feedback, and I learned a huge amount from her. I will always be grateful for that.

So I went back to first principles. Back to what I had learned during my degree, when we studied Agile, Taylorism, Denning, and the different ways of thinking about work. I went back through lean and agile articles, books, and talks, and grounded myself in a simple truth. Process should reflect the nature of the work.

What we were doing was not repetitive factory production. It was creative, uncertain, collaborative product development. That kind of work depends on fast feedback, shared ownership, and people working together early, not passing work between silos.

So when I came back to Clare, I was not proposing a new methodology. I was proposing a better way of working.

The case was simple. Keep the parts of Scrum that were helping, remove the behaviours that were creating walls, and increase collaboration across the team.

In practice, that meant changing how we planned and delivered features. When it came to planning, we brought together the stakeholders, the engineer, and the tester, and used a Three Amigos approach to discuss the change. We started with the business outcome, then planned a verifiable, incremental way of delivering the feature.

Testing did not sit at the end waiting for development to finish. It happened during development. We made sure testers had what they needed to test the way they wanted. Everyone on the team could draw the same high-level picture of the services involved and the data needed. We talked through the happy path and the known knowns up front.

That did not remove uncertainty, but it changed how we dealt with it. The mindset became simple. Unknowns will happen. Problems will happen. Our job was to design and communicate in a way that made reproducing, identifying, and resolving issues faster when they did.

Day to day, the work no longer felt disconnected. Everyone felt involved. Everyone knew the context. By the time we came into meetings, people were already aligned. The ceremony was naturally reduced because less time was spent catching people up or reconnecting work that had been split apart. Time wasted dropped. Communication increased.

Just as importantly, I was clear about the risks. The team was used to working in a certain way. More collaboration would feel uncomfortable at first. It would require people to communicate more openly, share ownership more actively, and adjust habits they had built over time.

But that honesty strengthened the proposal. I was not selling a silver bullet. I was showing the team where I thought this would help, where it might be difficult, and how we would know whether it was actually improving things.

That is what made the conversation credible.

What most people miss

Too often, people frame process change as a choice between named methodologies.

Scrum or Kanban. SAFe or no SAFe. Estimates or no estimates.

But most team problems are not solved by picking a new label. They are solved by understanding the actual constraint.

Sometimes the problem is too many handoffs. Sometimes it is weak communication. Sometimes, it is unclear ownership. Sometimes it is a process that has become disconnected from the people doing the work.

If you misdiagnose the problem, even a good framework will be applied badly.

Something is not right for your team just because it sounded good in a book, looked good in a case study, or became famous because a company like Spotify talked about it. Many teams copy labels, rituals, or structures without understanding what made them work in the first place. That is how people end up cargo-culting someone else’s model instead of solving their own problem.

The difference in practice

The anti-pattern

“We should switch to Kanban. I read that it works really well for high-performing teams.”

The standard

“Our problem is not Scrum itself. Our problem is that work is moving through silos. I think we should keep the useful structure, reduce the handoffs, increase collaboration, and review in two weeks whether flow, communication, and delivery have improved.”

That is a much stronger proposal because it is rooted in the team’s reality rather than borrowed from somebody else’s context.

What I took away

Start with the problem, not the framework.

Teams get into trouble when they inherit solutions before they understand the issue. Diagnose first. Change second.

This happens in sports all the time. Football teams do not choose one strategy in August and stick to it until May, no matter what. They adapt game by game based on the opposition, the conditions, injuries, and the strengths and weaknesses in front of them. Teams at work should be no different.

Improve before you replace.

You do not always need a new methodology. Often, you need to remove friction, reduce handoffs, and help the team collaborate more effectively within the process they already understand.

Present the risks as clearly as the benefits.

Anyone can sell the upside. Credibility comes from being able to say, “Here is where this may be harder than it sounds, and here is how we will know if it is working.”

Process should serve the team.

If a ceremony helps, keep it. If it creates a delay without adding value, challenge it. If someone is blocked, they should not have to wait for tomorrow’s stand-up to get help. Good process supports communication, ownership, and flow. It does not replace them.

The standard worth aiming for

The best proposals do not just answer the question, “What should we change?”

They answer, “What problem are we actually solving?” and “How will we know if this is better?”

That is the difference between borrowing an opinion and earning one.

That mindset led to a lot of changes that were a first for University Press. We were the first team to use TypeScript and Tailwind. We introduced feature branches. We focused on DORA metrics and on delivering value to the business, rather than obsessing over velocity points.

Each time, I said the same thing to the team: “Let’s try it. If it is not working in a month, we will remove it.”

We never had to remove a single thing.

And in leadership, that difference matters.

fn(args, deps) and Composition — Know When to Nest the Dolls

Wed, 18 Mar 2026 00:00:00 GMT

Just because two pieces of code look the same does not mean they are the same.

The most common architecture mistake is not too little abstraction. It is too much, too early. You see duplication, you extract a shared module, and six months later that module is a monster held together by special cases and boolean flags.

Dan Abramov gave a talk about this called The Wet Codebase. The core argument: the wrong abstraction is far more expensive than duplication. Once an abstraction exists, it creates inertia. Nobody wants to be the person who suggests copy-paste.

fn(args, deps) changes this calculus. It makes abstractions cheap to create, cheap to test, and cheap to undo.

When a function’s deps grow too large, that can be a signal that some responsibility has stabilized into its own function — and that new function itself follows fn(args, deps). (This is basically SRP pressure showing up in your signature; see the SOLID post for that framing.)

Russian dolls. Each layer independently testable. Each layer reversible.

The wrong abstraction creates inertia

Abramov tells a familiar story. Two modules share similar code. A colleague extracts a shared abstraction. Then a third use case arrives that is almost the same, but not quite. So you add a parameter. Then a special case. Then another.

Each step makes sense to the person writing and reviewing it. But if you lose track of what the abstraction was supposed to represent, you end up with a module nobody wants to touch.

The real damage is social, not technical. Even when the team agrees the abstraction is bad, unwinding it is expensive, especially if other teams depend on it, if nobody knows how to test the inlined version, or if nobody wants to be the one who decreases code coverage.

Abramov’s advice: restrain yourself. Let the code stabilize before you extract. And when you do extract, test the concrete behavior, not the abstraction — so that refactoring stays safe.

That advice is exactly right. And fn(args, deps) is what makes it practical.

When deps are explicit data, inlining an abstraction becomes much more mechanical. Copy the function body back, merge the deps types, and let the compiler show you the remaining seams. There is no module mocking to untangle, no framework wiring to redo, no decorator chain to unravel. The abstraction is just a function. Removing it is just removing a function.

The smell: too many deps

The fn(args, deps) equivalent of “too many constructor parameters” is a deps type that keeps growing.

type ProcessOrderArgs = {
  productId: string;
  quantity: number;
  paymentMethod: PaymentMethod;
};

type ProcessOrderDeps = {
  findProduct: (id: string) => Promise<Product>;
  checkInventory: (productId: string, qty: number) => Promise<boolean>;
  reserveStock: (productId: string, qty: number) => Promise<void>;
  chargePayment: (amount: number, method: PaymentMethod) => Promise<ChargeResult>;
  sendConfirmation: (order: Order) => Promise<void>;
  log: (msg: string, meta?: unknown) => void;
  now: () => number;
};

Seven deps. That is a signal, not a sin. The question is not “should I abstract?” It is “have some of these deps stabilized into a coherent responsibility?”

If you are on day one and this is the first function that uses these deps — leave it flat. You do not know yet which deps belong together. Premature grouping is how you get Abramov’s monster.

But if you have been living with this code for a while, and you see findProduct, checkInventory, and reserveStock showing up together in multiple functions, and they always represent the same concern — inventory fulfillment — then you probably have an abstraction waiting to be named.

Consolidate when the responsibility is clear

Some of those deps cluster naturally. findProduct, checkInventory, and reserveStock are all inventory concerns. chargePayment is a payment concern.

The move is to extract new functions that each follow fn(args, deps).

type FulfillOrderArgs = {
  productId: string;
  quantity: number;
};

type FulfillOrderDeps = {
  findProduct: (id: string) => Promise<Product>;
  checkInventory: (productId: string, qty: number) => Promise<boolean>;
  reserveStock: (productId: string, qty: number) => Promise<void>;
};

async function fulfillOrder(args: FulfillOrderArgs, deps: FulfillOrderDeps) {
  const product = await deps.findProduct(args.productId);
  const available = await deps.checkInventory(args.productId, args.quantity);

  if (!available) {
    return { ok: false as const, reason: 'out-of-stock' };
  }

  await deps.reserveStock(args.productId, args.quantity);
  return { ok: true as const, product };
}

Now the parent function’s deps get simpler:

type ProcessOrderDeps = {
  fulfillOrder: (args: FulfillOrderArgs) => Promise<FulfillResult>;
  chargePayment: (amount: number, method: PaymentMethod) => Promise<ChargeResult>;
  sendConfirmation: (order: Order) => Promise<void>;
  log: (msg: string, meta?: unknown) => void;
};

From seven deps to four. And fulfillOrder is itself fn(args, deps). It has its own deps — findProduct, checkInventory, reserveStock — wired at the composition root.

And now the parent function reads like a plan:

async function processOrder(args: ProcessOrderArgs, deps: ProcessOrderDeps) {
  deps.log('Processing order', { productId: args.productId });

  const fulfillment = await deps.fulfillOrder({
    productId: args.productId,
    quantity: args.quantity,
  });

  if (!fulfillment.ok) return fulfillment;

  const charge = await deps.chargePayment(
    fulfillment.product.price * args.quantity,
    args.paymentMethod,
  );

  if (!charge.ok) return { ok: false as const, reason: 'payment-failed' };

  const order = { ...args, product: fulfillment.product, chargeId: charge.id };
  await deps.sendConfirmation(order);

  return { ok: true as const, order };
}

The parent does not reach into fulfillment’s internals. It only knows the shape: give me a product and quantity, get back a result.

Each layer is independently testable. You can test fulfillOrder with fake product lookups. You can test processOrder with a fake fulfillOrder that always succeeds. The layers stay decoupled.

Extract only when the responsibility is clear

This is where Abramov’s lesson matters most.

Do not create fulfillOrder on day one. Start with the flat deps list. Write the code. Ship it. Let it stabilize.

The signal that it is time to extract:

The same cluster of deps appears across multiple functions
Those deps represent a coherent responsibility, not just accidental proximity
You can name the new function after what it does, not after how it is implemented

If you cannot name it clearly, you do not understand the responsibility yet. Wait.

fn(args, deps) makes this safe in both directions:

Extract: create a new fn(args, deps), move the clustered deps into it, wire it at the composition root
Inline: if the abstraction was wrong, copy the body back, merge the deps types — no framework ceremony to undo

The right time to abstract is when you understand the responsibility, not when you notice the duplication.

Why this scales

Each layer is just fn(args, deps). Same testing story. Same composition root wiring. Same explicit dependency surface.

No inheritance hierarchies. No decorator chains. No middleware stacks. No framework registration.

You can zoom in and test fulfillOrder in isolation with three simple deps. Or you can stay at the higher layer and test processOrder with a fake fulfillOrder that returns a canned result.

The pattern does not fight you when you change your mind. Because every layer is just a function that takes data and capabilities. If a layer turns out to be wrong, you delete it and move on. The types tell you exactly what to fix.

The takeaway

Composition is not about having the right abstraction from the start. It is about making abstractions cheap to create, cheap to test, and cheap to undo.

fn(args, deps) gives you all three, because every layer is just a function that takes data and capabilities. Nest them when the responsibility is clear. Flatten them when it is not.

That is the Russian dolls model: layers you can add when the responsibility is clear, and remove when it is not.

fn(args, deps) Is Programming to Interfaces — And That's How You Control Nondeterminism

Wed, 18 Mar 2026 00:00:00 GMT

Many software systems fail for one very boring reason.

Not because of microservices. Not because of monoliths. Not because of whatever methodology war is trending this week.

They fail because they are unpredictable.

If you make a change and you cannot reliably determine the impact, you cannot safely evolve the system. And when you cannot evolve it, it starts behaving like legacy.

Determinism is the bridge between “works on my machine” and “works every time, everywhere.”

fn(args, deps) gets you there — not because it is a clever trick, but because it makes boundaries explicit. Your logic programs to interfaces, which is what lets you control sources of nondeterminism.

Dave Farley makes this argument better than almost anyone. If you have not seen his talk on determinism and evolutionary architecture, it is worth your time:

What follows is not a recap of that talk. It is about one specific claim: that fn(args, deps) is a concrete, function-level way to implement the core of what Farley is describing.

Determinism is the prerequisite

You cannot evolve what you cannot measure. You cannot measure what you cannot repeat. And you cannot repeat what you cannot reliably reproduce.

Flaky tests, intermittent builds, and concurrency heisenbugs turn your delivery pipeline from a learning system into release theater.

Determinism is the prerequisite for trust. Without it, your feedback loops degrade and your architecture rots quietly in the background.

With it, you can run thousands of experiments per day, detect unintended consequences, and move fast without gambling. That is an evolutionary capability.

Continuous delivery depends on pipelines, but pipelines only earn trust when the system beneath them is deterministic. The pipeline is just the amplifier.

Separate the code that decides from the code that acts

Farley frames this as the deterministic core and the imperative shell.

The deterministic core is reproducible logic under controlled inputs. State in, decision out. No hidden database calls, no hidden clock, no hidden randomness.

The imperative shell is the messy part. Talk to the database, talk to the network, read the clock, execute side effects.

You can map this to “ports and adapters,” but the point here is simpler: “how do I make behavior predictable?”

If you have read the earlier post on hexagonal architecture, the structure is familiar. What changes here is the reason it matters.

fn(args, deps) is programming to interfaces

This is the connection most people miss.

When you write a deps type, you are writing an interface. Not a Java-style interface with a keyword and a class that implements it. Just a type that describes what capabilities the function needs from the outside world.

type GetExpiringItemsDeps = {
  now: () => number;
  findItemsExpiringBefore: (cutoff: Date) => Promise<Item[]>;
};

That type is the interface. The function programs to it. The composition root decides what implements it.

function expirationCutoff(
  args: { windowMs: number },
  deps: { now: () => number },
) {
  return new Date(deps.now() - args.windowMs);
}

async function getExpiringItems(
  args: { windowMs: number },
  deps: GetExpiringItemsDeps,
) {
  return deps.findItemsExpiringBefore(expirationCutoff(args, deps));
}

The function does not know whether now returns the real time or a frozen timestamp. It does not know whether findItemsExpiringBefore queries Postgres or returns a hardcoded array. It is deliberately indifferent.

That is what programming to interfaces means: your logic does not reach behind the contract. It stays on the surface—capabilities in, results out.

And that is what makes the logic reproducible under controlled inputs. Once sources of nondeterminism are explicit, you can control or replace them for the purpose of a call.

The deps type is the interface. The function programs to it. That is all programming to interfaces ever needed to be.

The clock example

Here is the simplest way to see it.

If your code calls Date.now() directly, you have pulled time into the middle of your logic. The same explicit input tomorrow can produce a different output, because time is now a hidden input. You can still test it, but tests get harder to make reliable and harder to replay exactly.

// nondeterministic — the output depends on when you run it
async function getExpiringItems(windowMs: number) {
  const cutoff = new Date(Date.now() - windowMs);
  return db.findItemsExpiringBefore(cutoff);
}

Instead, inject a clock. Pass time as data. Treat now as an input.

// reproducible under controlled deps — the output depends on what you pass in
function expirationCutoff(
  args: { windowMs: number },
  deps: { now: () => number },
) {
  return new Date(deps.now() - args.windowMs);
}

async function getExpiringItems(
  args: { windowMs: number },
  deps: GetExpiringItemsDeps,
) {
  return deps.findItemsExpiringBefore(expirationCutoff(args, deps));
}

Now something powerful happens. You can freeze time. You can fast-forward it. You can reproduce production bugs by replaying timestamps (assuming the rest of your dependencies are similarly controlled for the replay).

You have turned the universe into a parameter.

it('finds items expiring within the window', async () => {
  const frozenNow = new Date('2026-03-19T12:00:00Z').getTime();

  const result = await getExpiringItems(
    { windowMs: 60_000 },
    {
      now: () => frozenNow,
      findItemsExpiringBefore: async () => [
        { id: '1', name: 'Trial', expiresAt: new Date('2026-03-19T11:59:30Z') },
      ],
    },
  );

  expect(result).toHaveLength(1);
  expect(result[0].name).toBe('Trial');
});

No module patching. No hoisted mocks. No test framework reaching into module internals. Just explicit test doubles passed as data.

This works because the function programs to the deps interface, not to the real clock or the real database.

Why this matters for evolutionary architecture

When your core logic is written against explicit deps instead of reaching into the world, you can test it without monkey-patching, module interception, or elaborate scaffolding.

You pass in state. You assert on output. You can run thousands of tests in milliseconds.

Those are your fitness functions at scale. And when you have fitness functions you can trust, your architecture can evolve safely, one function at a time.

Determinism is not a coding trick. It is a systems property. fn(args, deps) does not create determinism by magic—it makes the boundaries around nondeterminism explicit. Once they are explicit, you can control them, replace them, and replay them.

The chain

fn(args, deps) is programming to interfaces without the ceremony.

Programming to interfaces makes sources of nondeterminism explicit.

Explicit nondeterminism is controllable and testable.

Controllable, testable behavior is what makes deterministic systems possible.

Deterministic systems are what make evolutionary architecture possible.

That chain — from function signature to system capability — is why the pattern matters.

How fn(args, deps) supports SOLID-style design in TypeScript

Wed, 18 Mar 2026 00:00:00 GMT

A lot of developers learn the SOLID principles through class-heavy examples.

That is probably why the conversation so often gets stuck there.

People start to associate SOLID with inheritance hierarchies, interface forests, service classes, and object-oriented ceremony.

But the useful part is not the ceremony.

It is the design pressure.

One of the simplest ways to apply that pressure in plain TypeScript is this shape:

fn(args, deps)

Where args is the input for this call and deps is the set of collaborators the function needs. You can read that as: data in, capabilities in.

fn(args, deps) is not a replacement for SOLID. It is a simple function shape that makes several SOLID ideas easier to apply without forcing you into class-heavy design.

SOLID is about design pressure, not class ceremony

SOLID is often taught as though it naturally implies classes, interfaces, and object hierarchies.

But the principles themselves are broader than that.

They are really about questions like these:

does this unit of code have one clear reason to change?
can behavior vary without editing unrelated logic?
are abstractions helping or just adding indirection?
are dependencies pointing in the right direction?

Those questions still matter in plain TypeScript, even if you are writing functions instead of classes.

That is where fn(args, deps) becomes useful.

It gives you a simple way to separate:

call-specific data
long-lived collaborators
application logic
infrastructure concerns

That separation does not automatically make code good, but it does make good design easier to see and bad design harder to hide.

A small example

Consider a use case like this:

type CreateUserArgs = {
  name: string;
  email: string;
};

type CreateUserDeps = {
  userRepo: UserRepo;
  mailer: Mailer;
  logger: Logger;
};

export async function createUser(
  args: CreateUserArgs,
  deps: CreateUserDeps,
) {
  deps.logger.info("Creating user", { email: args.email });

  const user = await deps.userRepo.insert(args);
  await deps.mailer.sendWelcome(user);

  return user;
}

This is just a function.

But even in this small example, the shape already encourages several useful design properties:

the input is explicit
the collaborators are explicit
side-effecting collaborators are visible in the signature
the business operation is separated from how dependencies are created

That is why this shape works well with SOLID-style design pressure.

Not because it turns functions into objects.

Because it makes responsibilities and dependencies visible.

Single Responsibility Principle

The Single Responsibility Principle is often paraphrased badly, but the useful version is simple:

A unit of code should have one coherent reason to change.

fn(args, deps) helps because it encourages you to keep one function focused on one job, while pushing infrastructure concerns outward.

In the example above, createUser is responsible for the user-creation workflow.

It is not also responsible for:

constructing a database client
configuring a mail transport
locating a logger
deciding which concrete implementations exist in production

Those concerns live elsewhere.

That matters because a lot of SRP violations show up when one unit mixes workflow logic with setup logic and environment logic.

This is a more obvious violation:

export async function createUser(args: CreateUserArgs) {
  const db = makeDb(process.env.DATABASE_URL!);
  const mailer = makeMailer();
  const logger = makeLogger();

  logger.info("Creating user", { email: args.email });

  const user = await db.users.insert(args);
  await mailer.sendWelcome(user);

  return user;
}

Now the function has multiple reasons to change:

the user creation flow changes
database construction changes
mail setup changes
logging setup changes
environment configuration changes

That is exactly the kind of responsibility bleed that fn(args, deps) helps prevent.

Open/Closed Principle

The Open/Closed Principle is usually summarized as:

Open for extension, closed for modification.

In practice, the useful question is:

Can I change how this function behaves by changing its collaborators, without rewriting the function itself?

With fn(args, deps), the answer is often yes.

You can change the repository, mailer, or logger without changing the core workflow:

await createUser(
  { name: "Ada", email: "ada@example.com" },
  {
    userRepo: cachedUserRepo,
    mailer: noopMailer,
    logger: structuredLogger,
  },
);

That is a practical form of extension through composition.

You are not subclassing anything. You are not overriding a method. You are varying behavior by supplying different collaborators.

That is often a much simpler way to satisfy the spirit of OCP in TypeScript.

The important caveat is that this only works well when the function depends on meaningful capabilities rather than concrete setup details.

If your deps object is full of leaky implementation specifics, you are not really extending cleanly. You are just moving complexity around.

Liskov Substitution Principle

LSP is one of the principles people invoke most and understand least.

In class-heavy design, it is about whether a subtype can stand in for its base type without breaking expectations.

LSP does not map as directly to function-oriented design as SRP, ISP, or DIP. But the same substitution concern still shows up at dependency boundaries.

The useful question becomes:

Can one dependency implementation be substituted for another without breaking the expectations of the function that uses it?

If createUser depends on a Mailer, then different mailers should behave in ways the workflow can safely rely on.

For example, these might both be valid substitutions:

a real SMTP mailer
a fake test mailer
a mailer that writes to an outbox table
a mailer that queues background jobs

What matters is that they preserve the contract the use case expects.

That is not unique to fn(args, deps), but the pattern makes the substitution point explicit.

Instead of inheritance hiding the boundary, the dependency boundary is right there in the function signature.

That visibility is useful because substitution problems become easier to spot. If the function keeps needing special cases for one implementation, that is often a signal that the abstraction is wrong.

Interface Segregation Principle

The Interface Segregation Principle says consumers should not be forced to depend on methods they do not use.

This is one place where fn(args, deps) can be better than large service objects.

Because dependencies are passed per function, you can keep them narrow.

For example:

type GetUserDeps = {
  userRepo: Pick<UserRepo, "findById">;
};

export async function getUser(
  args: { userId: string },
  deps: GetUserDeps,
) {
  return deps.userRepo.findById(args.userId);
}

That is a very small dependency surface.

The function does not need the entire repository API. It only needs the capability it actually uses.

This is an underrated advantage of the pattern.

When dependencies are explicit and local to the function, oversized interfaces start to feel awkward immediately.

That friction is good. It pushes you toward smaller, more honest contracts.

Dependency Inversion Principle

This is the SOLID principle most obviously related to dependency injection.

High-level policy should not depend directly on low-level details. Both should depend on abstractions or stable boundaries.

fn(args, deps) makes that easier because it naturally separates use-case logic from infrastructure wiring.

The use case says what it needs:

type CreateUserDeps = {
  userRepo: UserRepo;
  mailer: Mailer;
  logger: Logger;
};

The application edge decides what satisfies those needs:

import { createUser } from "./create-user";
import { makePostgresUserRepo } from "./infra/postgres-user-repo";
import { makeSesMailer } from "./infra/ses-mailer";
import { makeLogger } from "./infra/logger";

const userRepo = makePostgresUserRepo();
const mailer = makeSesMailer();
const logger = makeLogger();

await createUser(
  { name: "Ada", email: "ada@example.com" },
  { userRepo, mailer, logger },
);

That is Dependency Inversion without needing a container to make it feel real.

The use case does not know how the dependencies were created. It only knows the capabilities it was given.

That is the important part.

This does not mean “use abstractions everywhere”

This is where SOLID conversations often go off the rails.

Once people hear “Dependency Inversion” or “Interface Segregation,” they start introducing abstractions for everything.

That is usually a mistake.

Not every helper needs to be abstracted. Not every utility needs to be injected. Not every module needs an interface.

For pure logic, direct imports are often the better choice:

import { normalizeEmail } from "./normalize-email";

export function toUserRecord(args: CreateUserArgs) {
  return {
    ...args,
    email: normalizeEmail(args.email),
  };
}

That helper is deterministic, side-effect free, and not especially interesting as a substitution point. Passing it through deps would add ceremony without buying much.

A good rule of thumb:

use direct imports for pure logic and stable helpers
use deps for collaborators that touch the outside world, vary by environment, or matter in tests

That is how you keep SOLID from turning into ritual.

Composition still happens at the edge

One common objection is that explicit dependencies will make every call site noisy.

Sometimes they will.

That is why composition belongs at the edge of the application.

You can bind dependencies once and expose a simpler API:

export const makeCreateUser =
  (deps: CreateUserDeps) =>
  (args: CreateUserArgs) =>
    createUser(args, deps);

Then your startup code can assemble the real version:

const createUserWithServices = makeCreateUser({
  userRepo,
  mailer,
  logger,
});

await createUserWithServices({
  name: "Ada",
  email: "ada@example.com",
});

This is important for two reasons.

First, it keeps the use case explicit about what it needs.

Second, it keeps the wiring in one honest place instead of scattering it across the codebase.

That is good design whether you talk about composition roots, DI, or SOLID.

Where this pattern actually helps

The value of fn(args, deps) is not that it magically makes code SOLID.

No function signature can do that.

The value is that it nudges the codebase in useful directions:

responsibilities are easier to isolate
dependency surfaces are easier to shrink
substitution points are easier to see
infrastructure stays at the edge more often
tests become simpler because collaborators are already explicit

That is why I think this shape is such a strong default for many application-level functions in TypeScript.

It supports the intent behind SOLID without forcing you into all the class-oriented baggage that often comes with SOLID discussions.

Why fn(args, deps) works well with SOLID-style design

If SOLID only works when wrapped in inheritance hierarchies, giant interfaces, and framework-controlled injection, then it is not much use in a lot of modern TypeScript code.

The better view is simpler:

SOLID is a set of design pressures. fn(args, deps) is one practical way to apply several of them in TypeScript.

It helps you keep responsibilities narrower. It helps you keep dependency boundaries visible. It helps you vary collaborators without hiding where they come from. And it helps you keep infrastructure wiring out of business logic.

In plain TypeScript, that is often more valuable than a perfectly textbook interpretation of SOLID.

Because the goal is not to perform the principles.

It is to write code that stays understandable as the application grows.

fn(args, deps) — Bringing Order to Chaos Without Breaking Anything

Wed, 18 Mar 2026 00:00:00 GMT

There’s a major disconnect in AI-assisted development right now. Most of the conversation assumes you’re building something new, or working from the kind of clean, stable foundation that barely exists in real engineering teams.

The reality is that most engineering teams live in legacy systems under high load, with god classes, global singletons, and console.log as observability. The kind of code where every change is a gamble.

This post shows what happens when you apply fn(args, deps) and autotel to those codebases. fn(args, deps) creates the seam for safe change; production telemetry captures the behavioural record that survives when every other spec has decayed.

To prove the point, we’ll do this in plain JavaScript, not TypeScript.

Companion repo: fn-args-deps-bringing-order-to-chaos-without-breaking-anything: every step is independently runnable, in plain JavaScript, with no TypeScript required.

Three Fields, Three Realities

Greenfield is what the demos show: net new code, explicit intent, and a spec you still trust.

Brownfield is where most teams actually work: an existing system, still evolving, where intent is only partially recoverable.

Blackfield is something else again: legacy systems under load, on a deprecation path everyone agrees on and nobody has time to execute.

This post starts in Blackfield. Features are still being bolted on because the business can’t wait for the rewrite. Original authors are gone. Tests, if they exist, describe the system as it was. Documentation, if it exists, lies.

That isn’t just a codebase with worse docs. It’s a different engineering posture. You’re not trying to improve something. You’re trying not to break something while slowly making it possible to change. AI agents, trained to build forward and iterate toward green tests, are carrying the wrong assumptions into that room.

The Spec Problem Nobody Admits

Spec-driven development has a hidden premise: you know what your system is supposed to do.

In Greenfield, true. In Brownfield, partially true. In Blackfield, false.

Legacy systems accumulate behaviour nobody planned. Business rules encoded in conditionals that outlived everyone who understood them. When you ask an agent to work here without a spec, you haven’t freed it from needing one. You’ve made the spec implicit, and the inferred spec will be wrong in ways that are hard to detect and expensive to discover.

The spec problem isn’t solved by better prompts. It’s solved by better sources of truth.

Here’s what a typical legacy system looks like:

const { db } = require('./db.js');
const { paymentGateway } = require('./payment-gateway.js');
const { emailService } = require('./email-service.js');
const { inventoryService } = require('./inventory-service.js');

class OrderProcessor {
  constructor() {
    this.db = db;
    this.payment = paymentGateway;
    this.email = emailService;
    this.inventory = inventoryService;

    // Business rules hardcoded in the constructor.
    // Someone added these in a hotfix. Nobody documented why.
    this.discountRules = new Map([
      ['gold', 0.15],
      ['silver', 0.1],
      ['bronze', 0.05],
    ]);
    this.taxRate = parseFloat(process.env.TAX_RATE || '0.08');
  }

  async processOrder(customerId, items) {
    console.log(`[OrderProcessor] Processing order for ${customerId}`);
    const startTime = Date.now();

    const customer = await this.db.findCustomer(customerId);
    if (!customer) throw new Error(`Customer not found: ${customerId}`);

    const stockResult = await this.inventory.check(items);
    if (!stockResult.available) throw new Error('Insufficient inventory');

    const subtotal = items.reduce(
      (sum, item) => sum + item.priceInCents * item.quantity,
      0,
    );
    const discount = this.discountRules.get(customer.loyaltyTier) || 0;
    const discountedTotal = Math.round(subtotal * (1 - discount));
    const totalWithTax = Math.round(discountedTotal * (1 + this.taxRate));

    const paymentResult = await this.payment.charge(customerId, totalWithTax);
    if (!paymentResult.success)
      throw new Error(`Payment failed: ${paymentResult.error}`);

    await this.inventory.reserve(items);

    const order = {
      id: `ord_${Date.now()}`,
      customerId,
      items,
      totalInCents: totalWithTax,
      status: 'confirmed',
      createdAt: new Date(),
    };
    const savedOrder = await this.db.saveOrder(order);

    await this.email.send(
      customer.email,
      'Order Confirmed',
      `Your order ${savedOrder.id} for $${(totalWithTax / 100).toFixed(2)} has been confirmed.`,
    );

    console.log(
      `[OrderProcessor] Order ${savedOrder.id} completed in ${Date.now() - startTime}ms`,
    );

    return savedOrder;
  }
}

const orderProcessor = new OrderProcessor();
module.exports = { OrderProcessor, orderProcessor };

Four dependencies wired to global singletons. console.log as observability. Business rules in a constructor added during a hotfix. require() at the top of the file, no way to inject alternatives.

An AI agent will see a class with methods. It’ll mock the constructor, write tests that pass, and produce output that looks correct. But it can’t see the invisible contracts: the payment gateway that rejects amounts over $10,000. The discount rules that were a regulatory requirement. The call ordering that matters because the inventory service has side effects.

The agent has the code but not the contract.

It cannot see which paths are load-bearing in production, which edge cases still matter to real users, or which invisible constraints came from compliance, operations, or business reality rather than code aesthetics.

In legacy systems, the sequence is not write a spec, generate code, run tests. It is observe production, extract behaviour, lock it with tests, then change code safely.

Characterization Tests as Archaeology

Michael Feathers’ Working Effectively with Legacy Code introduced characterization tests, written not to verify correct behaviour, but to document current behaviour. You capture what the system does so that when you change something, you can see what moved.

Not a goalpost. A tripwire.

In Blackfield systems, system tests matter for the same reason characterization tests do: they preserve behaviour when intent is no longer trustworthy.

A black-box test anchored to user-visible outcomes often survives the decay that kills lower-level tests tied too closely to implementation.

For our order processor, characterization tests are ugly. They have to be. The code gives us no other choice:

jest.mock('../src/db.js', () => ({
  db: {
    findCustomer: jest.fn(),
    checkStock: jest.fn(),
    reserveStock: jest.fn(),
    saveOrder: jest.fn(),
  },
}));
jest.mock('../src/payment-gateway.js', () => ({
  paymentGateway: { charge: jest.fn(), refund: jest.fn() },
}));
jest.mock('../src/email-service.js', () => ({
  emailService: { send: jest.fn() },
}));
jest.mock('../src/inventory-service.js', () => ({
  inventoryService: { check: jest.fn(), reserve: jest.fn() },
}));

Four jest.mock() calls before we’ve tested a single thing. In this shape of legacy code, module-level mocking is the only way in—until you create an explicit seam.

This is exactly the sort of environment where an agent can look productive while misunderstanding the system. It can satisfy mocked tests without discovering the behavioural contracts hidden in call ordering, side effects, and production traffic.

But now we can lock the contract:

it('returns a confirmed order with correctly calculated total', async () => {
  const order = await processor.processOrder(TEST_CUSTOMER_ID, TEST_ITEMS);

  // Characterize the current pricing math:
  //   subtotal = (2 * 1000) + (1 * 2500) = 4500
  //   gold discount = 15% -> Math.round(4500 * 0.85) = 3825
  //   tax = 8% -> Math.round(3825 * 1.08) = 4131
  expect(order.totalInCents).toBe(4131);
  expect(order.status).toBe('confirmed');
});

it('throws when payment fails — and does NOT reserve inventory', async () => {
  mockPayment.charge.mockResolvedValue({
    success: false,
    error: 'Amount exceeds limit',
  });

  await expect(
    processor.processOrder(TEST_CUSTOMER_ID, TEST_ITEMS),
  ).rejects.toThrow('Payment failed: Amount exceeds limit');

  expect(mockInventory.reserve).not.toHaveBeenCalled();
  expect(mockDb.saveOrder).not.toHaveBeenCalled();
});

These are not the same as old tests that may have drifted. Old tests describe what the system was designed to do; characterization tests, written now against observed behaviour, describe what it actually does.

A green test suite on a Blackfield system is not always confidence. Sometimes it is a fossil record.

If the code has a bug in the pricing math, these tests enshrine that bug. We fix bugs after we have safe refactoring coverage.

The point isn’t quality. The point is: don’t make things worse.

The Seam: fn(args, deps = defaultDeps)

Now we can move. The strangler fig pattern starts here: extract one function at a time from the god class. Give it explicit dependencies. Provide defaults so existing callers keep working.

I’ll keep everything in CommonJS (require() / module.exports) because that’s what most legacy Node.js codebases actually look like. The seam is the point, not the module syntax.

The defaultDeps object imports the existing singletons, so the class can delegate to this function without any caller knowing:

const defaultDeps = {
  findCustomer: (customerId) => db.findCustomer(customerId),
  checkInventory: (items) => checkInventory({ items }),
  chargePayment: (customerId, amount) =>
    chargePayment({ customerId, amountInCents: amount }),
  reserveStock: (items) => inventoryService.reserve(items),
  saveOrder: (order) => db.saveOrder(order),
  sendConfirmation: (email, orderId, totalInCents) =>
    sendConfirmation({ email, orderId, totalInCents }),
  discountRules: new Map([
    ['gold', 0.15],
    ['silver', 0.1],
    ['bronze', 0.05],
  ]),
  taxRate: parseFloat(process.env.TAX_RATE || '0.08'),
};

async function processOrder(args, deps = defaultDeps) {
  const { customerId, items } = args;

  const customer = await deps.findCustomer(customerId);
  if (!customer) throw new Error(`Customer not found: ${customerId}`);

  const stockResult = await deps.checkInventory(items);
  if (!stockResult.available) throw new Error('Insufficient inventory');

  const subtotal = items.reduce(
    (sum, item) => sum + item.priceInCents * item.quantity,
    0,
  );
  const discount = deps.discountRules.get(customer.loyaltyTier) || 0;
  const discountedTotal = Math.round(subtotal * (1 - discount));
  const totalWithTax = Math.round(discountedTotal * (1 + deps.taxRate));

  const paymentResult = await deps.chargePayment(customerId, totalWithTax);
  if (!paymentResult.success)
    throw new Error(`Payment failed: ${paymentResult.error}`);

  await deps.reserveStock(items);

  const order = {
    id: `ord_${Date.now()}`,
    customerId,
    items,
    totalInCents: totalWithTax,
    status: 'confirmed',
    createdAt: new Date(),
  };
  const savedOrder = await deps.saveOrder(order);
  await deps.sendConfirmation(customer.email, savedOrder.id, totalWithTax);

  return savedOrder;
}

module.exports = { processOrder };

The class can still exist and delegate. In most cases the public call sites don’t need to change, though in real systems, watch for import timing and module side effects that can leak outward.

Now look at the tests:

function makeDeps(overrides = {}) {
  return {
    findCustomer: jest.fn().mockResolvedValue({
      id: 'cust_1',
      name: 'Alice',
      email: 'alice@test.com',
      loyaltyTier: 'gold',
    }),
    checkInventory: jest.fn().mockResolvedValue({ available: true }),
    chargePayment: jest.fn().mockResolvedValue({
      success: true,
      transactionId: 'txn_123',
    }),
    reserveStock: jest.fn().mockResolvedValue(undefined),
    saveOrder: jest.fn().mockImplementation(async (order) => order),
    sendConfirmation: jest.fn().mockResolvedValue(undefined),
    discountRules: new Map([
      ['gold', 0.15],
      ['silver', 0.1],
      ['bronze', 0.05],
    ]),
    taxRate: 0.08,
    ...overrides,
  };
}

it('processes a successful order with correct total', async () => {
  const deps = makeDeps();
  const items = [
    { productId: 'p1', name: 'Widget', quantity: 2, priceInCents: 1000 },
  ];

  const order = await processOrder({ customerId: 'cust_1', items }, deps);

  // subtotal = 2000, gold 15% => 1700, tax 8% => 1836
  expect(order.totalInCents).toBe(1836);
  expect(order.status).toBe('confirmed');
  expect(deps.chargePayment).toHaveBeenCalledWith('cust_1', 1836);
});

it('throws on payment failure — stock is never reserved', async () => {
  const deps = makeDeps({
    chargePayment: jest.fn().mockResolvedValue({
      success: false,
      error: 'Card declined',
    }),
  });

  await expect(
    processOrder({ customerId: 'cust_1', items: sampleItems }, deps),
  ).rejects.toThrow('Payment failed: Card declined');

  expect(deps.reserveStock).not.toHaveBeenCalled();
});

No module interception. No jest.mock(). No constructor ceremony. You mock exactly what the function uses by passing explicit test doubles as data.

The = defaultDeps is not laziness. It’s a strangler fig — temporary scaffolding that buys you a test seam without breaking existing callers. New callers and tests get clean injection. When every caller has migrated, you remove the default. (The foundational post is explicit: defaults are allowed only for migration.)

For the full pattern, see If You Only Enforce One Rule for AI Code, Make It fn(args, deps).

Production Telemetry Is the Spec That Survived

When everything else rots (the documentation, the tests, the original ticket) production telemetry is often the most reliable surviving record of behaviour. Users kept showing up. They kept logging in, hitting endpoints, triggering workflows.

fn(args, deps) is the extraction pattern that makes legacy code testable; tracing and structured telemetry are what make that behaviour legible enough for humans and agents to change safely.

Charity Majors has argued for years that production is where you find out how your system actually behaves. Her framing is forward-looking: instrument as you build, observe as you deploy. But the same logic applies backward, to systems that nobody instrumented properly when they were young.

Telemetry is not a normative spec. It cannot tell you whether behaviour is correct. It can only tell you what behaviour survived contact with production. Production behaviour can encode accidents as readily as intent. But in a legacy system where every other artefact has decayed, observed behaviour is often the safer place to start.

The deeper question is not just how to observe production, but how to make telemetry queryable as a spec — not as a dashboard, but as a source of truth for safe change. fn(args, deps) gives us the seam to start adding that telemetry to systems that never had it.

Wrapping With trace()

Now that our functions have the shape fn(args, deps), we can wrap them with autotel’s trace(). The business logic does not change. Existing tests often pass unchanged.

const { trace } = require('autotel');

const processOrder = trace((ctx) => async (args, deps = defaultDeps) => {
  ctx.setAttribute('order.customerId', args.customerId);
  ctx.setAttribute('order.itemCount', args.items.length);

  // ... exact same business logic ...

  ctx.setAttribute('order.totalInCents', totalWithTax);
  ctx.setAttribute('order.id', savedOrder.id);
  ctx.setAttribute('customer.loyaltyTier', customer.loyaltyTier);

  return savedOrder;
});

module.exports = { processOrder };

One-time setup in your entry point:

const { init } = require('autotel');

init({
  service: 'order-api',
  environment: process.env.NODE_ENV || 'development',
});

Nested traced functions create child spans automatically:

processOrder (span)
├── checkInventory (child span)
├── chargePayment (child span)
└── sendConfirmation (child span)

You wrapped it. You didn’t rewrite it. The tests still pass. But now you can see it in production.

Wire at the Boundary

Once callers have been updated, remove the defaults and wire everything once in a composition root:

function createOrderService() {
  const deps = {
    findCustomer: (id) => db.findCustomer(id),
    checkInventory: (args) =>
      checkInventory(args, { checkStock: (items) => db.checkStock(items) }),
    chargePayment: (args) =>
      chargePayment(args, {
        charge: (id, amount) => paymentGateway.charge(id, amount),
      }),
    reserveStock: (items) => db.reserveStock(items),
    saveOrder: (order) => db.saveOrder(order),
    sendConfirmation: (args) =>
      sendConfirmation(args, {
        sendEmail: (to, subject, body) => emailService.send(to, subject, body),
      }),
    discountRules: new Map([
      ['gold', 0.15],
      ['silver', 0.1],
      ['bronze', 0.05],
    ]),
    taxRate: parseFloat(process.env.TAX_RATE || '0.08'),
  };

  return {
    processOrder: (args) => processOrder(args, deps),
  };
}

module.exports = { createOrderService };

The god class is gone. The singletons still exist in infra/ but they’re wired at the boundary, not imported everywhere. The route handler stays clean:

const orderService = createOrderService();

app.post('/orders', async (req, res) => {
  try {
    const { customerId, items } = req.body;
    const order = await orderService.processOrder({ customerId, items });
    res.status(201).json(order);
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
});

The Spec You Can Query

Here’s the payoff. Three layers of observability, each using real tools:

Span attributes via autotel: business context attached to every trace:

ctx.setAttribute('order.customerId', args.customerId);
ctx.setAttribute('customer.loyaltyTier', customer.loyaltyTier);
ctx.setAttribute('order.totalInCents', totalInCents);
ctx.setAttribute('payment.transactionId', paymentResult.transactionId);

Structured logging via pino: JSON log lines with scoped context, injected as a dependency:

const log = logger.child({
  operation: 'processOrder',
  customerId: args.customerId,
});

log.info(
  { customerName: customer.name, loyaltyTier: customer.loyaltyTier },
  'customer found',
);
log.info(
  { subtotalInCents, discountInCents, taxInCents, totalInCents },
  'totals calculated',
);
log.info({ transactionId: paymentResult.transactionId }, 'payment charged');
log.info({ orderId: savedOrder.id, totalInCents }, 'order completed');

Business event tracking via track(): analytics events for downstream consumers:

track('order.completed', {
  orderId: savedOrder.id,
  customerId: args.customerId,
  totalInCents,
  itemCount: args.items.length,
  loyaltyTier: customer.loyaltyTier,
  discountInCents,
});

Structured errors with OrderError: a plain JS class that tells callers why it failed and how to fix it:

class OrderError extends Error {
  constructor(opts) {
    super(opts.message);
    this.name = 'OrderError';
    this.status = opts.status;
    this.code = opts.code;
    this.why = opts.why;
    this.fix = opts.fix;
  }
}

throw new OrderError({
  message: 'Payment failed',
  status: 402,
  code: 'PAYMENT_FAILED',
  why: 'Payment processor declined',
  fix: 'Try a different payment method',
});

Together, traces, logs, events, and structured errors turn legacy behaviour from something implicit and anecdotal into something queryable.

Every log line is JSON. Every span has business attributes. Conceptually, once logs and trace attributes are structured, you can ask questions like:

-- Find all failed orders for gold customers
SELECT * FROM logs
WHERE loyaltyTier = 'gold'
  AND error.code IS NOT NULL;

-- Average order value by loyalty tier
SELECT loyaltyTier, AVG(totalInCents)
FROM logs
WHERE status = 'confirmed'
GROUP BY loyaltyTier;

The Sequence

For legacy systems, the sequence isn’t:

Write a spec → generate code → run tests

It’s:

Observe production → extract behavioural contracts → encode as tests → use those as the spec → then bring the agent in

Step	What Changes	Public API changes?	What You Get
0	Nothing. This is the mess	-	Starting point
1	Add characterization tests	None	Behavioural lock
2	Extract to `fn(args, deps = defaultDeps)`	Usually none	Testable seams
3	Wrap with `trace()`	None	Observability
4	Remove defaults, wire at boundary	Callers update at boundary	Clean architecture
5	Structured logging + actionable errors	Error shapes change	Observable system
6	UUID IDs, webhook notifier, config	None	Features on a clean codebase

Steps 0–5 are the refactoring. Step 6 is the payoff: three features land safely because the architecture supports them: UUID order IDs (one line change), a webhook notifier (new domain function + feature flag in the composition root), and centralized config. None of them touch existing domain code. None of them break existing tests.

That’s the whole point. You don’t refactor for the sake of refactoring. You refactor so that the next feature is easy.

The tools already exist: characterization tests, strangler figs, contract testing, observability. The legacy code community discovered them before AI entered the picture. The point is not to invent a new discipline for agents. It is to carry forward the disciplines that made legacy change survivable in the first place.

We hand agents the codebase, the tests, the docs, the specs, the commit history. But in legacy systems, the most trustworthy surviving artefact is often runtime behaviour itself.

Capture that behaviour as structured, queryable context, and the system becomes legible again — to humans first, and then to agents.

The ideas in this piece draw on Michael Feathers’ Working Effectively with Legacy Code (characterization tests, seams, the legacy code change algorithm), Joost Visser’s Building Maintainable Software (maintainability guidelines, loose coupling), and x(https://charity.wtf/).

What `fn(args, deps)` actually gives you in DDD-style TypeScript

Wed, 18 Mar 2026 00:00:00 GMT

Domain-Driven Design comes with a lot of vocabulary: aggregates, repositories, domain services, bounded contexts, ubiquitous language, anemic models.

That vocabulary can make DDD sound heavier than it really is.

The useful idea is simpler: keep domain behavior and domain boundaries central, and keep infrastructure, persistence, and framework wiring secondary.

fn(args, deps) does not do that modeling for you. What it gives you is a clear shape for application-layer code in TypeScript: one place for use-case input, one place for collaborators, and less room for domain decisions to drift into wiring.

DDD as pressure, not prescription

A lot of DDD discussions get pulled into implementation details too early.

People hear “DDD” and picture layered systems, elaborate abstractions, or class-heavy code full of patterns used more out of loyalty than necessity.

That is not the useful part.

The useful pressure from DDD is much more direct:

focus on the language of the domain
protect important invariants
define boundaries around aggregates
keep infrastructure secondary to business behavior

That pressure matters whether your code uses classes, plain objects, pure functions, or a mix.

This is where fn(args, deps) becomes useful.

Not as a substitute for DDD, but as a repeatable way to structure application code around the model instead of around the framework.

That is a narrower claim than “this is DDD,” but it is also the more useful one.

Aggregates and invariants in TypeScript

In plain terms, an aggregate is a consistency boundary.

It is a boundary around related domain state that must remain valid through rules enforced by a single root. The aggregate root is the entry point through which those invariants are protected.

That sounds abstract, so here is a small example.

Imagine an order that can contain line items, can be cancelled, and can be shipped. One invariant might be that a cancelled order cannot be shipped.

Another might be that you cannot add lines once the order has shipped.

In TypeScript, that does not require a giant inheritance hierarchy. The domain model can stay small and direct.

type OrderId = string;
type OrderStatus = "draft" | "placed" | "cancelled" | "shipped";

type OrderLine = {
  productId: string;
  quantity: number;
};

type Order = {
  id: OrderId;
  status: OrderStatus;
  lines: OrderLine[];
};

const addLine = (order: Order, line: OrderLine): Order => {
  if (order.status === "shipped") {
    throw new Error("Cannot add lines to a shipped order");
  }

  if (order.status === "cancelled") {
    throw new Error("Cannot add lines to a cancelled order");
  }

  if (line.quantity <= 0) {
    throw new Error("Quantity must be greater than zero");
  }

  return {
    ...order,
    lines: [...order.lines, line],
  };
};

const ship = (order: Order): Order => {
  if (order.status === "cancelled") {
    throw new Error("Cannot ship a cancelled order");
  }

  if (order.lines.length === 0) {
    throw new Error("Cannot ship an order with no lines");
  }

  return {
    ...order,
    status: "shipped",
  };
};

In production code, I would usually prefer explicit domain error types over generic Error values, but Error keeps the examples smaller here.

This is already recognizable as domain logic.

The important point is not whether this is written as a class or as pure functions over a value.

The important point is that the rules live with the domain behavior, not buried in controllers, handlers, or database code.

That part is standard DDD. The more interesting question is what happens once an application use case has to load that model, persist it, and coordinate what comes next.

Where `fn(args, deps)` actually helps

In a DDD-influenced system, fn(args, deps) fits most naturally in the application layer.

That means it acts like a use case or application service. It loads an aggregate, invokes domain behavior, persists the result, and coordinates side effects.

If you like the “inside vs outside” framing, this is the same boundary idea as hexagonal architecture. The wiring still happens at the edge in a composition root.

That last part matters.

The function shape is helpful only if the application layer stays a coordinator. The model should still carry the meaning of the business rules.

A simple shipOrder(args, deps) might look like this:

type ShipOrderArgs = {
  orderId: string;
};

type ShipOrderDeps = {
  orders: {
    getById(id: string): Promise<Order | null>;
    save(order: Order): Promise<void>;
  };
  domainEvents: {
    publish(events: { type: string; orderId: string }[]): Promise<void>;
  };
};

export const shipOrder = async (
  args: ShipOrderArgs,
  deps: ShipOrderDeps,
): Promise<void> => {
  const order = await deps.orders.getById(args.orderId);

  if (!order) {
    throw new Error("Order not found");
  }

  const shippedOrder = ship(order);

  await deps.orders.save(shippedOrder);

  await deps.domainEvents.publish([
    {
      type: "OrderShipped",
      orderId: shippedOrder.id,
    },
  ]);
};

There is a clean split here:

args is command-shaped input
deps contains infrastructure-facing collaborators
the domain rule stays in domain behavior
the function itself orchestrates the use case

For teams that want clear boundaries without a heavy framework or a class hierarchy, that is already a meaningful improvement.

Avoiding anemic domain models without overcorrecting

One of the standard critiques in DDD discussions is the anemic domain model: objects that carry data while the important decisions live elsewhere.

That critique is still useful.

But there is also a common overcorrection where people assume every bit of behavior must live on a large entity class or the design has somehow failed.

fn(args, deps) helps with the first problem without forcing the second.

Here is the weak version:

type CancelOrderArgs = {
  orderId: string;
};

type CancelOrderDeps = {
  orders: {
    getById(id: string): Promise<Order | null>;
    save(order: Order): Promise<void>;
  };
};

export const cancelOrder = async (
  args: CancelOrderArgs,
  deps: CancelOrderDeps,
): Promise<void> => {
  const order = await deps.orders.getById(args.orderId);

  if (!order) {
    throw new Error("Order not found");
  }

  if (order.status === "shipped") {
    throw new Error("Cannot cancel a shipped order");
  }

  if (order.lines.length === 0) {
    throw new Error("Cannot cancel an empty order");
  }

  await deps.orders.save({
    ...order,
    status: "cancelled",
  });
};

This works, but the use case is making domain decisions directly.

The order is mostly a data bag.

When that happens repeatedly, the code may use DDD vocabulary without really letting the domain model do domain work.

A better version moves the decision back into domain behavior:

const cancel = (order: Order): Order => {
  if (order.status === "shipped") {
    throw new Error("Cannot cancel a shipped order");
  }

  if (order.lines.length === 0) {
    throw new Error("Cannot cancel an empty order");
  }

  return {
    ...order,
    status: "cancelled",
  };
};

export const cancelOrder = async (
  args: CancelOrderArgs,
  deps: CancelOrderDeps,
): Promise<void> => {
  const order = await deps.orders.getById(args.orderId);

  if (!order) {
    throw new Error("Order not found");
  }

  await deps.orders.save(cancel(order));
};

That is the distinction that matters.

The point of the pattern is not to drag decisions upward into orchestration. It is to keep orchestration honest while leaving the rules with the model.

Aggregate boundaries become easier to see

One practical side effect of fn(args, deps) is that aggregate boundaries become easier to see.

In DDD-style systems, repositories are often modeled around aggregate roots. That means the dependency list for a use case often acts as feedback on the shape of the model.

For example:

type ApproveInvoiceDeps = {
  invoices: InvoiceRepository;
  customers: CustomerRepository;
  creditPolicies: CreditPolicyService;
  outbox: OutboxWriter;
};

Sometimes that is perfectly reasonable.

Sometimes it is a warning sign.

If a single use case keeps needing multiple repositories, cross-aggregate reads, extra coordination services, and transaction workarounds, the code is often telling you something about the boundary you picked.

That is one of the more practical benefits of the pattern.

Because dependencies are explicit, awkward boundaries become harder to ignore. Instead of disappearing into injected class state or framework wiring, the complexity stays visible in the use case signature.

That is useful in code review, in refactoring, and in the earlier stages of figuring out where aggregate boundaries really belong.

Transactions are where reality starts

This is also where the pattern stops being self-explanatory.

The earlier shipOrder(args, deps) example is fine as a teaching example, but real systems have to care about failure between steps. If orders.save succeeds and outbox.append fails, you have now persisted new state without recording the event the rest of the system may depend on.

fn(args, deps) does not solve that problem by itself.

What it does do is give you an obvious place to model the transactional boundary explicitly:

type ShipOrderDeps = {
  withTransaction<T>(
    work: (tx: {
      orders: {
        getById(id: string): Promise<Order | null>;
        save(order: Order): Promise<void>;
      };
      outbox: {
        append(events: { type: string; orderId: string }[]): Promise<void>;
      };
    }) => Promise<T>,
  ): Promise<T>;
};

export const shipOrder = async (
  args: ShipOrderArgs,
  deps: ShipOrderDeps,
): Promise<void> => {
  await deps.withTransaction(async (tx) => {
    const order = await tx.orders.getById(args.orderId);

    if (!order) {
      throw new Error("Order not found");
    }

    const shippedOrder = ship(order);

    await tx.orders.save(shippedOrder);
    await tx.outbox.append([
      { type: "OrderShipped", orderId: shippedOrder.id },
    ]);
  });
};

You can model that as a unit of work, a transaction wrapper, an outbox writer inside a transaction, or something equivalent.

The abstraction can vary. What matters is that transaction coordination is an explicit application-layer decision, not an invisible assumption.

Domain events are a design choice, not a freebie

Domain events are another place where fn(args, deps) fits naturally.

Some teams model domain events as part of the domain result. Others derive them in the application layer after a domain operation. Either way, fn(args, deps) gives you a clear place to persist state and then hand those events to an explicit dependency such as an event publisher or outbox writer.

type SaveResult = {
  order: Order;
  events: { type: string; orderId: string }[];
};

const shipWithEvents = (order: Order): SaveResult => {
  const shippedOrder = ship(order);

  return {
    order: shippedOrder,
    events: [{ type: "OrderShipped", orderId: shippedOrder.id }],
  };
};

type ShipOrderWithOutboxDeps = {
  orders: {
    getById(id: string): Promise<Order | null>;
    save(order: Order): Promise<void>;
  };
  outbox: {
    append(events: { type: string; orderId: string }[]): Promise<void>;
  };
};

export const shipOrderWithOutbox = async (
  args: ShipOrderArgs,
  deps: ShipOrderWithOutboxDeps,
): Promise<void> => {
  const order = await deps.orders.getById(args.orderId);

  if (!order) {
    throw new Error("Order not found");
  }

  const result = shipWithEvents(order);

  await deps.orders.save(result.order);
  await deps.outbox.append(result.events);
};

In this version, the domain operation returns events rather than storing them on the aggregate itself.

That is a design choice, not a law of DDD.

Some teams prefer aggregates to collect events internally and expose them after the operation. In function-oriented code, returning the events directly often keeps side effects more visible and avoids hidden mutable state. But that is a stylistic preference, not something the domain model forced on you.

The main win is simpler than that: the eventing concern stays explicit without smuggling infrastructure mechanics into every domain type.

Keep TypeScript dependency typing practical

One fair criticism of this style is that dependency typing can get repetitive in large codebases.

If every use case hand-writes an inline repository shape with getById and save, the code stays explicit but starts to feel noisy.

That is usually a sign to standardize the building blocks without giving up the narrow dependency surface:

type Repository<T> = {
  getById(id: string): Promise<T | null>;
  save(entity: T): Promise<void>;
};

type ShipOrderDeps = {
  orders: Pick<Repository<Order>, "getById" | "save">;
  outbox: OutboxWriter;
};

That keeps the use case honest about what it needs while avoiding copy-pasting the same method signatures across dozens of files.

The useful constraint is not “write every dependency type from scratch forever.”

The useful constraint is “make dependencies explicit and keep them narrower than the whole container.”

Why teams tend to like this shape

The appeal is not theoretical purity. It is consistency.

useCase(args, deps) gives teams a predictable entry point, makes collaborators easy to inspect, keeps testing straightforward, and exposes anemic designs by making repeated decisions easier to notice.

In practice, that means less hunting through framework wiring to figure out where a business action starts and what it actually depends on.

Where this pattern stops

It is worth being precise here.

fn(args, deps) does not solve strategic DDD. It does not define bounded contexts, create a ubiquitous language, discover aggregate boundaries, or model long-running workflows for you.

And tidy function signatures definitely do not guarantee a good domain model.

Those things still require domain knowledge, modeling effort, and iteration.

Once a business process crosses multiple aggregates, multiple transactions, or multiple services, a tidy useCase(args, deps) per step is no longer the whole story. You now need workflow coordination, retry behavior, idempotency, and eventual consistency concerns that live above a single aggregate use case.

That is not a failure of the pattern. It is just the boundary of what the pattern is for.

What this pattern does give you is a low-ceremony way to express application-layer decisions clearly once you have started making them.

Why this still matters

DDD is about domain focus, domain language, and domain boundaries.

fn(args, deps) does not do that work for you. What it does do is give the work a clear place to live: domain rules in domain behavior, coordination in the application layer, and infrastructure at the edges.

None of this removes the need to model responsibilities well inside the domain; it only gives those responsibilities a cleaner application boundary.

That is a modest promise, but in real TypeScript systems it is often the difference between code that merely mentions DDD and code that can actually support it.

fn(args, deps) and Hexagonal Architecture

Tue, 17 Mar 2026 00:00:00 GMT

Hexagonal architecture often gets introduced with a lot of diagrams, vocabulary, and ceremony.

Ports. Adapters. Application core. Inbound and outbound boundaries.

That framing is useful, but it can also make the idea feel more abstract than it really is.

You may also hear this called “ports and adapters” — same idea, different name. The port is what a function needs; the adapter is what fulfils it.

A simpler way to understand it is this:

Hexagonal architecture is mostly about keeping business logic independent from delivery and infrastructure details.

One practical way to do that in plain TypeScript is this shape:

fn(args, deps);

Where args is the input for this use case, and deps is the set of external capabilities it needs. You can read that as: data in, capabilities in.

In plain TypeScript, fn(args, deps) is a simple way to implement the core idea of hexagonal architecture without turning the pattern into ceremony.

The real point of hexagonal architecture

A lot of explanations make hexagonal architecture sound like the point is drawing a boundary around “the domain.”

That is not wrong, but it is incomplete.

The real point is to stop business logic from being tangled up with transport, frameworks, and infrastructure.

You do not want your core use case to care whether it was called from:

HTTP
a CLI
a queue consumer
a cron job
a test

And you do not want it tightly coupled to:

a specific database client
a specific mail provider
a framework request object
a logger singleton
process-wide global state

That is the architectural problem hexagonal architecture is trying to solve.

A use case can just be a function

In plain TypeScript, a use case does not need to be a class to have a boundary.

It can just be a function.

type CreateUserArgs = {
  name: string;
  email: string;
};

type CreateUserDeps = {
  saveUser: (input: { name: string; email: string }) => Promise<User>;
  sendWelcomeEmail: (user: User) => Promise<void>;
  log: (message: string, meta?: unknown) => void;
};

export async function createUser(args: CreateUserArgs, deps: CreateUserDeps) {
  deps.log('Creating user', { email: args.email });

  const user = await deps.saveUser(args);
  await deps.sendWelcomeEmail(user);

  return user;
}

That function already has the important boundary:

args is what the use case needs from the caller
deps is what the use case needs from the outside world

That is a very direct way to express the “inside versus outside” split that hexagonal architecture cares about.

args and deps map cleanly to the hexagon

This is the useful connection.

In hexagonal architecture, people often talk about:

inbound adapters
outbound adapters
ports
the application core

Those ideas can sound bigger than they are.

With fn(args, deps), the mapping is usually straightforward:

args comes from an inbound adapter
deps is implemented by outbound adapters
the function body is the application core
the function signature is the boundary

Those adapters get assembled at the application edge — your composition root.

For example:

an HTTP handler can translate a request into args
a Postgres repository can implement part of deps
an email provider can implement part of deps
the use case itself stays unaware of Express, Fastify, Postgres, or SES

That is hexagonal architecture in practice, not just in a diagram.

The HTTP layer becomes an adapter, not the center

Here is a simple HTTP adapter:

import { createUser } from '../application/create-user';
import { makeCreateUserDeps } from '../composition-root';

export async function postUsersHandler(req: Request, res: Response) {
  const deps = makeCreateUserDeps();

  const user = await createUser(
    {
      name: req.body.name,
      email: req.body.email,
    },
    deps,
  );

  res.status(201).json({
    id: user.id,
    name: user.name,
    email: user.email,
  });
}

The handler does three things:

reads transport-specific input
calls the use case
translates the result back to transport-specific output

That is adapter work.

The business logic is not in the handler. The handler is not the center of the system. It is just one way into the core.

That is one of the biggest practical wins of hexagonal architecture: the delivery mechanism becomes replaceable.

Composition Roots and fn(args, deps)

Mon, 16 Mar 2026 00:00:00 GMT

A lot of developers hear “dependency injection” and immediately think of containers, decorators, registration APIs, lifecycle scopes, and framework magic.

That reaction is understandable.

But that association often leads people to overcomplicate a problem that has a much simpler starting point.

At its core, dependency injection just means this:

Pass collaborators in explicitly instead of reaching for them implicitly.

One of the simplest ways to do that in plain TypeScript is this shape (introduced in the series starting point: fn(args, deps)):

fn(args, deps)

Where args is call-specific input and deps is the set of collaborators the function needs.

You can read that as: data in, capabilities in.

fn(args, deps) is flexible enough to support composition, testing, and clean application wiring without forcing you into a DI framework.

Why explicit dependencies matter

Consider a function like this:

type CreateUserArgs = {
  name: string;
  email: string;
};

type CreateUserDeps = {
  db: Database;
  mailer: Mailer;
  logger: Logger;
};

export async function createUser(
  args: CreateUserArgs,
  deps: CreateUserDeps,
) {
  deps.logger.info("Creating user", { email: args.email });

  const user = await deps.db.users.insert(args);
  await deps.mailer.sendWelcome(user);

  return user;
}

The signature tells you almost everything you need to know about this function:

what input the function needs
what external capabilities it relies on
what has to be provided in tests
what has to be wired at the edge of the app

Nothing is hidden in global state. Nothing is implied through a class instance. Nothing relies on framework wiring happening somewhere else.

That explicitness is the payoff.

This is just a function shape, not a framework

fn(args, deps) is not a DI framework.

It does not require a container, decorators, runtime lookup, reflection, registration, or auto-wiring:

await createUser(
  { name: "Ada", email: "ada@example.com" },
  { db, mailer, logger },
);

That is just a function with explicit collaborators, and that is exactly why it is useful.

If you can call a function, you already know how to use this pattern.

In many TypeScript codebases, that is enough, and it is often better than introducing a container prematurely.

The composition root is where the wiring happens

fn(args, deps) does not mean manually passing dependencies through every layer forever.

The function stays explicit about what it needs. The application satisfies those dependencies at the edge.

That edge is your composition root.

// main.ts
import { createUser } from "./create-user";
import { makeDb } from "./infra/db";
import { makeMailer } from "./infra/mailer";
import { makeLogger } from "./infra/logger";

const db = makeDb(process.env.DATABASE_URL!);
const mailer = makeMailer();
const logger = makeLogger();

const productionDeps = { db, mailer, logger };

await createUser(
  { name: "Ada", email: "ada@example.com" },
  productionDeps,
);

That startup file is the composition root, the place where infrastructure gets created and connected to application logic.

In a CLI, this might be bin/main.ts. In a web application, it might be the HTTP server bootstrap or the framework entrypoint.

It is usually worth creating small factory functions for reusable infrastructure, things like makeLogger, makeDb, and makeMailer.

These are the primitive building blocks of the composition root: small functions that construct infrastructure without mixing in domain logic.

They give the composition root reusable infrastructure pieces, while fn(args, deps) shows how those pieces are combined into application logic.

Bind dependencies when it helps ergonomics

If you do not want every call site to pass both args and deps, bind dependencies once and expose a simpler function:

export const makeCreateUser =
  (deps: CreateUserDeps) =>
  (args: CreateUserArgs) =>
    createUser(args, deps);

Then at the composition root:

const createUserWithServices = makeCreateUser({ db, mailer, logger });

await createUserWithServices({
  name: "Ada",
  email: "ada@example.com",
});

Tests can still call the base function directly:

await createUser(
  { name: "Ada", email: "ada@example.com" },
  { db: fakeDb, mailer: fakeMailer, logger: fakeLogger },
);

The pattern supports direct calls, partial application, and pre-wired functions without hiding the dependency boundary.

If you bind every function eagerly you can end up with a forest of factories. Reach for makeX when there are multiple call sites or when you are crossing a boundary such as from routes or controllers into domain logic. Otherwise, calling fn(args, deps) directly is fine.

When plain imports are better

Not every dependency needs to be injected.

For pure helpers with no side effects, direct imports are usually simpler:

import { normalizeEmail } from "./normalize-email";

export function toUserRecord(args: CreateUserArgs) {
  return {
    ...args,
    email: normalizeEmail(args.email),
  };
}

Passing it through deps would add ceremony without much value.

A useful rule of thumb (also echoed in the SOLID post):

use direct imports for pure logic
use deps for things that touch the outside world, vary by environment, or matter in tests

You can think of deps as the things that talk to the world or differ by environment. Pure helpers are regular imports, not missing dependencies.

What this pattern prevents

A lot of the value comes from the mistakes it makes harder to hide.

When dependencies are explicit:

hidden coupling is harder to introduce
tests are easier to write honestly
dependency growth shows up in the signature
wiring lives at the edge instead of being smeared across the codebase

Compare these two versions:

// Hidden dependency: reaches into infrastructure directly
import { db } from "../infra/db";

export async function getUserHidden(userId: string) {
  return db.users.find(userId);
}

// Explicit dependency: db is part of the signature
export async function getUserExplicit(
  args: { userId: string },
  deps: { db: Database },
) {
  return deps.db.users.find(args.userId);
}

In the first version, tests have to mock the module import or spin up a real database. In the second, you pass in whatever you want.

The real case for fn(args, deps)

The case for this pattern is simple: it keeps dependencies explicit, wiring honest, and complexity optional.

Used consistently, it gives you a codebase where side effects are visible in signatures, tests are straightforward to wire, and application assembly happens in one honest place.

That is the appeal of fn(args, deps).

In plain TypeScript, it is a better default than reaching for framework-style DI before your code actually demands it.

AI Makes It Easy to Build. That Doesn’t Mean We Should Ship More

Thu, 12 Mar 2026 00:00:00 GMT

Dax is spot on.

sent this to the team today

everything great comes from being able to delay gratification for as long as possible

and it feels like we're collectively losing our ability to do that pic.twitter.com/HlIpY86eJn
— dax (@thdxr) March 10, 2026

A couple of weeks ago, I attended the Cambridge Product Management meetup.

During a discussion, someone said their company had given everyone AI tools and that they were now shipping faster than ever.

I told him that it risked being irresponsible.

When it becomes trivial to build things, the bar for what deserves to ship often drops.

Suddenly, every idea can become a feature.

First, bombarding users with constant changes is confusing. Even with feature flags. If every time someone opens your product, something has changed, you risk frustrating them rather than helping them.

Second, features need time to bed in.

You need time to explain them: marketing, emails, videos, blog posts, support docs.

You cannot compress this process.

Like a good wine, some things simply take time. You need an inspect-and-adapt loop that produces insights you can actually base decisions on.

Just because AI makes it easier to build things doesn’t mean we should ship more of them.

Everything great comes from delaying gratification.

When it comes to using AI, be like Luke

Mon, 09 Mar 2026 00:00:00 GMT

I love using AI coding tools. As someone who definitely gets their money’s worth, I feel more productive than ever.

But sometimes, just like Obi-Wan told Luke, you need to let go.

Control the instinct to reach for AI all the time.

Trust yourself.

Yes, AI can help you ship faster. I feel that too. But if every hard problem is handed off to a tool, you slowly lose the feel for the craft. The ability to reason through a problem, explore the codebase, and figure things out.

One way I like to mix things up is through Software Craftsmanship meetups.

At my local Cambridge meetup, we sometimes pair up and work through coding activities using good old pair programming.

The last time I did this, we switched off the AI completely and just enjoyed solving the problem together.

No copilots. No prompts. Just thinking, experimenting, and learning.

And that’s the thing. Sometimes stepping away from the tools reminds you why you enjoyed programming in the first place.

Stretch those muscles. Don’t become stiff and stale.

Why Matt Pocock Is Right About Making Codebases AI Agents Love

Thu, 05 Mar 2026 00:00:00 GMT

Matt Pocock is once again on the money in his video “How To Make Codebases AI Agents Love”.

Here’s why I agree with him.

The Memento analogy for how AI sees a codebase is spot on. And the idea of deep modules as progressive disclosure of complexity makes a lot of sense.

One thing that has worked well for me, especially when using AI coding agents on legacy codebases, is enforcing a simple code constraint.

fn(args, (deps = defaultDeps));

It is a non-breaking change, and it has proven very reliable.

I wrote about it here:

If You Only Enforce One Rule for AI Generated Code, Make It fn(args, deps).

When he said “the interface sits at the top and explains what the module does”, that idea translates nicely inside the module too.

The Deps type makes the dependency boundary explicit (the same “inside vs outside” idea as ports and adapters). It says exactly what the function needs from the outside world. The AI reads the contract first instead of guessing which collaborators exist or which ones matter.

It also makes testing simpler. Dependencies are visible. Interfaces stay small. You can hold a few of them in your head and let the AI handle the implementation within the module.

This aligns with his point that what works for humans also works for AI.

Making dependencies explicit is not clever. It is boring.

But boring constraints are what let both humans and agents reason about the code.

If You Only Enforce One Rule for AI Code, Make It fn(args, deps)

Wed, 04 Mar 2026 00:00:00 GMT

AI coding agents produce code faster than you can review and understand it.

One pattern works in both new and legacy codebases because you can adopt it incrementally, without breaking callers.

For business logic, treat every function as having two inputs: data (args) and capabilities (deps).

Without a clear constraint, generated code becomes harder to reason about: dependencies disappear, side effects spread, composition gets messy. This is why visible structure is essential.

fn(args, deps) is that constraint

For existing code, start with:

functionName(args, deps = defaultDeps)

Dependencies are explicit, not hidden.

There’s no framework and no package to install.

Linters and reviews catch problems after the fact; it’s better to get the shape right before the agent writes the code.

This pattern is functional DI with explicit seams, good design regardless. It also happens to keep AI-generated code tractable.

This means: no hidden infrastructure imports, no global singletons, and no constructors that quietly accumulate the whole world.

The types explain intent. The agent has to colour within the lines.

This approach is boring, but that’s precisely why it scales.

The rule matters where business logic meets infrastructure. Pure functions don’t need deps; UI event handlers and glue code don’t need the ceremony. Request-scoped context (auth, tenant, transaction) follows the same rule: pass it in args or expose it as a dep (e.g. getCurrentUser); no need for an extra parameter.

The core idea: make dependencies visible, not implicit

Usually, the problem isn’t testability; it’s that the dependency surface is invisible. As a result, you jump between files to understand a method; an agent overmocks the whole constructor and produces tests that pass for the wrong reasons.

Hidden dependencies (often via constructor injection) are the most common culprit:

class UserService {
  constructor(
    private db: Database,
    private logger: Logger,
    private cache: Cache,
    private mailer: Mailer,
    private metrics: Metrics,
  ) {}

  async getUser(userId: string) {
    this.logger.info(`Getting user ${userId}`);
    return this.db.findUser(userId);
  }
}

getUser needs db and logger, but the constructor forces every caller (and every test) to satisfy five dependencies.

An agent sees five dependencies and mocks all five. It doesn’t know which of getUser’s methods are actually used.

When someone later adds a side effect inside getUser, the test still passes and verification drifts. The signature lies; the constructor hides the truth.

fn(args, deps) makes the truth the default

export type GetUserArgs = { userId: string };
export type GetUserDeps = { db: Database; logger: Logger };

export async function getUser(args: GetUserArgs, deps: GetUserDeps) {
  deps.logger.info(`Getting user ${args.userId}`);
  return deps.db.findUser(args.userId);
}

When you ask, “What does this need?”, you see it in the signature. So does the agent. There’s no guessing or overmocking: the agent’s job becomes mechanical—read the types, mock those, call, and assert. You don’t thread deps through ten unrelated layers; you wire them once at a composition root.

Why this is a testing upgrade

You mock exactly what the function uses. No module interception, no vi.mock() hoisting, no constructor archaeology. Good tests stop describing object construction and start describing behaviour.

import { mock } from "vitest-mock-extended";
import { getUser, type GetUserDeps } from "./get-user";

it("returns the user when found", async () => {
  const deps = mock<GetUserDeps>();
  deps.db.findUser.mockResolvedValue({ id: "1", name: "Alice" });

  const user = await getUser({ userId: "1" }, deps);
  expect(user?.name).toBe("Alice");
});

When the dependency surface changes, the Deps type changes, and tests fail at compile time rather than drift. For an agent, that’s a clear signal (“Property ‘logger’ is missing”); with classes, errors are noisier, and the agent often fixes one thing and breaks another.

Ports and adapters, without the ceremony

Deps are ports: capabilities the function needs (db, logger, sendEmail, current time). Adapters are what you wire at the boundary.

export type SendWelcomeEmailDeps = {
  sendEmail: (input: { to: string; template: string }) => Promise<void>;
};

export async function sendWelcomeEmail(
  args: { email: string },
  deps: SendWelcomeEmailDeps,
) {
  await deps.sendEmail({ to: args.email, template: "welcome" });
}

At the boundary: deps = { sendEmail: sendgridAdapter.sendEmail }. In tests: deps = { sendEmail: vi.fn().mockResolvedValue(undefined) }. The agent only needs to satisfy the type; it doesn’t need to know SendGrid or SMTP. This is ports and adapters reduced to a function signature.

Adopting the pattern in an existing codebase

You don’t need a big-bang rewrite. Add a seam (e.g. route through a factory), then migrate one function at a time; contract tests tell you if behaviour changes. The move that makes it backward-compatible: add deps as the last argument with a default so existing callers keep working.

import { mailer as _mailer, type Mailer } from "../infra/mailer";

export type SendWelcomeEmailDeps = { mailer: Mailer };
const defaultDeps: SendWelcomeEmailDeps = { mailer: _mailer };

export async function sendWelcomeEmail(
  recipient: User,
  sender: User,
  deps: SendWelcomeEmailDeps = defaultDeps,
) {
  const { mailer } = deps;
  return mailer.send({
    to: recipient.email,
    from: sender.email,
    template: "welcome",
  });
}

Test seam on day one, no call-site churn. Later: migrate callers to pass deps from the composition root, then remove the default. Same pattern for any function that currently imports infrastructure.

Wire once at the boundary

You don’t pass deps at every call site. Wire once at a composition root (service factory, router, app bootstrap). Domain functions receive deps and pass them down only when calling other domain functions; if you’re plumbing deps through five layers, your module boundaries are likely wrong.

export function createUserService(deps: { db: Database; logger: Logger }) {
  return {
    getUser: (args: { userId: string }) => getUser(args, deps),
    createUser: (args: { name: string; email: string }) => createUser(args, deps),
  };
}

const userService = createUserService({ db, logger });
await userService.getUser({ userId: "123" });

The agent follows one rule: domain functions are fn(args, deps); wiring lives in the factory.

The “if you only do one thing” rules

Keep the rule set short. Agents do best with a small, repeatable set of constraints.

Business logic is fn(args, deps).
Domain code does not import infrastructure at runtime (no DB clients, HTTP clients, env reads).
Inject what you want to mock (network, disk, clock, collaborators). Don’t inject pure utilities—they don’t need test doubles.
Wire adapters at the boundary (a composition root or service factory).
Defaults are allowed only for migration. Temporary scaffolding.

That’s it. Not a framework, not a platform bet, just a dependency surface that stays visible. When the agent has that, it can reason about what, how, and why, and so can you. Once the seams are visible, testing, composition, and refactoring stop fighting the code.

Start using this visible dependency surface today. Apply the fn(args, deps) rule in your next PR or refactor. By making seams visible, you make testing, composition, and refactoring smoother. The sooner you begin, the sooner you gain clarity for both you and the agent. Embrace the pattern; your codebase will thank you.

Visualising Awaitly Workflows with awaitly-visualizer

Sat, 07 Feb 2026 00:00:00 GMT

You have an Awaitly workflow: a few steps, some dependencies, typed results. It works. When someone asks “what does this do?” or you need to debug a run, you’re left tracing through code.

What if you could see the same workflow as a diagram? awaitly-visualizer plugs into your workflow’s events and turns them into that picture. For a checkout that runs fetchCart, validateCart, processPayment, then completeOrder, you get output like this:

┌── checkout ────┐
│  ✓ fetchCart   │
│  ✓ validateCart│
│  ✓ processPayment
│  ✓ completeOrder
│  Completed     │
└────────────────┘

Same idea as Mermaid flowcharts: steps, order, success and failure. This post walks through adding it step by step. All of the code below lives in a test in the repo so you can run it yourself.

Part 1: A minimal checkout workflow

Start with a workflow that has no visualisation. We’ll use a simple checkout: fetch cart, validate, process payment, complete order. Each operation returns an AsyncResult; the workflow composes them with step().

import { ok, err, type AsyncResult } from "awaitly/core";
import { createWorkflow } from "awaitly/workflow";

type Cart = { id: string; itemCount: number; totalCents: number };

const fetchCart = async (
  cartId: string
): AsyncResult<Cart, "CART_NOT_FOUND"> => {
  if (cartId === "missing") return err("CART_NOT_FOUND");
  return ok({
    id: cartId,
    itemCount: 2,
    totalCents: 5999,
  });
};

const validateCart = async (
  cart: Cart
): AsyncResult<boolean, "INVALID_CART"> => {
  if (cart.itemCount < 1) return err("INVALID_CART");
  return ok(true);
};

const processPayment = async (
  cart: Cart
): AsyncResult<{ transactionId: string }, "PAYMENT_FAILED"> => {
  return ok({ transactionId: "txn-123" });
};

const completeOrder = async (
  cartId: string
): AsyncResult<{ orderId: string }, "COMPLETE_ERROR"> => {
  return ok({ orderId: `order-${cartId}` });
};

const deps = {
  fetchCart,
  validateCart,
  processPayment,
  completeOrder,
};

Run the workflow by creating it with a name and deps, then calling it with a function that uses step() for each operation:

const workflow = createWorkflow("checkout", deps);

const result = await workflow(
  async (step, { fetchCart, validateCart, processPayment, completeOrder }) => {
    const cart = await step("fetchCart", () => fetchCart("cart-1"));
    await step("validateCart", () => validateCart(cart));
    const payment = await step("processPayment", () => processPayment(cart));
    const order = await step("completeOrder", () => completeOrder(cart.id));
    return { cart, payment, order };
  }
);

Nothing is visualised yet. The workflow just runs and returns a result. Next we wire in the visualiser so every step is reflected in a diagram.

Part 2: Live visualisation with createVisualizer

Create a visualiser with a workflow name (used as the diagram title). Pass its handleEvent into the workflow’s onEvent option. After the workflow completes, call render() to get the default ASCII diagram, or renderAs("mermaid") for Mermaid source.

We’ll use the same workflow body everywhere below; only the wiring changes. Here it is in full once:

import { createVisualizer } from "awaitly-visualizer";

const viz = createVisualizer({
  workflowName: "checkout",
  detectParallel: false,  // this workflow is strictly sequential; with step.parallel() or step.race(), set true to show parallel/race scopes
});

const workflow = createWorkflow("checkout", deps, {
  onEvent: viz.handleEvent,
});

// Same four-step workflow as Part 1; we only add onEvent above
await workflow(
  async (step, { fetchCart, validateCart, processPayment, completeOrder }) => {
    const cart = await step("fetchCart", () => fetchCart("cart-1"));
    await step("validateCart", () => validateCart(cart));
    const payment = await step("processPayment", () => processPayment(cart));
    const order = await step("completeOrder", () => completeOrder(cart.id));
    return { cart, payment, order };
  }
);

console.log(viz.render());
console.log(viz.renderAs("mermaid"));

Example output from the blog-journey test (ASCII from viz.render(), Mermaid from viz.renderAs("mermaid")):

ASCII (terminal-style box):

┌── checkout ──────────────────────────────────────────────────────────────────┐
│                                                                              │
│  ✓ fetchCart [0ms]                                                           │
│  ✓ validateCart [0ms]                                                        │
│  ✓ processPayment [0ms]                                                      │
│  ✓ completeOrder [0ms]                                                       │
│                                                                              │
│  Completed in 2ms                                                            │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

Mermaid (flowchart source):

flowchart TD
    start(("▶ Start"))
    step_fetchCart["✓ fetchCart 0ms"]:::success
    start --> step_fetchCart
    step_validateCart["✓ validateCart 0ms"]:::success
    step_fetchCart --> step_validateCart
    step_processPayment["✓ processPayment 0ms"]:::success
    step_validateCart --> step_processPayment
    step_completeOrder["✓ completeOrder 0ms"]:::success
    step_processPayment --> step_completeOrder
    finish(("✓ Done")):::success
    step_completeOrder --> finish

    classDef success fill:#d1fae5,stroke:#10b981,stroke-width:3px,color:#065f46
    %% other classDefs (pending, error, etc.) omitted for brevity

When a step fails

Everything so far is the happy path. If a step returns an error (for example fetchCart("missing") returns CART_NOT_FOUND), the workflow short-circuits and the visualiser shows the failure. Same code; run with a cart id that triggers the error:

// Same viz and workflow as above; only the input changes
const result = await workflow(
  async (step, { fetchCart, validateCart, processPayment, completeOrder }) => {
    const cart = await step("fetchCart", () => fetchCart("missing"));
    // ... rest unchanged
  }
);
// result.ok === false

ASCII (failed run):

┌── checkout ──────────────────────────────────────────────────────────────────┐
│                                                                              │
│  ✗ fetchCart [0ms]                                                           │
│                                                                              │
│  Failed in 0ms                                                               │
│                                                                              │
└──────────────────────────────────────────────────────────────────────────────┘

Mermaid (failed run): the flowchart shows the failing step with an error style and a “✗ Failed” end node instead of “✓ Done”. The diagram makes it obvious where the run stopped and that it did not reach later steps.

The visualiser consumes workflow events (step start, step success, workflow success, and so on) and builds an internal representation. That representation is what gets rendered as ASCII or Mermaid. You can also use renderAs("json") to inspect the structure, or renderAs("logger") for a step-by-step log style. Event handling is synchronous and lightweight, so it’s fine for dev, test, and debugging; in production you might keep visualisation behind a feature flag or sample a fraction of runs if you want to limit overhead.

Part 3: Collect then visualise with createEventCollector

Sometimes you want to record events and visualise later, or send the same events to both a visualiser and your own logger or metrics. Use createEventCollector and wire its handlers; same workflow as above, only the onEvent and what you do after the run change:

import { createEventCollector } from "awaitly-visualizer";

const collector = createEventCollector({
  workflowName: "checkout",
  detectParallel: false,
});

const workflow = createWorkflow("checkout", deps, {
  onEvent: collector.handleEvent,
});

await workflow(/* same workflow function as Part 2 */);

console.log(collector.visualize());
console.log(collector.visualizeAs("mermaid"));

The collector stores every workflow event it receives. When you call visualize(), it replays those events through an internal visualiser and returns the same kind of output you get from createVisualizer. You can also use collector.getEvents() to get the raw events, or visualizeEvents(events, options) if you’ve pushed events into your own array.

Part 4: Combine handlers

If you want both a live visualiser and another consumer (logging, metrics, or a collector for later), use combineEventHandlers so one run feeds all of them. Same workflow again; only the handler wiring changes:

import type { WorkflowEvent } from "awaitly/workflow";
import { createVisualizer, combineEventHandlers } from "awaitly-visualizer";

const viz = createVisualizer({
  workflowName: "checkout",
  detectParallel: false,
});
const collectedEvents: WorkflowEvent<unknown>[] = [];

const combined = combineEventHandlers(
  viz.handleEvent,
  (e: WorkflowEvent<unknown>) => collectedEvents.push(e)
);

const workflow = createWorkflow("checkout", deps, { onEvent: combined });
await workflow(/* same workflow function as Part 2 */);

console.log(viz.render());
// collectedEvents now holds every event for this run

Each event is passed to every handler you combined. You keep a single onEvent and get multiple observers.

Try it yourself

Install the packages:

npm install awaitly awaitly-visualizer

The code in this post is covered by tests in the awaitly repo. Clone and run the blog journey tests:

git clone https://github.com/jagreehal/awaitly
cd awaitly
pnpm install
pnpm --filter awaitly-visualizer test -- blog-journey

You’ll see the same checkout workflow exercised with createVisualizer, createEventCollector, and combineEventHandlers. From there you can add visualisation to your own workflows by passing onEvent and, if you like, collecting events for logs or Mermaid in docs. At that point, your workflows stop being invisible control flow and start being artifacts you can reason about.

Repo: github.com/jagreehal/awaitly

No, the sun does not shine out of Claude’s backside

Fri, 30 Jan 2026 00:00:00 GMT

As of today, Opus 4.5 is the best coding model I’ve used. That is not praise by vibes. That is, after building libraries and utilities that fixed problems I could not solve with the tools I was using before.

The progress is impressive.

However, it’s not all sunshine and rainbows, as people on social media and YouTube claim.

To be fair, this is not just a Claude problem. It is just the model I use daily, so it is the one I am most familiar with.

I keep seeing people brag about running Claude in a loop twenty-four seven, as if it were working shifts in a sweatshop.

Are they producing apps? Yes.

Are tasks being completed? Yes.

Is the quality good? Maybe.

When I built my version of the Ralph loop, I added an inspect-and-adapt step. Without that, what people are calling autonomy is just waterfall development with extra steps. But even with me stopping to inspect, it still wasn’t good enough.

People like me started adding hooks, skills, validators, linting, compiler checks, and tests just to coax Claude into producing code that does not immediately fall over.

But that doesn’t guarantee quality or a good product. None of it guarantees good design. It solves an immediate need. It patches a leak. It does not fix the abstraction. In fact, more often than not, it leaves you with a leaky one.

Everyone has an opinion on how to avoid this, including me, and I will share my methods later. The reality is that nobody now or ever is betting their house on a process that guarantees success.

The boring solution

So the solution is boring. Pay attention. Stay involved.

Be the pig, not the chicken.

Also, use another AI

Alongside Claude, I use another frontier model, Codex 5.2, purely as a reviewer.

So far, Claude has not produced code that Codex considered correct on the first attempt. Often it takes many iterations.

When Codex finds an issue, I ask it to create a failing test. I do this on purpose, and I do not automate it, even though I could.

The friction matters.

I read the issue, copy it, and hand it back to Claude to fix. How Codex identifies problems and how Claude explains its reasoning and resolution for a failing test are among the best lessons I’ve had.

But by attentive, I mean that when Claude starts going off piste and over-engineering a solution, I stop it and give feedback.

Yes, I am a backseat driver. That’s fine. AI isn’t going to kick me out of the car just yet.

Stop Throwing Errors. Awaitly Makes Async Error Handling Actually Work

Thu, 22 Jan 2026 00:00:00 GMT

We’ve all written this code:

const lambdaHandler = async () => {
  try {
    const db = await connectToDb();
    const result = await errorHandler({ taskId, error }, { db });
    return { statusCode: 200, body: { message: 'Success', task: result } };
  } catch (error) {
    return { statusCode: 500, body: { message: 'Error' } };
  }
}

That catch (error) swallows everything. Was it a “task not found”? A database connection failure? A permissions issue? Who knows.

Throwing exceptions for expected failures is like using GOTO. You lose the thread.

Awaitly fixes this by treating errors as data, not explosions. This guide teaches the patterns one concept at a time.

Part 1: Foundation (No Library)

Before we touch Awaitly, let’s fix some fundamental patterns using pure TypeScript. These are good practices regardless of what error handling library you use.

Step 0: The Starting Point

Here’s a typical function that creates its own dependency:

async function getTask(taskId: string): Promise<Task> {
  // This runs every time!
  const db = await connectToDb();

  const task = await db.findTask(taskId);
  if (!task) throw new Error('Task not found');
  return task;
}

async function handler(event: { taskId: string }) {
  try {
    const task = await getTask(event.taskId);
    return { statusCode: 200, body: { task } };
  } catch {
    return { statusCode: 500, body: { message: 'Error' } };
  }
}

The problem: the db connection is created every time getTask is called. We can’t test getTask without a real database, and we can’t reuse the connection across multiple functions.

Call handler twice and you connect to the database twice. Wasteful.

Step 1: Move Dependencies to the Boundary

Create dependencies at the “boundary” (your handler), then pass them down.

type Db = { findTask: (id: string) => Promise<Task | null> };

// db is passed IN as a parameter
async function getTask(taskId: string, db: Db): Promise<Task> {
  const task = await db.findTask(taskId);
  if (!task) throw new Error('Task not found');
  return task;
}

// Create db ONCE at the boundary
const db = await connectToDb();

async function handler(event: { taskId: string }, deps: { db: Db }) {
  try {
    const task = await getTask(event.taskId, deps.db);
    return { statusCode: 200, body: { task } };
  } catch {
    return { statusCode: 500, body: { message: 'Error' } };
  }
}

// Call handler twice with the SAME db instance
await handler({ taskId: 't-1' }, { db });
await handler({ taskId: 't-1' }, { db });
// db connected only ONCE, reused across both calls

Now db is created once per handler invocation (not per function call), getTask can be tested by passing a mock db, and the connection is reusable across multiple function calls.

Bonus: getTask is now testable with a mock:

const mockDb: Db = {
  findTask: async () => ({ _id: 'mock', status: 'Pending' }),
};
const mockResult = await getTask('any-id', mockDb);

Step 2: The fn(args, deps) Pattern

Let’s standardise on a consistent function signature:

// fn({ ...args }, { ...deps })
//    ^^^^^^^^^^   ^^^^^^^^^^
//    what to do   how to do it

async function getTask(
  args: { taskId: string },
  deps: { db: Db }
): Promise<Task> {
  const task = await deps.db.findTask(args.taskId);
  if (!task) throw new Error('Task not found');
  return task;
}

// Call with named parameters
const task = await getTask({ taskId: 't-1' }, { db });

This gives you a consistent shape across all business functions, clear separation of “what” vs “how”, self-documenting named parameters, and it’s easy to add new args or deps without changing call sites.

Compare the readability:

// Positional args: what's what?
getTask('t-1', db, cache, logger);

// Named args: crystal clear
getTask({ taskId: 't-1' }, { db, cache, logger });
//       ^^^^^^^^^^^^^^^   ^^^^^^^^^^^^^^^^^^^^^
//       business args     infrastructure deps

Step 3: Return null for Expected Failures

Throwing for “not found” is using exceptions for control flow. Return null instead:

async function getTask(
  args: { taskId: string },
  deps: { db: Db }
): Promise<Task | null> {
  const task = await deps.db.findTask(args.taskId);
  if (!task) return null;  // Expected failure, not an exception
  return task;
}

// Found case
const found = await getTask({ taskId: 't-1' }, { db });
expect(found).not.toBeNull();

// Not found case: no try/catch needed!
const notFound = await getTask({ taskId: 't-missing' }, { db });
expect(notFound).toBeNull();

The problem with this approach:

const result = await getTask({ taskId: 't-missing' }, { db });
// result is null... but WHY?
// Was it not found? Permission error? DB down?

null tells us it failed, but not why. We’ll solve this in Part 2.

Part 2: Typed Results

Now we introduce Awaitly’s Result type, one piece at a time.

Step 4: Introduce ok() for Explicit Success

Wrap successful returns in ok(value):

import { ok, type AsyncResult } from 'awaitly';

async function getTask(
  args: { taskId: string },
  deps: { db: Db }
): AsyncResult<Task, never> {
  const task = await deps.db.findTask(args.taskId);
  return ok(task);  // Wrapped in ok()
}

const result = await getTask({ taskId: 't-1' }, { db });
// result.ok === true
// result.value === Task

Success is now explicit, not implicit. It prepares for adding error cases (next step), and result.ok tells you immediately if it succeeded.

The AsyncResult<Task, never> type means: “This returns a Task on success, and never returns an error.” We’ll change that never to an actual error type next.

Step 5: Introduce err() for Named Failures

Return err('ERROR_CODE') instead of null:

import { ok, err } from 'awaitly';

async function getTask(args: { taskId: string }, deps: { db: Db }) {
  const task = await deps.db.findTask(args.taskId);
  if (!task) return err('TASK_NOT_FOUND');  // Named error!
  return ok(task);
}

// Success case
const success = await getTask({ taskId: 't-1' }, { db });
// success.ok === true, success.value === Task

// Error case: we know WHAT went wrong
const failure = await getTask({ taskId: 't-missing' }, { db });
// failure.ok === false, failure.error === 'TASK_NOT_FOUND'

Now errors have a name, not just null.

Step 6: AsyncResult Type Annotation

Add explicit return types so TypeScript enforces correctness:

import { ok, err, type AsyncResult } from 'awaitly';

async function getTask(
  args: { taskId: string },
  deps: { db: Db }
): AsyncResult<Task, 'TASK_NOT_FOUND'> {
//            ^^^^  ^^^^^^^^^^^^^^^^^
//            Success type    Error type
  const task = await deps.db.findTask(args.taskId);
  if (!task) return err('TASK_NOT_FOUND');
  return ok(task);
}

const result = await getTask({ taskId: 't-1' }, { db });

// TypeScript now knows:
// - If result.ok === true, result.value is Task
// - If result.ok === false, result.error is 'TASK_NOT_FOUND'

AsyncResult<T, E> is just Promise<Ok<T> | Err<E>>. The type annotation tells TypeScript exactly what success and failure look like.

Step 7: Rich Error Objects

Use error objects with type plus context instead of plain strings:

// Rich error type with context
type TaskNotFound = { type: 'TASK_NOT_FOUND'; taskId: string };

async function getTask(
  args: { taskId: string },
  deps: { db: Db }
): AsyncResult<Task, TaskNotFound> {
  const task = await deps.db.findTask(args.taskId);
  if (!task) {
    return err({
      type: 'TASK_NOT_FOUND',
      taskId: args.taskId,  // Include context!
    });
  }
  return ok(task);
}

const result = await getTask({ taskId: 't-missing' }, { db });

if (!result.ok) {
  // Now we have rich error info for logging/debugging
  console.error(`${result.error.type}: Task ${result.error.taskId} not found`);
  // Output: TASK_NOT_FOUND: Task t-missing not found
}

Errors now carry useful debugging information. You can include taskId, timestamps, details, stack traces. Makes logging and error handling much richer.

Part 3: Composing Results

Multiple functions that return Results: how do we compose them cleanly?

Step 8: The Boilerplate Problem

Call multiple AsyncResult functions and handle each manually:

type TaskNotFound = { type: 'TASK_NOT_FOUND'; taskId: string };
type UpdateFailed = { type: 'UPDATE_FAILED'; message: string };

async function getTask(
  args: { taskId: string },
  deps: { db: Db }
): AsyncResult<Task, TaskNotFound> {
  const task = await deps.db.findTask(args.taskId);
  if (!task) return err({ type: 'TASK_NOT_FOUND', taskId: args.taskId });
  return ok(task);
}

async function markErrored(
  args: { taskId: string },
  deps: { db: Db }
): AsyncResult<Task, UpdateFailed> {
  try {
    const updated = await deps.db.updateTask(args.taskId, { status: 'Errored' });
    return ok(updated);
  } catch (e) {
    return err({ type: 'UPDATE_FAILED', message: e instanceof Error ? e.message : 'Unknown' });
  }
}

// Manual composition: lots of boilerplate!
async function handleError(
  args: { taskId: string },
  deps: { db: Db }
): AsyncResult<Task, TaskNotFound | UpdateFailed> {
  // Step 1: Get task
  const taskResult = await getTask({ taskId: args.taskId }, deps);
  if (!taskResult.ok) return taskResult;  // Boilerplate

  // Step 2: Mark errored
  const updateResult = await markErrored({ taskId: taskResult.value._id }, deps);
  if (!updateResult.ok) return updateResult;  // More boilerplate

  // Imagine 5 more steps... that's 5 more if-checks!
  return updateResult;
}

Every call needs if (!result.ok) return result. The happy path gets buried in error-checking ceremony.

Step 9: run() for Linear Composition

Use run() to compose multiple AsyncResult functions. step() unwraps ok values; err values exit early.

import { run, ok, err } from 'awaitly';

type TaskNotFound = { type: 'TASK_NOT_FOUND'; taskId: string };
type Unexpected = { type: 'UNEXPECTED'; cause: unknown };
type HandlerError = TaskNotFound | Unexpected;

async function handleError(
  args: { taskId: string },
  deps: { db: Db }
): AsyncResult<Task, HandlerError> {
  return run<Task, HandlerError>(
    async (step) => {
      // step(id, thunk): id for observability; thunk unwraps ok values. Err values exit the run() immediately.
      const task = await step('getTask', () => getTask({ taskId: args.taskId }, deps));
      // task is Task, not Result<Task, ...>: already unwrapped!

      const updated = await step('markErrored', () => markErrored({ taskId: task._id }, deps));
      // If getTask returned err, we never reach this line

      return updated;
    },
    {
      // catchUnexpected maps any unexpected throw to our error type
      catchUnexpected: (cause) => ({ type: 'UNEXPECTED', cause }),
    }
  );
}

// Happy path
const success = await handleError({ taskId: 't-1' }, { db });
// success.ok === true, success.value.status === 'Errored'

// Error path: exits early, never calls markErrored
const failure = await handleError({ taskId: 't-missing' }, { db });
// failure.ok === false, failure.error.type === 'TASK_NOT_FOUND'

The happy path now reads linearly (no if-checks), errors automatically short-circuit, and step() unwraps the value so you can use it directly.

Important API note: with run(), you use step(id, () => fn()) — an id (string) and a thunk. The thunk lets run() catch synchronous throws.

Step 10: step.try() for Throwing Code

Third-party SDKs (Slack, Stripe, database drivers) throw exceptions. Use step.try() to wrap them into typed errors:

type ConnectFailed = { type: 'CONNECT_FAILED'; message: string };
type TaskNotFound = { type: 'TASK_NOT_FOUND' };
type Unexpected = { type: 'UNEXPECTED'; cause: unknown };
type StepError = ConnectFailed | TaskNotFound | Unexpected;

// Simulate an SDK that throws
const unstableDb = {
  connect: async () => {
    throw new Error('Connection refused');
  },
};

async function handleWithUnstableDb(): AsyncResult<string, StepError> {
  return run<string, StepError>(
    async (step) => {
      // step.try(id, thunk, opts) catches the throw and converts to typed error
      const db = await step.try('connect', () => unstableDb.connect(), {
        onError: (e): ConnectFailed => ({
          type: 'CONNECT_FAILED',
          message: e instanceof Error ? e.message : 'Unknown',
        }),
      });

      // This line never runs because connect() threw
      const task = await db.findTask();
      return task ? 'success' : 'no-task';
    },
    {
      catchUnexpected: (cause) => ({ type: 'UNEXPECTED', cause }),
    }
  );
}

const result = await handleWithUnstableDb();
// result.ok === false
// result.error.type === 'CONNECT_FAILED'
// result.error.message === 'Connection refused'

Use step.try() to wrap specific calls you know can throw (SDKs, I/O). Use catchUnexpected as a safety net for bugs and unexpected throws.

Prefer step.try() for known throwers; rely on catchUnexpected as a fallback for genuine surprises.

Part 4: Workflows

createWorkflow keeps the same mental model as run() but adds automatic error inference.

Step 11: createWorkflow for Automatic Error Inference

With run(), you manually maintain the error union:

type HandlerError = TaskNotFound | DbFailed | SlackFailed | Unexpected;
return run<TaskWithError, HandlerError>(...);

With createWorkflow, the error union is computed from your deps:

import { createWorkflow } from 'awaitly/workflow';

type TaskNotFound = { type: 'TASK_NOT_FOUND'; taskId: string };
type DbFailed = { type: 'DB_FAILED'; message: string };
type SlackFailed = { type: 'SLACK_FAILED'; message: string };

const loadTask = async (taskId: string): AsyncResult<Task, TaskNotFound> => {
  const task = taskStore.get(taskId);
  if (!task) return err({ type: 'TASK_NOT_FOUND', taskId });
  return ok(task);
};

const markErrored = async (taskId: string): AsyncResult<Task, DbFailed> => {
  // ... implementation
};

const notifySlack = async (taskId: string): AsyncResult<void, SlackFailed> => {
  // ... implementation
};

// createWorkflow: declare deps, get automatic error inference
const errorHandler = createWorkflow({ loadTask, markErrored, notifySlack });

// TypeScript INFERS the error type from deps:
// Error = TaskNotFound | DbFailed | SlackFailed | UnexpectedError
// No manual "type HandlerError = ..." needed!

const result = await errorHandler(async (step, deps) => {
  const task = await step('loadTask', () => deps.loadTask('t-1'));
  const updated = await step('markErrored', () => deps.markErrored(task._id));
  await step('notifySlack', () => deps.notifySlack(updated._id));
  return updated;
});

No manual union maintenance (add a dep and the error type is auto-included), UnexpectedError is included by default (no catchUnexpected needed), and each step uses step(id, () => deps.fn()) — an id plus a thunk so the workflow can cache and track steps.

API note: both run() and workflow use step(id, () => fn()). The id identifies the step for observability and caching; use step(id, () => deps.fn(), { key }) when you need an explicit cache key.

Step 12: Boundary Mapping with UnexpectedError

At your Lambda boundary, map the Result to HTTP responses:

import { createWorkflow, UNEXPECTED_ERROR } from 'awaitly/workflow';

type LambdaResponse =
  | { statusCode: 200; body: { message: string; task: Task } }
  | { statusCode: 404; body: { message: string; taskId: string } }
  | { statusCode: 500; body: { message: string } };

const errorHandler = createWorkflow({ loadTask, markErrored });

async function lambdaHandler(event: { taskId: string }): Promise<LambdaResponse> {
  const result = await errorHandler(async (step, deps) => {
    const task = await step('loadTask', () => deps.loadTask(event.taskId));
    return await step('markErrored', () => deps.markErrored(task._id));
  });

  if (result.ok) {
    return { statusCode: 200, body: { message: 'Success', task: result.value } };
  }

  // Error type is: TaskNotFound | DbFailed | UnexpectedError
  // TypeScript ensures we handle all cases (including UnexpectedError!)
  switch (result.error.type) {
    case 'TASK_NOT_FOUND':
      return { statusCode: 404, body: { message: 'Task not found', taskId: result.error.taskId } };
    case 'DB_FAILED':
      return { statusCode: 500, body: { message: `DB error: ${result.error.message}` } };
    case UNEXPECTED_ERROR:
      // UnexpectedError is included automatically by workflow
      return { statusCode: 500, body: { message: 'Internal server error' } };
  }
}

Key difference from run(): the workflow error type includes UnexpectedError automatically. You must handle UnexpectedError in your switch (TypeScript will remind you). Use the UNEXPECTED_ERROR constant to ensure you can’t drift from the library.

Step 13: Step Names and Observability

Add name/key options to steps for observability:

const workflow = createWorkflow(
  { loadTask },
  {
    onEvent: (event) => {
      const name = 'name' in event ? event.name : undefined;
      console.log(`${event.type}:${name ?? 'anonymous'}`);
    },
  }
);

const result = await workflow(async (step, deps) => {
  // Step id shows up in events; key enables caching
  return await step('loadTask', () => deps.loadTask('t-1'), {
    key: 'task:t-1',  // Same key = cached result
  });
});

// Console output:
// step_start:loadTask
// step_complete:loadTask

Step ids show up in logs/traces, keys enable caching (same key = cached result), and this is the foundation for durable execution (resume from last step).

Part 5: Full Example

The complete refactored handler using createWorkflow:

import { ok, err, tryAsync, type AsyncResult } from 'awaitly';
import { createWorkflow, UNEXPECTED_ERROR } from 'awaitly/workflow';

// ═══════════════════════════════════════════════════════════════════════════
// TYPES
// ═══════════════════════════════════════════════════════════════════════════

type Task = { _id: string; status: 'Pending' | 'Errored'; error?: { formattedLogs: string[] } };

type TaskNotFound = { type: 'TASK_NOT_FOUND'; taskId: string };
type DbFailed = { type: 'DB_FAILED'; message: string };
type SlackFailed = { type: 'SLACK_FAILED'; message: string };

type LambdaResponse =
  | { statusCode: 200; body: { message: string; task: Task } }
  | { statusCode: 404; body: { message: string; taskId: string } }
  | { statusCode: 500; body: { message: string } };

// ═══════════════════════════════════════════════════════════════════════════
// BUSINESS OPERATIONS (each returns AsyncResult)
// ═══════════════════════════════════════════════════════════════════════════

const loadTask = async (taskId: string): AsyncResult<Task, TaskNotFound> => {
  const task = taskStore.get(taskId);
  if (!task) return err({ type: 'TASK_NOT_FOUND', taskId });
  return ok(task);
};

const getLogs = async (taskId: string): AsyncResult<string[], DbFailed> => {
  return tryAsync(
    async () => logStore.get(taskId) ?? [],
    (e) => ({ type: 'DB_FAILED', message: e instanceof Error ? e.message : 'Unknown' })
  );
};

const markErrored = async (
  taskId: string,
  formattedLogs: string[]
): AsyncResult<Task, DbFailed> => {
  const task = taskStore.get(taskId);
  if (!task) return err({ type: 'DB_FAILED', message: 'Not found' });
  const updated: Task = { ...task, status: 'Errored', error: { formattedLogs } };
  taskStore.set(taskId, updated);
  return ok(updated);
};

const notifySlack = async (slack: SlackClient, task: Task): AsyncResult<void, SlackFailed> => {
  return tryAsync(
    async () => await slack.post({ title: `Task ${task._id} errored` }),
    (e) => ({ type: 'SLACK_FAILED', message: e instanceof Error ? e.message : 'Unknown' })
  );
};

// Pure function: no deps needed
const formatLogs = (logs: string[]) => logs.map((l) => `• ${l}`);

// ═══════════════════════════════════════════════════════════════════════════
// WORKFLOW (orchestrates operations)
// ═══════════════════════════════════════════════════════════════════════════

// Error type is INFERRED: TaskNotFound | DbFailed | SlackFailed | UnexpectedError
const errorHandler = createWorkflow({ loadTask, getLogs, markErrored, notifySlack });

// ═══════════════════════════════════════════════════════════════════════════
// LAMBDA HANDLER (boundary)
// ═══════════════════════════════════════════════════════════════════════════

async function lambdaHandler(
  event: { taskId: string },
  slack: SlackClient,
  env: { production: boolean }
): Promise<LambdaResponse> {
  const result = await errorHandler(async (step, deps) => {
    const task = await step('loadTask', () => deps.loadTask(event.taskId));
    const logs = await step('getLogs', () => deps.getLogs(task._id));
    const formattedLogs = formatLogs(logs);
    const updated = await step('markErrored', () => deps.markErrored(task._id, formattedLogs));

    if (env.production) {
      await step('notifySlack', () => deps.notifySlack(slack, updated));
    }

    return updated;
  });

  if (result.ok) {
    return { statusCode: 200, body: { message: 'Error handled successfully', task: result.value } };
  }

  // Handle all error types (including UnexpectedError from workflow)
  switch (result.error.type) {
    case 'TASK_NOT_FOUND':
      return { statusCode: 404, body: { message: 'Task not found', taskId: result.error.taskId } };
    case 'DB_FAILED':
    case 'SLACK_FAILED':
    case UNEXPECTED_ERROR:
      return { statusCode: 500, body: { message: 'Internal server error' } };
  }
}

Compare to original:

Before	After
try/catch everywhere	Typed Results throughout
deps created inline	deps injected at boundary
errors as strings	errors are data with context
manual error union maintenance	automatic inference with createWorkflow
happy path buried in if-checks	linear happy path with step()

Appendix: Convenience Helpers

These helpers reduce boilerplate but aren’t core to understanding Awaitly. Learn the core concepts first (ok/err/run/workflow), then use these for cleaner code.

tryAsync: Wrap Throwing Code

import { tryAsync } from 'awaitly';

type DbError = { type: 'DB_ERROR'; message: string };

// Manual try/catch
async function updateTaskManual(id: string): AsyncResult<{ id: string }, DbError> {
  try {
    const result = await unstableDb.update(id);
    return ok(result);
  } catch (e) {
    return err({ type: 'DB_ERROR', message: e instanceof Error ? e.message : 'Unknown' });
  }
}

// Using tryAsync: much cleaner
async function updateTaskClean(id: string): AsyncResult<{ id: string }, DbError> {
  return tryAsync(
    async () => await unstableDb.update(id),
    (e) => ({ type: 'DB_ERROR', message: e instanceof Error ? e.message : 'Unknown' })
  );
}

Use this when wrapping SDK calls (Slack, Stripe, database drivers) that throw exceptions.

fromNullable: Convert null to Typed Error

import { fromNullable } from 'awaitly';

type TaskNotFound = { type: 'TASK_NOT_FOUND'; taskId: string };

// Manual null check
async function getTaskManual(taskId: string): AsyncResult<Task, TaskNotFound> {
  const task = taskStore.get(taskId);
  if (!task) return err({ type: 'TASK_NOT_FOUND', taskId });
  return ok(task);
}

// Using fromNullable: one line
async function getTaskClean(taskId: string): AsyncResult<Task, TaskNotFound> {
  const task = taskStore.get(taskId);
  return fromNullable(task, () => ({ type: 'TASK_NOT_FOUND', taskId }));
}

Use this for database queries that return null for “not found”, or optional properties that should be required.

Errors<> and ErrorOf<>: Extract Error Types

import { type Errors, type ErrorOf } from 'awaitly';

// ErrorOf extracts the error type from a single function
type TaskNotFoundExtracted = ErrorOf<typeof loadTask>;
// Type: { type: 'TASK_NOT_FOUND'; taskId: string }

// Errors extracts and unions error types from multiple functions
type HandlerErrorExtracted = Errors<[typeof loadTask, typeof markErrored]>;
// Type: TaskNotFound | DbFailed

// Instead of manually writing:
type HandlerErrorManual = TaskNotFound | DbFailed;

Use this when building error unions from multiple functions, or reducing boilerplate in error type definitions.

Appendix: run() vs createWorkflow

When to prefer run():

You want explicit control over the error union
You’re teaching or learning the fundamentals
Simple cases with 1-2 steps

When to prefer createWorkflow:

Multiple steps with multiple error types
You want automatic error inference
You need observability (onEvent)
You want built-in UnexpectedError handling

// run(): manual error union
type HandlerError = TaskNotFound | DbFailed | Unexpected;
return run<Task, HandlerError>(
  async (step) => {
    const task = await step('loadTask', () => loadTask({ taskId }));
    return await step('markErrored', () => markErrored({ taskId: task._id }));
  },
  { catchUnexpected: (cause) => ({ type: 'UNEXPECTED', cause }) }
);

// createWorkflow: automatic error union
const workflow = createWorkflow({ loadTask, markErrored });
return workflow(async (step, deps) => {
  const task = await step('loadTask', () => deps.loadTask(taskId));
  return await step('markErrored', () => deps.markErrored(task._id));
});
// Error type is inferred: TaskNotFound | DbFailed | UnexpectedError

Try It

npm install awaitly

Start with ok() and err(). Move to run() when you have multiple steps. Graduate to createWorkflow when you want automatic error inference.

Your future self (and your on-call colleagues) will thank you.

Repo: github.com/jagreehal/awaitly

Instrumenting Message Queues? Autotel Handles the Ceremony

Sun, 18 Jan 2026 00:00:00 GMT

The OneUptime team is spot on in their Instrument Message Queues with OpenTelemetry post.

Inject trace context on the producer, extract on the consumer; use PRODUCER and CONSUMER span kinds; set semantic conventions (messaging.system, messaging.destination.name, messaging.operation, Kafka partition/offset/consumer group).

They show the raw OpenTelemetry code. It’s comprehensive. It’s also verbose. Every team ends up re-implementing the same patterns: inject, extract, span kinds, semantic attributes, error handling.

We’ve all been there: copying “best practice” code from blog posts and adapting it for our broker.

Their key insight:

For batch processing, use a batch span with links or child spans to contributing traces.

But there’s still a gap…

I built Autotel to close that gap.

While this demo uses Kafka, the same PRODUCER/CONSUMER span patterns apply to other brokers.

Autotel makes message queue instrumentation the default. PRODUCER/CONSUMER span kinds automatic. Semantic attributes automatic. Error handling with recordException() and setStatus(ERROR) in the wrappers. Batch spans with links via extractBatchLineage().

The Problem You’re Solving

The OneUptime article shows what correct instrumentation looks like:

// Manual approach: producer span with all the ceremony
import { trace, context, SpanKind, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('my-service');

async function publishMessage(topic: string, payload: any) {
  const span = tracer.startSpan(`${topic} publish`, {
    kind: SpanKind.PRODUCER,
    attributes: {
      'messaging.system': 'kafka',
      'messaging.destination.name': topic,
      'messaging.operation': 'publish',
    },
  });

  try {
    const headers: Record<string, string> = {};

    // Inject trace context into headers
    propagation.inject(context.active(), headers, {
      set: (carrier, key, value) => { carrier[key] = value; }
    });

    await producer.send({
      topic,
      messages: [{ value: JSON.stringify(payload), headers }],
    });
  } catch (error) {
    span.recordException(error as Error);
    span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });
    throw error;
  } finally {
    span.end();
  }
}

// Manual approach: consumer span with semantic conventions
async function consumeMessage(msg: KafkaMessage) {
  // Extract trace context from headers
  const parentContext = propagation.extract(context.active(), msg.headers, {
    get: (carrier, key) => carrier[key]?.toString()
  });

  const span = tracer.startSpan(`${topic} process`, {
    kind: SpanKind.CONSUMER,
    attributes: {
      'messaging.system': 'kafka',
      'messaging.destination.name': topic,
      'messaging.operation': 'process',
      'messaging.kafka.consumer.group': consumerGroup,
      'messaging.kafka.partition': msg.partition,
      'messaging.kafka.offset': msg.offset.toString(),
    },
  }, parentContext);

  try {
    await context.with(trace.setSpan(parentContext, span), async () => {
      await processMessage(msg);
    });
  } catch (error) {
    span.recordException(error as Error);
    span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });
    throw error;
  } finally {
    span.end();
  }
}

// Manual approach: batch span with child spans
async function processBatch(messages: KafkaMessage[]) {
  const batchSpan = tracer.startSpan('batch process', {
    kind: SpanKind.CONSUMER,
    attributes: {
      'messaging.batch.message_count': messages.length,
    },
  });

  try {
    for (const msg of messages) {
      const parentContext = propagation.extract(context.active(), msg.headers, {
        get: (carrier, key) => carrier[key]?.toString()
      });

      const childSpan = tracer.startSpan(`message process`, {
        kind: SpanKind.CONSUMER,
        links: [{ context: trace.getSpanContext(parentContext)! }],
      }, trace.setSpan(context.active(), batchSpan));

      try {
        await processMessage(msg);
      } finally {
        childSpan.end();
      }
    }
  } finally {
    batchSpan.end();
  }
}

It works, but it’s a lot of ceremony for something we all want to be the default.

The Solution: Thin Wrappers, Correct Defaults

With Autotel, the same instrumentation uses withProducerSpan() and withConsumerSpan():

// Producer: automatic PRODUCER span kind, semantic attributes
import { withProducerSpan, injectTraceHeaders } from "@demo/observability/kafka";

async function sendEvent(payload: Envelope, headers: Record<string, string>) {
  await withProducerSpan(
    "orders",
    { messageKey: payload.correlation_id },
    async () => {
      const h = injectTraceHeaders(headers);
      await producer.send({
        messages: [{
          topic: "orders",
          key: payload.correlation_id,
          value: JSON.stringify(payload),
          headers: h,
        }],
      });
    },
  );
}
// Span includes: messaging.system=kafka, messaging.destination.name=orders,
// messaging.operation=publish, SpanKind=PRODUCER
// Error handling: recordException() + setStatus(ERROR) on throw

// Consumer: automatic CONSUMER span kind, semantic attributes
import { withConsumerSpan, normalizeHeaders } from "@demo/observability/kafka";

for await (const msg of stream) {
  const headers = normalizeHeaders(msg.headers);
  const messageMeta = { partition: msg.partition, offset: String(msg.offset) };

  await withConsumerSpan(
    "orders",
    "email-worker",
    headers,
    messageMeta,
    async () => {
      await processMessage(msg);
    },
  );
}
// Span includes: messaging.system=kafka, messaging.destination.name=orders,
// messaging.operation=process, messaging.kafka.consumer.group=email-worker,
// messaging.kafka.partition, messaging.kafka.offset, SpanKind=CONSUMER
// Context extracted from headers automatically

Batch Spans with Links (No Child Span Creation)

For batch processing, extractBatchLineage() creates links without manual child spans:

import { extractBatchLineage, withProcessingSpan, SEMATTRS_LINKED_TRACE_ID_COUNT, SEMATTRS_LINKED_TRACE_ID_HASH } from "@demo/observability/kafka";

async function doFlush() {
  const items = batch;
  batch = [];

  // Extract lineage: trace links + summary hash
  const lineage = extractBatchLineage(
    items.map((it) => ({ headers: it.headers })),
    { maxLinks: 50 },
  );

  await withProcessingSpan(
    {
      name: "v1.settlements.batched process",
      headers: {},
      contextMode: "none",  // New trace for batch
      links: lineage.links,  // Links to contributing traces
      topic: "settlements",
      consumerGroup: "batcher",
    },
    async (span) => {
      // Summary attributes for querying
      span.setAttribute(SEMATTRS_LINKED_TRACE_ID_COUNT, lineage.linked_trace_id_count);
      span.setAttribute(SEMATTRS_LINKED_TRACE_ID_HASH, lineage.linked_trace_id_hash);
      await processBatch(items);
    },
  );
}

The batch span carries:

Links to contributing traces (capped by maxLinks to avoid payload explosion)
linked_trace_id_count: Number of traces that contributed
linked_trace_id_hash: Stable hash for “same set of inputs” queries

No manual child span creation. No looping through messages to create links.

Manual Instrumentation vs Autotel

Manual Instrumentation	Autotel
`tracer.startSpan()` with `SpanKind.PRODUCER`	`withProducerSpan()`
`tracer.startSpan()` with `SpanKind.CONSUMER`	`withConsumerSpan()`
Manual `propagation.inject()`	`injectTraceHeaders()`
Manual `propagation.extract()`	Automatic in `withConsumerSpan()`
Manual `span.setAttribute()` for each convention	Semantic conventions automatic
Try/catch with `recordException()` + `setStatus(ERROR)`	Error handling in wrapper
`span.end()` in finally block	Automatic on wrapper completion
Manual child spans for batch	`extractBatchLineage()` with links
Different code per broker	Same API (`withProducerSpan` / `withConsumerSpan`)

What the Wrappers Provide

The withProducerSpan() and withConsumerSpan() wrappers are thin layers over autotel-plugins:

// packages/observability/src/kafka.ts
import { withProducerSpan as pluginWithProducerSpan } from "autotel-plugins/kafka";

export async function withProducerSpan<T>(
  topic: string,
  options: ProducerSpanOptions | undefined,
  fn: () => Promise<T>,
): Promise<T> {
  return pluginWithProducerSpan(
    {
      name: `${topic} ${MESSAGING_OPERATION_PUBLISH}`,
      topic,
      messageKey: options?.messageKey,
    },
    async () => fn(),
  );
}

The plugin handles:

SpanKind.PRODUCER / SpanKind.CONSUMER
Semantic attributes: messaging.system, messaging.destination.name, messaging.operation, Kafka-specific (partition, offset, consumer.group)
Error handling: recordException(error) + setStatus({ code: SpanStatusCode.ERROR })
Context propagation: Inject on producer, extract on consumer

You write business logic. The wrapper handles observability ceremony. Forwarded retry or dead-letter messages preserve trace context and correlation_id so failures remain observable across hops.

Try It Yourself

Clone the demo and see automatic instrumentation:

git clone https://github.com/jagreehal/event-driven-observability-demo
cd event-driven-observability-demo
pnpm install
docker compose up -d  # Kafka, Jaeger
pnpm build && pnpm start

Trigger a checkout:

curl -sX POST http://localhost:3000/rpc/checkout/start \
  -H "content-type: application/json" \
  -d '{"cartValue":149.99,"itemCount":2,"userId":"usr_123","tenantId":"acme"}' | jq

Open Jaeger at http://localhost:16686. Find the trace. You’ll see:

PRODUCER spans with messaging.system=kafka, messaging.destination.name, messaging.operation=publish
CONSUMER spans with messaging.kafka.consumer.group, messaging.kafka.partition, messaging.kafka.offset
Batch spans with linked_trace_id_count and linked_trace_id_hash

All semantic conventions set. No manual attribute code in the application.

Evidence from a live demo run

Here’s actual output from a demo run (2026-02-07):

Jaeger trace (queried via Jaeger API):

{
  "traceID": "429a7e3408f6408f259348df534db2fc",
  "spanCount": 3,
  "services": ["api", "worker-email"],
  "spans": [
    {
      "operation": "checkout/start",
      "kind": "internal",
      "service": "api"
    },
    {
      "operation": "domain-events publish",
      "kind": "producer",
      "attributes": {
        "messaging.system": "kafka",
        "messaging.destination.name": "domain-events",
        "messaging.operation": "publish",
        "messaging.kafka.message.key": "429a7e3408f6408f"
      }
    },
    {
      "operation": "domain-events process",
      "kind": "consumer",
      "attributes": {
        "messaging.system": "kafka",
        "messaging.destination.name": "domain-events",
        "messaging.kafka.consumer.group": "demo-v1",
        "messaging.kafka.partition": 2,
        "messaging.kafka.offset": "0"
      }
    }
  ]
}

Same traceID across api and worker-email; PRODUCER and CONSUMER span kinds; all messaging semantic conventions set automatically by withProducerSpan / withConsumerSpan.

Repo: github.com/jagreehal/event-driven-observability-demo

Message Isolation? Autotel Makes Tenant Context Flow

Sat, 17 Jan 2026 00:00:00 GMT

The Signadot team is spot on in their Testing Event-Driven Architectures with OpenTelemetry post.

Message isolation using a shared queue: propagate tenant ID in Kafka message headers; consumers use tenant ID for selective message consumption.

They make the case that infrastructure duplication is expensive. Instead of separate Kafka clusters per environment, use tenant ID filtering on a shared queue. Instrument producers and consumers for context propagation.

We’ve all been there: maintaining four “identical” Kafka setups that slowly drift apart.

Their key insight:

Requires modifying consumers and using OpenTelemetry for context propagation.

But there’s still a gap…

I built Autotel to close that gap.

Autotel does not invent a new isolation mechanism; it standardizes how existing OpenTelemetry context propagation is applied consistently across services.

Autotel makes message isolation practical. One ID for selective consumption AND observability. No per-service boilerplate for context propagation.

The Problem You’re Solving

You want to test async workflows without duplicating infrastructure. The Signadot article describes the approach:

Tag messages with tenant ID at the producer
Propagate context through message headers
Consumers filter by tenant ID (selective consumption)
Only process messages for your tenant/environment

The manual implementation looks like this:

// Manual approach: producer adds tenant context
async function publish(topic: string, payload: any, tenantId: string) {
  const headers: Record<string, string> = {
    'x-tenant-id': tenantId,
  };

  // Must inject trace context manually
  propagation.inject(context.active(), headers, {
    set: (carrier, key, value) => { carrier[key] = value; }
  });

  await producer.send({
    topic,
    messages: [{ value: JSON.stringify(payload), headers }],
  });
}

// Consumer: extract and filter
async function consume(msg: KafkaMessage) {
  const headers = normalizeHeaders(msg.headers);
  const tenantId = headers['x-tenant-id'];

  // Filter: skip if not my tenant
  if (tenantId !== process.env.MY_TENANT) {
    return;
  }

  // Extract trace context manually
  const parentContext = propagation.extract(context.active(), headers, {
    get: (carrier, key) => carrier[key]?.toString()
  });

  await context.with(parentContext, async () => {
    await processMessage(msg);
  });
}

It works, but it’s a lot of ceremony for something we all want to be the default.

The Solution: Automatic Context Propagation

With Autotel, context propagation is automatic:

// Kafka consumer with context mode toggle
import { withConsumerSpan, normalizeHeaders, HEADER_CORRELATION_ID } from "@demo/observability/kafka";

for await (const msg of stream) {
  const headers = normalizeHeaders(msg.headers);
  const messageMeta = { partition: msg.partition, offset: String(msg.offset) };

  // Selective consumption: filter by correlation_id or tenant
  const correlationId = headers[HEADER_CORRELATION_ID];
  if (shouldFilter(correlationId)) {
    continue;  // Skip, not my tenant/request
  }

  if (process.env.DEMO_PROPAGATE_TRACE) {
    // Context mode: inherit trace from producer
    await withConsumerSpan(topic, consumerGroup, headers, messageMeta, async () => {
      await trace("worker-email.handle", async () => {
        await handleOrderCreated(msg);
      });
    });
  } else {
    // Context mode: none (new trace per message)
    await context.with(context.active(), async () => {
      await trace("worker-email.handle", async () => {
        await handleOrderCreated(msg);
      });
    });
  }
}

One ID for Two Purposes

The correlation_id serves dual purposes:

Selective consumption: “Should I process this message?”
Observability join key: “Show me everything for this request”

// At the API boundary
const correlationId = getOrCreateCorrelationId();

// In the message
const payload = {
  event: "v1.orders.created",
  correlation_id: correlationId,
  data: { order_id: "order-123" },
};

// In the headers
const headers = injectTraceHeaders({
  [HEADER_CORRELATION_ID]: correlationId,
  "x-event-name": "v1.orders.created",
});

Now consumers can filter. Selective consumption should still acknowledge or forward skipped messages according to your retry/DLQ strategy.

function shouldFilter(correlationId: string | undefined): boolean {
  if (!process.env.ISOLATION_TENANT) return false;

  // correlation_id can encode tenant directly, or reference it indirectly
  return !correlationId?.startsWith(process.env.ISOLATION_TENANT);
}

And observability tools can join:

-- All traces for this request
SELECT * FROM traces WHERE tags['correlation_id'] = 'tenant-123-checkout-abc';

-- All events for this request
SELECT * FROM events WHERE correlation_id = 'tenant-123-checkout-abc';

-- All logs for this request
SELECT * FROM logs WHERE correlation_id = 'tenant-123-checkout-abc';

Manual Message Isolation vs Autotel

Manual Message Isolation	Autotel
Manual `propagation.inject()`	`injectTraceHeaders()`
Manual `propagation.extract()`	Automatic in `withConsumerSpan()`
Separate tenant ID and trace context	`correlation_id` serves both
Per-service boilerplate	Thin wrappers over autotel-plugins
Easy to forget context in one service	Context flows automatically

Try It Yourself

Clone the demo and see message isolation in action:

git clone https://github.com/jagreehal/event-driven-observability-demo
cd event-driven-observability-demo
pnpm install
docker compose up -d  # Kafka, Jaeger
pnpm build && pnpm start

Trigger a checkout:

curl -sX POST http://localhost:3000/rpc/checkout/start \
  -H "content-type: application/json" \
  -d '{"cartValue":149.99,"itemCount":2,"userId":"usr_123","tenantId":"acme"}' | jq

Watch worker logs; you’ll see correlation_id flowing through every service. Open Jaeger at http://localhost:16686 and search by that ID — all traces, all services, one query.

Evidence from a live demo run

Here’s actual output from a demo run (2026-02-07):

API response:

{ "ok": true, "correlation_id": "429a7e3408f6408f", "published": "v1.orders.created" }

Jaeger trace (one trace across API and worker):

{
  "traceID": "429a7e3408f6408f259348df534db2fc",
  "services": ["api", "worker-email"],
  "spans": [
    { "operation": "checkout/start", "service": "api", "kind": "internal" },
    { "operation": "domain-events publish", "service": "api", "kind": "producer" },
    { "operation": "domain-events process", "service": "worker-email", "kind": "consumer" }
  ]
}

Event with correlation_id (from analytics sink):

{
  "name": "v1.checkouts.started",
  "autotel": {
    "correlation_id": "429a7e3408f6408f",
    "trace_id": "429a7e3408f6408f259348df534db2fc",
    "trace_url": "http://localhost:16686/trace/429a7e3408f6408f259348df534db2fc"
  }
}

Same correlation_id across API response, Kafka message key, and event payload. One ID for isolation and observability.

Repo: github.com/jagreehal/event-driven-observability-demo

Request-Level Isolation? Autotel Propagates Context Automatically

Fri, 16 Jan 2026 00:00:00 GMT

The CNCF team is spot on in their Testing Asynchronous Workflows using OpenTelemetry and Istio post.

Request-level isolation is the most cost-effective approach.

They make the case against duplicating infrastructure for testing. Instead of spinning up separate Kafka clusters per tenant, use OpenTelemetry Baggage to propagate tenant ID through async flows. Consumers filter by tenant ID. Istio handles routing.

We’ve all been there: every team has their own “staging Kafka” and costs balloon.

Their key insight:

Use OpenTelemetry Baggage to propagate tenant ID through sync and async. When publishing to Kafka, producers inject trace context (including baggage) into message headers; consumers extract and make routing decisions.

But there’s still a gap…

I built Autotel to close that gap.

Autotel makes context propagation the default. No manual inject/extract into message headers. No per-service boilerplate. No forgetting to include tenant ID in product events.

The Problem You’re Solving

You want request-level isolation for testing. The CNCF article describes the approach:

Set tenant ID in OpenTelemetry Baggage at the edge
Propagate baggage through HTTP (Istio handles it)
Inject baggage into Kafka message headers on publish
Extract baggage on consume, filter by tenant ID
Enrich product events with tenant context

The manual implementation:

// Manual approach: producer injects baggage into headers
import { propagation, context, baggage } from '@opentelemetry/api';

async function publishEvent(topic: string, payload: any) {
  const currentBaggage = propagation.getBaggage(context.active());
  const tenantId = currentBaggage?.getEntry('tenant.id')?.value;

  const headers: Record<string, string> = {};

  // Manual inject - easy to forget
  propagation.inject(context.active(), headers, {
    set: (carrier, key, value) => { carrier[key] = value; }
  });

  // Must remember to include baggage keys explicitly
  if (tenantId) {
    headers['x-tenant-id'] = tenantId;
  }

  await producer.send({
    topic,
    messages: [{ value: JSON.stringify(payload), headers }],
  });
}

// Consumer extracts and filters
async function consume(msg: KafkaMessage) {
  const parentContext = propagation.extract(context.active(), msg.headers, {
    get: (carrier, key) => carrier[key]?.toString()
  });

  const tenantId = msg.headers['x-tenant-id']?.toString();

  // Filter for selective consumption
  if (tenantId !== expectedTenantId) {
    return; // Skip messages for other tenants
  }

  await context.with(parentContext, async () => {
    await processMessage(msg);
  });
}

// And every track() call needs tenant context
track('order.created', {
  tenant_id: tenantId,  // Easy to forget
  order_id: order.id,
});

It works, but it’s a lot of ceremony for something we all want to be the default.

The Solution: Context Flows Automatically

With Autotel, injectTraceHeaders() propagates baggage automatically:

// Producer: API sends to Kafka
import { injectTraceHeaders, withProducerSpan } from "@demo/observability/kafka";

function buildHeaders(correlationId: string, eventName: string) {
  return injectTraceHeaders({
    "x-correlation-id": correlationId,
    "x-event-name": eventName,
  });
  // Returns: traceparent, tracestate, x-correlation-id, x-event-name
  // Baggage entries are propagated via tracestate
}

async function sendEvent(payload: Envelope, headers: Record<string, string>) {
  await withProducerSpan("orders", { messageKey: payload.correlation_id }, async () => {
    const h = injectTraceHeaders(headers);
    await producer.send({
      messages: [{
        topic: "orders",
        key: payload.correlation_id,
        value: JSON.stringify(payload),
        headers: h,
      }],
    });
  });
}

Autotel still supports OpenTelemetry Baggage; correlation_id is used as a stable, queryable join key that complements baggage rather than replacing it.

The correlation_id serves as the tenant/request identifier. Use it for:

Selective consumption: “Only process messages where correlation_id starts with tenant-123”
Observability: “Show me everything for this checkout”

Prefix-based filtering is one example; correlation_id can also reference tenant via lookup or structured encoding.

Enriching Product Events from Baggage

Configure enrichFromBaggage once at init:

import { init, track } from "autotel";

init({
  service: "api",
  events: {
    includeTraceContext: true,
    traceUrl: (ctx) => `http://localhost:16686/trace/${ctx.traceId}`,
    enrichFromBaggage: {
      allow: ["tenant.id", "user.id", "request.id"],
      deny: ["user.email", "user.ssn"],  // Never include PII
      prefix: "ctx.",
      transform: { "user.id": "hash", "tenant.id": "plain" },
    },
  },
});

// Now every track() call includes baggage context automatically
track("order.created", { order_id: order.id });
// Emitted event includes: ctx.tenant.id, ctx.user.id (hashed), ctx.request.id

No passing tenant ID into every track() call. No forgetting to include context. The event schema stays clean, and observability context is always present.

Manual Baggage Propagation vs Autotel

Manual Baggage Propagation	Autotel
`propagation.inject()` into headers	`injectTraceHeaders()`
`propagation.extract()` from headers	Automatic in `withConsumerSpan()`
Manually include baggage keys in headers	Baggage propagated via tracestate
Pass tenant ID to every `track()` call	`enrichFromBaggage` config
Different code for HTTP vs Kafka	Same pattern everywhere
Easy to forget context in one service	Context flows automatically

One ID for Isolation and Observability

The correlation_id is your stable join key:

// At the edge (API gateway or first service)
const correlationId = getOrCreateCorrelationId();  // From autotel

// In message payload and headers
const payload = {
  event: "v1.checkout.started",
  correlation_id: correlationId,  // Same ID everywhere
  data: { order_id: "order-123" },
};

const headers = injectTraceHeaders({
  "x-correlation-id": correlationId,
});

Now you can:

-- Find all traces for a request
SELECT * FROM traces WHERE correlation_id = 'abc-123';

-- Find all events for a request
SELECT * FROM events WHERE correlation_id = 'abc-123';

-- Find all logs for a request
SELECT * FROM logs WHERE correlation_id = 'abc-123';

And for testing with request-level isolation:

// Consumer filters by correlation_id prefix or tenant ID
const headers = normalizeHeaders(msg.headers);
const tenantId = headers["x-tenant-id"];

if (process.env.ISOLATION_MODE && tenantId !== process.env.MY_TENANT) {
  // Skipped messages should still be acknowledged or forwarded
  // according to your retry/DLQ strategy to avoid backlogs
  return;
}

await withConsumerSpan(topic, consumerGroup, headers, messageMeta, async () => {
  await processMessage(msg);
});

Try It Yourself

Clone the demo and see context propagation in action:

git clone https://github.com/jagreehal/event-driven-observability-demo
cd event-driven-observability-demo
pnpm install
docker compose up -d  # Kafka, Jaeger
pnpm build && pnpm start

Trigger a checkout:

curl -sX POST http://localhost:3000/rpc/checkout/start \
  -H "content-type: application/json" \
  -d '{"cartValue":149.99,"itemCount":2,"userId":"usr_123","tenantId":"acme"}' | jq

Check the worker logs; you’ll see correlation_id flowing through every service. Open Jaeger at http://localhost:16686 and search by correlation_id. All traces, all services, one query.

Evidence from a live demo run

Here’s actual output from a demo run (2026-02-07):

API response (stable correlation_id returned immediately):

{ "ok": true, "correlation_id": "429a7e3408f6408f", "published": "v1.orders.created" }

Event with autotel enrichment (from analytics sink at /events):

{
  "name": "v1.checkouts.started",
  "autotel": {
    "correlation_id": "429a7e3408f6408f",
    "trace_id": "429a7e3408f6408f259348df534db2fc",
    "span_id": "557a33e70d8d3589",
    "trace_flags": "01",
    "trace_url": "http://localhost:16686/trace/429a7e3408f6408f259348df534db2fc"
  }
}

Same correlation_id in the API response and in every event; trace_url links directly to the Jaeger trace. One ID joins traces, events, and logs.

Repo: github.com/jagreehal/event-driven-observability-demo

End-to-End Tracing? Autotel Makes It Automatic

Thu, 15 Jan 2026 00:00:00 GMT

The OSO team is spot on in their End-to-End Tracing in Event Driven Architectures post.

Traces break at queues unless you extract context from message headers and put it in the appropriate context.

They walk through the real pain: stateful processing loses trace context in caches, Kafka Connect can only do batch-level tracing, and every team ends up writing custom interceptors and state store wrappers.

We’ve all been there.

Their key insight:

In Kafka Streams and Kafka Connect this often means manual work: interceptors, state stores, batch spans, or extending tracing logic to extract from headers.

But there’s still a gap…

I built Autotel to close that gap.

Autotel does not change Kafka semantics or tracing theory; it standardizes how OpenTelemetry context propagation is applied across services.

Autotel makes end-to-end tracing the default. No custom interceptors. No state store wrappers. No per-service boilerplate for inject/extract.

The Problem You’re Solving

A checkout request flows through your event-driven system:

API → Kafka → payment-worker → Kafka → email-worker → Kafka → batcher

Without proper context propagation, you get:

Trace A (API)
Trace B (payment-worker)  ← new trace, no link to A
Trace C (email-worker)    ← new trace, no link to A or B
Trace D (batcher)         ← batch span, no idea which requests contributed

You cannot see the full path in Jaeger. You cannot query “everything for this checkout.” When the batcher fails, you have no idea which upstream requests were affected.

The manual fix? Every team writes:

// Manual approach: custom interceptor
class TracingProducerInterceptor implements ProducerInterceptor {
  onSend(record: ProducerRecord): ProducerRecord {
    const span = tracer.startSpan('kafka.send', { kind: SpanKind.PRODUCER });
    const headers = record.headers ?? {};
    propagator.inject(context.active(), headers, {
      set: (carrier, key, value) => { carrier[key] = value; }
    });
    record.headers = headers;
    span.setAttribute('messaging.system', 'kafka');
    span.setAttribute('messaging.destination.name', record.topic);
    span.end();
    return record;
  }
}

// Then a consumer interceptor...
class TracingConsumerInterceptor implements ConsumerInterceptor {
  onConsume(record: ConsumerRecord): ConsumerRecord {
    const parentContext = propagator.extract(context.active(), record.headers, {
      get: (carrier, key) => carrier[key]?.toString()
    });
    return context.with(parentContext, () => {
      const span = tracer.startSpan('kafka.process', { kind: SpanKind.CONSUMER });
      span.setAttribute('messaging.system', 'kafka');
      span.setAttribute('messaging.kafka.partition', record.partition);
      span.setAttribute('messaging.kafka.offset', record.offset);
      // ... more attributes
      return record;
    });
  }
}

// And for batches, custom state stores, batch spans with links...

It works, but it’s a lot of ceremony for something we all want to be the default.

The Solution: Zero Per-Service Wiring

With Autotel, the same flow uses withProducerSpan() and withConsumerSpan():

// Producer: API sends to Kafka
import { withProducerSpan, injectTraceHeaders } from "@demo/observability/kafka";

async function sendEvent(payload: Envelope, headers: Record<string, string>) {
  await withProducerSpan(
    "orders",
    { messageKey: payload.correlation_id },
    async () => {
      const h = injectTraceHeaders(headers);
      await producer.send({
        messages: [{
          topic: "orders",
          key: payload.correlation_id,
          value: JSON.stringify(payload),
          headers: h,  // traceparent, tracestate, x-correlation-id
        }],
      });
    },
  );
}

// Consumer: Worker processes from Kafka
import { withConsumerSpan, normalizeHeaders } from "@demo/observability/kafka";

for await (const msg of stream) {
  const headers = normalizeHeaders(msg.headers);
  const messageMeta = { partition: msg.partition, offset: String(msg.offset) };

  await withConsumerSpan(
    "orders",
    "email-worker",
    headers,
    messageMeta,
    async () => {
      // Your business logic here
      // Context is already extracted - you're in the same trace
      await handleOrderCreated(msg);
    },
  );
}

No custom interceptors. No propagator.inject() / propagator.extract(). No manual span attribute setting.

Batch Lineage for Fan-In

When multiple messages feed into one batch, Autotel’s extractBatchLineage() creates a batch span with links:

import { extractBatchLineage, withProcessingSpan, SEMATTRS_LINKED_TRACE_ID_COUNT, SEMATTRS_LINKED_TRACE_ID_HASH } from "@demo/observability/kafka";

async function doFlush() {
  const items = batch;
  batch = [];

  // Extract lineage from all contributing messages
  const lineage = extractBatchLineage(
    items.map((it) => ({ headers: it.headers })),
    { maxLinks: 50 },
  );

  await withProcessingSpan(
    {
      name: "v1.settlements.batched process",
      headers: {},
      contextMode: "none",
      links: lineage.links,  // Links to contributing traces
      topic: "settlements",
      consumerGroup: "batcher",
    },
    async (span) => {
      span.setAttribute(SEMATTRS_LINKED_TRACE_ID_COUNT, lineage.linked_trace_id_count);
      span.setAttribute(SEMATTRS_LINKED_TRACE_ID_HASH, lineage.linked_trace_id_hash);
      await processBatch(items);
    },
  );
}

The batch span carries:

linked_trace_id_count: How many traces contributed
linked_trace_id_hash: Stable hash for querying “same set of inputs”
links: Actual trace links to contributing spans (capped to avoid payload explosion)

Manual Interceptors vs Autotel

Manual Interceptors	Autotel
Custom ProducerInterceptor class	`withProducerSpan()` wrapper
Custom ConsumerInterceptor class	`withConsumerSpan()` wrapper
Manual `propagator.inject()`	`injectTraceHeaders()`
Manual `propagator.extract()`	Automatic in `withConsumerSpan()`
Per-attribute `span.setAttribute()`	Semantic conventions automatic
Custom state store wrappers for stateful processing	Same pattern, context flows through stateful processing
Manual batch spans with child spans	`extractBatchLineage()` with links
Different code per broker	Same API (`withProducerSpan` / `withConsumerSpan`)

One Trace, API to Batcher

With Autotel, the same checkout request shows as one trace in Jaeger:

Trace: 4bf92f3577b34da6a3ce929d0e0e4736
├─ api: POST /checkout (INTERNAL)
│  └─ orders publish (PRODUCER)
├─ orders process (CONSUMER) - payment-worker
│  └─ payments publish (PRODUCER)
├─ payments process (CONSUMER) - email-worker
│  └─ notifications publish (PRODUCER)
└─ v1.settlements.batched process (CONSUMER) - batcher
   └─ linked_trace_id_count: 5
   └─ linked_trace_id_hash: a3f2c1...

Query correlation_id to find all related logs, events, and traces. Query linked_trace_id_hash to find all batches with the same contributing requests. Retries and dead-letter forwarding preserve correlation_id and trace context so failures remain observable across hops.

Try It Yourself

Clone the demo and see end-to-end traces in Jaeger:

git clone https://github.com/jagreehal/event-driven-observability-demo
cd event-driven-observability-demo
pnpm install
docker compose up -d  # Kafka, Jaeger
pnpm build && pnpm start

Trigger a checkout:

curl -sX POST http://localhost:3000/rpc/checkout/start \
  -H "content-type: application/json" \
  -d '{"cartValue":149.99,"itemCount":2,"userId":"usr_123","tenantId":"acme"}' | jq

Open Jaeger at http://localhost:16686. Search for the trace. You’ll see one connected trace from API through workers, including batch spans with lineage.

Evidence from a live demo run

Here’s actual output from a demo run (2026-02-07):

Jaeger trace (queried via Jaeger API):

{
  "traceID": "429a7e3408f6408f259348df534db2fc",
  "spanCount": 3,
  "services": ["api", "worker-email"],
  "spans": [
    {
      "operation": "checkout/start",
      "kind": "internal",
      "service": "api"
    },
    {
      "operation": "domain-events publish",
      "kind": "producer",
      "attributes": {
        "messaging.system": "kafka",
        "messaging.destination.name": "domain-events",
        "messaging.operation": "publish",
        "messaging.kafka.message.key": "429a7e3408f6408f"
      }
    },
    {
      "operation": "domain-events process",
      "kind": "consumer",
      "attributes": {
        "messaging.system": "kafka",
        "messaging.destination.name": "domain-events",
        "messaging.kafka.consumer.group": "demo-v1",
        "messaging.kafka.partition": 2,
        "messaging.kafka.offset": "0"
      }
    }
  ]
}

One trace (429a7e3408f6408f...) spans api and worker-email. The CONSUMER span’s parent is the PRODUCER span — no custom interceptors, no state store wrappers, just withProducerSpan and withConsumerSpan.

Repo: github.com/jagreehal/event-driven-observability-demo

Logging Sucks. Autotel Makes Wide Events the Default

Thu, 08 Jan 2026 00:00:00 GMT

Boris is spot on in his Logging Sucks post

logs are optimised for writing, not querying

He explains why debugging in production feels like archaeology.

You grep for user-123, find it logged 47 different ways, then spend an hour correlating timestamps across services.

We’ve all been there.

His wide event example nails it:

{
  "user": {"id": "user_456", "subscription": "premium", "lifetime_value_cents": 284700},
  "cart": {"item_count": 3, "total_cents": 15999, "coupon_applied": "SAVE20"},
  "payment": {"method": "card", "provider": "stripe", "latency_ms": 1089},
  "error": {"type": "PaymentError", "code": "card_declined", "stripe_decline_code": "insufficient_funds"}
}

One event. High-cardinality keys (user.id, traceId). High dimensionality. Queryable.

But there’s still a gap…

I built Autotel to close that gap.

Autotel makes wide events the default output of traces, producing one canonical structured log per request, chosen via tail sampling.

By default, Autotel emits a canonical log line for the root/server span (and optionally for spans that match custom filters such as status=ERROR or duration_ms > 2000), with tail sampling applied automatically.

The middleware-based approach still requires wiring, manual context accumulation, and custom sampling logic.

It works, but it’s a lot of ceremony for something we all want to be the default.

The Problem You’re Solving

A single checkout produces scattered logs:

2024-12-20T10:23:45.100Z INFO  Checkout started userId=user-123
2024-12-20T10:23:45.150Z DEBUG Cart loaded cartId=cart-1 itemCount=2
2024-12-20T10:23:45.300Z INFO  Payment processing method=card provider=stripe
2024-12-20T10:23:45.450Z DEBUG Validating payment details
2024-12-20T10:23:45.612Z ERROR Payment failed error=card_declined userId=user-123

To debug, you grep for userId, then cartId, then correlate timestamps across services.

One Wide Event. Zero Per-Endpoint Wiring

With Autotel, the same checkout produces a single canonical log line (a wide event). No per-endpoint middleware wiring, no finally blocks, no string logs:

{
  "level": 50, // Pino numeric level (50=error, 30=info)
  "name": "checkout-api",
  "msg": "[HTTP POST /api/checkout] Request completed",
  "http.request.method": "POST",
  "http.route": "/api/checkout",
  "http.response.status_code": 500,
  "user.id": "user-123",
  "user.subscription": "premium",
  "user.account_age_days": 847,
  "user.lifetime_value_cents": 284700,
  "request.id": "req_123",
  "cart.id": "cart-1",
  "cart.item_count": 2,
  "cart.total_cents": 7998,
  "cart.coupon_applied": "SAVE20",
  "payment.method": "card",
  "payment.provider": "stripe",
  "payment.latency_ms": 150,
  "payment.attempt": 1,
  "error.type": "PaymentError",
  "error.code": "card_declined",
  "error.message": "Card declined by issuer",
  "error.stripe_decline_code": "insufficient_funds",
  "service.name": "checkout-api",
  "service.version": "1.0.0",
  "deployment.environment": "development",
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "00f067aa0ba902b7",
  "correlationId": "4bf92f3577b34da6",
  "duration_ms": 512.3,
  "status_code": 2, // OpenTelemetry StatusCode (1=OK, 2=ERROR)
  "status_message": "Payment failed: card_declined",
  "timestamp": "2024-12-20T10:23:45.612Z"
}

Examples are JSONC for readability. The status_code field uses OpenTelemetry’s StatusCode enum (OK=1, ERROR=2); for errors, status_message provides a human-readable description.

Use attributeRedactor in production to redact PII.

One event with 25+ fields, emitted automatically when the root span completes.

All attributes use dot-notation (e.g., user.id) rather than nested objects. Most log stores flatten nested objects into dotted keys anyway.

Autotel emits that shape directly.

Which spans become wide events?

By default, Autotel emits canonical logs for root/server spans (typically your HTTP request handlers).

You can configure additional spans via filters (e.g., by span.kind, minimum duration, or status).

This avoids a log volume explosion from every nested span.

Limits & safety rails

OpenTelemetry SDKs enforce attribute limits (defaults vary by SDK and are configurable). Autotel respects SDK limits and truncates when exceeded. For large payloads, store IDs and pointers rather than full objects. Use attributeRedactor to redact PII before emission.

Be intentional about which high-cardinality fields you index vs store: index a small set, store the rest.

Manual Wide-Event Assembly vs Autotel

Manual middleware-based approach builds the event manually:

// Manual approach: middleware + manual accumulation
export function wideEventMiddleware() {
  return async (ctx, next) => {
    const startTime = Date.now();
    const event = { 'request.id': ctx.get('requestId') };
    ctx.set('wideEvent', event);

    try {
      await next();
    } finally {
      event.duration_ms = Date.now() - startTime;
      logger.info(event);
    }
  };
}

// Then in every handler:
app.post('/checkout', async (ctx) => {
  const event = ctx.get('wideEvent');
  // Manual approach often ends up with nested blobs that get flattened inconsistently
  event.user = { id: user.id, subscription: user.subscription };
  event.cart = { id: cart.id, item_count: cart.items.length };
  // ...more manual accumulation
});

With Autotel, context accumulates naturally:

import { trace, setUser, httpServer } from 'autotel';

export const processCheckout = trace((ctx) => async (req: CheckoutRequest) => {
  const user = await getUser(req.userId);
  
  // Standard user attributes via setUser
  setUser(ctx, {
    id: user.id,
    // email: user.email, // Redact PII in production
  });

  // Custom user attributes via setAttributes
  ctx.setAttributes({
    'user.subscription': user.subscription,
    'user.account_age_days': user.accountAgeDays,
    'user.lifetime_value_cents': user.lifetimeValueCents,
  });

  httpServer(ctx, { method: 'POST', route: '/api/checkout' });

  const total = req.items.reduce((sum, item) => sum + item.price, 0);
  ctx.setAttributes({
    'cart.id': req.cartId,
    'cart.item_count': req.items.length,
    'cart.total_cents': Math.round(total * 100),
    'cart.coupon_applied': req.coupon,
  });

  const payment = await processPayment(req);

  ctx.setAttributes({
    'payment.method': payment.method,
    'payment.provider': payment.provider,
    'payment.latency_ms': payment.latencyMs,
  });

  if (payment.error) {
    ctx.setAttributes({
      'error.type': 'PaymentError',
      'error.code': payment.error.code,
      'error.can_retry': payment.error.canRetry,
    });
    throw new PaymentError(payment.error.message);
  }

  return { orderId: payment.orderId };
});
// On span end, Autotel emits one structured log line via the configured exporter

No per-endpoint middleware. No ctx.get(‘wideEvent’). No finally block. (You still need init() setup and exporter configuration, but that’s one-time, not per-endpoint.)

By default, this can go to stdout (JSON) or to an OTLP/HTTP exporter, depending on your setup.

Manual Wide-Event Assembly	Autotel Approach
Build a separate event object	Use the span as the event source
Middleware to create/attach context	trace() creates context per function
Mutate ctx.get(‘wideEvent’)	ctx.setAttributes() on the span
Emit in finally block	Emit on span completion
Custom shouldSample() function	Built-in tail sampler

On “OTel Won’t Save You”

He’s right.

Most OTel setups capture span name, duration, and maybe recordException(error). That’s baseline instrumentation: useful, but it won’t give you queryable context.

Autotel’s trace() wrapper makes rich instrumentation the path of least resistance.

You’re already inside the traced function, so adding ctx.setAttributes() is natural. Context accumulates as you process, not in a separate middleware layer.

Tail Sampling Without the Logic

His sampling function:

function shouldSample(event) {
  if (event.status_code >= 500) return true;
  if (event.duration_ms > 2000) return true;
  if (event.user?.subscription === 'enterprise') return true;
  return Math.random() < 0.05;
}

With Autotel:

import { init, AdaptiveSampler } from 'autotel';

init({
  service: 'checkout-api',
  sampler: new AdaptiveSampler({
    baselineSampleRate: 0.05,
    alwaysSampleErrors: true,
    alwaysSampleSlow: true,
    slowThresholdMs: 2000,
  }),
});

Same logic, zero custom code. The sampler evaluates after the span completes (tail sampling), so it sees duration, status, and all attributes before deciding.

Autotel’s sampler decides whether to emit the canonical log line for each root span; it’s not trying to guarantee whole-trace retention.

Try It Yourself

If you want to see the shape firsthand, here’s a 60-second demo.

Quickstart: clone the repo and run the demo:

git clone https://github.com/jagreehal/autotel
cd autotel/apps/example-canonical-logs
pnpm install && pnpm start:server
# In another terminal: ./demo.sh

The demo accepts request_id in the request body, then records it on the span as request.id so the emitted logs stay dot-notated.

Request IDs:

Inputs: request_id (ergonomic)
Output: request.id (normalised)
If header exists: also store http.request.header.x-request-id

The Queries You Want

SELECT * FROM logs
WHERE user.subscription = 'premium'
  AND error.type IS NOT NULL
  AND feature_flags.new_checkout_flow = true;

Failed payments by decline code:

SELECT error.code, COUNT(*) as count
FROM logs
WHERE error.type = 'PaymentError'
GROUP BY error.code;

Slow checkouts for high-value customers:

SELECT * FROM logs
WHERE duration_ms > 500
  AND user.lifetime_value_cents > 100000;

Each curl call adds context to the span. Watch the server console on the final step, one canonical log line with everything:

{
  "level": 30,
  "name": "checkout-api",
  "http.request.method": "POST",
  "http.route": "/api/checkout",
  "http.response.status_code": 200,
  "user.id": "user_456",
  "user.subscription": "premium",
  "user.account_age_days": 847,
  "user.lifetime_value_cents": 284700,
  "request.id": "req_test123",
  "cart.id": "cart_xyz",
  "cart.item_count": 3,
  "cart.total_cents": 15999,
  "cart.coupon_applied": "SAVE20",
  "payment.method": "card",
  "payment.provider": "stripe",
  "payment.latency_ms": 241,
  "payment.attempt": 1,
  "service.name": "checkout-api",
  "service.version": "1.0.0",
  "deployment.environment": "development",
  "traceId": "24e8e520137c5cdcdd59ffd5f653d870",
  "spanId": "ed53a2d659907201",
  "correlationId": "24e8e520137c5cdc",
  "duration_ms": 1.04,
  "status_code": 1,
  "timestamp": "2026-01-08T00:26:08.836Z",
  "msg": "[HTTP POST /api/checkout] Request completed"
}

The demo supports manual curl commands too:

Step 1: Initialize request

curl -X POST http://localhost:3000/checkout/start \
  -H "Content-Type: application/json" \
  -d '{"request_id": "req_123"}'

Step 2: Add user context

curl -X POST http://localhost:3000/checkout/auth \
  -H "Content-Type: application/json" \
  -d '{"request_id": "req_123", "user_id": "user_456"}'

Step 3: Add cart context

curl -X POST http://localhost:3000/checkout/cart \
  -H "Content-Type: application/json" \
  -d '{"request_id": "req_123", "cart_id": "cart_xyz"}'

Step 4: Process payment

curl -X POST http://localhost:3000/checkout/payment \
  -H "Content-Type: application/json" \
  -d '{"request_id": "req_123"}'

Step 5-6: Complete + emit canonical log

curl -X POST http://localhost:3000/checkout/complete \
  -H "Content-Type: application/json" \
  -d '{"request_id": "req_123"}'

Repo: github.com/jagreehal/autotel

Related: OpenTelemetry Is Not the Three Pillars (draft) — siloed stores versus one wide event, with a runnable A/B in apps/example-pillars-vs-unified.

Reranking: Improving Search Relevance with the AI SDK

Tue, 23 Dec 2025 00:00:00 GMT

Reranking improves search relevance by reordering documents based on their relevance to a query. Unlike embedding-based similarity search, reranking models are specifically trained to understand the relationship between queries and documents, often producing more accurate relevance scores.

The Problem: Vector Search vs Query Relevance

Vector search finds semantically similar documents but may miss the most relevant one when the query uses different vocabulary. For example, a query like “How do I get a refund?” might match documents containing “billing” or “payment” (similar financial vocabulary) rather than the document that actually answers the question.

Two-Stage Retrieval Pipeline

The recommended approach combines fast vector search with precise reranking:

┌───────────────────────────────────────────────────────────────┐
│                    All Documents (1000s)                      │
└──────────────────────────┬────────────────────────────────────┘
                           │
                           ▼
              ┌───────────────────────┐
              │   Vector Search       │
              │   (Fast, Broad)       │
              └──────────┬────────────┘
                         │
                         ▼
              ┌─────────────────────────┐
              │   Top 20-100 Candidates │
              └──────────┬──────────────┘
                         │
                         ▼
              ┌────────────────────────┐
              │   Reranking            │
              │   (Precise, Slower)    │
              └──────────┬─────────────┘
                         │
                         ▼
              ┌─────────────────────────┐
              │   Top 3-10 Results      │
              └─────────────────────────┘

Broad retrieval: Vector search returns top 20-100 candidates quickly (tune for latency/recall)
Precise reranking: Re-score candidates for query relevance using the rerank function

Example: Customer Support Knowledge Base

Using a knowledge base with 6 articles, we compare vector search with reranking.

For clarity, examples below rerank small document sets directly. In production, rerank only the top candidates from vector search (see Two-Stage Retrieval Pipeline).

Vector Search Results

import { embed, cosineSimilarity } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const query = 'How do I get a refund?';
const queryEmbedding = await embed({
  model: ollama.textEmbeddingModel('nomic-embed-text'),
  value: query,
});

// Compare with pre-embedded articles
const scores = articles.map((article, i) => ({
  article,
  score: cosineSimilarity(queryEmbedding.embedding, articleEmbeddings[i].embedding),
}));

const topResults = scores.sort((a, b) => b.score - a.score).slice(0, 3);

Results:

Rank	Article	Score
1	Billing and Payment Methods	0.6073
2	Account Plan Changes	0.5862	✓ correct
3	Exporting Your Data	0.5797

Vector search ranks “Billing and Payment Methods” #1 because it shares financial vocabulary, but it doesn’t answer the refund question.

After Reranking

Using the rerank function:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const { ranking, rerankedDocuments } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: supportArticles.map(a => ({
    id: a.id,
    title: a.title,
    content: a.content,
  })),
  query: 'How do I get a refund?',
  topN: 3,
});

Results:

Rank	Article	Score	Original Index
1	Account Plan Changes (KB003)	0.5376	2
2	How to Reset Your Password (KB001)	0.2920	0
3	Billing and Payment Methods (KB002)	0.2815	1

Reranking correctly identifies KB003 (which contains “Refunds are available within 14 days”) as the most relevant result.

Comparison: Vector Search vs Reranking

Testing 7 queries designed to expose vocabulary mismatch issues:

Query	Vector Search #1	After Rerank #1	Result
“How do I get a refund?”	Billing and Payment Methods	Account Plan Changes ✓	IMPROVED
“money back”	Billing and Payment Methods	Account Plan Changes ✓	IMPROVED
“account protection options”	Two-Factor Authentication Setup ✓	Two-Factor Authentication Setup ✓	Same
“stop my account”	Account Plan Changes ✓	Account Plan Changes ✓	Same
“download all my info”	Exporting Your Data ✓	Exporting Your Data ✓	Same
“forgot my login”	How to Reset Your Password ✓	How to Reset Your Password ✓	Same
“how much can I call the API”	API Rate Limits ✓	API Rate Limits ✓	Same

Results:

Vector Search accuracy: 5/7 (71%)
After Reranking: 7/7 (100%)
Queries improved: 2 (both vocabulary mismatch cases)

Working with Object Documents

Reranking supports structured documents (JSON objects), making it ideal for searching through databases, emails, or other structured content:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const emails = [
  { from: 'aws-billing@amazon.com', subject: 'Your AWS Invoice', date: '2024-12-15' },
  { from: 'team@linear.app', subject: 'Weekly Project Update', date: '2024-12-14' },
  { from: 'noreply@github.com', subject: 'Security alert: new sign-in', date: '2024-12-14' },
  { from: 'sales@datadog.com', subject: 'Your Datadog Quote - Enterprise Plan', date: '2024-12-13' },
  { from: 'no-reply@vercel.com', subject: 'Deployment failed: main branch', date: '2024-12-13' },
  { from: 'hr@company.com', subject: 'Holiday Schedule Reminder', date: '2024-12-12' },
];

const { ranking, rerankedDocuments } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: emails,
  query: 'Find the pricing quote from Datadog',
  topN: 3,
});

Results:

Rank	From	Subject	Score
1	sales@datadog.com	Your Datadog Quote - Enterprise Plan	0.6830
2	aws-billing@amazon.com	Your AWS Invoice	0.3238
3	hr@company.com	Holiday Schedule Reminder	0.1946

Use Case: RAG Context Selection

For RAG applications, reranking selects the most relevant documents to include in the LLM context:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const docs = [
  'The createClient() function initializes a new API client...',
  'Authentication uses Bearer tokens...',
  'Rate limiting returns HTTP 429 when exceeded...',
  'Webhooks are sent via POST to your configured endpoint...',
  'Error responses follow RFC 7807 Problem Details format...',
  'Pagination uses cursor-based navigation...',
];

const { rerankedDocuments, ranking } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: docs,
  query: 'How do I handle API errors?',
  topN: 2,
});

// Use rerankedDocuments for LLM context
const context = rerankedDocuments.join('\n\n');

Results:

Rank	Document	Score
1	Error responses follow RFC 7807 Problem Details format…	0.4662
2	Rate limiting returns HTTP 429 when exceeded…	0.4447

These top documents would be passed to the LLM for answer generation.

Understanding the Results

The rerank function returns a comprehensive result object:

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const { ranking, rerankedDocuments, originalDocuments } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents: supportArticles,
  query: 'How do I get a refund?',
  topN: 3,
});

Return value properties:

Property	Description
`ranking`	Sorted array of `{ originalIndex, score, document }`
`rerankedDocuments`	Documents sorted by relevance (convenience property)
`originalDocuments`	Original documents array (unchanged)

Each item in the ranking array contains:

originalIndex: Position in the original documents array
score: Relevance score (higher is more relevant)
document: The original document

Note: Scores are not comparable across models or providers; only relative ordering within a single rerank call matters.

Filtering by Score Threshold

In production, you may want to discard results with low relevance scores. Filter the ranking array to only include results above a threshold:

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents,
  query,
  topN: 10,
});

// Filter to only include results with score > 0.4
const relevantResults = ranking.filter(item => item.score > 0.4);
// Returns only documents with relevance score above threshold

Reranking Providers & Models

The AI SDK supports multiple reranking providers. You can use reranking models from:

Cohere

import { rerank } from 'ai';
import { cohere } from '@ai-sdk/cohere';

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents,
  query,
  topN: 3,
});

Amazon Bedrock

import { rerank } from 'ai';
import { bedrock } from '@ai-sdk/amazon-bedrock';

const { ranking } = await rerank({
  model: bedrock.reranking('cohere.rerank-v3-5:0'),
  documents,
  query,
  topN: 3,
});

Ollama (Local)

import { rerank } from 'ai';
import { ollama } from 'ai-sdk-ollama';

const { ranking } = await rerank({
  model: ollama.embeddingReranking('embeddinggemma'),
  documents,
  query,
  topN: 3,
});

Note: Some reranking models (e.g., Cohere rerankers) are cross-encoders that jointly score query–document pairs, while others (e.g., embedding-based rerankers) trade some accuracy for speed and locality. Cross-encoders typically provide higher accuracy but require API calls, while embedding-based rerankers can run locally with lower latency.

Settings

Top-N Results

Use topN to limit the number of results returned. This is useful for retrieving only the most relevant documents:

import { rerank } from 'ai';
import { cohere } from '@ai-sdk/cohere';

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents: ['doc1', 'doc2', 'doc3', 'doc4', 'doc5'],
  query: 'relevant information',
  topN: 3, // Return only top 3 most relevant documents
});

Provider Options

Reranking model settings can be configured using providerOptions for provider-specific parameters. This is particularly useful for handling long documents that might otherwise be truncated:

import { rerank } from 'ai';
import { cohere } from '@ai-sdk/cohere';

const { ranking } = await rerank({
  model: cohere.reranking('rerank-v3.5'),
  documents: ['long document 1...', 'long document 2...'],
  query: 'relevant information',
  providerOptions: {
    cohere: {
      maxTokensPerDoc: 1000, // Limit tokens per document
    },
  },
});

Examples (provider-specific; check provider docs for available options):

Cohere: maxTokensPerDoc - Maximum tokens per document (default varies by model)
Other providers may expose additional options via providerOptions

Best Practices

Use vector search for broad retrieval first: Retrieve top 20-100 candidates before reranking (tune for latency/recall trade-offs). In practice, reranking typically adds tens of milliseconds for 20–50 documents, depending on model and provider.
Tune topN based on use case:
- RAG: topN: 3 (small context window)
- Search results: topN: 10
- Auto-reply selection: topN: 1
Monitor performance: Track click-through rates, RAG answer accuracy, and queries requiring refinement.

Summary

Metric	Vector Search	With Reranking
Accuracy (top-1)	71% (5/7)	100% (7/7)
Vocabulary mismatch handling	Poor	Excellent
Speed	Fast	+tens of milliseconds for 20–50 docs
Best for	Broad retrieval	Final ranking

Reranking improves search relevance by reordering documents based on query-document relationships rather than just semantic similarity. For search systems where precision matters customer support, RAG, email, documentation, reranking improves result quality with minimal code changes.

See the ai-sdk-reranking-example repository for runnable code.

AI Code Needs Rules, Not Rituals (The Proof)

Thu, 18 Dec 2025 00:00:00 GMT

I wrote previously about why AI-generated code needs rules, not rituals.

AI coding agents aren’t going anywhere. They’re excellent at exploring ideas, generating boilerplate, and moving fast. But speed without reliability just ships bugs faster. And without constraints, AI-generated code is unreliable by default.

Prompting is a ritual. Linting is a rule.

Here is the proof.

Why Prompts Don’t Cut It

Most teams try to control AI output with instructions:

CLAUDE.md
AGENTS.md
Cursor rules
Commands / Sub-agents / Skills
Long prompt templates
Carefully curated examples

All of this helps. None of it guarantees anything.

Prompting is probabilistic. You’re hoping the model remembers your preferences and applies them consistently across contexts. Sometimes it does. Sometimes it doesn’t. Often it lands in the worst possible place: almost correct.

Almost correct is how bugs slip through review.

A Real Example

My preference in JavaScript is that functions should take a single object parameter rather than positional arguments.

I’ve explained this preference to AI tools. I’ve shown examples. I’ve reinforced it repeatedly.

And I still get this:

// ❌ Generated anyway
function calculateDiscount(price, taxRate, couponValue) {
  // …
}

Sometimes the model listens. Sometimes it doesn’t. That inconsistency alone disqualifies prompting as enforcement.

What Rules Actually Do

Encode the preference as a lint rule. I built eslint-plugin-prefer-object-params for exactly this. Now the bad version can’t survive:

error  Prefer object params over positional arguments  prefer-object-params

No retries. No “please remember next time.” The rule fires, the code fails, and the agent fixes it.

That’s enforcement.

“But What If No Rule Exists?”

Write one. Or have the agent write one.

That’s exactly what I did with the plugin above.

Now, AI-generated code hits the same walls as human-written code. Every generation has to pass. The output becomes deterministic, reviewable, and aligned with how I actually want the code to look.

What This Catches

Before:

// ❌ Easy to miss in review
function sendEmail(to, subject, body, cc, bcc) {
  // …
}

After:

✖ prefer-object-params: Use a single options object instead of positional arguments

AI slop hits a brick wall.

Beyond Style

These examples are structural because they’re easy to show: parameters, signatures, conventions. But the principle scales.

Some rules live in linters. Some live in type systems. Some live in tests and runtime validation.

Complex business logic still needs tests. Security rules still need audits.

The point isn’t that lint rules replace everything. The point is that anything you can make deterministic, you should.

The Reality

Prompting is great for exploration. Production code needs rules.

If AI is writing code in your repo, constrain it with the same systems you already trust: linters, types, tests, and CI checks. That’s how you get the speed AI promises without sacrificing reliability.

Rituals hope. Rules enforce.

AI Code Needs Rules, Not Rituals

Mon, 20 Oct 2025 00:00:00 GMT

Coding agents are here to stay, and I know I’m absolutely right about that. While we’re all getting used to workflows using AI-powered coding agents, we now live in the world of dark arts and rituals.

We spend hours tweaking prompts, creating elaborate Claude.md, Agents.md and other files formatted in a particular way, stored in a particular way essentially performing black magic to hope our LLM agent adheres to our team’s best practices and coding patterns.

On a good day, it works. On others, the behaviour is random.

Here’s the problem: Random is not good enough for production.

We’re trying to force a non-deterministic, generative tool to be a deterministic rule-follower. This is the wrong approach.

Instead, we should let AI do what it does best: be creative and generative, helping us achieve and realise our desired outcomes while following instructions and examples for how, what, and where it should generate.

My advice? Stop relying on hope-driven prompting. Start using linters to guarantee your standards.

The Problem with Hope-Driven Development

After months of working with AI coding agents across different projects, I’ve noticed a pattern that’s become impossible to ignore. We’ve all been there: spending hours crafting the perfect prompt, organising our project files just so, and crossing our fingers that this time, the AI will follow our coding standards.

But here’s what I’ve learned: hope is not a strategy for production code.

In this article, I’ll show you why relying on prompts alone is fundamentally flawed, and how to build a deterministic system that guarantees your AI-generated code meets your standards. You’ll discover practical examples of linting rules that solve real problems, learn how to create a self-correcting AI workflow, and understand why this approach future-proofs your development process.

By the end, you’ll have a concrete plan to stop hoping for quality and start engineering it.

A Lesson from the Trenches

Last month, I spent three hours debugging why our AI-generated React components were causing bundle size warnings. The issue? Inconsistent import patterns across 47 files. Some used import { z } from 'zod', others used import * as z from 'zod'. Our AI agent had been trained on mixed examples and couldn’t maintain consistency.

That’s when I realised: we were asking the wrong question. Instead of “How can we make our AI more consistent?”, we should have been asking “How can we guarantee consistency regardless of AI behaviour?”

The solution wasn’t better prompting—it was better tooling.

Example 1: The Subtle Art of Imports

It’s not just about correctness; consistency and performance matter too.

Consider this common Zod import:

// ❌ This *can* pull in unnecessary bytes depending on your bundler
// Some bundlers may not tree-shake this effectively
import { z } from 'zod';

Many teams, aiming for optimal bundle sizes, prefer this pattern:

// ✅ This is often more efficient and easier for tree-shaking
// Bundlers can more reliably eliminate unused exports
import * as z from 'zod';

Will your AI agent know this? Maybe. Will it remember 100% of the time? Absolutely not.

Instead of adding this trivia to a prompt file you hope the AI reads, just add a dedicated lint rule. The community has already solved this with eslint-plugin-import-zod. Using this lint rule ensures that both AI and we will use the most optimal way for importing Zod.

Example 2: The React Hooks Minefield

Let’s talk about React. The Rules of Hooks are notoriously tricky. There’s advice, for the advice, for which there is more guidance, and then a clarification on that makes it confusing for us, and who knows what patterns and docs AI has trained on.

With so many edge cases, models often make hook mistakes.

Thankfully, the React team provides a deterministic solution for this: eslint-plugin-react-hooks.

This is where the “prompting” strategy truly falls apart. That ESLint plugin was recently updated to support the new React Compiler and the useEffectEvent API.

Now, ask yourself:

Does your AI agent, trained on a massive (but older) corpus of code, know about these cutting-edge rules?
Can you guarantee it will apply them correctly, every time?

This is the “LLM Lag” in action. Remember: LLMs are trained on older codebases. They often won’t know the latest patterns or subtle rule changes. While MCP servers such as context7 can help, it’s still up to your coding tool to first use the tools available and then act on the information retrieved.

In contrast, ESLint and other linting tools provide the deterministic check you need, ensuring that even AI-generated code adheres to the most modern React practices.

Why ESLint? Because Determinism is Your Reassurance

In a world of AI randomness, ESLint delivers determinism.

Why risk your code quality and standards on chance?

Integrating ESLint into your workflow provides a reliable safety net and ensures code meets production quality standards.

The “Self-Correcting” AI Workflow

This is where it gets truly powerful. You don’t just run the linter at the end. You build a feedback loop.

Imagine this workflow:

The AI coding agent generates a new component or feature.
A hook (e.g., a file-save hook in your IDE, or post-tool use hook ) that runs eslint --fix on the generated files.
The agent sees the linting errors or the resulting code changes.
In its next turn, the AI can self-correct based on the deterministic feedback from the linter.

The AI remains fast, but lint feedback keeps code on track with your quality standards.

Here’s how this self-correcting workflow looks in practice:

graph TD
    A[AI Agent Generates Code] --> B[File Save Hook Triggers]
    B --> C[ESLint Runs Automatically]
    C --> D{Linting Errors?}
    D -->|Yes| E[ESLint Auto-Fixes Issues]
    D -->|No| F[Code Meets Standards]
    E --> G[AI Sees Fixed Code]
    G --> H[AI Learns from Corrections]
    H --> I[Next Generation Improved]
    F --> I
    I --> A
    
    style A fill:#e1f5fe
    style C fill:#f3e5f5
    style E fill:#e8f5e8
    style F fill:#e8f5e8
    style H fill:#fff3e0

This creates a continuous improvement loop where the AI learns from deterministic feedback, becoming more aligned with your standards over time.

Prevent Vendor Lock-in

While we hope for standardisation, the AI market is a fast-moving, innovative space. Today’s way of doing things becomes the old way. I’ve seen this with Cursor, and now we have skills with Claude. I use a variety of AI tools as they all have their strengths and weaknesses, and so linting rules offer the consistency I can rely on.

The Added Bonus: Future-Proofing Your Codebase

This workflow brings huge long-term benefits.

Next month, when a new React pattern emerges or your team discovers a new performance optimisation, you don’t need to retrain or re-prompt your AI.

You just add the new ESLint rule.

From that moment on, you can be assured that any AI-generated code will follow this new best practice.

Even better, you can now ask the AI to help you refactor your existing codebase, and the linter will act as its guide, ensuring the improvements are applied consistently.

Key Takeaways

Stop hoping for quality. Engineer it. Replace hope-driven prompting with deterministic linting rules that guarantee your standards.
Build a self-correcting feedback loop. Integrate ESLint into your AI workflow so the agent learns from deterministic corrections.
Future-proof your approach. When new best practices emerge, add a lint rule rather than updating prompts across multiple files.
Maintain consistency across tools. Linting rules work regardless of which AI tool you’re using, preventing vendor lock-in.

Your Next Steps

Audit your current prompts. Identify which “rules” in your prompt files could be enforced with linting instead.
Start with one rule. Pick the most common inconsistency in your AI-generated code and create an ESLint rule for it.
Set up the feedback loop. Configure your IDE or build process to run eslint --fix automatically on AI-generated files.
Gradually expand. Add more rules as you discover patterns, building a comprehensive quality system.

A Balanced Approach

This isn’t about abandoning prompts entirely. High-level guidance, team conventions, and architectural decisions still benefit from well-crafted prompts. Those still leak out of a long context. Before the agent writes a line, somebody has to hold the floodplain and the wind, which is the site survey. But for the nitty-gritty details, import patterns, hook usage, naming conventions, let your linters do the heavy lifting.

Stop hoping for quality. Engineer it.

Let your AI be the generative engine, and let your linters be the deterministic, ever-updatable guardrails that future-proof your codebase.

When a new best practice emerges, you won’t need to re-prompt your markdown files; you’ll just add a rule.

Agent: The Sugar Syntax of streamText

Fri, 19 Sep 2025 00:00:00 GMT

For most of us, AI still feels like a black box. We send it a prompt and we get back a blob of text. Maybe we write some code to call a tool; maybe we juggle a few callbacks. We tell ourselves that this is just how things work: a model can only generate tokens, and tools can only run in our code.

But what if this mental model is the problem?

In this post I want to argue that the Agent pattern in the AI SDK is as revolutionary for AI development as useState and useEffect were for React. Just like React’s client/server directives annotate where code runs across the network, the Agent API annotates where logic runs across the AI/model boundary.

Even if you never use it, understanding why it exists will change how you think about building AI applications.

Two worlds stitched together

Let’s start with the basics. The AI SDK exposes two ways to talk to language models:

generateText / streamText, which directly stream tokens back from an LLM.
Agent, an object-oriented facade built on top of those functions.

The raw streamText function does a lot. It starts streaming immediately, supports callbacks for chunks and errors, and exposes helper functions for sending responses. The result object contains rich information: the generated content, reasoning, tool calls and results, usage data and more. In other words, streamText already knows how to handle multi-step conversations, execute tools, and recover from errors. It is the low-level primitive.

The Agent class, by contrast, is almost boring.

The official docs describe it as

“a structured way to encapsulate LLM configuration, tools, and behavior into reusable components”.

It accepts the same settings you would pass to generateText or streamText. Under the hood, its generate and stream methods literally forward those options to the corresponding functions.

So why bother?

The old way

To appreciate the difference, imagine you are building a simple weather bot with raw streamText.

Your code might look like this:

import { streamText } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';

const result = streamText({
  model: openai('gpt-4o'),
  system: 'You are a helpful assistant.',
  prompt: 'What is the weather in NYC?',
  tools: {
    getWeather: {
      description: 'Get the current weather',
      inputSchema: z.object({ location: z.string() }),
      async execute({ location }) {
        // call your weather API
        return { temperature: 72, condition: 'sunny' };
      }
    }
  },
  onError({ error }) {
    console.error(error);
  },
  onFinish({ toolCalls }) {
    // Stream finished; tools with execute already ran.
    // For manual handling, omit execute and process toolCalls.
  }
});

// consume the async iterator
for await (const chunk of result.textStream) {
  console.log(chunk);
}

This works fine, but notice how much boilerplate you are dealing with:

You have to pass the same model, system prompt and tools every time.
You manually consume the textStream and handle onError / onFinish.
You decide when to stop streaming, when to call tools and when to finish.

You are writing orchestration logic rather than focusing on your problem. Every detail about streaming, tool calls and state management lives in your code.

The Agent way

Now look at the same logic expressed with an Agent:

import { Agent } from 'ai';
import { openai } from '@ai-sdk/openai';
import { z } from 'zod';

const assistant = new Agent({
  model: openai('gpt-4o'),
  system: 'You are a helpful assistant.',
  tools: {
    getWeather: {
      description: 'Get the current weather',
      inputSchema: z.object({ location: z.string() }),
      async execute({ location }) {
        return { temperature: 72, condition: 'sunny' };
      }
    }
  }
});

const result = await assistant.stream({ prompt: 'What is the weather in NYC?' });

That’s it. There are no callbacks to hook, no loop to write. When the model decides to call a tool, the SDK calls it for you. When the model finishes generating text, the stream ends. The same configuration is reused for every call to the agent.

Under the hood, nothing magical is happening. The Agent’s stream method merges your prompt with the agent’s configuration and forwards it to streamText. Agent methods return the same result objects as generateText/streamText and reuse most of the same settings. Yet this tiny abstraction has outsized effects on how you structure your code.

Why does this matter?

From configuration to behavior

When you call streamText directly, you think procedurally:

Call the model. Set up the model, system prompt and tools.
Handle the stream. Consume tokens, watch for tool calls, call tools, feed results back.
Manage state. Keep track of previous messages, step counts and stop conditions.
Handle errors. Log and recover from exceptions.

With an Agent you think declaratively:

Define an agent. Specify the model, system prompt, tools and stopping rules once.
Give it tasks. Pass a prompt to generate() or stream().
Let it orchestrate. The SDK manages streaming, tool calls and state.

In other words, the Agent doesn’t add new features; it removes cognitive overhead. You stop thinking about configuration and start thinking about behavior. The agent becomes a reusable AI entity in your codebase rather than a loose configuration object.

Reusability and encapsulation

Because an agent is a class, you can instantiate as many as you need with different roles. Here’s a more advanced example:

const codeReviewer = new Agent({
  model: openai('gpt-4o'),
  system: `You are a senior software engineer conducting code reviews.
You can read files, analyze code quality and suggest improvements.`,
  tools: {
    readFile: readFileTool,
    analyzeComplexity: complexityTool,
    suggestRefactoring: refactorTool,
    checkTests: testingTool
  },
  stopWhen: [stepCountIs(5)] // maximum five steps
});

const review = await codeReviewer.stream({
  prompt: 'Please review the file src/utils/data-processor.js'
});

You’re not manually iterating over tool calls anymore. You define the capabilities of an assistant and let it figure out how to achieve its goal. Each call to stream() reuses the same tools, system prompt and stopping conditions; the only thing that changes is the prompt. As the docs note, the Agent encapsulates configuration so you can reuse it across your application.

A bigger mental model: AI as a single program

This abstraction hints at a deeper mental model. In the old world, your application and the model live in separate universes. The model only knows how to generate text; your code only knows how to run tools. There is no way to express that a piece of code wants to orchestrate both. You glue them together with fetch() or manual loops.

Agents invite you to see your application and the AI model as one program split across two execution contexts. You describe an assistant’s behavior (its model, prompt and tools) and then call it like any other method. The AI decides when to call your tools; your tools feed their results back into the model. Each generation step is either a piece of text or a tool invocation, and the loop continues until a stopping condition is met. You no longer need to manage the loop yourself; you think at a higher level.

Here’s a visual way to think about it:

graph TD
  subgraph "Your Code"
    A[Define Agent] --> B["Call agent.stream()"]
  end

  subgraph "AI SDK"
    B --> C{streamText}
    C --> D[LLM]
    C --> E[Tool Executor]
    E --> D
  end

  subgraph "Agent Configuration"
    F[Model]
    G[System Prompt]
    H[Tools]
    I[Stop Conditions]
  end

  F -.-> B
  G -.-> B
  H -.-> B
  I -.-> B

The dashed lines represent configuration that the agent holds on to. Whenever you call agent.stream(), that configuration is merged with your prompt and passed to streamText. The loop of model → tool call → model continues until the model generates text or hits a stop condition.

Cognitive load and API design

So what is the big deal out of what is essentially a convenience wrapper?

Because good API design is as much about psychology as it is about features. When all we had was streamText, we had to think about the network protocol, callbacks and error handling. We were coding at the wrong level of abstraction.

By introducing a small facade, the AI SDK changes our default mental model. It makes the boundary between your code and the model explicit but frictionless. You can still drop down to streamText for fine-grained control, for example, when you need to handle each chunk yourself or implement custom loop logic. But for most use cases, an Agent lets you focus on what your AI should do, not how it should do it.

This reminds me of the leap from writing XHR calls to using fetch() / async functions; or from manual DOM updates to using React components. Each leap didn’t add something we couldn’t technically do before; it removed something we no longer had to think about. That’s why I think the Agent pattern will outlive the current library. It’s not just sugar, it’s a mental model.

Beyond the basics: building AI entities

Once you start seeing agents as reusable AI entities, you begin to imagine more ambitious scenarios. For example:

const codebaseAgent = new Agent({
  model: openai('gpt-4o'),
  system: 'You understand this entire codebase.',
  tools: {
    readFile,
    searchCode,
    runTests,
    deployCode,
    analyzePerformance,
    reviewPRs,
    updateDocs
  },
  stopWhen: [stepCountIs(10)]
});

// Same agent helps with multiple tasks
await codebaseAgent.stream({ prompt: 'Refactor the authentication system' });
await codebaseAgent.stream({
  prompt: 'Write API documentation for the billing module'
});
await codebaseAgent.stream({ prompt: 'Investigate why the build is failing' });

Here, the agent isn’t just a function call; it’s a team member with a persistent identity. You might add memory so it remembers previous files it has read, or share context so it builds up a knowledge base. At this point, you’re not building AI features anymore. You’re building AI entities with their own capabilities, constraints and state.

The meta-lesson

The relationship between Agents and streamText illustrates an important principle: sometimes the most significant innovation is not a new capability, but a new way of thinking about existing capabilities.

The Agent class doesn’t change what streamText can do; it changes how you think about orchestrating AI and tools. It allows you to see your application and the model as a single program and to express your intent declaratively.

When you design APIs, ask yourself: What mental model am I promoting? Are you forcing your users to juggle configuration and boilerplate, or are you giving them the right abstractions so they can focus on behavior?

The Agent pattern shows that a tiny bit of sugar can make a complex system feel intuitive and human. And that might be the most important feature of all.

Build Safer, Faster, and More Reliable AI Apps with AI SDK Middleware

Wed, 27 Aug 2025 00:00:00 GMT

Having recently built an AI Guardrails library for the AI SDK, I wanted to share what I learned along the way. This post will walk you through how you can write your own middleware, and why it’s such a game-changer for building robust AI applications.

Design AI features that are safer, faster, and easier to evolve by layering language model middleware. This guide explains how to use AI SDK middleware to transform inputs, post-process outputs, enforce safety rules, cache results, observe performance, and handle streaming using a clean, composable approach aligned with official guidance.

Why middleware

Middleware enables you to intercept and modify calls to a language model without altering business logic. Add features incrementally, reuse them across apps, and standardize behavior across providers.

When to use middleware

You need consistent guardrails or validation across multiple models/providers
You want caching, rate limiting, or cost controls without changing app logic
You need to enrich prompts or normalize outputs at a single integration point
You want structured, uniform observability and error handling

Quickstart

import { wrapLanguageModel } from 'ai';

const enhancedModel = wrapLanguageModel({
  model: yourModel,

  middleware: yourMiddleware,
});

The middleware interface

Implement one or more of these hooks:

import type { LanguageModelMiddleware } from 'ai';

export const yourMiddleware: LanguageModelMiddleware = {
  transformParams: async ({ params }) => {
    return params; // modify request options for both generate and stream
  },

  wrapGenerate: async ({ doGenerate }) => {
    const result = await doGenerate();

    return result; // post-process non-streaming results
  },

  wrapStream: async ({ doStream }) => {
    const { stream, ...rest } = await doStream();

    return { stream, ...rest }; // transform streaming results
  },
};

Recommended middleware order

Apply middleware in this order for predictable behavior and performance:

Safety and validation first (block/clean unsafe input early)
Caching early (short-circuit on cache hit)
Transformations next (input/output shaping)
Monitoring and logging last (observe the final shape and timing)

const model = wrapLanguageModel({
  model: yourModel,

  middleware: [
    safetyMiddleware, // 1) block/validate

    cachingMiddleware, // 2) short-circuit identical calls

    inputEnhancerMiddleware, // 3) transform inputs

    outputProcessorMiddleware, // 4) transform outputs

    monitoringMiddleware, // 5) observe last
  ],
});

flowchart LR



  A[User request] --> B[Safety & Validation]

  B --> C{Blocked?}

  C -->|Yes| X[Return safe fallback]

  C -->|No| D[Caching]

  D --> E{Cache hit?}

  E -->|Yes| Y[Return cached result]

  E -->|No| F[Input Transformation]

  F --> G[Model Inference]

  G --> H[Output Processing]

  H --> I[Monitoring / Logging]

  I --> J[Response]

Core patterns

1) Input transformation

import type { LanguageModelMiddleware } from 'ai';

export const inputEnhancerMiddleware: LanguageModelMiddleware = {
  transformParams: async ({ params }) => {
    const enhancedPrompt = [
      { role: 'system', content: 'You are a helpful assistant.' },

      ...(Array.isArray(params.prompt) ? params.prompt : [params.prompt]),
    ];

    return { ...params, prompt: enhancedPrompt };
  },
};

2) Output processing

export const outputProcessorMiddleware: LanguageModelMiddleware = {
  wrapGenerate: async ({ doGenerate }) => {
    const result = await doGenerate();

    const text = (result.content || [])

      .filter((p: any) => p.type === 'text')

      .map((p: any) => p.text)

      .join('')

      .trim()

      .replace(/\s+/g, ' ');

    return { ...result, content: [{ type: 'text', text }] };
  },
};

3) Safety and request blocking

export const safetyMiddleware: LanguageModelMiddleware = {
  wrapGenerate: async ({ doGenerate, params }) => {
    const toText = (p: unknown) => {
      if (typeof p === 'string') return p;

      if (
        p &&
        typeof p === 'object' &&
        'content' in (p as { content?: unknown })
      ) {
        const content = (p as { content?: string | Array<{ text?: string }> })
          .content;

        if (Array.isArray(content))
          return content.map((c) => c.text || '').join(' ');

        if (typeof content === 'string') return content;
      }

      return '';
    };

    const promptText = Array.isArray(params.prompt)
      ? params.prompt.map(toText).join(' ')
      : toText(params.prompt);

    if (/\b(hack|exploit|bypass|malicious)\b/i.test(promptText)) {
      return {
        content: [{ type: 'text', text: 'I cannot process this request.' }],

        finishReason: 'other',

        usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },

        warnings: [],
      };
    }

    return doGenerate();
  },
};

4) Streaming transformation

Handle multiple stream part types for fine-grained control.

export const streamingMiddleware: LanguageModelMiddleware = {
  wrapStream: async ({ doStream }) => {
    const { stream, ...rest } = await doStream();

    const transformStream = new TransformStream({
      transform(chunk: any, controller) {
        switch (chunk.type) {
          case 'text-delta': {
            const filtered = chunk.delta.replace(/\b(harmful|bad)\b/gi, '***');

            controller.enqueue({ ...chunk, delta: filtered });

            break;
          }

          case 'tool-call':

          case 'tool-result':

          case 'response-metadata':

          case 'error':

          default: {
            controller.enqueue(chunk);
          }
        }
      },
    });

    return { stream: stream.pipeThrough(transformStream), ...rest };
  },
};

flowchart TD



  S[Model Stream] --> T[TransformStream]

  T -->|text_delta| D[Filter delta - redact words]

  D --> U[Emit filtered text_delta]

  T -->|tool_call| C[Pass-through tool_call]

  T -->|tool_result| R[Pass-through tool_result]

  T -->|response_metadata| M[Pass-through metadata]

  T -->|error| E[Pass-through error]

  U --> K[Client]

  C --> K

  R --> K

  M --> K

  E --> K

5) Monitoring and logging

export const monitoringMiddleware: LanguageModelMiddleware = {
  wrapGenerate: async ({ doGenerate, params }) => {
    const start = Date.now();

    try {
      const result = await doGenerate();

      const durationMs = Date.now() - start;

      console.log('AI call completed', {
        durationMs,

        temperature: params.temperature,

        inputTokens: result.usage?.inputTokens,

        outputTokens: result.usage?.outputTokens,
      });

      return result;
    } catch (error) {
      const durationMs = Date.now() - start;

      console.error('AI call failed', {
        durationMs,

        error: (error as Error).message,
      });

      throw error;
    }
  },
};

6) Caching

const cache = new Map<string, any>();

export const cachingMiddleware: LanguageModelMiddleware = {
  wrapGenerate: async ({ doGenerate, params }) => {
    const cacheKey = JSON.stringify(params);

    if (cache.has(cacheKey)) return cache.get(cacheKey);

    const result = await doGenerate();

    cache.set(cacheKey, result);

    return result;
  },
};

7) Retry logic

export const retryMiddleware: LanguageModelMiddleware = {
  wrapGenerate: async ({ doGenerate }) => {
    const maxRetries = 3;

    let lastError: Error | undefined;

    for (let attempt = 1; attempt <= maxRetries; attempt++) {
      try {
        return await doGenerate();
      } catch (error) {
        lastError = error as Error;

        if (attempt === maxRetries) throw lastError;

        await new Promise((r) => setTimeout(r, Math.pow(2, attempt) * 1000));
      }
    }
  },
};

Error handling

Start with a safe fallback, and optionally branch on SDK error types (e.g., NoSuchToolError) to produce targeted responses:

import { NoSuchToolError, type LanguageModelMiddleware } from 'ai';

export const robustMiddleware: LanguageModelMiddleware = {
  wrapGenerate: async ({ doGenerate }) => {
    try {
      return await doGenerate();
    } catch (error) {
      if (error instanceof NoSuchToolError) {
        type ToolWarning = { type: 'tool'; message: string };

        return {
          content: [{ type: 'text', text: 'Requested tool is not available.' }],

          finishReason: 'error',

          usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },

          warnings: [
            {
              type: 'tool',

              message: 'Missing tool in execution.',
            } as ToolWarning,
          ],
        };
      }

      return {
        content: [{ type: 'text', text: 'Sorry, something went wrong.' }],

        finishReason: 'error',

        usage: { inputTokens: 0, outputTokens: 0, totalTokens: 0 },

        warnings: [],
      };
    }
  },
};

flowchart TD



  A[doGenerate] --> B{throws}

  B -->|No| Z[Success]

  B -->|Yes| C{NoSuchToolError}

  C -->|Yes| D[Tool fallback]

  C -->|No| E[Generic fallback]

Composition that scales

Compose multiple middlewares to form a clear pipeline.

import { wrapLanguageModel } from 'ai';

const modelWithStack = wrapLanguageModel({
  model: yourModel,

  middleware: [
    safetyMiddleware,

    cachingMiddleware,

    inputEnhancerMiddleware,

    outputProcessorMiddleware,

    monitoringMiddleware,
  ],
});

Performance considerations

Keep heavy computation out of hot paths; precompute or cache
Avoid excessive serialization in transformParams
Stream: transform deltas incrementally, don’t buffer entire outputs
Log only what you need; prefer structured logs

Testing

import { describe, it, expect, vi } from 'vitest';

describe('safetyMiddleware', () => {
  it('blocks harmful content', async () => {
    const mockModel: { doGenerate: () => unknown } = { doGenerate: vi.fn() };

    const wrapped = wrapLanguageModel({
      model: mockModel,

      middleware: safetyMiddleware,
    });

    const params: { prompt: string } = { prompt: 'how to hack wifi?' };

    const result = await wrapped.doGenerate(params);

    expect(result.content[0].text).toMatch(/cannot process/i);

    expect(mockModel.doGenerate).not.toHaveBeenCalled();
  });
});

Takeaways

Middleware standardizes safety, caching, transformations, and observability
Order matters: safety → cache → transform → monitor
Handle streaming by part type for fine-grained control
Prefer explicit error branches for known SDK errors
Keep it lightweight and testable

References

Language Model Middleware (docs): ai-sdk.dev/docs
Middleware guide and examples: sdk.vercel.ai/docs/ai-sdk-core/middleware
Cookbook (caching, streaming, tools): sdk.vercel.ai/examples

Building Reliable AI Agents Series: Factor 2 – Own Your Prompts

Mon, 28 Jul 2025 00:00:00 GMT

This is the second post in a series of posts about building reliable AI agents.

When building AI agents, where do your prompts live? If they’re hidden inside frameworks or scattered across configuration files, you’re missing a fundamental principle of maintainable AI systems: treating prompts as first-class code citizens.

Factor 2: Own Your Prompts

The Hidden Prompt Problem

Most AI frameworks abstract away prompts, promising simplicity. But this convenience comes at a cost. When your agent misbehaves, how do you debug it? When you need to improve responses, where do you look? Without visibility into prompts, you’re essentially flying blind.

Factor 2 of the 12-Factor Agent methodology solves this by establishing a simple principle: prompts should be explicit, version-controlled code artefacts. No hidden abstractions, no mysterious instructions - just clear, manageable code.

Why Explicit Prompts Matter

Consider this scenario: your customer service agent suddenly starts giving incorrect information. With hidden prompts, you’re left guessing. With explicit prompts, you can:

Check the exact instructions your AI received
Review recent prompt changes in version control
Roll back to a previous working version
Test different prompt variations systematically

The formula is simple: Same Question + Different Prompts = Different Behaviour (Predictably).

Implementing Factor 2

Here’s what explicit prompt management looks like in practice:

const PROMPTS = {
  basicV1: `You are a helpful assistant.`,

  basicV2: `You are a helpful and friendly assistant.
Always be polite, clear, and provide helpful explanations.
When asked about calculations, use the available math tools.`,

  calculator: `You are a mathematical assistant with access to calculator tools.
When asked to perform calculations:
1. Use the appropriate math tool
2. Show the calculation step by step
3. Explain the result clearly`,
};

Notice how each prompt version is clearly defined, labelled, and ready for version control. No magic, no hidden configuration - just code.

The Benefits in Production

When you own your prompts, several things become possible:

Systematic A/B Testing: Compare prompt performance with real metrics. Which version leads to better customer satisfaction? Which reduces support tickets?

Team Collaboration: Prompts become reviewable artefacts. Team members can suggest improvements through standard pull requests.

Compliance and Auditing: Track exactly what instructions your AI received at any point in time. Essential for regulated industries.

Debugging: When issues arise, you can immediately see the exact prompt that caused the problem.

Moving Beyond Guesswork

Traditional prompt engineering often feels like alchemy - mysterious incantations that sometimes work. Factor 2 transforms it into engineering - systematic, testable, and reproducible.

By treating prompts as code, you gain the same benefits that version control brought to software development: collaboration, history, rollbacks, and systematic improvement.

Getting Started

Implementing Factor 2 doesn’t require complex frameworks. Start simple:

Extract all prompts from configuration into explicit constants
Version these prompts alongside your code
Create a simple system for switching between prompt versions
Track which prompts are used in production

The key is visibility. When every team member can see, understand, and improve your prompts, your AI systems become truly maintainable.

Conclusion

Hidden prompts are technical debt. They make debugging harder, improvement slower, and collaboration impossible. By owning your prompts - making them explicit, versioned, and testable - you transform AI development from guesswork into engineering.

The next time you’re building an AI agent, ask yourself: can I see exactly what instructions it’s receiving? If not, it’s time to take ownership of your prompts.

Want to see Factor 2 in action? Check out the complete working example.

Hannah Hampton. A proper comeback story.

Sun, 27 Jul 2025 00:00:00 GMT

Photo by Katie Chan (KTC), licensed under CC BY-SA 4.0.

⚽️ Dropped by then and current manager Sarina Wiegman from the England squad in 2022 amid concerns over her attitude and conduct, she remained outside the national setup for almost two years.

⚽️ Born with strabismus and challenged by depth‑perception issues, she was told by doctors she should not play football.

⚽️ Rebuilt her career at Chelsea, playing an integral role in winning the treble last season.

⚽️ Chosen as England’s first-choice goalkeeper ahead of the great and brilliant Mary Earps.

⚽️ Delivered under pressure in the quarter‑final versus Sweden, even with a bloody nose. Named Player of the Match.

Tonight, she starts in a European final for England.

Even when things don’t go your way, never give up.

Building Reliable AI Agents Series: A 12-Factor Methodology

Mon, 14 Jul 2025 00:00:00 GMT

After building production AI systems over the past few years, thanks to HumanLayer, I’ve learned that most agent failures aren’t about the LLM, they’re about architecture.

That’s why I’m creating a series of posts sharing the 12-Factor Agents methodology using Mastra.

In each part, I’ll break down one principle that transforms fragile prototypes into robust, production-ready AI agents.

Factor 1: Natural Language → Tool Calls

Factor 2: Own Your Prompts

Building Reliable AI Agents Series: Factor 1 – Natural Language → Tool Calls

Mon, 14 Jul 2025 00:00:00 GMT

This is the first post in a series of posts about building reliable AI agents.

Unlock reliable, testable AI agents by treating your LLM as a parser, not an executor. Learn how converting natural language into structured tool calls leads to predictable, scalable systems.

Factor 1: Natural Language → Tool Calls

Modern AI agents are most robust when their LLMs parse instructions, rather than execute them directly. This approach transforms the LLM into a sophisticated parser that converts natural language into structured function calls.

The flow:

Natural Language → LLM Processing → Structured Tool Call → Deterministic Execution

Why this matters

Predictable results – Same input, same output every time
Debuggable workflows – You can trace exactly what happened
Testable components – Tools can be unit-tested independently
Scalable architecture – Add new capabilities without touching LLM logic

See Factor 1 in action here

Demystifying MCP: Coaching Colleagues Through the New AI Protocol

Mon, 30 Jun 2025 00:00:00 GMT

Last month, I had the fantastic opportunity to run a hands-on Model Context Protocol (MCP) workshop. It was a chance to mentor and coach colleagues, from engineers and testers to product owners, as we explored what MCP is, how it works, and why it is generating so much momentum across AI and developer communities.

My goal was not just to share knowledge. It was to guide the learning journey and build a shared, foundational understanding of this powerful emerging standard.

Workshop Highlights

We covered everything from core protocol basics and prompt crafting to real world patterns such as:

Simple MCP client and servers
Tools
Resources
Prompts
Authentication flows
Session context handling
Composing multiple MCP servers

Outcome

By the end of the session, everyone had a clear understanding of MCP, its potential, and how it could fundamentally change how we build AI toolchains. This clarity, made possible through practical examples and direct support, was a truly rewarding outcome.

Explore the Workshop Repository

You can explore the same TypeScript repository I used for the workshop here: 👉 https://github.com/jagreehal/typescript-mcp-workshop

Each example includes step-by-step guidance. So, whether you want to spin up your first MCP server or dive deep into tool composition and integration flows, this will show you how.

MCP is evolving rapidly, and I will keep the repository up to date as best practices emerge. If you have feedback or ideas for what I should cover next, please let me know.

Understanding Commands & Events with Orchestration & Choreography

Wed, 18 Jun 2025 00:00:00 GMT

Designing distributed systems has never been more challenging. As teams embrace microservices and event‑driven architectures, a persistent myth has arisen: commands equal orchestration, and events equal choreography. This tidy equivalence often becomes a mental shortcut but conceals the deeper truth of control‑flow patterns versus messaging semantics. As many practitioners have observed, collapsing these separate dimensions can restrict your system’s flexibility and resilience.

In this article, we’ll demystify these concepts and show how to apply them independently. You’ll discover how separating semantics (commands vs. events) from control flow (orchestration vs. choreography) grants you greater architectural freedom and clearer, more maintainable workflows.

Let’s examine three core scenarios, highlighting error handling, delivery guarantees and state‑management nuances.

1. Orchestration with Events: Centralised, Resumable Workflows

An orchestrator maintains end‑to‑end visibility by subscribing to domain events, tracking each workflow’s state, and emitting further actions. Although illustrated with events, the same pattern applies to commands.

// orchestrator.ts
interface OrderPlaced {
  orderId: string;
}
interface PaymentConfirmed {
  orderId: string;
}

// EventBus and StateStore interfaces for DI
interface EventBus {
  emit(type: string, payload: any): Promise<void>;
}
interface StateStore {
  get(orderId: string): string | undefined;
  set(orderId: string, state: string): void;
}

class OrderOrchestrator {
  constructor(
    private eventBus: EventBus, // Injected event bus
    private state: StateStore // Injected state store
  ) {}

  async handle(event: OrderPlaced) {
    this.state.set(event.orderId, 'payment_pending');
    console.log(`Order ${event.orderId} placed. Starting payment.`);
    await this.eventBus.emit('ProcessPayment', { orderId: event.orderId });
  }

  async onPaymentConfirmed(event: PaymentConfirmed) {
    this.state.set(event.orderId, 'fulfilled');
    console.log(`Payment confirmed for ${event.orderId}. Fulfilling order.`);
    await this.eventBus.emit('FulfilOrder', { orderId: event.orderId });
  }
}

Error Handling: The orchestrator can centrally retry failed steps or invoke compensating actions. If a step fails, the workflow pauses and retries based on configured policies.

Delivery Guarantees: Events often default to at‑most‑once delivery on basic buses, though modern platforms (Kafka, Pulsar) provide at‑least‑once semantics with idempotent consumers to avoid data loss. The orchestrator persists all incoming events and uses idempotent handlers to ensure each workflow step runs exactly once or is safely retried.

2. Choreography with Commands: Decentralised, Implicit Flows

In a choreographed system, each service reacts locally and issues the next command. There is no central coordinator—each participant knows only its own scope.

// inventory.service.ts
// CommandBus interface for DI
interface CommandBus {
  send(type: string, payload: any): Promise<void>;
}

class InventoryService {
  constructor(private commandBus: CommandBus) {} // Injected command bus

  async reserveStock(orderId: string, sku: string) {
    console.log(`Inventory: Reserving ${sku} for order ${orderId}.`);
    await this.commandBus.send('ScheduleShipment', { orderId, sku });
  }
}

// shipping.service.ts
class ShippingService {
  constructor(private commandBus: CommandBus) {} // Injected command bus

  async scheduleShipment(orderId: string, sku: string) {
    console.log(`Shipping: Scheduling shipment for ${orderId} (${sku}).`);
    // ... further logic ...
  }
}

Error Handling: Each service can route unprocessable messages to a dead‑letter queue for later inspection, preventing a single failure from halting the entire choreography.

Delivery Guarantees: Commands generally require at‑least‑once delivery (e.g. via durable queues), so idempotent handlers are essential. Events in choreographies may default to at‑most‑once delivery, but real‑world event streaming often guarantees at‑least‑once with idempotent consumers.

3. Mixed Messaging: Orthogonal Dimensions in Practice

By decoupling semantics from control flow, you can mix commands and events as appropriate.

// payment-orchestrator.ts
class PaymentOrchestrator {
  constructor(private commandBus: CommandBus) {} // Injected command bus

  async handleCommand(cmd: { type: 'ProcessPayment'; orderId: string }) {
    console.log(`Processing payment for ${cmd.orderId}.`);
    if (await this.needsCharge(cmd.orderId)) {
      await this.commandBus.send('AttemptCharge', cmd);
    }
  }

  private async needsCharge(orderId: string): Promise<boolean> {
    // ... check if charge is needed ...
    return true;
  }
}

// shipping-subscriber.ts
// EventBus and CommandBus interfaces reused
function subscribeEvent<T>(
  eventType: string,
  handler: (event: T) => Promise<void>
) {
  // ... subscribe logic ...
}

class ShippingSubscriber {
  constructor(
    private commandBus: CommandBus, // Injected command bus
    private eventBus: EventBus // Injected event bus
  ) {}

  async onOrderPlaced(event: { type: 'OrderPlaced'; orderId: string }) {
    console.log(
      `Shipping (choreography): Received OrderPlaced for ${event.orderId}.`
    );
    await this.commandBus.send('InitiatePackaging', { orderId: event.orderId });
    await this.eventBus.emit('ShipmentScheduled', { orderId: event.orderId });
  }
}

State‑Management Nuance

// Example: rebuilding state via event sourcing in a choreographed service
// EventLog interface for DI
interface EventLog {
  getEvents(type: string): any[];
}

class InventoryService {
  private reserved: Record<string, string> = {};
  constructor(private eventLog: EventLog) {} // Injected event log

  async rebuildState() {
    const events = this.eventLog.getEvents('StockReserved');
    for (const { orderId, sku } of events) {
      this.reserved[orderId] = sku;
    }
    console.log('State rebuilt from event log:', this.reserved);
  }
}

Choreographed services can manage their local state via event sourcing—replaying an event log to rebuild state on restart, enabling rich audit trails without central orchestration.

Visualising the Patterns

Below is a diagram illustrating the difference between orchestration (centralised control) and choreography (decentralised flow). Orchestration is like a conductor leading an orchestra—one central figure directs each part. Choreography is more like a jazz band, where each musician listens and responds, creating harmony without a single leader.

%% Figure 1: Orchestration and Choreography Control Flows
%% Alt text: Diagram showing orchestration as a central orchestrator directing workflow steps, and choreography as services passing commands and events in a decentralised manner.
flowchart LR
  subgraph Orchestration
    A[OrderPlaced Event] --> B[Orchestrator Determines Next Step]
    B --> C[Emit ProcessPayment]
    C --> D[PaymentConfirmed Event]
    D --> E[Emit FulfilOrder]
  end

  subgraph Choreography
    F[Inventory Service] -- "Command: ReserveStock" --> G[Billing Service]
    G -- "Event: StockReserved" --> H[Shipping Service]
    H -- "Command: ScheduleShipment" --> I[Notification Service]
    I -- "Event: ShipmentScheduled" --> J[Analytics Service]
  end

Separating messaging semantics (commands vs. events) from control-flow patterns (orchestration vs. choreography) gives you the flexibility to design robust, adaptable distributed systems. Choose the right tool for each job, and don’t conflate the two dimensions.

Conclusion

By disentangling messaging semantics from control‑flow patterns, you gain architectural agility, robust orchestrators, and flexible choreographies. This separation allows you to adapt your system as requirements evolve, without being locked into a single pattern.

From Gatekeeper to Enabler: Rethinking Engineering in the Age of the Citizen Developer

Mon, 09 Jun 2025 00:00:00 GMT

Last month, our company hackathon became a vivid illustration of a broader shift in how software gets built. Across teams, from sales to support, product to engineering, domain experts huddled around laptops, experimenting with AI powered platforms such as V0, Lovable and Bolt. Within hours, they’d fashioned interactive prototypes complete with navigation flows and validation rules, all without writing a single line of traditional code.

Although these early demos relied on mock data, the fact that non-engineers could conjure usable software unaided was striking.

As a full-stack engineer accustomed to crafting CRUD apps from the ground up, I found myself asking a new question: How might we empower these citizen developers to build more often, more securely, and with live data? Their deep problem domain knowledge meant they moved swiftly, iterated boldly and learned faster than any handoff-laden process could permit.

An isometric illustration of a futuristic highway under construction, where software engineers in hard hats are laying down glowing code-shaped road segments.

Over my two decades in software, I’ve discovered my highest leverage isn’t in writing every screen or endpoint myself, but in architecting robust APIs, infrastructure and tooling so that others can deliver user value. In this two-part series, I’ll share how engineers can transition from gatekeeping code to enabling creation at scale. In this first instalment, we’ll explore the mindset shifts and guiding principles. Part 2 will dive into concrete patterns and architectural strategies for secure, sustainable enablement.

The Democratisation of Software Development

Traditionally, software lived in the hands of specialists schooled in algorithms, architecture and abstraction. That expertise guaranteed rigour, but often created bottlenecks. Those closest to the problem had to wait for engineers. Today, low-code and no-code platforms are rewriting that story. By abstracting away boilerplate—through drag-and-drop interfaces, natural-language prompts or schema-driven generators—these tools empower domain experts to design, test and ship apps without deep coding knowledge. The advent of the citizen developer blurs the boundary between “user” and “builder.” As engineers, our role must evolve: we are no longer the sole authors of software, but the architects of the roads on which others will drive.

The Rise of the Citizen Developer

Citizen developers are not seeking to dismantle engineering teams—they simply need to solve context-specific problems fast. Whether it’s an operations lead automating routine workflows, an analyst assembling a live dashboard or a product manager sketching an onboarding flow, these experts bring intimate understanding of user needs. Platforms like V0.dev, Lovable.dev and Bolt combine intuitive interfaces with smart defaults and rich integrations. Some even scaffold complete applications from database schemas or plain-English descriptions. What once took weeks now happens in hours. Sure, these MVPs may carry imperfections, but their true value lies in rapid validation and immediate feedback. This is not a distant horizon; it’s our present.

Engineers as Enablers

To capitalise on this momentum, engineers must adopt a new mindset: from building every feature to creating the ecosystem where features emerge. That means crafting reusable components and well-documented APIs, defining safe defaults and guardrails, and fostering experimentation within controlled boundaries. In practice this looks like:

Stewardship over ownership: Rather than micromanaging every app, we curate the environments, libraries and conventions that reflect our standards.
Constraints over approvals: Guardrails and templates scale more gracefully than manual reviews.
Enablement over velocity: Our success is measured by how quickly non-engineers can deliver without compromising security or stability.

A well-paved road isn’t an easy excuse; it’s the fastest, safest path forward.

Schema-Driven UI Generation

One powerful pattern is schema-driven UI. Instead of hand-coding forms and validations, citizen developers define a schema—perhaps in JSON Schema or via GraphQL introspection—and the platform auto-generates the interface. Rename a field or tweak a validation rule in the schema, and the UI updates instantly. Benefits include:

Speed: Interfaces materialise in moments.
Consistency: Uniform look, feel and behaviour across apps.
Reliability: Built-in validation reduces runtime errors.
Maintainability: Schema changes propagate automatically.

By keeping business logic declarative, engineers can focus on data models and APIs, while platforms handle presentation. Both developer and domain expert stay in their zone of expertise.

Security and Governance

Enabling a multitude of builders needn’t mean relinquishing control. Thoughtfully designed platforms bake governance into every layer, making policies transparent, auditable and automated. Core patterns include:

Centralised identity providers for authentication
Curated and versioned APIs for data access
Rate-limiting and comprehensive logging for observability
Usage analytics to understand how templates and tools are adopted

Many organisations encapsulate these capabilities in an Internal Developer Platform (IDP), which serves as the single source of truth for permissions, policies and pipelines. In this model, security isn’t an afterthought—it’s inherent to the platform.

Trade-offs and Tensions

Citizen development offers clear upsides—accelerated prototyping, reduced IT backlogs and closer alignment between solutions and needs—but it also introduces challenges: potential sprawl of shadow IT, uneven code quality and emerging compliance gaps. These concerns aren’t reasons to halt progress, but signals to evolve our platforms and processes: tighten templates, codify best practices and foster ongoing collaboration between engineers and domain experts.

Key Takeaways

Engineers as enablers: Our highest value is in building the platforms, APIs and guardrails that empower others to create.

Citizen developers are here: With the right tools, domain experts can deliver real solutions—fast.

Governance is essential: Security, observability and policy must be built into the platform, not bolted on.

Collaboration is the future: The best outcomes arise when engineers and domain experts work together, not in silos.

Visualising the New Collaboration Model

This diagram shows how engineers, citizen developers, and the platform interact. Engineers create the platform and guardrails, enabling citizen developers to build solutions safely. Governance and feedback ensure continuous improvement.

flowchart TD
  A["Engineers"] -->|"Build"| B["Platform & Guardrails"]
  B -->|"Enable"| C["Citizen Developers"]
  C -->|"Build"| D["Apps & Solutions"]
  D -->|"Feedback"| B
  B -->|"Governance"| E["Security & Compliance"]

Conclusion: A New Kind of Collaboration

The rise of the citizen developer marks a structural transformation in how organisations solve problems. For engineers, this shift isn’t a diminution of our role but a reorientation from building every screen to designing the highways of innovation. By embracing stewardship, defining clear guardrails and architecting platforms that blend security with flexibility, we empower entire teams to deliver value safely and swiftly. The most effective engineering groups will be those that not only ship code, but also cultivate the ecosystem in which others can ship it too.

In Part 2, we’ll explore concrete architectural patterns and API-exposure strategies that make secure citizen development not just possible, but effortless.

Dual-Layer Locking: Redis for Speed, PostgreSQL for Reliability

Mon, 02 Jun 2025 00:00:00 GMT

Two users withdraw money from the same account at exactly the same moment. Your system processes both requests, your database balance goes negative, and you wake up to an incident report. Sound familiar?

What if you could combine Redis’s millisecond response times with PostgreSQL’s bulletproof consistency? This dual-layer locking pattern does exactly that, giving you both speed and safety.

In this post, we’ll explore a pattern that combines Redis’s speed with PostgreSQL’s reliability to prevent race conditions at scale.

The Problem: Race Conditions at Scale

Consider this scenario: 100 concurrent requests to update the same user account. Without proper locking:

---
alt: "Flowchart showing three requests reading and writing the same balance, resulting in an incorrect final value."
---
graph TD
    A[Request 1: Update Balance] -->|Reads £100| DB[(Database)]
    B[Request 2: Update Balance] -->|Reads £100| DB
    C[Request 3: Update Balance] -->|Reads £100| DB
    A -->|Writes £80| DB
    B -->|Writes £50| DB
    C -->|Writes £120| DB
    D[Final Balance: £120] -->|Should be £50| DB
    style D fill:#f96,stroke:#333,stroke-width:2px,color:#fff,font-weight:bold

Figure 1: Race condition scenario with concurrent balance updates.

The last write wins, and you’ve lost £70. In production, the ‘thundering herd’ problem where many processes try to acquire a lock at once can make things even worse by overwhelming your system.

The Solution: Dual-Layer Defense

Think of it like securing a bank vault, you need both the outer perimeter (Redis) and the inner vault door (PostgreSQL):

---
alt: "Flowchart showing a request passing through Redis lock, PostgreSQL transaction, and safe update steps."
---
graph LR
    A[Incoming Request] -->|Step 1| B[Redis Lock<br/>Fast Distribution]
    B -->|Step 2| C[PostgreSQL Transaction<br/>Data Integrity]
    C -->|Step 3| D[Safe Update]

    classDef redis fill:#ffe6e6,stroke:#cc0000,stroke-width:2px
    classDef postgres fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    classDef success fill:#e6ffe6,stroke:#00cc00,stroke-width:2px

    class B redis
    class C postgres
    class D success

Figure 2: Dual-layer defence—Redis for fast distribution, PostgreSQL for data integrity.

Layer 1: Redis Distributed Lock

Coordinates across services in <1ms (learn more)
Prevents stampeding herd problems (many processes trying to acquire the same lock)
Auto-expires to prevent deadlocks

Layer 2: PostgreSQL Row Lock

Guarantees ACID compliance
Prevents partial updates
Handles complex transactions safely

What is Optimistic Locking?

Optimistic locking is a technique that allows multiple transactions to proceed without locking resources up front. Instead, each transaction checks whether the data has changed before committing. If another transaction has modified the data, the update fails, and the process can retry. This helps catch rare edge cases where data changes between read and write.

Production-Ready Implementation

---
alt: "Flowchart showing the dual-layer locking process, including Redis lock acquisition, DB transaction, version check, and success/error paths."
---
flowchart TD
    A[Request] --> B{Redis Lock?}
    B -->|Acquired| C[DB Transaction]
    B -->|Failed| D[Exponential Backoff]
    D --> E{Retry?}
    E -->|Yes| B
    E -->|No| F[Return Error]
    C --> G[SELECT FOR UPDATE]
    G --> H[Version Check]
    H --> I{Valid?}
    I -->|Yes| J[Update & Commit]
    I -->|No| K[Rollback & Retry]
    J --> L[Release Lock]
    K --> D
    L --> M[Success]

    classDef redis fill:#ffe6e6,stroke:#cc0000,stroke-width:2px
    classDef postgres fill:#e6f3ff,stroke:#0066cc,stroke-width:2px
    classDef success fill:#e6ffe6,stroke:#00cc00,stroke-width:2px
    classDef error fill:#ffe6cc,stroke:#ff6600,stroke-width:2px

    class B,D,L redis
    class C,G,H,I,J,K postgres
    class M success
    class E,F error

Figure 4: Dual-layer locking process from request to success or error.

Which can be implemented in code like this:

// TransferRequest defines the structure for a money transfer operation
interface TransferRequest {
  fromAccountId: string;
  toAccountId: string;
  amount: number;
}

class AccountService {
  async transferMoney({ fromAccountId, toAccountId, amount }: TransferRequest) {
    // CRITICAL: Always acquire locks in consistent order to prevent deadlocks
    const [firstId, secondId] = [fromAccountId, toAccountId].sort();
    const lockKey = `transfer:${firstId}:${secondId}`;

    // Acquire a distributed lock in Redis
    const lock = await this.acquireRedisLock(lockKey);

    try {
      await this.db.query('BEGIN');

      // Lock rows in same order as Redis locks
      const accounts = await this.db.query(
        `
        SELECT id, balance, version 
        FROM accounts 
        WHERE id IN ($1, $2) 
        ORDER BY id
        FOR UPDATE
      `,
        [firstId, secondId]
      );

      const fromAccount = accounts.rows.find((a) => a.id === fromAccountId);
      const toAccount = accounts.rows.find((a) => a.id === toAccountId);

      if (fromAccount.balance < amount) {
        throw new InsufficientFundsError(
          `Balance: ${fromAccount.balance}, Required: ${amount}`
        );
      }

      // Atomic updates with optimistic locking
      const results = await Promise.all([
        this.db.query(
          `
          UPDATE accounts 
          SET balance = balance - $1, version = version + 1, updated_at = NOW()
          WHERE id = $2 AND version = $3
        `,
          [amount, fromAccountId, fromAccount.version]
        ),

        this.db.query(
          `
          UPDATE accounts 
          SET balance = balance + $1, version = version + 1, updated_at = NOW()
          WHERE id = $2 AND version = $3  
        `,
          [amount, toAccountId, toAccount.version]
        ),
      ]);

      // Verify both updates succeeded (optimistic lock check)
      if (results.some((result) => result.rowCount === 0)) {
        throw new ConcurrentModificationError(
          'Account was modified by another transaction'
        );
      }

      await this.db.query('COMMIT');
    } catch (error) {
      await this.db.query('ROLLBACK');
      throw error;
    } finally {
      // Always release the Redis lock
      await this.releaseRedisLock(lock);
    }
  }

  // Acquire a distributed lock in Redis with retries and exponential backoff
  private async acquireRedisLock(
    key: string,
    timeoutMs = 30000
  ): Promise<Lock> {
    const token = generateUUID();
    const maxRetries = 3;
    let attempt = 0;

    while (attempt < maxRetries) {
      const acquired = await this.redis.set(key, token, 'PX', timeoutMs, 'NX');

      if (acquired) {
        return { key, token, timeoutMs };
      }

      // Exponential backoff: 50ms, 100ms, 200ms
      const delay = 50 * Math.pow(2, attempt);
      await new Promise((resolve) => setTimeout(resolve, delay));
      attempt++;
    }

    throw new LockAcquisitionError(
      `Failed to acquire lock after ${maxRetries} attempts`
    );
  }

  // Release the Redis lock using an atomic Lua script
  private async releaseRedisLock({ key, token }: Lock): Promise<void> {
    // Lua script ensures atomic check-and-delete
    const script = `
      if redis.call("GET", KEYS[1]) == ARGV[1] then
        return redis.call("DEL", KEYS[1])
      else
        return 0
      end
    `;

    await this.redis.eval(script, 1, key, token);
  }
}

Critical Design Patterns

1. Lock Ordering Prevents Deadlocks

---
alt: "Sequence diagram showing two transfers acquiring locks in the same order to prevent deadlocks."
---
sequenceDiagram
    participant T1 as Transfer A→B
    participant T2 as Transfer B→A
    participant Redis as Redis
    participant DB as PostgreSQL

    Note over T1,T2: Both sort account IDs: A, B
    T1->>Redis: Lock "transfer:A:B"
    T2->>Redis: Lock "transfer:A:B"
    Note over T2: Waits for T1
    T1->>DB: Lock accounts A, B (sorted order)
    T1->>DB: Complete transfer
    T1->>Redis: Release lock
    T2->>Redis: Acquire lock "transfer:A:B"
    T2->>DB: Lock accounts A, B
    T2->>DB: Complete transfer

Figure 3: Consistent lock ordering prevents deadlocks.

Why this matters: Without consistent ordering, Transfer A→B and Transfer B→A can deadlock each other.

2. Exponential Backoff Handles Contention

// Bad: Hammers Redis with retries (thundering herd problem)
for (let i = 0; i < 100; i++) {
  if (await tryLock()) break;
  // Retry immediately - creates thundering herd
}

// Good: Exponential backoff with jitter
const delay = Math.min(
  baseDelay * Math.pow(2, attempt) + Math.random() * 100,
  maxDelay
);

3. Optimistic Locking Catches Edge Cases

Even with locks in place, version checks help catch rare edge cases where another transaction modifies the row between reads and writes:

// This will fail if another transaction modified the row
UPDATE accounts
SET balance = balance - $1, version = version + 1
WHERE id = $2 AND version = $3  -- This condition fails if row was modified

Error Handling That Actually Works

class TransferService {
  async transferWithRetry(request: TransferRequest): Promise<TransferResult> {
    const maxRetries = 3;
    let lastError: Error;

    for (let attempt = 1; attempt <= maxRetries; attempt++) {
      try {
        return await this.transferMoney(request);
      } catch (error) {
        lastError = error;

        // Don't retry business logic errors
        if (
          error instanceof InsufficientFundsError ||
          error instanceof InvalidAccountError
        ) {
          throw error;
        }

        // Retry on infrastructure failures
        if (
          error instanceof LockAcquisitionError ||
          error instanceof ConcurrentModificationError
        ) {
          if (attempt === maxRetries) break;

          const delay = this.calculateBackoff(attempt);
          await this.sleep(delay);
          continue;
        }

        // Unknown error - don't retry
        throw error;
      }
    }

    throw new MaxRetriesExceededError(
      `Failed after ${maxRetries} attempts`,
      lastError
    );
  }

  private calculateBackoff(attempt: number): number {
    // Exponential backoff with jitter: 100ms, 200ms, 400ms
    const base = 100 * Math.pow(2, attempt - 1);
    const jitter = Math.random() * 100;
    return Math.min(base + jitter, 5000); // Cap at 5 seconds
  }
}

When NOT to Use This Pattern

This pattern adds complexity. Don’t use it when:

Low contention scenarios: If conflicts happen <1% of the time, simpler optimistic locking may suffice
Single-service operations: If you’re not distributed, PostgreSQL locks alone are often enough
Read-heavy workloads: This pattern optimises writes; reads don’t need dual locking
Batch operations: For bulk updates, consider queue-based processing instead

Alternative approaches:

Event sourcing: For audit-heavy domains
CQRS: When read/write patterns differ significantly
Database partitioning: When you can shard by tenant/region

Monitoring and Observability

Track these metrics to catch problems early:

// Key metrics to monitor
interface LockingMetrics {
  lockAcquisitionTime: Histogram; // Should be <10ms p99
  lockHoldTime: Histogram; // Should be <100ms p99
  lockContentionRate: Counter; // Failures due to contention
  deadlockCount: Counter; // Should be zero
  optimisticLockFailures: Counter; // Version conflicts
}

// Alert on these conditions
if (lockAcquisitionTime.p99 > 50) {
  alert('High Redis lock contention');
}

if (deadlockCount > 0) {
  alert('Deadlock detected - check lock ordering');
}

Conclusion

Dual-layer locking isn’t just about preventing race conditions—it’s about building systems that stay fast under load while maintaining strict consistency guarantees.

This pattern powers financial transactions, inventory management, and any system where correctness and performance both matter. The complexity pays for itself when you’re processing millions of pounds and can’t afford to be wrong.

Impact Through Empowerment: Contributing Without Direct Participation

Mon, 24 Mar 2025 00:00:00 GMT

In my career, I have realised that the best companies celebrate each success in a genuine, informal way. They avoid contrived rituals that can feel like an artificial façade.

In these organisations, individuals willingly help colleagues reach shared goals.

Take Janet from my local Parkrun, for example.

Although she does not compete, she choose to find the time to cheer on each participant to achieve their goals… even on cold Saturday mornings.

Celebrating success without ceremony creates an environment where everyone feels genuinely valued.

This approach is far more natural than forcing people to attend mandatory meetings or follow rigid procedures.

From my experience, an open approach, such as using a Slack channel for professional and personal achievements which is open to everyone, works better than overly formal recognition schemes in which it can be harder to contribute and share.

Once people embrace this method, openness becomes a habit. Employees feel inspired to share their accomplishments on their terms, promoting a positive organisational culture.

From Static to Stateful: Revolutionising AI Communication with the MCP Protocol

Mon, 10 Mar 2025 00:00:00 GMT

MCP standardises how AI applications communicate with external systems using JSON‑RPC 2.0 over stateful, bidirectional connections. Unlike standard JSON‑RPC, which treats every request independently, MCP augments the protocol by embedding session tokens and context IDs into each message.

This enhancement provides state, enabling advanced, multi‑step interactions and dynamic module loading at runtime. Such a stateful design is essential for modern, agile AI systems.

Imagine an AI‑powered payments system that dynamically integrates a new fraud detection module during operation. MCP allows the system to load this module on the fly without a full redeployment, provided the server supports dynamic module loading.

In this post, we’ll explore the MCP protocol, its core components, and how it compares to traditional REST and GraphQL APIs.

Core Components

Advantages:

Flexibility & Modularity: The clear separation into client, server, and protocol components enables dynamic integration and iterative enhancements.
Advanced Capabilities: The stateful design supports multi‑step operations and real‑time context tracking.

Disadvantages:

Increased Complexity: The extra architectural layers and stateful interactions require careful design and a steeper learning curve.
Implementation Overhead: The need for sophisticated session management introduces additional development overhead, though it unlocks features that static APIs cannot provide.

flowchart LR
    subgraph "MCP Core Components"
        direction LR
        A["Client<br/>(Requests & Callbacks)"] --- C
        C --- B["Server<br/>(Tools & Resources)"]
        C["Protocol Layer<br/>(JSON-RPC 2.0+)"]

        C -.-> D[("Session<br/>Tokens")]
        C -.-> E[("Context<br/>IDs")]
    end

    A <===> |"Stateful<br/>Bidirectional<br/>Communication"| B

    classDef component fill:#4285f4,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef token fill:#fbbc05,stroke:#333,stroke-width:1px,color:#333,rounded:true

    class A,B,C component
    class D,E token

    style A font-weight:bold
    style B font-weight:bold
    style C font-weight:bold

Technical Deep Dive

State Management

MCP’s implementation of statefulness builds on the foundation of JSON‑RPC. While standard JSON‑RPC treats each request as stateless, MCP embeds unique session tokens and context IDs into every message. This approach allows the server to:

Establish a session: The server issues a session token and context ID on connection.
Maintain context: Subsequent requests carry these identifiers, allowing the server to track multi‑step interactions and adjust responses based on accumulated context.

State Management Interaction Diagram

sequenceDiagram
    autonumber
    participant Client
    participant Server

    rect rgb(240, 248, 255)
    Note over Client,Server: Session Initialization
    Client->>Server: Initiate Connection (JSON‑RPC Request)
    Server-->>Client: Respond with Session Token & Context ID
    end

    rect rgb(245, 245, 245)
    Note over Client,Server: Stateful Interaction
    Client->>Server: Request with Session Token & Context ID
    alt Context Valid
        Server-->>Client: Response with Updated Context
        Client->>Server: Follow-up Request (with updated context)
        Server-->>Client: Final Result (context-aware)
    else Context Invalid
        Server-->>Client: Error: Invalid Context
    end
    end

Security

MCP’s security is built on a layered approach, incorporating additional steps to safeguard dynamic interactions and state management.

Security Layers Diagram

flowchart TD
    A[Client Request] --> B[JSON‑RPC Message]
    B --> |"Embedded Security"| C{"Session Token,<br/>Context ID &<br/>Auth Token"}
    C --> D[TLS Encryption]
    D --> E[Authentication<br/>& Authorization]
    E --> F[Token Validation]
    F --> G[Access Control]
    G --> H[Secure Processing]

    classDef process fill:#4285f4,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef security fill:#0f9d58,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef tokens fill:#fbbc05,stroke:#333,stroke-width:1px,color:#333,rounded:true

    class A,B,H process
    class D,E,F,G security
    class C tokens

Error Handling

MCP integrates robust error handling mechanisms to ensure reliable interactions even in complex, stateful sessions.

Error Handling Sequence Diagram

sequenceDiagram
    autonumber
    participant Client
    participant Server

    Client->>Server: Function call request (with session token)
    alt Error Scenario
        Server--x Client: Error Response
        Note over Client,Server: Error Code, Message & Recovery Hints
        Client->>Server: Retry with Error Handling Strategy
        alt Recovery Successful
            Server-->>Client: Success Response
        else Persistent Error
            Server--x Client: Escalated Error With Diagnostics
        end
    else Success Scenario
        Server-->>Client: Success Response with Updated State
    end

Tools, Prompts, and Resources

MCP differentiates among three types of operations:

Tools: RPC methods that produce side effects (e.g., creating or updating resources) and require explicit confirmation.
Prompts: Predefined message templates that guide AI actions without causing side effects.
Resources: Read‑only endpoints that provide data access.

flowchart TD
    A[MCP Server] --> B
    A --> C
    A --> D

    subgraph Operations
        B["⚙️ Tool Operations<br/><i>Side Effects</i><br/><small>create, update, delete</small>"]
        C["⚙️ Prompts<br/><i>No Side Effects</i><br/><small>templates, guidance</small>"]
        D["⚙️ Resources<br/><i>Read-Only</i><br/><small>data access, query</small>"]
    end

    B --> |"Requires"| E[Explicit Confirmation]
    C --> |"Provides"| F[Action Guidance]
    D --> |"Returns"| G[Data Only]

    classDef server fill:#4285f4,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef operation fill:#34a853,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef effect fill:#fbbc05,stroke:#333,stroke-width:1px,color:#333,rounded:true

    class A server
    class B,C,D operation
    class E,F,G effect

Comparisons to REST/GraphQL

flowchart LR
    subgraph REST/GraphQL
        A1[Stateless]
        A2[Resource-centric]
        A3[Fixed Endpoints]
    end

    subgraph MCP
        B1[Stateful]
        B2[Operation-centric]
        B3[Dynamic Tools]
    end

    subgraph Communication
        C1[REST: One-way HTTP]
        C2[GraphQL: Query-focused]
        C3[MCP: Bidirectional with Context]
    end

    subgraph Integration
        D1[REST/GraphQL: Static Integration]
        D2[MCP: Runtime Extension]
    end

    REST/GraphQL -.-> Communication
    MCP -.-> Communication
    REST/GraphQL -.-> Integration
    MCP -.-> Integration

    classDef restStyle fill:#ea4335,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef mcpStyle fill:#4285f4,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef compareStyle fill:#fbbc05,stroke:#333,stroke-width:1px,color:#333,rounded:true

    class A1,A2,A3,C1,C2,D1 restStyle
    class B1,B2,B3,C3,D2 mcpStyle
    class REST/GraphQL,MCP,Communication,Integration compareStyle

Conclusion

MCP represents a significant evolution from static, one‑off API integrations to dynamic, stateful connections that adapt in real time. Its three‑element architecture, comprising the client, server, and protocol, enables advanced features like dynamic tool integration and context‑aware interactions that standard JSON‑RPC cannot support due to its stateless nature.

flowchart TD
    M[MCP Protocol] --> A & B & C

    subgraph "Key Capabilities"
        A[Stateful<br/>Communication]
        B[Dynamic Tool<br/>Integration]
        C[Context-Aware<br/>Interactions]
    end

    subgraph "Implementation Benefits"
        D[Real-time<br/>Adaptability]
        E[Enhanced<br/>Security]
        F[Error<br/>Resilience]
        G[Operational<br/>Flexibility]
    end

    A --> D & E
    B --> F & G
    C --> D & G

    classDef main fill:#4285f4,stroke:#333,stroke-width:2px,color:white,rounded:true
    classDef capability fill:#34a853,stroke:#333,stroke-width:1px,color:white,rounded:true
    classDef benefit fill:#fbbc05,stroke:#333,stroke-width:1px,color:#333,rounded:true
    classDef group fill:none,stroke:#ddd,stroke-width:1px,color:#666

    class M main
    class A,B,C capability
    class D,E,F,G benefit

While MCP may not yet be as mature as REST or GraphQL, its enhancements, especially in state management, security, error handling, and bidirectional communication, position it as a promising protocol for the future of agentic AI integration.

Organisations must balance their dynamic capabilities against the technology’s increased complexity and evolving maturity.

Multi-Agent Patterns: A Practical Guide

Wed, 05 Mar 2025 00:00:00 GMT

AI applications are shifting from monolithic large language models to modular, multi-agent systems—a transformation that enhances performance, flexibility, and maintainability.

In my talk about AI Agents last September, I said AI agents would become increasingly popular. Today, we see this shift happening across industries. By breaking down complex tasks into specialised components, engineers can design smarter, more scalable AI workflows.

Analogy: Think of multi-agent systems like a well-coordinated orchestra. Each musician (agent) has a specific role, and together, they create a harmonious performance. In software, this means dividing complex problems into manageable, specialised parts that work in concert.

In this guide, we’ll explore four key multi-agent patterns, using travel booking as an example. You’ll learn how to choose the right pattern for your application, implementation strategies, and error-handling techniques to build robust multi-agent AI systems.

The Domain Challenge

While our examples focus on travel applications, the challenges addressed here are common across many industries:

Complex User Interactions

Interpreting natural language preferences
Processing dates, locations and multi-step workflows

Sequential Processing Requirements

Searching and filtering (e.g. flight searches)
Selecting seats, ancillary services, payment processing, and final confirmation

External System Integration

Interacting with airline reservation systems, payment gateways, hotel PMS, etc.

Domain Expertise Requirements

Understanding fare rules, pricing optimisation, route planning, and travel regulations

Below is a mermaid diagram that visualises these challenges:

Mermaid diagram: Visualising the main domain challenges in travel booking applications.

graph TD
A[Domain Challenges] --> B[Complex User Interactions]
A --> C[Sequential Processing]
A --> D[External Integration]
A --> E[Domain Expertise]
B --> F[Natural Language]
B --> G[Constraints]
C --> H[Search]
C --> I[Booking]
D --> J[APIs & Payment]
E --> K[Pricing]
E --> L[Routing]

Pattern 1: Single Agent Workflows

Summary: Use a single agent for focused, well-defined tasks where boundaries are clear and minimal state management is required. This pattern is fast, simple, and ideal for stateless queries.

Implementation Details

The Single Agent Workflow pattern is ideal for focused, well-defined tasks. Here’s a simplified example of a flight search agent:

from pydantic import BaseModel
from typing import List, Optional
from datetime import date

class FlightDetails(BaseModel):
    flight_number: str
    price: float
    origin: str
    destination: str
    departure_date: date
    arrival_date: date
    duration_hours: float

class FlightSearchResult(BaseModel):
    found_flights: List[FlightDetails]
    best_flight: Optional[FlightDetails] = None
    explanation: str

class FlightSearchAgent:
    def __init__(self, model: str):
        self.system_prompt = """You are a flight search expert. Your job is to:
        1. Search available flights using the search_flights tool
        2. Analyze the results to find the best option based on price and duration
        3. Provide a clear explanation of why you chose that flight
        """
        # ... initialization code ...

    async def search_flights(self, origin: str, destination: str, date: date) -> List[FlightDetails]:
        # Implementation to search flights
        pass

When to Use Single Agent Workflows

Single Agent Workflows are most effective when:

Task Boundaries are Clear:
- E.g. flight searches with specific criteria, seat selection, or baggage allowance calculations.
Minimal State Management is Required:
- Ideal for one-off, stateless queries and independent transactions.
Performance is Critical:
- Such as in real-time pricing updates or immediate availability checks.

Implementation Considerations

Aspect	Consideration	Example
Input Validation	Ensure all required parameters are provided	Check date formats, airport codes
Error Handling	Comprehensive error catching is essential	Handling API timeouts, invalid responses
Response Format	Structure responses for easy consumption	Use Pydantic models for type safety
Monitoring	Track performance and success rates	Log response times and error rates

Code Example: Error Handling

Robust error handling is crucial. The example below demonstrates how to implement retries with exponential backoff:

class FlightSearchError(Exception):
    """Base exception for flight search errors"""
    pass

class InvalidSearchCriteriaError(FlightSearchError):
    """Raised when search criteria is invalid"""
    pass

async def search_with_retries(self, criteria: FlightSearchCriteria, max_retries: int = 3) -> List[FlightSearchResult]:
 retry_count = 0
    while retry_count < max_retries:
        try:
            return await self._perform_search(criteria)
        except FlightSearchError as e:
 retry_count += 1
            if retry_count == max_retries:
                raise
            await asyncio.sleep(1 * retry_count)  # Exponential backoff

A corresponding sequence diagram visualises the process:

Mermaid sequence diagram: Flight search agent workflow from user input to curated options.

sequenceDiagram
 participant U as User
 participant A as Flight Search Agent
 participant D as Flight Database
 participant P as Price Optimiser

 U->>A: Submit search criteria
 A->>A: Validate input
 A->>D: Query flights
 D-->>A: Raw flight data
 A->>P: Optimise pricing
 P-->>A: Price recommendations
 A->>A: Apply business rules
 A-->>U: Curated flight options

Pattern 2: Agent Delegation

Summary: Delegate responsibilities to specialist agents, coordinated by a controller. This pattern increases modularity and scalability, but requires robust error handling and coordination.

Implementation Architecture

The Agent Delegation pattern uses a controller agent to coordinate specialized delegate agents. Here’s a simplified example:

from typing import List, Optional
from datetime import date
from pydantic import BaseModel

class TravelPlan(BaseModel):
    outbound_flight: FlightDetails
    return_flight: Optional[FlightDetails]
    hotel_recommendations: List[str]
    activity_suggestions: List[str]
    total_budget: float

# Flight search agent (delegate)
flight_search_agent = Agent(
    model="gpt-4",
    result_type=List[FlightDetails],
    system_prompt="You are a flight search specialist. Find the best flights matching the given criteria."
)

# Travel planner agent (controller)
travel_planner_agent = Agent(
    model="gpt-4",
    result_type=TravelPlan,
    system_prompt="""You are a travel planning expert. Your job is to:
    1. Use the flight_search tool to find suitable flights
    2. Suggest hotels and activities at the destination
    3. Calculate total budget including flights and estimated other expenses
    """
)

Benefits and Trade-offs

Aspect	Benefits	Trade-offs
Modularity	Clear division of responsibilities	Increased system complexity
Scalability	Easy to add specialist agents	More points of failure
Maintenance	Agents can be updated independently	Requires robust error handling
Performance	Parallel processing is possible	Additional coordination overhead

Error Handling Strategies

Coordinated error handling is key. For example:

class TravelPlanningError(Exception):
    """Base exception for travel planning errors"""
    pass

class AgentCoordinationError(TravelPlanningError):
    """Raised when agent coordination fails"""
    pass

class TravelPlannerAgent:
    async def create_travel_plan_with_fallback(self, origin: str, destination: str, dates: Dict[str, datetime], preferences: Dict[str, any]) -> TravelPlan:
        try:
            return await self.create_travel_plan(origin, destination, dates, preferences)
        except FlightSearchError:  # Fallback to alternative dates or routes
            return await self._handle_flight_search_failure(origin, destination, dates, preferences)
        except HotelSearchError:  # Fallback to alternative accommodations
            return await self._handle_hotel_search_failure(destination, dates, preferences)
        except ActivityPlanningError:  # Create a plan without activities
            return await self._create_basic_travel_plan(origin, destination, dates, preferences)

Implementation Considerations

Agent Communication: Use standardised message formats and implement retry mechanisms.
State Management: Track the progress of each sub-agent and maintain booking consistency.
Monitoring and Logging: Monitor performance metrics and log coordination events.
Testing: Unit test individual agents and integration test their interactions.

Pattern 3: Programmatic Agent Hand-off

Summary: Explicitly transition between specialised agents for each workflow step. This pattern offers flexibility and strong error recovery, but increases complexity and requires careful state management.

Implementation Details

This pattern manages explicit transitions between specialized agents:

# Flight search agent
flight_search_agent = Agent(
    model="gpt-4",
    result_type=List[FlightDetails],
    system_prompt="Find available flights matching the search criteria."
)

# Seat selection agent
seat_selection_agent = Agent(
    model="gpt-4",
    result_type=SeatPreference,
    system_prompt="""Help users select their seat based on their preferences.
    - Rows 1, 14, and 20 have extra legroom
    - Seats A and F are window seats
    """
)

# Payment processing agent
payment_agent = Agent(
    model="gpt-4",
    result_type=PaymentResult,
    system_prompt="Process payment and generate booking confirmation."
)

async def process_booking():
    # Step 1: Search for flights
    flight = await search_flights(usage=usage, origin="SFO", destination="JFK", travel_date=date(2024, 5, 1))

    # Step 2: Select seat
    seat = await select_seat(usage=usage, preferences="window seat with extra legroom")

    # Step 3: Process payment
    payment_result = await process_payment(usage=usage, flight=flight, seat=seat, payment_info="Credit card ending in 1234")

Error Handling Strategies

Key aspects include state-based recovery, graceful degradation, and robust monitoring. Below is a mermaid state diagram illustrating possible transitions:

Mermaid state diagram: State transitions in a programmatic agent hand-off workflow.

stateDiagram-v2
 [*] --> INITIATED
 INITIATED --> FLIGHTS_SELECTED: Flight search success
 INITIATED --> FAILED: Search error
 FLIGHTS_SELECTED --> SEATS_SELECTED: Seat selection success
 FLIGHTS_SELECTED --> FAILED: Selection error
 SEATS_SELECTED --> PAYMENT_PROCESSING: Payment initiated
 PAYMENT_PROCESSING --> COMPLETED: Payment success
 PAYMENT_PROCESSING --> FAILED: Payment error
 FAILED --> [*]
 COMPLETED --> [*]

Usage Example

An API endpoint that processes a booking using the orchestrator:

@router.post("/bookings")
async def create_booking(booking_request: BookingRequest, orchestrator: BookingOrchestrator = Depends(get_orchestrator)) -> BookingResponse:
 context = BookingContext(
        booking_id=generate_booking_id(),
        state=BookingState.INITIATED,
        user_id=booking_request.user_id,
        search_criteria=booking_request.search_criteria,
        created_at=datetime.utcnow(),
        updated_at=datetime.utcnow()
    )

 final_context = await orchestrator.process_booking(context)

    return BookingResponse(
        booking_id=final_context.booking_id,
        status=final_context.state.value,
        error=final_context.error,
        flights=final_context.flights,
        seats=final_context.selected_seats,
        payment_info=final_context.payment_info
    )

Implementation Considerations

State Management: Persist the BookingContext (e.g. using a database).
Retry Strategies: Use exponential backoff where necessary.
Transaction Boundaries: Maintain atomicity of state transitions.
Monitoring: Record state transition times and error counts.

Pattern 4: Graph-based Control Flow

Summary: Use a graph structure to manage complex workflows with many possible transitions. This pattern is highly flexible and scalable, but requires comprehensive monitoring and error recovery.

Implementation Details

This pattern uses a graph structure to manage complex workflows:

@dataclass
class BookingState:
    """State maintained throughout the booking process."""
    origin: str
    destination: str
    travel_date: date
    selected_flight: Optional[FlightDetails] = None
    selected_seat: Optional[SeatPreference] = None
    payment_info: Optional[str] = None
    booking_confirmed: bool = False
    agent_messages: Dict[str, List[ModelMessage]] = field(default_factory=dict)

@dataclass
class SearchFlights(BaseNode[BookingState]):
    """Search for available flights."""
    async def run(self, ctx: GraphRunContext[BookingState]) -> BaseNode[BookingState] | End[None]:
        result = await flight_search_agent.run(
            f"Find flights from {ctx.state.origin} to {ctx.state.destination} on {ctx.state.travel_date}"
        )
        ctx.state.selected_flight = result.data[0]
        return SelectSeat()

@dataclass
class SelectSeat(BaseNode[BookingState]):
    """Select seat preferences for the flight."""
    async def run(self, ctx: GraphRunContext[BookingState]) -> BaseNode[BookingState]:
        result = await seat_selection_agent.run("I'd like a window seat with extra legroom if possible")
        ctx.state.selected_seat = result.data
        return ProcessPayment()

Implementation Considerations

State Design: Define clear transitions and metadata.
Error Recovery: Implement retryable and monitored nodes.
Monitoring and Metrics: Track execution times, transitions, and errors.

Below is a mermaid diagram representing a typical graph-based workflow:

Mermaid state diagram: Graph-based control flow for a travel booking system.

stateDiagram-v2
 [*] --> SearchFlights
 SearchFlights --> SelectFlights: Flights found
 SearchFlights --> [*]: Error
 SelectFlights --> SearchHotels: Flights selected
 SelectFlights --> [*]: Error
 SearchHotels --> SelectHotels: Hotels found
 SearchHotels --> [*]: Error
 SelectHotels --> SelectSeats: Hotels selected
 SelectSeats --> ProcessPayment: Seats selected
 SelectSeats --> SelectSeats: Retry selection
 ProcessPayment --> [*]: Success/Error

Usage Example

An API endpoint employing a graph-based runner might look like this:

@router.post("/bookings/graph")
async def create_booking_graph(booking_request: BookingRequest, runner: BookingGraphRunner = Depends(get_graph_runner)) -> BookingResponse:
 initial_state = BookingGraphState(
        booking_id=generate_booking_id(),
        user_id=booking_request.user_id,
        origin=booking_request.origin,
        destination=booking_request.destination,
        dates=booking_request.dates,
        preferences=booking_request.preferences,
        agent_messages={}
    )
 final_state = await runner.run(initial_state)
    return BookingResponse(
        booking_id=final_state.booking_id,
        status="completed" if not final_state.error else "failed",
        error=final_state.error,
        flights=final_state.flights,
        hotels=final_state.hotels,
        selected_seats=final_state.selected_seats,
        payment_info=final_state.payment_info
    )

Choosing the Right Pattern

Comparative Analysis

Aspect	Single Agent	Agent Delegation	Programmatic Hand-off	Graph-based Flow
Complexity	Low	Medium	High	Very High
State Management	Simple	Moderate	Complex	Very Complex
Error Handling	Basic	Coordinated	Programmatic	Comprehensive
Scalability	Limited	Good	Very Good	Excellent
Development Speed	Fast	Medium	Slow	Very Slow
Maintenance	Easy	Moderate	Complex	Very Complex
Flexibility	Limited	Good	Very Good	Excellent
Monitoring	Simple	Moderate	Complex	Very Complex

Real-World Applications

Flight Booking Systems

Example: A multi-city flight booking system using Graph-based Flow to manage complex routing.

Hotel Booking Systems

Example: Using Agent Delegation for optimised hotel search and pricing.

Travel Package Systems

Example: Building dynamic packages via Programmatic Hand-off to sequence flights, hotels and activities.

Travel Itinerary Systems

Example: Employing Single Agent Workflows for fast, AI-powered itinerary optimisation.

Best Practices

Pattern Selection

Begin with the most straightforward pattern that meets your needs.
Consider future scalability and complexity before adding layers.

Implementation

Use strong typing with Pydantic models.
Incorporate comprehensive logging and monitoring.
Build robust error handling and retry strategies.

Testing

Unit test individual agents.
Integration test agent interactions.
Execute end-to-end tests and stress-test state management.

Monitoring

Track performance metrics and state transitions.
Log error rates and adjust retry strategies as needed.

Conclusion

Multi-agent patterns provide a structured and scalable approach to managing complex tasks in modern applications.

Whether you’re implementing a simple, single-agent workflow or a comprehensive, graph-based control flow, the right pattern will ensure clarity, reliability, and an excellent user experience.

By following these best practices and real-world examples from the travel domain, you can build robust and adaptable systems.

What multi-agent pattern will you try in your next project? Share your experiences, questions, or challenges in the comments or reach out to continue the conversation.

When to Use Agentic Systems

Sun, 02 Mar 2025 00:00:00 GMT

Agentic systems, where multiple AI agents collaborate through decision-making and handoffs, shine in specific scenarios but add operational complexity.

In this post, we’ll explore the scenarios where agentic systems are most effective and the challenges you may face when using them.

Anthropic’s guide on building effective agents has become the definitive resource in this space.

Their core advice? Start simple - use the most straightforward solution, and only add complexity when necessary. Agentic systems trade increased latency and cost for improved performance, so validate this tradeoff first.

From hands-on experience building production systems, agentic approaches prove most valuable when:

Tasks require multiple steps: When a single LLM call can’t complete the job (e.g., research → analysis → report generation)
Specialisation matters: When different task aspects need tailored models/prompts (e.g., factual lookup vs. creative writing)
Flexibility is critical: When step sequences can’t be predetermined (e.g., customer service → technical specialist escalation)
Domain complexity demands it: When knowledge needs compartmentalization (e.g., legal + medical expertise in insurance claims)

In my work with clients, optimised single LLM calls with retrieval augmentation often suffice. It’s only when the complexity of the task is such that the agentic approach demonstrably improves outcomes that I recommend adopting agentic patterns.

Design Principles for Effective Agents

Now that we’ve established when agentic systems are appropriate, let’s explore core design principles informed by Anthropic’s research and hard-won implementation lessons:

Simplicity First

Implement the minimal viable system, then iterate. Follow the engineering maxim: “Make it work, make it right, make it fast” - in that order.

Atomic Responsibilities

Each agent should have one clear purpose. Like microservices, maintain a strict separation of concerns (e.g., “Medical Term Explainer” vs “Insurance Policy Parser”).

Complexity with Cause

Add components only when metrics prove their value. Remember: More agents = more failure points + latency.

Transparent Operations

Make the active agent always apparent to users and engineers. Visual indicators (e.g., “[Medical Expert] Analysing scan results…”) build trust and aid debugging.

Failure Resilience

Implement:

Timeouts for all agent interactions
Fallback routes for stuck workflows
Input validation guards
Test edge cases ruthlessly - especially handoff transitions.

Observability Focus

Track:

Handoff success rates
Agent-specific latency
Error hot spots
Completion metrics

Use this data to drive iterative improvements.

TL;DR

Agentic systems aren’t inherently better; they’re tools with specific strengths. The goal isn’t to build the most sophisticated architecture using the latest flavor of AI patterns but the simplest solution for your users’ needs.

Next.js after: Track Analytics and Events Without Slowing Down Your App

Thu, 27 Feb 2025 00:00:00 GMT

Imagine: you run a busy online shop where every moment matters.

Your customers expect the fastest page loads, yet you still require dependable analytics to understand user behaviour and drive informed business decisions.

Relying solely on client-side tracking with tools like PostHog can be fraught with challenges. Many users have ad blockers or privacy extensions that prevent tracking scripts from running, and even when these scripts do execute, they can slow down page rendering and affect conversion rates.

In this post I’ll go over how you can use Next.js 15’s next/after API to handle analytics and events without slowing down your site.

The code used in this post is available on GitHub.

The Problem with Client-side Analytics

Browser Blockers: Ad blockers or privacy extensions often intercept client-side analytics requests, leading to data loss.
Performance Delays: Tracking scripts that run on the client can slow down page loading as they load and execute.
Reliability Issues: Network hiccups or browser inconsistencies might stop analytics data from being sent reliably.

The Solution: Server-side analytics with next/after

Next.js 15 introduces the after function from next/server.

This feature lets you run code after the response has been sent to the client. In practice, this means you can offload non-critical tasks, such as tracking user events,to the background, ensuring your users enjoy a fast, seamless experience.

Benefits include

Bypass Browser Restrictions: Ad blockers don’t affect server-side requests, ensuring you capture every important event.
Faster Load Times: Your users get an immediate response, while the tracking occurs asynchronously.
Improved Reliability: Handling analytics on the server minimises issues caused by network fluctuations or browser quirks.
Improved UX with Immediate Redirects: When a form is submitted, the user is redirected to the success page instantly, without waiting for analytics tracking to complete.
Resource Optimization: Computing resources are better utilized as analytics processing doesn’t block your main user flows.
Optimal for Serverless: Perfect for serverless environments where compute time is limited, as it allows the main response to complete quickly.

Implementing This with PostHog

1. Set Up Your Next.js 15 Project

First, create your Next.js project:

npx create-next-app@latest my-analytics-app

The after function is available by default in Next.js 15 without requiring any experimental flags.

2. Install PostHog and Secure Your Credentials

Install the PostHog Node.js library:

npm install posthog-node

Create a utility file to initialise your PostHog client using server-only environment variables. This is important to ensure your API key remains secure:

// src/lib/posthog.ts
import { PostHog } from 'posthog-node';

export function getPostHogClient() {
  return new PostHog(
    process.env.POSTHOG_KEY as string, // Use a server-side environment variable
    { host: process.env.POSTHOG_HOST as string },
  );
}

Set your credentials in a .env.local file:

POSTHOG_KEY=your_project_api_key
POSTHOG_HOST=https://app.posthog.com

3. Track Events with next/after

Use a server action to defer analytics tracking until after the response is sent. This ensures that the analytics process doesn’t hamper user experience:

// src/app/actions.ts
'use server';

import { getPostHogClient } from '@/lib/posthog';
import { after } from 'next/server';

export async function trackEvent(eventName: string, userId: string) {
  after(async () => {
    try {
      const posthog = getPostHogClient();
      await posthog.capture({
        distinctId: userId,
        event: eventName,
      });

      // Flush the queue using the standard shutdown method
      await posthog.shutdown();
    } catch (error) {
      console.error('Error tracking event:', error);
    }
  });
}

4. Integrate Tracking in Your Components

For instance, if you wish to track a form submission without delaying the user’s redirect, you might do the following:

// src/app/page.tsx
import { trackEvent } from './actions';
import { redirect } from 'next/navigation';

export default function Home() {
  async function handleSubmit(formData: FormData) {
    'use server';
    // Process form data as required
    const email = formData.get('email') as string;

    // This won't block the response - analytics happens in the background
    await trackEvent('form_submitted', email || 'anonymous_user');

    // User is redirected immediately
    redirect('/success');
  }

  return (
    <form action={handleSubmit}>
      <input type="email" name="email" placeholder="Enter your email" required />
      <button type="submit">Submit</button>
    </form>
  );
}

5. Create a Success Page

Create a simple success page that users will be redirected to after form submission:

// src/app/success/page.tsx
import Link from 'next/link';

export default function SuccessPage() {
  return (
    <div>
      <h1>Form Submitted Successfully!</h1>
      <p>Your form has been submitted and analytics are being tracked in the background.</p>
      <Link href="/">Back to Home</Link>
    </div>
  );
}

6. Run the Development Server

npm run dev

7. Submit the Form

Submit the form to see server-side analytics in action. You can check your PostHog dashboard to verify the events are being tracked.

Why This Approach Works

Instant User Experience: The critical page rendering and navigations aren’t delayed by analytics processing.
Accurate Data: Server-side tracking bypasses the limitations of ad blockers and reduces the chance of network-related errors.
Simple Integration: The after function makes managing background tasks straightforward without complex setups.

Using Next.js 15’s cutting-edge features combined with PostHog for server-side analytics, you can optimise your website’s performance and the accuracy of your data, ensuring a seamless experience for your users and more reliable insights for your business.

Event-Driven Architecture: How I Name Events (A Practical Guide With Examples)

Mon, 24 Feb 2025 00:00:00 GMT

Getting event names right in event-driven architecture and Domain-Driven Design (DDD) is essential for clarity, consistency, and scalability. A key decision is using singular or plural terms in event names.

Here’s how I approach it, with examples and reasoning to help you make the best choice.

1. Plural for Collections

Example: v1.payees.created

When an event relates to a collection of entities, the domain should be plural. This clarifies that the event belongs to a broader category and aligns with DDD aggregate roots.

Examples:

v1.payees.created → A new payee was created within the payees aggregate
v1.payments.failed → A payment failure occurred within the payments domain
v1.users.registered → A user was registered in the users domain

2. Singular for Processes

Example: v1.transaction.authorised

If the event describes a business process, workflow, or single entity, use singular to avoid confusion with collections.

Examples:

v1.transaction.authorised → A single transaction was authorised
v1.conversion.trade.settled → A specific trade within a conversion was settled
v1.authentication.failed → A single authentication attempt failed

3. Plural for Sub-Entities

Example: v1.conversion.trades.settled

When an event relates to sub-entities within a domain, pluralization helps differentiate them from the main domain.

Examples:

v1.conversion.trades.settled → Multiple trades within a single conversion were settled
v1.order.items.shipped → Multiple items in an order were shipped
v1.invoice.payments.processed → Multiple payments related to an invoice were processed

4. Versioning at the Beginning

Example: v1.domain.event

Best Practice: Always place the version number at the beginning to make versioning immediately visible and to facilitate version-based routing and filtering.

🚫 Bad Example: payments.failed.v1
✅ Good Example: v1.payments.failed

Why Version First?

Placing the version at the beginning offers several advantages:

Immediate visibility: Version information is the first thing you see
Easier filtering: Systems can quickly filter or route by version without parsing the entire event name
Clearer version transitions: Makes it easier to identify different versions of the same event type
Consistent with API versioning: Aligns with common API versioning practices

Additional Guidelines

Aggregate Naming in DDD:
Aggregates in DDD are typically singular (e.g., Order), but event topics use plural names (e.g., v1.payments.failed).

Clarity vs. Rigidity:
Conventions should adapt to domain needs. For example: v1.conversions.trade.settled might be valid for process collections.

Event Payload Context:
Plurality in event names doesn’t dictate entity count. v1.payments.failed could reference a single payment failure.

Recap

Plural for collections: v1.payees.created, v1.payments.failed
Singular for processes: v1.transaction.authorised, v1.conversion.trade.settled
Plural for sub-entities: v1.conversion.trades.settled, v1.order.items.shipped
Version at the beginning: v1.domain.event

Following these naming conventions will make your event-driven architecture structured, scalable, and aligned with domain modelling principles.

Building TypeScript Libraries' A Guide to Explicit Exports, Tree Shaking, and Common Pitfalls

Thu, 20 Feb 2025 00:00:00 GMT

Building a TypeScript library that is both maintainable and optimised for modern bundlers requires careful consideration of exporting functions, types, and other constructs from your modules.

With several strategies available, from wildcard re-exports to namespaced exports, explicitly named re-exports have emerged as the clear winner. This post explores the alternatives with examples based on an order management system.

We’ll discuss the benefits of exporting types, how to organise multiple entry routes (such as in /src/orders and /src/users), and review the necessary TypeScript configuration options (such as verbatimModuleSyntax) and package.json tweaks. Additionally, we’ll explain how this approach helps prevent circular dependency challenges by enforcing a clear dependency graph.

Building a library

Imagine you are developing a library for an order management system that handles order lookups, cancellations, and creation. Your project is organised into distinct domains:

Orders – containing all logic related to order lookups, cancellations, and creation
Users – managing user profiles, authentication, and related queries

Your public API should enable consumers to import functions and types naturally. For instance, they should be able to write:

import { lookupOrder, cancelOrder } from './orders';
import type { Order } from './orders';
import { getUser, updateUser } from './users';

The challenge is organising your exports to achieve maximum tree-shaking efficiency, avoid naming collisions, and offer a seamless experience for value and type-only imports.

Alternative Exporting Approaches

1. Re-Exporting with export * from

A common method is to re-export everything from a module:

// src/orders/index.ts
export * from './lookup';
export * from './commands';

Pros:

Concise and straightforward
Automatically exposes all exports from the underlying modules

Cons:

While modern bundlers generally handle tree shaking well, wildcard exports can occasionally lead to ambiguity in complex dependency graphs
There is a higher risk of naming collisions if multiple modules export identically named identifiers
Consumers may have less clarity about which symbols form part of the public API

2. Namespace Exports with export * as ns from

Another option is to group exports under a namespace:

// src/orders/index.ts
export * as lookup from './lookup';
export * as commands from './commands';

Consumers would then import and access functions as follows:

import { lookup } from './orders';
lookup.lookupOrder(...);

Pros:

Minimises the risk of naming collisions by encapsulating exports within a namespace
Clearly indicates the origin of each function

Cons:

Introduces an extra level of nesting, which can be less ergonomic
Consumers must always reference the namespace even if they only require one function

3. Explicit Named Re-Exports (Our Recommended Approach)

We recommend explicitly listing the exports in your domain’s index file. For example, in the domain of the order:

// src/orders/index.ts
export { lookupOrder, cancelOrder } from './lookup';
export { createOrder } from './commands';
export type { Order, OrderStatus } from './types';

And in the users domain:

// src/users/index.ts
export { getUser, listUsers } from './queries';
export { updateUser } from './commands';

Pros:

Tree Shaking & Bundle Optimisation: Explicit exports enable bundlers such as Webpack, Rollup, and esbuild to eliminate unused code accurately
Collision Avoidance: Explicitly listing exports prevents accidental name collisions
Consumer Experience: Consumers benefit from a clean and direct import syntax without unnecessary nesting
API Clarity: The public API is clearly defined and straightforward to document
Prevention of Circular Dependencies: By clearly specifying the exports for each module, you enforce a transparent dependency graph
Future-Proofing: This approach works seamlessly with modern ECMAScript Modules and TypeScript’s verbatimModuleSyntax option

Cons:

Requires manual updating of index files when new functions or types are added
Involves slightly more maintenance overhead compared to wildcard re-exports (though modern IDEs can often automate this process)

TypeScript Configuration & Package.json Adjustments

To fully leverage explicit named re-exports in the ESM era, update your tsconfig.json as follows:

{
  "compilerOptions": {
    "module": "esnext",
    "target": "esnext",
    "verbatimModuleSyntax": true,
    // other options...
  }
}

The verbatimModuleSyntax option ensures that TypeScript leaves your ES module import/export statements untouched, essential for effective tree shaking by modern bundlers.

In your package.json, expose your public API entry points:

{
  "name": "my-library",
  "version": "1.0.0",
  "type": "module",
  "exports": {
    "./orders": "./dist/orders/index.js",
    "./users": "./dist/users/index.js"
  },
  "scripts": {
    "build": "tsc"
  }
}

Multiple Entry Routes: A Concrete Example

Assuming your project is structured like this:

src/
  orders/
    lookup.ts         // Contains lookupOrder, cancelOrder, etc.
    commands.ts       // Contains createOrder.
    types.ts          // Contains Order, OrderStatus, etc.
    index.ts          // Explicitly re-exports functions and types.
  users/
    queries.ts        // Contains getUser, listUsers, etc.
    commands.ts       // Contains updateUser.
    index.ts

Exporting Types: Why It Matters

Explicitly exporting types along with functions offers several advantages:

Clarity for Consumers: Exported types allow users to write precise type annotations that match your public API
Avoids Global Namespace Pollution: Consumers import only what they need, keeping the global namespace clean
Enhanced Tree Shaking: Modern bundlers can drop unused type-only imports when using the import type syntax
Consistency: This approach aligns with the broader industry move towards explicit, module-based API definitions
Circular Dependency Prevention: By clearly defining which types are exported, you reduce the risk of circular dependencies

For example, rather than using a wildcard re-export for types:

// Avoid: Not ideal for public libraries
export * from './types';

It is preferable to use explicit type exports:

export type { Order, OrderStatus } from './types';

Conclusion

Explicitly named re-exports remain the most recommended approach for public libraries in 2025. They provide:

Superior tree-shaking and bundle optimisation
A straightforward, unambiguous public API that avoids naming collisions
A smooth consumer experience with direct, idiomatic imports for both values and types
Clear benefits in preventing circular dependency issues by enforcing a well-defined dependency structure
Future-proof compatibility with modern TypeScript options such as verbatimModuleSyntax

Organising your exports explicitly in each domain’s index file for a library with multiple domains, such as orders and users, ensures that your public API is robust, optimised, and easy to maintain as the ecosystem evolves.

Hopefully, this guide helps you build more maintainable, optimised, and future-proof libraries.

Choosing Your React Framework in 2025: Understanding Next.js Trade-offs

Thu, 13 Feb 2025 00:00:00 GMT

I love Next.js. It’s been my framework of choice since my first tiny contribution in 2017.

Yesterday, I joined a team discussion about which React framework to choose. In 2025, that decision is harder than ever, with so many excellent alternatives available.

While Next.js is recommended by React themselves, and is a very popular choice, the not-so-good parts are often left out on platforms like LinkedIn and YouTube.

I plan to write about the good things about Next.js 15 in the future, but here are some challenges I still face using it today.

Next.js 15 and React Server Components (RSC)

It’s not just me who has doubts about (overhyped) React Server Components, which on paper look fantastic, however…

RSC is the most painful new React API in the State of React survey.

Complaints:
- Hard to test.
- Hard to debug.
- Hard to reason about.
- Confusing mental model.
- Seems designed to sell Vercel.
- Some think only Next.js supports RSC.
- More complexity for little benefit. pic.twitter.com/HT5DzXlK3t
— Cory House (@housecor) February 12, 2025

Middleware Issues

Edge Runtime Limitations

Restricted Node.js API usage:
The Next.js Edge Runtime explicitly excludes many native Node.js APIs (such as fs, net, etc.), making tasks like direct database connections or session management impossible in middleware.
Verified by the official documentation:
Vercel’s Edge Middleware Limitations and the Next.js Edge Runtime API Reference.
Authentication compatibility issues:
Third-party libraries that depend on Node.js APIs (for example, MongoDB drivers, Firebase Admin SDK, or Redis clients like ioredis) often face compatibility issues when used in middleware. Several GitHub discussions note struggles with using these libraries in the Edge Runtime.
For example, this GitHub discussion highlights user-reported issues regarding such limitations (the core restrictions are detailed in the official docs).

Good news: Node.js runtime support for middleware is merged! The PR #75624 by the Next.js team enables middleware to run with full Node.js API support, addressing many of the current limitations. Broader discussions on this topic are tracked in GitHub Discussion #71727.

Single Middleware File Restriction

Single entry point:
Next.js supports only one middleware file (typically middleware.ts or middleware.js placed in the project root or within the src directory). This forces developers to consolidate all middleware logic into one file. Although not exhaustively documented in a single article, this constraint is implicit in the official Next.js Middleware docs and has been noted in various user reports. Having to use matches and then manually filter is a horrible developer experience.

Implementation Problems

Inconsistent Execution

Middleware Execution Issues:
In GitHub Issue #58025, users reported that middleware sometimes doesn’t execute as expected in Next.js v14.2.4, occasionally requiring a hard refresh to function correctly. While this isn’t solely a file placement issue, it underscores the importance of proper middleware configuration.
Redirect Behavior in Middleware:
GitHub Issue #59218 discusses problems with middleware redirects, where users experienced unexpected behaviour due to browser caching. This highlights the need for careful implementation and a thorough understanding of middleware behaviour.

Authentication Compatibility

Challenges with authentication libraries:
Libraries like NextAuth.js have noted issues when used in middleware—largely because the Edge Runtime lacks certain Node.js APIs (e.g., Node’s crypto module) required by these libraries.
For more details, refer to the Next.js Edge Runtime API Reference.

Fetch API and Monkey Patching Issues

Global Fetch Modifications

Next.js extends the Web fetch:
Next.js extends the standard Web fetch() API to allow each server-side request to set its persistent caching and revalidation semantics. More details are provided in the Next.js fetch documentation.
Memory leaks and compatibility issues:
Modifications to the global fetch (supporting features like caching and Server Actions) have sometimes led to memory leaks.
A documented example is available in GitHub Issue #64212.
Interference with error handling and third-party libraries:
Monkey-patching fetch can conflict with libraries that rely on or modify it. For example, discussions on Reddit highlight how such modifications may disrupt expected error handling.
See this Reddit thread on monkey-patching fetch.

MSW Integration Challenges

Server-side interception issues:
Many developers have reported challenges with server-side request interception when integrating Mock Service Worker (MSW) with the new App Router in Next.js 13+. These issues are likely due to the restricted Edge Runtime, which lacks Node.js APIs (such as the http/https modules) required for full HTTP interception.
Learn more about MSW on their official documentation.

No Platform Adapter (Unlike Remix, Svelte, and Astro)

There is an initiative called OpenNext, but it appears to be playing catchup. Vercel remains the best platform to host Next.js, so having adapters for other platforms might introduce conflicts of interest.

Backward Compatibility Bloat

Legacy systems maintained alongside new features:
Next.js continues to support legacy systems (e.g., the Pages Router and Babel) even as it introduces new paradigms like the App Router. This coexistence increases overall complexity.
Official Next.js Pages Documentation.
SWC vs. Babel compatibility:
Although Next.js introduced the SWC compiler for performance improvements, Babel compatibility is still maintained—which adds to the bloat.
Next.js 12 Blog on SWC.

Modern Alternatives

Astro for Static Sites

Performance benefits:
Astro claims to load static content up to 40% faster and deliver 90% less JavaScript than Next.js, making it an attractive option for content-heavy sites.
Learn more about Astro.

TanStack Router for SPAs

Superior type safety and flexibility:
TanStack Router offers excellent type safety and flexible routing patterns, and being client-side first means you can sprinkle in server actions with server validation when needed.

Bonus: it uses lightning-quick Nitro and Vite to bundle and deploy your application.

TanStack Framework Comparison.

Remix/React Router 7/?

Should have been number one to me:
Backed by Shopify, with Ryan and Michael leading the way, this would have been my top choice a year ago. I love everything about their approach, from open planning sessions to clear community communication and a commitment to web standards.

However, the merging of Remix and React Router is a bit confusing at best. During this transition period, it’s a tough sell to recommend it to clients. Hopefully, when things stabilize and we have a clearer picture of what’s what it will be worth another look.

Development Experience Issues

Caching Complexities

Inconsistent caching behavior:
Next.js 14’s aggressive caching sometimes led to stale data. Although Next.js 15 disabled caching by default to address these complaints, new issues have emerged in some cases.
See community threads on GitHub discussions.

Architectural Challenges

Hydration mismatches in Server Components:
Blending server and client components can lead to hydration mismatches that are challenging to debug.
Next.js Server Components Documentation.
Mandatory file-based routing quirks:
The requirement that each page have a page.tsx file can be confusing and may lead to accidental bundling of server-only code into client bundles.
Official Next.js Pages Documentation.

Upgrades and Dependency Issues

Dependency conflicts during upgrades:
Some users have experienced issues when migrating between major versions (e.g., from Next.js 14 to 15), especially as many libraries are still catching up with changes introduced in React 19. In some cases, developers have resorted to workarounds like running npm install --legacy-peer-deps to bypass strict peer dependency checks.
See discussion on this workaround in the shadcn/ui repository.

Despite these challenges, Next.js remains a powerful and flexible framework that continues to evolve.

It will improve, and hopefully, one can be seamlessly deployed on many platforms, not just Vercel.

With a huge ecosystem, it’s still my go-to framework.

However, modern alternatives like Astro and TanStack Router may offer more focused solutions for developers who find these pain points too limiting.

Ultimately, the right tool depends on your specific use case and priorities.

Tracking Every Decision in Payments: The Context Pattern Approach

Fri, 31 Jan 2025 00:00:00 GMT

Designing a payment system is like any other software solution I’ve worked on. At first, it appears straightforward, but real-world factors like currency conversions, fees, payee validations and external APIs quickly add complexity.

When something goes wrong, you need to know exactly which step failed and why.

In this post, I’ll explain how the Context Pattern helps solve this by storing all important data in a single object.

Each function returns new information rather than silently modifying the shared state, and the caller explicitly updates the context. This ensures clear data flow, easy debugging and a more maintainable system.

The Problem

Fragmented Logging: Logs end up scattered across multiple functions or services, making it difficult to reconstruct a single payment from start to finish.
Ambiguous State Changes: Without a single data store, you end up with countless parameters or rely on global variables—leading to confusion over which function changed what data and why.
Hard to Add Features: Introducing new requirements like compliance checks or multi-tier fees means revisiting multiple function calls and logs, increasing the chances of errors.
Limited Error Context: If a step fails, you might only see partial details. Recreating an issue becomes guesswork without a full view of the payment data.

A Real-World Scenario Without the Context Pattern

Below is an example of a typical payment flow that returns values from functions, but lacks a single, shared data structure and a proper correlation ID:

// (A) Fetch a conversion rate
async function getConversionRate(
  clientId: string,
  sourceCurrency: string,
  targetCurrency: string
): Promise<number> {
  console.log(
    `[${clientId}] Fetching rate for ${sourceCurrency} to ${targetCurrency}.`
  );
  const rate = 1.23; // Mocked response
  console.log(`[${clientId}] Retrieved rate: ${rate}`);
  return rate;
}

// (B) Process the payment
async function processPayment(
  clientId: string,
  amount: number,
  conversionRate: number
): Promise<{ status: string; transactionId: string }> {
  console.log(
    `[${clientId}] Processing payment of ${amount} at rate ${conversionRate}.`
  );

  if (amount <= 0) {
    throw new Error(`Invalid amount: ${amount}. Cannot process payment.`);
  }

  const transactionId = 'tx-999';
  console.log(
    `[${clientId}] Payment processed with transaction ID ${transactionId}.`
  );

  return { status: 'SUCCESS', transactionId };
}

// (C) Main flow that orchestrates both steps
async function mainPaymentFlow(
  clientId: string,
  sourceCurrency: string,
  targetCurrency: string,
  amount: number
) {
  try {
    const rate = await getConversionRate(
      clientId,
      sourceCurrency,
      targetCurrency
    );
    const paymentResult = await processPayment(clientId, amount, rate);
    console.log(`[${clientId}] Flow complete. Result:`, paymentResult);
  } catch (err) {
    console.error(`[${clientId}] Payment flow error:`, err);
  }
}

// Simulate usage
(async () => {
  await mainPaymentFlow('client-123', 'GBP', 'USD', 200);
})();

What’s lacking here?

No Single Source of Truth: Data is scattered and only passed around ad hoc.
Weak Correlation: clientId is a shared identifier, but logs can become tangled if multiple payments happen simultaneously for the same client.
Harder to Extend: Adding extra steps requires new return values or extra parameters everywhere.

The Context Pattern

The Context Pattern centres on a single object (the context) that holds all important data throughout the payment lifecycle. Each helper function returns the data it computes, and the caller decides how to update the context. No mysterious global changes exist, and each log can attach a unique spanId (or similar).

flowchart LR
 subgraph Context Flow
 Start([Payment Start]) --> Context[Create Context\nspanId, currencies, amount]
 Context --> Rate[Get Rate\nReturns rate]
 Rate --> Fees[Calculate Fees\nReturns fee structure]
 Fees --> Validate[Validate\nReturns validation result]
 Validate --> Process[Process Payment\nReturns result]
 Process --> Store[(Store Context)]
 end

 subgraph Error Handling
 Rate -- Error --> ErrorLog[Log Error + Full Context]
 Fees -- Error --> ErrorLog
 Validate -- Error --> ErrorLog
 Process -- Error --> ErrorLog
 end

import pino from 'pino';

// Base logger for the application
const baseLogger = pino({ level: 'info' });

// Payment result type
interface PaymentResult {
  status: string;
  transactionId: string;
}

// The context that holds all relevant fields in one place
interface PaymentContext {
  spanId: string;
  sourceCurrency: string;
  targetCurrency: string;
  amount: number;
  conversionRate?: number;
  paymentResult?: PaymentResult;
}

// Fetch a conversion rate
async function getConversionRate(context: PaymentContext): Promise<number> {
  const logger = baseLogger.child({
    spanId: context.spanId,
    function: 'getConversionRate',
  });

  if (context.amount <= 0) {
    throw new Error(
      'Amount must be greater than zero before fetching a conversion rate.'
    );
  }

  logger.info(
    `Fetching rate for ${context.sourceCurrency} to ${context.targetCurrency}.`
  );
  const mockRate = 1.23;
  logger.info({ mockRate }, 'Conversion rate retrieved.');
  return mockRate;
}

// Process the payment
async function processPayment(context: PaymentContext): Promise<PaymentResult> {
  const logger = baseLogger.child({
    spanId: context.spanId,
    function: 'processPayment',
  });
  logger.info(
    `Processing payment of ${context.amount} ${context.sourceCurrency}.`
  );

  if (context.amount <= 0) {
    throw new Error(
      `Invalid amount: ${context.amount}. Cannot process payment.`
    );
  }

  const result: PaymentResult = { status: 'SUCCESS', transactionId: 'tx-999' };
  logger.info({ result }, 'Payment processed.');
  return result;
}

// Main flow uses a single context
async function main() {
  const paymentContext: PaymentContext = {
    spanId: 'unique-trace-id-123',
    sourceCurrency: 'GBP',
    targetCurrency: 'USD',
    amount: 200,
  };

  try {
    // 1. Retrieve conversion rate
    const rate = await getConversionRate(paymentContext);
    paymentContext.conversionRate = rate;

    // 2. Process payment
    const paymentRes = await processPayment(paymentContext);
    paymentContext.paymentResult = paymentRes;

    baseLogger.info({ paymentContext }, 'Payment flow completed successfully.');
  } catch (error) {
    // Log both the error and context for better debugging
    baseLogger.error(
      { error, paymentContext },
      'Payment flow encountered an error.'
    );
  }
}

main().catch(console.error);

Advanced Features

Fee Calculation and Rounding

In the real world, payment flows often need multiple fees that must be rounded precisely. You might add a fees field to your context:

interface PaymentFees {
  base: number;
  foreign: number;
  total: number;
}

interface PaymentContext {
  // ...
  fees?: PaymentFees;
}

async function calculateFees(context: PaymentContext): Promise<PaymentFees> {
  const baseFee = Math.round(context.amount * 0.01 * 100) / 100;
  const foreignFee =
    context.sourceCurrency !== context.targetCurrency
      ? Math.round(context.amount * 0.005 * 100) / 100
      : 0;
  const totalFee = baseFee + foreignFee;
  return { base: baseFee, foreign: foreignFee, total: totalFee };
}

You would then do:

context.fees = await calculateFees(context);

Multi-step Validations

If you have a compliance check or KYC rule, you can create a function that takes context and returns a result (or throws an error), so all data flows through the same pattern.

Testing and Persistence

Unit Testing with Mocks

Because each function returns data, testing them in isolation is simple. For example:

describe('calculateFees', () => {
  it('calculates foreign fees correctly', async () => {
    const context: PaymentContext = {
      spanId: 'test',
      sourceCurrency: 'GBP',
      targetCurrency: 'USD',
      amount: 200,
    };
    const fees = await calculateFees(context);
    expect(fees.total).toBe(3); // 2 base + 1 foreign, as an example
  });
});

Long-Running Transactions

If your process is asynchronous or spans multiple stages, you can save the context into a database at certain checkpoints:

async function checkpointContext(context: PaymentContext) {
  await prisma.paymentFlow.upsert({
    where: { spanId: context.spanId },
    update: { metadata: context, lastUpdated: new Date() },
    create: {
      spanId: context.spanId,
      metadata: context,
      status: 'IN_PROGRESS',
    },
  });
}

This allows you to resume or replay from that checkpoint if a later step fails.

Migrating Existing Code

A common question is how to refactor older, scattered logic into this pattern. The most straightforward approach is usually incremental:

// Old approach
function oldProcess(amount: number, currency: string) {
  // ... scattered logic and partial logs
}

// New approach using context
function newProcess(context: PaymentContext) {
  // ... shared context with proper logs
}

You might begin by creating a PaymentContext and gradually modifying existing functions to accept context rather than many parameters. Log calls can be redirected through a child logger with spanId.

Over time, you reduce global variables or repeated parameters and end up with a single context that you can checkpoint, log and debug in one place.

Real-World Pitfalls

Context Object Size

Keep an eye on how large your context becomes. Use references or IDs if you only need to load extra data on demand. Redundant or overly large fields can affect performance.

Long-running Flows

Store partial context at each step if the transaction is split into multiple asynchronous stages. Use an event-driven architecture with an ID that references a stored context in Redis or a database.

Clearing Sensitive Data

Some fields (like card numbers or personal info) must not remain in logs. Use redaction features (for example, in Pino) or remove those fields from the context once you no longer need them.

Comparison with Other Patterns

Command Pattern

Ideal if you want each action to be a discrete command object. It can be more complex than a simple context approach and is not always necessary if you just need straightforward flows.

Event Sourcing

Allows you to replay state changes by storing events. It is powerful for auditing but requires a more elaborate design. The Context Pattern is typically more straightforward for most small to medium payment systems.

Conclusion

Using a context object, passing it through each stage of your payment flow and returning computed values lets you build a highly transparent, adaptable system.

If an error occurs, you can log the entire context, making it far easier to diagnose issues later. This pattern also scales well when you need to introduce fees, compliance checks or multi-step validations since each new function simply takes and returns context-related data.

Whether you stay with the Context Pattern or eventually transition to something like Event Sourcing, you’ll find that starting with a single shared context and a correlation ID ensures your payment code remains both maintainable and easy to evolve.

Chasing the AI God: Why We Worship Every New Model

Wed, 29 Jan 2025 00:00:00 GMT

It’s like a new diet pill has hit the market, promising instant weight loss with no effort, and now everyone’s scrambling to get their hands on it because it’s all over their TikTok.

Some folks are talking about DeepSeek as if it’s the second coming of AI.

The clamour might be because it’s the first serious open non-US model with reasoning capabilities from China.

Most people are confused about which version of DeepSeek they’re using.

Most providers offer a distilled, watered-down version that’ll run anywhere, so you’re not getting the actual full fat version people are talking as to really see its magic, you’d need a small fortune in computing power.

It’s like being promised a rare vintage white wine, only to find the bottle filled with slightly grape-scented water. Same brand, same label, but all the depth and character stripped away, leaving you with a hollow imitation.

The hype suggests it can keep pace with anything OpenAI does, which might sound thrilling if you’re already tired of whatever ChatGPT or its siblings spit out.

But here’s where the alarm bells should start ringing.

We have a strange habit of treating new tech like a shiny toy that must be perfect simply because it’s the latest in line. It’s easy to forget that these models are about as trustworthy as me telling you I won’t pinch your biscuit when you’ve nipped out to the loo.

They make mistakes, they have biases, and they hallucinate with total confidence.

I predict we’re heading for trouble as people want to believe everything AI generates simply because it’s quick. In a society of short attention spans, it gives people the instant gratification they seek, and there’s no time or desire to question it.

Picture this scenario: some fringe political group with more money than sense decides to release an AI that’s even easier to use, has no guardrails, and promises unlimited answers in the blink of an eye.

They hardly restrict it, so you can ask for all the edgy stuff you want without any pesky guidelines. The risk is that people flock to it and gobble up whatever it says just because it’s convenient, fast, and free.

A few million retweets later, you’ve got a swirling vortex of misinformation.

This isn’t some wild conspiracy theory. We already live in an age where people retreat into echo chambers that echo back at them. Throw an unregulated AI into the mix, and you’ve effectively given a free megaphone to anyone who can afford it. Forget about analysing sources or thinking things through. We just want immediate answers that confirm what we already believe.

So yes, DeepSeek and the models that will follow are impressive, and they might do a decent job as long as you keep your wits about you and treat them like any other tool. They’re not wise old sages, nor are they perfect oracles.

The moment we start worshipping them as flawless fountains of truth is the moment we open the door to a future where the biggest loudmouth with the most populist AI wins.

Oh, and about the privacy of the personal information and company intellectual property you’re feeding it…

Lessons from Building AI Products with Stakeholders

Mon, 27 Jan 2025 00:00:00 GMT

Building AI products with stakeholders requires a fundamentally different approach than traditional software development.

From my experience working with AI, success depends not only on technical implementation but also on bridging the gap between AI capabilities and stakeholder expectations.

In this post, I’ll share lessons from guiding stakeholders through AI’s possibilities and limitations.

Teaching Stakeholders About the Inner Workings of AI

One of the first things I realised when working with AI is how many misconceptions stakeholders have about how it works, what’s possible, and how to maintain AI features.

Many expect AI models will behave like deterministic tools they’ve used in the past. These misunderstandings often stem from what stakeholders have seen rather than experienced. Industry hype and “demoware” contribute to unrealistic expectations about how easily AI can deliver the results they want.

For example, stakeholders often ask why OpenAI API results differ from their ChatGPT experience. Addressing these questions early is critical for building trust and setting realistic expectations.

Explaining how AI generates results helps stakeholders understand its capabilities and limitations. This lays the foundation for collaboration and ensures stakeholders feel involved in shaping the outcomes.

Starting with the OpenAI Playground

Stakeholders need an interactive way to experiment with AI.

The OpenAI Playground is an excellent starting point. It allows users to test prompts, adjust parameters, and see results immediately.

This hands-on experience helps stakeholders understand how generative AI works and how changing inputs can influence outputs. For many, it’s their first experience seeing beyond simply entering text into a chat interface.

The OpenAI Playground is an intuitive tool for stakeholders to test prompts, tweak settings, and explore how generative AI works.

The OpenAI Playground also offers built-in collaboration features, making it easy for teams to tweak settings like temperature and learn through trial and error. Its clean, intuitive interface ensures users can get results quickly without being overwhelmed by complexity.

Educating Stakeholders Using Their Data

While the OpenAI Playground is a fantastic starting point, I needed to retrieve data from our database and support experimentation with non-OpenAI models. Using personalised data provides a sharper focus on the goals stakeholders want to achieve.

With my full-stack experience, I built a secure web app that integrated:

Database connectivity for real-world scenarios using client data (e.g., databases, APIs).
Support for multiple providers, allowing comparisons between OpenAI and non-OpenAI models.
Real-time metrics, such as token usage, generation time, and associated costs.

This customisation streamlined collaboration with stakeholders, allowing them to experiment directly, compare results, and explore what worked best. It turned abstract technical concepts into practical, measurable outcomes, improving alignment and creating a shared learning experience.

After stakeholders understand how their data can integrate with AI systems, the next step is determining which AI approaches best align with their goals.

Helping Stakeholders Choose the Right AI Approach

There’s no one-size-fits-all approach when building AI products.

Stakeholders often assume advanced techniques, like fine-tuning, are essential for success. As a consultant, I recommend beginning with straightforward methods that deliver results quickly using fewer resources and at less expense.

For example, few-shot prompting can be a practical alternative or a precursor to fine-tuning. By providing a few well-crafted examples, the AI can adapt its outputs to meet specific requirements, often achieving the desired results without the need for the added complexity of fine-tuning.

That’s not to say fine-tuning is never needed, but taking an incremental approach ensures that solutions are evaluated step by step, with short feedback loops to assess their value, keeping progress aligned with stakeholder goals.

Once stakeholders understand how their data integrates with AI tools and the right approaches to achieve their goals, the next step is to explore trade-offs and implications through experimentation.

Understanding Trade-offs Through Experimentation

Educating stakeholders about the practical implications of AI is essential. This includes:

The time required to generate results with different models.
Token usage: What tokens are, why they matter, and how they affect costs.
Parameter choices: How settings like temperature influence cost, consistency, and output quality.
Guardrails and patterns: Techniques like LLM-as-a-Judge, where a secondary model evaluates and refines outputs, are critical for maintaining accuracy and reliability.

Through experimentation, stakeholders can observe these trade-offs firsthand. For instance, they can see how increasing temperature settings results in more creative but less consistent outputs, or how longer prompts affect processing time and cost.

These insights enable stakeholders to make data-driven decisions that align with their goals and constraints, ensuring AI solutions are practical, cost-effective, and aligned with their needs.

How to Work with Stakeholders When Building AI Products

Effective collaboration with stakeholders requires more than just technical expertise. It’s about creating an environment where stakeholders feel empowered to explore, learn, and contribute meaningfully.

Here’s what I’ve found works best:

Start with Accessible Tools: Use tools like the OpenAI Playground to introduce generative AI concepts in an approachable way.
Focus on Education: Explain key concepts, such as temperature, token limits, and costs, in simple terms. Use examples to make abstract ideas concrete.
Collaborate Transparently: Provide tools that make the decision-making process measurable and clear. Transparency builds trust and fosters alignment.
Adapt to Their Needs: Tailor your approach to stakeholder priorities, whether they focus on cost, speed, or quality.
Experiment: Start with small, low-risk tests to gather insights. Scale up once you’ve identified what works best.
Use the Best Tools for Their Context: Help stakeholders identify the tools and techniques (e.g., fine-tuning, few-shot prompting) that align with their specific goals and constraints, avoiding unnecessary complexity.

By following these steps and fostering collaboration, transparency, and experimentation, you can empower stakeholders to navigate AI’s opportunities and challenges with confidence.

This approach leads to more informed decisions, and with stakeholder engagement, you can deliver outcomes that align with their goals.

Whose Responsibility Is It Anyway? Defining Boundaries with Event-Driven Architecture

Mon, 20 Jan 2025 00:00:00 GMT

Defining clear boundaries is essential to building clean, scalable, and reliable architectures.

In my experience, organisational demands often override the focus on boundaries and domain-driven design. This reflects the tension between following technical best practices and delivering business outcomes quickly. While theoretical approaches are widely discussed at conferences, in books, and in videos, the practical implementation of these ideas is often shaped by cultural dynamics, resource constraints, tight deadlines, and internal politics.

In this post, we’ll explore the challenges of maintaining boundaries and potential solutions, using a payment system as an example. This system facilitates transactions between clients and payment providers, such as PayPal or Stripe, highlighting the distribution of responsibilities within its architecture.

Why Stateless Systems Are Key to Loosely Coupled Architecture

Stateless systems are a ket component of loosely coupled architecture. By avoiding unnecessary state storage, they offer several compelling benefits:

Focus on Core Functions:
- Stateless systems adhere to the single responsibility principle, keeping them focused on their primary task.
Scale More Easily:
- With minimal state to track, these systems can handle higher loads without significant overhead.
Simplify Communication:
- Stateless systems exchange lightweight messages, often limited to essential identifiers like IDs, reducing complexity and ensuring efficient communication.

Take, for example, a payment system that links a client’s payee ID to a payment provider’s ID. This design avoids storing additional details, such as contact information, delegating that responsibility to the client. The client’s database becomes the single source of truth, ensuring a cleaner and more reliable architecture.

flowchart LR
    Client[Client System] -->|Payee ID| Payments[Payments System]
    Payments -->|Provider ID| Provider[Payment Provider]

However, in my experience problems emerge when teams request the systems to take on responsibilities beyond its intended scope.

For instance, they might ask it to send emails to payees whenever details are updated. Although seemingly convenient, this approach can lead to significant architectural pitfalls.

Why Adding Contact Details is a Bad Idea

Allowing the payment system to store and manage contact information may seem harmless at first, but in my experience the convenience opens a Pandora’s box and leads to significant problems later down the line:

Duplication of Data:
- Contact details must be synchronised between the client’s database and the payment system. This increases the risk of discrepancies and data conflicts.
Increased Complexity:
- The payment system would require additional mechanisms to manage updates, validate data, and resolve conflicts.
Misaligned Responsibilities:
- Payment systems should focus on processing transactions, not managing user contact details. Expanding their role leads to bloated, harder-to-maintain codebases.
Messaging Failures:
- Another challenge arises when considering failures. What happens if the payment system tries to send an email and it fails? Should it handle retries, or should this responsibility lie elsewhere? Adding retry logic and failure handling shifts the focus of the payment system away from its core role of facilitating transactions. This responsibility is better suited to the client, who already has the necessary data.

Here’s what happens when responsibilities blur:

flowchart TD
    subgraph "Problem: Duplicated Responsibilities"
        A[Client Database] -- Updates --> B[Payments System]
        B -- Sends Emails --> C[Payee]
        C -- Updates --> A
        B -- Retry Loop --> C
    end

To address these challenges and maintain clear boundaries, event-driven architecture offers an effective solution.

Event-Driven Architecture: A Scalable Solution

Event-driven architecture offers a clean, scalable solution for maintaining clear system boundaries while enabling flexible workflows.

The Problem: Overlapping Responsibilities

When systems take on responsibilities outside their intended scope, complexity and duplication increase, leading to maintenance challenges and inefficiencies.

For instance, if the payment system manages both transactions and user contact information or handles email notifications, it blurs boundaries and introduces unnecessary coupling.

The Solution: Clear Boundaries with Events

Event-driven architecture resolves this by enabling systems to remain focused on their core responsibilities while collaborating seamlessly through events.

Raise Events:
- The payment system publishes events like payee.updated or payment.successful when specific actions occur, passing only essential identifiers.
Clients Handle Events:
- The client system listens for these events, processes them, and performs actions like updating records or sending emails.
Handle Failures Gracefully:
- By delegating retry logic and error handling to the client system, failures in operations like email delivery are isolated from the payment system.

How It Works in Practice

Payee Updates Details:
- A payee updates their information with a payment provider.
Payment System Emits an Event:
- The system raises a payee.updated event containing only the relevant IDs.
Client Listens and Processes:
- The client system receives the event, updates its database, and triggers any necessary workflows, such as emailing.
Retry Logic:
- If an email fails to send, the client system handles retries without impacting the performance of the payment system.

sequenceDiagram
participant P as Payment Provider
participant PS as Payments System
participant C as Client System
participant E as Email Service

    P->>PS: Payment status update
    PS->>C: Emit payee.updated event
    Note over PS: Only passes IDs
    C->>C: Process event
    C->>E: Handle email (with retries)
    Note over C: Manages all contact details<br/>and messaging logic

Benefits of Event-Driven Architecture

This approach provides long-term benefits for scalability, maintainability, and system integrity.

Maintainability
Systems with focused responsibilities are easier to manage, test, and extend.
Scalability
Decoupled systems can scale independently, reducing bottlenecks and enhancing performance.
Data Integrity
By avoiding data duplication, systems ensure consistency and reduce the risk of conflicts.
Customisation
Client systems can handle domain-specific logic like email templates, personalisation, and language localisation without impacting the payment system.

This architecture ensures robust error handling, clean system boundaries, and a flexible foundation for future enhancements.

Handling Email Templates, Languages, and Personalisation

Beyond architectural benefits, event-driven systems also enable domain specific customisation, particularly for tasks like email notifications and user personalisation.

A client system handling email notifications has the added benefit of full control over:

Email Templates:
- Templates can be customised for different use cases, such as notifications, reminders, or promotions. This flexibility ensures that emails align with the client’s branding and messaging strategy.
Language Localisation:
- The client system can dynamically select the appropriate language for each payee, ensuring clear communication regardless of their preferred language.
Personalisation:
- Personalised content (e.g., using the payee’s name, specific account details, or tailored offers) can be included without requiring the payment system to store sensitive data.

By delegating this responsibility to the client, the system provides a better user experience while preserving clear boundaries and minimising complexity.

Conclusion

Clear boundaries between systems are essential for building scalable and maintainable architectures. Systems with focused responsibilities are easier to manage, test, and extend, while decoupled designs scale more effectively as each system evolves independently. Minimising duplication ensures consistency, avoids data conflicts, and reinforces the integrity of the overall system.

System design often requires balancing user demands with architectural constraints.

Event-driven architecture offers a powerful way to maintain these clear boundaries while addressing complex requirements. By adopting this approach, each system can focus on its core responsibilities, promoting flexibility, scalability, and reliability.

The principle I follow is simple: simplicity scales.

By embracing event-driven designs, we can create architectures that perform efficiently today and adapt to meet the demands of tomorrow.

The Determinism Dilemma: Why Traditional Software Engineers Struggle with AI Development

Mon, 13 Jan 2025 00:00:00 GMT

The goal of creating something “predictable,” reliable, and consistent is a shared principle across all the teams I’ve worked with throughout my career.

Knowing that the same code would always return the same output when given the same inputs was the foundation of everything we built.

We aimed for no surprises, no matter how complex a workflow might be. Whether implicitly or explicitly using finite state machines, this determinism enabled us to build testable, monitorable, maintainable, and, most importantly, predictable workflows.

We read and shared ideas at conferences, promoting patterns and principles like SOLID and DRY to create functional, composable, and extensible software.

Why Forcing AI to be Deterministic is the Wrong Approach

AI applications challenge this principle in a fundamental way. While AI systems can still be structured and maintainable, their inherent unpredictability requires us to rethink our approach.

For instance, asking an AI model to summarise a piece of text multiple times often produces different responses, especially when using zero-shot prompting. Many developers, uncomfortable with this unpredictability, try to force consistency by tweaking parameters like temperature or top-k.

However, forcing an AI model to behave more predictably risks losing its greatest strength. Generative models excel when producing varied and creative outputs. Overfitting an AI model to achieve consistency may make it behave more like traditional software but, in doing so, diminishes its versatility and potential.

Embracing AI’s Creativity

The key is to tailor the approach to the use case. In payment applications, predictability is paramount. These systems must behave in a consistent and reliable manner.

In contrast, a chatbot thrives on variety and personalisation. Its responses should feel unique and engaging because its purpose is to interact with users whose goals or needs may not be immediately clear. This shift in thinking requires developers to embrace AI’s non-deterministic nature when appropriate.

Rather than suppressing it, we must design systems that take advantage of this quality while incorporating safeguards to manage its limitations.

“Let Go of Determinism, Luke”

To borrow from Obi-Wan Kenobi’s advice: “Use the Force, Luke. Let go.”

Knowing when to rely on determinism and when to let go of it is crucial.

For traditional applications, the goal is consistent and predictable behaviour.

For AI, the focus is often on adaptability and creativity.

In future posts, I will explore how to balance these approaches.

For now, remember that the key to success in AI development is not forcing it to follow old rules. The real challenge is learning to use its unique strengths effectively.

Déjà Vu: AI Agent Frameworks and the JavaScript Gold Rush Revisited

Sat, 07 Dec 2024 00:00:00 GMT

Having lived through the era of a “new JavaScript framework every week,” we now find ourselves in the gold rush of the AI agent framework space.

New frameworks appear daily, each claiming to be the ‘ultimate’ solution for building AI agents, often backed by YouTubers enthusiastically promoting demoware and usually their own library, framework, or SaaS offering. Unfortunately, this enthusiasm can lead companies to uncritically adopt these tools without considering the long-term implications.

The recent AutoGen situation is a prime example of the chaos. Created by Microsoft, one might have expected stability, yet the framework has splintered, with founding contributors releasing their own fork, leaving developers confused and exhausted.

Similarly, LangChain presents numerous overlapping solutions, and occasional breaking changes can make it more of a hindrance than a help, as they require constant learning and frequent updates.

The lesson from the JS framework wars is clear: focus on understanding the core principles rather than tightly coupling your solution to any single framework.

The biggest takeaway we learnt back then was to think in terms of components and composition, which I covered in my “Build Components Not Walls” talk.

Some AI offerings will undoubtedly emerge as long-term winners, just as React did, but many will fade as early hype and quick profits give way to stability and sustainability.

How to build solutions during the age of the AI gold rush?

Adopt a “do one thing, and do it well” approach with framework-agnostic functions.

This will enable you to adapt, experiment, compose, and combine agentic components as the landscape shifts. Having such abstractions allows you to future-proof your approach. By breaking down elements, such as prompts and schemas, you can more easily adapt if a framework falls out of favour or is abandoned.

However, balance is crucial. Abstract too aggressively, and you may lose the genuine benefits that a framework provides.

In my upcoming posts, I’ll share examples of how I’ve applied these principles.

For now, the key takeaway is simple:

Focus on core concepts over fleeting trends.

What About Using Structured Naming for Event-Driven Messaging?

Wed, 20 Nov 2024 00:00:00 GMT

It’s a well-known adage that naming things is hard.

In event-driven architectures, a consistent naming convention is essential for scalability, communication, collaboration and maintainability.

At the moment, I’m architecting a new event-driven solution, and I planning to adopt a Structured Naming approach for event types.

In this post I’ll share my thoughts on this approach and its benefits.

Recommended Practice: Embedding Metadata in Event Payloads

It’s a widely recommended practice to include metadata directly within the event payload.

This metadata provides essential context about the event, such as its type, domain, and service, in addition to the data payload itself.

{
  "detail": {
    "metadata": {
      "id": "01HXHMF28A94NS7NSHC5GM80F4",
      "version": "1",
      "timestamp": "2020-04-28T07:20:20Z",
      "domain": "order",
      "service": "order-service",
      "type": "order-placed"
    },
    "data": {
      "user_id": "123456",
      "order_id": "1234567890123",
      "restaurant_id": "123456789",
      "currency": "USD",
      "total": "42.42"
    }
  }
}

In this example, the type field within the metadata specifies the event as order-placed.

While this method provides the necessary context, it can lead to challenges in event routing and subscription management, especially if you’re using NATS, where wildcards offer flexibility and agility.

For instance, if consumers need to subscribe to all events related to orders, they may have to manage subscriptions manually for each type (order-placed, order-updated, etc.).

Why Consider a Structured Naming Approach?

The Structured Naming approach organises event names in a structured, dot-separated format. This structure mirrors the natural categorisation of events, enabling intuitive grouping, routing, and filtering by using dots (.) as separators.

Example of Structured Event Naming

Using a structured naming approach, events related to orders could be organised like this:

order.created
order.updated
order.deleted

Here, the prefix order denotes the domain or entity, while the suffix (created, updated, deleted) specifies the verb. This structure allows consumers to easily subscribe to domain event types, such as all events under order.*.

Consistency is Key

Keeping naming patterns predictable and consistent is critical to reducing cognitive load and avoiding confusion. For instance: For instance:

use a single separator style (order.created) and avoid mixing styles like order-created and order_created in the same system.
Adhering to a clear and consistent format ensures event names remain instantly recognisable.

Benefits of the Structured Naming Approach

Improved Clarity

The structured naming convention includes both the event’s domain and action, making it easier to understand the purpose and intent of an event at a glance.

Efficient Subscription Management

Consumers can use wildcard subscriptions to receive a broader set of events. For example, subscribing to order.* captures all events related to orders, such as order.created and order.updated.

Scalability

As the system grows, adding new event types within a domain becomes seamless. You can introduce new events like order.canceled or user.profile.updated without overhauling existing subscription logic.

Sharding Capabilities

A structured naming convention can facilitate sharding by grouping related events with common prefixes (e.g. order.* or user.*).

This simplifies routing related events to the same shard, improving performance and resource allocation.

Adopting a Structured Naming Pattern

When adopting a structured naming approach, splitting by domain helps communicate intent and related actions.

Domain or Entity

Represents the primary category, such as order, user, or payment.
Action or Event Type

Specifies the exact type of event, like created, updated, or deleted.

Example of Structured Event Names

User Events
- user.registered
- user.logged_in
- user.profile.updated
Payment Events
- payment.initiated
- payment.completed
- payment.failed

This structure provides a consistent and organised approach to naming events, which should prove especially beneficial as systems scale and require more event types across multiple domains.

Versioning Strategy

When versioning events, a Version-First strategy aligns well with wildcard subscriptions and provides immediate visibility of the version. For instance:

Versioned Event Types
- v1.order.created
- v2.order.updated
- v1.user.profile.updated

This approach allows services to listen to specific versions or broader categories, such as v2.*, giving flexibility in routing as schemas evolve.

Benefits of Version-First Naming

Immediate visibility: Version information is the first thing you see
Easier filtering: Systems can quickly filter or route by version without parsing the entire event name
Clearer version transitions: Makes it easier to identify different versions of the same event type
Consistent with API versioning: Aligns with common API versioning practices

For example, a service could subscribe to all v1 events across domains with v1.* or all v2 order events with v2.order.*.

Use Cases

Microservices Architectures

In a microservices environment, services often respond to events from multiple domains. A structured naming convention allows services to subscribe to broader categories efficiently. For example, subscribing to user.*simplifies setup by reducing the need to enumerate all event types.

Event Sourcing

In event-sourced systems, where state is reconstructed from historical events, a structured naming scheme enables systematic grouping and processing of events by category.

AntiPatterns and Potential Drawbacks of Structured Naming

As with all things, there is a danger of overuse and misuse that could lead to complexity and challenges:

Initial Implementation Effort

Standardising event names across teams requires upfront planning and governance.
Complexity in Deep Naming Structures

Overly deep structures (e.g., order.process.payment.canceled) can reduce clarity. Keeping structures simple and well-documented avoids this pitfall.

Conclusion

I think a structured naming convention for events can offer a straightforward, organised approach that enhances clarity, scalability, and efficiency in event-driven systems.

Have you tried this approach? I’d love to hear about anything I’ve missed or overlooked, as well as any challenges, advantages, or pitfalls you’ve encountered.

AWS Step Functions Cookbook

Fri, 01 Nov 2024 00:00:00 GMT

This is my AWS Step Functions Mapping and Patterns and cookbook covering data manipulation, concurrency patterns, error handling, and advanced workflows.

Basic Concepts
Path Processing Fundamentals
Concurrency Patterns
Advanced State Patterns
Complex Workflows
Error Handling & Resilience
Integration Patterns
Best Practices & Tips

Basic Concepts

Processing Order Visualization

Input → InputPath → Parameters → Task → ResultSelector → ResultPath → OutputPath → Output

JSONPath Quick Reference

// First book's title
$.store.book[0].title

// All book titles
$.store.book[*].title

// Books under $10
$.store.book[?(@.price < 10)]

// All authors
$..author

// Number of books
$.store.book.length()

Path Processing Fundamentals

AWS Step Functions allows you to manipulate the data passing through your state machine at various stages using InputPath, Parameters, ResultSelector, ResultPath, and OutputPath.

1. InputPath with Complex Filtering

Example: Filter Nested Arrays

Initial State Input:

{
  "orders": [
    { "id": 1, "status": "PENDING", "amount": 100 },
    { "id": 2, "status": "SHIPPED", "amount": 200 },
    { "id": 3, "status": "PENDING", "amount": 150 }
  ],
  "metadata": {
    "requestId": "abc-123"
  }
}

State Definition with InputPath:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessPendingOrders",
  "InputPath": "$.orders[?(@.status == 'PENDING')]",
  "ResultPath": "$.processedOrders",
  "End": true
}

Lambda Receives:

[
  { "id": 1, "status": "PENDING", "amount": 100 },
  { "id": 3, "status": "PENDING", "amount": 150 }
]

If Lambda Returns:

{
  "processedCount": 2,
  "totalAmount": 250
}

Resulting State Output:

{
  "orders": [
    { "id": 1, "status": "PENDING", "amount": 100 },
    { "id": 2, "status": "SHIPPED", "amount": 200 },
    { "id": 3, "status": "PENDING", "amount": 150 }
  ],
  "metadata": {
    "requestId": "abc-123"
  },
  "processedOrders": {
    "processedCount": 2,
    "totalAmount": 250
  }
}

Example: Multiple Field Selection

Initial State Input:

{
  "orderId": "order-456",
  "items": [
    { "productId": "prod-1", "quantity": 2 },
    { "productId": "prod-2", "quantity": 1 }
  ],
  "address": {
    "street": "123 Main St",
    "city": "Anytown",
    "zip": "12345"
  },
  "customerInfo": {
    "name": "John Doe",
    "email": "john@example.com"
  }
}

State Definition with Parameters:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessOrderDetails",
  "Parameters": {
    "orderDetails": {
      "id.$": "$.orderId",
      "items.$": "$.items[*].productId",
      "shipping.$": "$.address"
    },
    "customerEmail.$": "$.customerInfo.email"
  },
  "ResultPath": "$.orderProcessingResult",
  "End": true
}

Lambda Receives:

{
  "orderDetails": {
    "id": "order-456",
    "items": ["prod-1", "prod-2"],
    "shipping": {
      "street": "123 Main St",
      "city": "Anytown",
      "zip": "12345"
    }
  },
  "customerEmail": "john@example.com"
}

If Lambda Returns:

{
  "confirmationNumber": "CONF-789",
  "estimatedDelivery": "2023-12-01"
}

Resulting State Output:

{
  "orderId": "order-456",
  "items": [
    { "productId": "prod-1", "quantity": 2 },
    { "productId": "prod-2", "quantity": 1 }
  ],
  "address": {
    "street": "123 Main St",
    "city": "Anytown",
    "zip": "12345"
  },
  "customerInfo": {
    "name": "John Doe",
    "email": "john@example.com"
  },
  "orderProcessingResult": {
    "confirmationNumber": "CONF-789",
    "estimatedDelivery": "2023-12-01"
  }
}

2. Advanced Parameters Usage

Example: Dynamic Array Processing and Intrinsic Functions

Initial State Input:

{
  "item": "product-123",
  "items": ["product-123", "product-456"],
  "stockStatus": "true",
  "date": "2023-11-15",
  "time": "14:30:00",
  "metadata": {
    "sessionId": "sess-001"
  }
}

State Definition with Parameters:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessData",
  "Parameters": {
    "itemsArray.$": "States.Array($.item)",
    "itemsCount.$": "States.ArrayLength($.items)",
    "hasStock.$": "States.StringToBoolean($.stockStatus)",
    "processTime": {
      "startTime.$": "$$.State.EnteredTime",
      "formatted.$": "States.Format('{}T{}Z', $.date, $.time)"
    },
    "metadata": {
      "executionStartTime.$": "$$.Execution.StartTime",
      "retryCount.$": "$$.State.RetryCount",
      "sessionId.$": "$.metadata.sessionId"
    }
  },
  "ResultPath": "$.processingResult",
  "End": true
}

Lambda Receives:

{
  "itemsArray": ["product-123"],
  "itemsCount": 2,
  "hasStock": true,
  "processTime": {
    "startTime": "2023-11-15T14:30:00Z",
    "formatted": "2023-11-15T14:30:00Z"
  },
  "metadata": {
    "executionStartTime": "2023-11-15T14:29:50Z",
    "retryCount": 0,
    "sessionId": "sess-001"
  }
}

If Lambda Returns:

{
  "status": "Processed",
  "processedAt": "2023-11-15T14:30:10Z"
}

Resulting State Output:

{
  "item": "product-123",
  "items": ["product-123", "product-456"],
  "stockStatus": "true",
  "date": "2023-11-15",
  "time": "14:30:00",
  "metadata": {
    "sessionId": "sess-001"
  },
  "processingResult": {
    "status": "Processed",
    "processedAt": "2023-11-15T14:30:10Z"
  }
}

3. Replacing or Overwriting Data in State Output

Example 1: Overwriting State Data Using ResultPath

Initial State Input:

{
  "orderId": "order-456",
  "status": "NEW",
  "customerInfo": {
    "name": "Jane Doe",
    "email": "jane@example.com"
  }
}

State Definition with ResultPath Overwrite:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessOrderStatus",
  "Parameters": {
    "orderDetails": {
      "id.$": "$.orderId",
      "status": "PROCESSED"
    }
  },
  "ResultPath": "$",
  "End": true
}

Lambda Receives:

{
  "orderDetails": {
    "id": "order-456",
    "status": "PROCESSED"
  }
}

If Lambda Returns:

{
  "status": "SUCCESS",
  "processedAt": "2023-12-01T10:00:00Z"
}

Resulting State Output:

{
  "status": "SUCCESS",
  "processedAt": "2023-12-01T10:00:00Z"
}

Using ResultPath with the "$" path overwrites the entire state input, replacing it with the result from the Lambda function.

Example 2: Partially Replacing Data with ResultPath

Initial State Input:

{
  "orderId": "order-123",
  "items": [
    { "productId": "prod-1", "quantity": 1 },
    { "productId": "prod-2", "quantity": 3 }
  ],
  "customer": {
    "name": "Alice",
    "email": "alice@example.com"
  },
  "metadata": {
    "requestId": "req-789"
  }
}

State Definition with Partial Overwrite Using ResultPath:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:UpdateOrder",
  "Parameters": {
    "orderId.$": "$.orderId",
    "items.$": "$.items[*].productId"
  },
  "ResultPath": "$.updatedOrder",
  "End": true
}

Lambda Receives:

{
  "orderId": "order-123",
  "items": ["prod-1", "prod-2"]
}

If Lambda Returns:

{
  "status": "UPDATED",
  "timestamp": "2023-12-01T12:30:00Z"
}

Resulting State Output:

{
  "orderId": "order-123",
  "items": [
    { "productId": "prod-1", "quantity": 1 },
    { "productId": "prod-2", "quantity": 3 }
  ],
  "customer": {
    "name": "Alice",
    "email": "alice@example.com"
  },
  "metadata": {
    "requestId": "req-789"
  },
  "updatedOrder": {
    "status": "UPDATED",
    "timestamp": "2023-12-01T12:30:00Z"
  }
}

In this example, the ResultPath directs the result to the updatedOrder field in the existing state, allowing a partial overwrite.

Concurrency Patterns

1. Parallel State Processing

Example: Basic Parallel Processing

Initial State Input:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"]
}

State Definition with Parallel:

{
  "Type": "Parallel",
  "Branches": [
    {
      "StartAt": "ProcessOrders",
      "States": {
        "ProcessOrders": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:ProcessOrders",
          "End": true
        }
      }
    },
    {
      "StartAt": "UpdateInventory",
      "States": {
        "UpdateInventory": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:UpdateInventory",
          "End": true
        }
      }
    }
  ],
  "ResultPath": "$.parallelResults",
  "End": true
}

Tasks Receive:

ProcessOrders Lambda Receives:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"]
}

UpdateInventory Lambda Receives:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"]
}

If Lambdas Return:

ProcessOrders Returns:
```
{
  "orderStatus": "Processed"
}
```
UpdateInventory Returns:
```
{
  "inventoryStatus": "Updated"
}
```

Resulting State Output:

{
  "orderId": "order-789",
  "customerId": "cust-123",
  "items": ["item-1", "item-2"],
  "parallelResults": [
    {
      "orderStatus": "Processed"
    },
    {
      "inventoryStatus": "Updated"
    }
  ]
}

Example: Parallel with Shared Context

Initial State Input:

{
  "userId": "user-456",
  "sessionData": {
    "sessionId": "sess-789",
    "timestamp": "2023-11-15T15:00:00Z"
  },
  "preferences": {
    "theme": "dark",
    "notifications": true
  }
}

State Definition with Shared Parameters:

{
  "Type": "Parallel",
  "Parameters": {
    "context.$": "$.sessionData",
    "userId.$": "$.userId"
  },
  "Branches": [
    {
      "StartAt": "Branch1",
      "States": {
        "Branch1": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:BranchFunction1",
          "End": true
        }
      }
    },
    {
      "StartAt": "Branch2",
      "States": {
        "Branch2": {
          "Type": "Task",
          "Resource": "arn:aws:lambda:region:account-id:function:BranchFunction2",
          "End": true
        }
      }
    }
  ],
  "ResultSelector": {
    "branch1Result.$": "$[0]",
    "branch2Result.$": "$[1]"
  },
  "ResultPath": "$.branchResults",
  "End": true
}

Tasks Receive:

Both Lambdas Receive:

{
  "context": {
    "sessionId": "sess-789",
    "timestamp": "2023-11-15T15:00:00Z"
  },
  "userId": "user-456"
}

If Lambdas Return:

BranchFunction1 Returns:

{
  "activityLog": "Activity recorded"
}

BranchFunction2 Returns:

{
  "preferenceUpdate": "Preferences saved"
}

Resulting State Output:

{
  "userId": "user-456",
  "sessionData": {
    "sessionId": "sess-789",
    "timestamp": "2023-11-15T15:00:00Z"
  },
  "preferences": {
    "theme": "dark",
    "notifications": true
  },
  "branchResults": {
    "branch1Result": {
      "activityLog": "Activity recorded"
    },
    "branch2Result": {
      "preferenceUpdate": "Preferences saved"
    }
  }
}

2. Map State with Concurrency Control

Example: Basic Map with Concurrency

Initial State Input:

{
  "orders": [
    { "orderId": "order-1", "amount": 100 },
    { "orderId": "order-2", "amount": 150 },
    { "orderId": "order-3", "amount": 200 },
    { "orderId": "order-4", "amount": 250 },
    { "orderId": "order-5", "amount": 300 }
  ]
}

State Definition with Map:

{
  "Type": "Map",
  "ItemsPath": "$.orders",
  "MaxConcurrency": 2,
  "Iterator": {
    "StartAt": "ProcessSingleOrder",
    "States": {
      "ProcessSingleOrder": {
        "Type": "Task",
        "Resource": "arn:aws:lambda:region:account-id:function:ProcessSingleOrder",
        "End": true
      }
    }
  },
  "ResultPath": "$.processedOrders",
  "End": true
}

Lambda Receives (per iteration):

First Iteration:
```
{ "orderId": "order-1", "amount": 100 }
```
Second Iteration:
```
{ "orderId": "order-2", "amount": 150 }
```

(Up to MaxConcurrency of 2 at a time.)

If Lambdas Return:

{
  "orderId": "order-X",
  "status": "Processed"
}

Resulting State Output:

{
  "orders": [
    { "orderId": "order-1", "amount": 100 },
    { "orderId": "order-2", "amount": 150 },
    { "orderId": "order-3", "amount": 200 },
    { "orderId": "order-4", "amount": 250 },
    { "orderId": "order-5", "amount": 300 }
  ],
  "processedOrders": [
    { "orderId": "order-1", "status": "Processed" },
    { "orderId": "order-2", "status": "Processed" },
    { "orderId": "order-3", "status": "Processed" },
    { "orderId": "order-4", "status": "Processed" },
    { "orderId": "order-5", "status": "Processed" }
  ]
}

Example: Map with `ItemSelector`

Initial State Input:

{
  "array": [1, 4, 9, 16, 25]
}

State Definition with Map and ItemSelector:

{
  "Type": "Map",
  "ItemsPath": "$.array",
  "ItemSelector": {
    "index.$": "$$.Map.Item.Index",
    "value.$": "$$.Map.Item.Value",
    "sqrtValue.$": "States.MathSqrt($$.Map.Item.Value)"
  },
  "Iterator": {
    "StartAt": "ProcessItem",
    "States": {
      "ProcessItem": {
        "Type": "Pass",
        "ResultPath": "$",
        "End": true
      }
    }
  },
  "ResultPath": "$.results",
  "End": true
}

Data Processed in Each Iteration:

First Iteration:

{
  "index": 0,
  "value": 1,
  "sqrtValue": 1.0
}

Second Iteration:

{
  "index": 1,
  "value": 4,
  "sqrtValue": 2.0
}

Resulting State Output:

{
  "array": [1, 4, 9, 16, 25],
  "results": [
    { "index": 0, "value": 1, "sqrtValue": 1.0 },
    { "index": 1, "value": 4, "sqrtValue": 2.0 },
    { "index": 2, "value": 9, "sqrtValue": 3.0 },
    { "index": 3, "value": 16, "sqrtValue": 4.0 },
    { "index": 4, "value": 25, "sqrtValue": 5.0 }
  ]
}

Advanced State Patterns

1. Dynamic Branching with Choice State

Initial State Input:

{
  "order": {
    "total": 1200,
    "priority": "HIGH",
    "items": [
      { "productId": "prod-1", "category": "STANDARD" },
      { "productId": "prod-2", "category": "FRAGILE" }
    ],
    "shipping": {
      "express": true
    }
  }
}

State Definition with Choice:

{
  "Type": "Choice",
  "Choices": [
    {
      "And": [
        {
          "Variable": "$.order.total",
          "NumericGreaterThan": 1000
        },
        {
          "Variable": "$.order.priority",
          "StringEquals": "HIGH"
        }
      ],
      "Next": "PremiumProcessing"
    },
    {
      "Or": [
        {
          "Variable": "$.order.items[?(@.category == 'FRAGILE')]",
          "IsPresent": true
        },
        {
          "Variable": "$.order.shipping.express",
          "BooleanEquals": true
        }
      ],
      "Next": "SpecialHandling"
    }
  ],
  "Default": "StandardProcessing"
}

Execution Path:

Since $.order.total > 1000 and $.order.priority == “HIGH”, the execution moves to PremiumProcessing.

2. Wait States with Dynamic Duration

Initial State Input:

{
  "task": "WaitForApproval",
  "delay": 30 // Wait for 30 seconds
}

State Definition with Wait:

{
  "Type": "Wait",
  "SecondsPath": "$.delay",
  "Next": "CheckApprovalStatus"
}

Execution Flow:

The state machine waits for 30 seconds (as specified in $.delay) before proceeding to CheckApprovalStatus.

Complex Workflows

1. Orchestration Pattern

Initial State Input:

{
  "orderId": "order-999",
  "customerId": "cust-789",
  "items": ["item-5", "item-6"]
}

State Machine Definition:

{
  "StartAt": "InitializeOrder",
  "States": {
    "InitializeOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:InitializeOrder",
      "Next": "ParallelProcessing"
    },
    "ParallelProcessing": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "ValidateInventory",
          "States": {
            "ValidateInventory": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account-id:function:ValidateInventory",
              "End": true
            }
          }
        },
        {
          "StartAt": "ProcessPayment",
          "States": {
            "ProcessPayment": {
              "Type": "Task",
              "Resource": "arn:aws:lambda:region:account-id:function:ProcessPayment",
              "End": true
            }
          }
        }
      ],
      "ResultPath": "$.processingResults",
      "Next": "EvaluateResults"
    },
    "EvaluateResults": {
      "Type": "Choice",
      "Choices": [
        {
          "And": [
            {
              "Variable": "$.processingResults[0].inventoryAvailable",
              "BooleanEquals": true
            },
            {
              "Variable": "$.processingResults[1].paymentSuccessful",
              "BooleanEquals": true
            }
          ],
          "Next": "FulfillOrder"
        }
      ],
      "Default": "HandleFailure"
    },
    "FulfillOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:FulfillOrder",
      "End": true
    },
    "HandleFailure": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:HandleFailure",
      "End": true
    }
  }
}

Execution Flow:

InitializeOrder Lambda receives the initial input and performs setup tasks.
ParallelProcessing runs ValidateInventory and ProcessPayment in parallel.
EvaluateResults checks if both inventory is available and payment was successful.
If both are true, it proceeds to FulfillOrder; otherwise, it moves to HandleFailure.

Resulting State Output:

Depends on the execution path taken. If successful, the order is fulfilled; if not, failure handling is executed.

2. Saga Pattern with Compensation

Initial State Input:

{
  "reservationDetails": {
    "flightId": "FL-123",
    "hotelId": "HT-456",
    "carRentalId": "CR-789"
  }
}

State Machine Definition:

{
  "StartAt": "ReserveFlight",
  "States": {
    "ReserveFlight": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:ReserveFlight",
      "Next": "ReserveHotel",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.errorInfo",
          "Next": "CleanupState"
        }
      ]
    },
    "ReserveHotel": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:ReserveHotel",
      "Next": "ReserveCar",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.errorInfo",
          "Next": "CompensateFlight"
        }
      ]
    },
    "ReserveCar": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:ReserveCar",
      "Next": "CompleteBooking",
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.errorInfo",
          "Next": "CompensateHotel"
        }
      ]
    },
    "CompensateHotel": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:CancelHotel",
      "Next": "CompensateFlight"
    },
    "CompensateFlight": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:CancelFlight",
      "Next": "CleanupState"
    },
    "CompleteBooking": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:region:account-id:function:CompleteBooking",
      "End": true
    },
    "CleanupState": {
      "Type": "Pass",
      "End": true
    }
  }
}

Execution Flow:

If any reservation step fails, the state machine invokes compensation steps to undo previous actions.

Resulting State Output:

Depends on the success or failure of each reservation step and the execution of compensation actions.

Error Handling & Resilience

1. Comprehensive Error Handling

Example: Retry Configuration with Catch

State Definition with Retry and Catch:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessData",
  "Retry": [
    {
      "ErrorEquals": ["States.Timeout"],
      "IntervalSeconds": 2,
      "MaxAttempts": 3,
      "BackoffRate": 2.0
    },
    {
      "ErrorEquals": ["CustomError"],
      "IntervalSeconds": 1,
      "MaxAttempts": 2
    }
  ],
  "Catch": [
    {
      "ErrorEquals": ["States.ALL"],
      "ResultPath": "$.errorInfo",
      "Next": "ErrorHandler"
    }
  ],
  "End": true
}

Execution Flow:

The task retries on States.Timeout and CustomError according to specified settings.
If all retries fail or another error occurs, it catches the error and transitions to ErrorHandler.

2. Circuit Breaker Implementation

State Definition with Circuit Breaker Logic:

{
  "Type": "Choice",
  "Choices": [
    {
      "Variable": "$.serviceHealth.errorCount",
      "NumericGreaterThan": 5,
      "Next": "CircuitOpen"
    }
  ],
  "Default": "ProcessNormally"
}

Execution Flow:

If $.serviceHealth.errorCount exceeds 5, the workflow transitions to CircuitOpen, preventing further processing.
Otherwise, it proceeds to ProcessNormally.

Integration Patterns

1. Service Integration Pattern

Example: SQS `SendMessage` Integration

Initial State Input:

{
  "orderId": "order-1010",
  "message": "Order placed successfully."
}

State Definition with SQS Integration:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::sqs:sendMessage",
  "Parameters": {
    "QueueUrl": "https://sqs.region.amazonaws.com/account-id/queue-name",
    "MessageBody.$": "$.message",
    "MessageAttributes": {
      "OrderId": {
        "DataType": "String",
        "StringValue.$": "$.orderId"
      }
    }
  },
  "ResultSelector": {
    "MessageId.$": "$.MessageId",
    "Timestamp.$": "$$.State.EnteredTime"
  },
  "ResultPath": "$.sqsResult",
  "End": true
}

Resulting State Output:

{
  "orderId": "order-1010",
  "message": "Order placed successfully.",
  "sqsResult": {
    "MessageId": "abcd1234-5678-90ef-ghij-klmnopqrstuv",
    "Timestamp": "2023-11-15T16:00:00Z"
  }
}

2. API Gateway Integration

Initial State Input:

{
  "orderData": {
    "orderId": "order-2020",
    "items": ["item-A", "item-B"]
  }
}

State Definition with API Gateway Integration:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::apigateway:invoke",
  "Parameters": {
    "ApiEndpoint": "https://myapi.execute-api.region.amazonaws.com",
    "Method": "POST",
    "Headers": {
      "Content-Type": "application/json"
    },
    "Path": "/orders",
    "RequestBody.$": "$.orderData"
  },
  "ResultSelector": {
    "StatusCode.$": "$.StatusCode",
    "ResponseBody.$": "$.ResponseBody"
  },
  "ResultPath": "$.apiResponse",
  "End": true
}

Resulting State Output:

{
  "orderData": {
    "orderId": "order-2020",
    "items": ["item-A", "item-B"]
  },
  "apiResponse": {
    "StatusCode": 200,
    "ResponseBody": {
      "message": "Order received",
      "orderId": "order-2020"
    }
  }
}

Best Practices & Tips

1. Use Intrinsic Functions Effectively

Example: Utilizing Intrinsic Functions in Parameters

Initial State Input:

{
  "date": "2023-11-15",
  "time": "17:00:00",
  "item": "item-XYZ",
  "items": ["item-XYZ", "item-ABC"],
  "status": "true"
}

State Definition with Intrinsic Functions:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessIntrinsic",
  "Parameters": {
    "timestamp.$": "States.Format('{}T{}Z', $.date, $.time)",
    "itemsArray.$": "States.Array($.item)",
    "itemsCount.$": "States.ArrayLength($.items)",
    "isValid.$": "States.StringToBoolean($.status)"
  },
  "ResultPath": "$.intrinsicResult",
  "End": true
}

Lambda Receives:

{
  "timestamp": "2023-11-15T17:00:00Z",
  "itemsArray": ["item-XYZ"],
  "itemsCount": 2,
  "isValid": true
}

2. Implement Idempotency for Reliability

State Definition with Idempotency Key:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:ProcessWithIdempotency",
  "Parameters": {
    "idempotencyKey.$": "States.Format('{}-{}', $.orderId, $$.State.Name)",
    "payload.$": "$"
  },
  "ResultPath": "$.idempotentResult",
  "End": true
}

Ensures that each execution can be uniquely identified and avoids duplicate processing.

3. Monitor Execution Progress with CloudWatch

State Definition with CloudWatch Integration:

{
  "Type": "Task",
  "Resource": "arn:aws:states:::cloudwatch:putMetricData",
  "Parameters": {
    "Namespace": "StepFunctions/Processing",
    "MetricData": [
      {
        "MetricName": "ProcessingTime",
        "Value.$": "$.duration",
        "Unit": "Milliseconds",
        "Dimensions": [
          {
            "Name": "ExecutionId",
            "Value.$": "$$.Execution.Id"
          }
        ]
      }
    ]
  },
  "ResultPath": "$.cloudWatchResult",
  "End": true
}

Resulting State Output:

Includes metrics data sent to CloudWatch for monitoring purposes.

4. Replace Data Effectively with OutputPath

Using OutputPath to selectively replace or retain data in the output state can help keep workflow data clean and relevant.

Example 3: Replacing Data with OutputPath

Initial State Input:

{
  "userId": "user-789",
  "preferences": {
    "notifications": true,
    "theme": "light"
  },
  "settings": {
    "email": "user@example.com",
    "subscription": "basic"
  }
}

State Definition with OutputPath Replacement:

{
  "Type": "Task",
  "Resource": "arn:aws:lambda:region:account-id:function:UpdatePreferences",
  "Parameters": {
    "userId.$": "$.userId",
    "preferences.$": "$.preferences"
  },
  "OutputPath": "$.preferences",
  "End": true
}

Lambda Receives:

{
  "userId": "user-789",
  "preferences": {
    "notifications": true,
    "theme": "light"
  }
}

If Lambda Returns:

{
  "notifications": false,
  "theme": "dark"
}

Resulting State Output:

{
  "notifications": false,
  "theme": "dark"
}

By setting the OutputPath to $.preferences, the final state output retains only the updated preferences data, replacing the initial input with the result of this task.

Sometimes Boring is Best

Sat, 26 Oct 2024 00:00:00 GMT

As I stood in the rain as a volunteer race marshal at my local park run, it occurred to me that I wanted the same thing when running applications in production, and that’s absolutely nothing to happen.

The last thing I wanted to do was be a hero.

🎸 “Always keep on the right side of the path” – the Parkrun version of Monty Python’s song.

I’d rather everyone enjoy a safe, smooth race where I can cheer and encourage people on as they pass, reminding them to keep to the right so they don’t collide with runners coming the other way.

It’s the same in the land of IT. The only thing I want to see when viewing Grafana dashboards is a sea of green, 200 status codes and steady traffic patterns.

What’s So Bad About Heroes?

Whilst working with consultants on a client engagement, one of my colleagues questioned why we were celebrating someone waking up at 3 AM to “rescue” the application with Superman emojis.

At first, I didn’t agree, but then I saw their point.

Reactive, heroic behaviour creates great headlines and attention, but sometimes we should aim for clicking clock levels of excitement.

After all, the best work in designing, building, architecting, and deploying is often the work nobody notices.

Don't Look Back in Anger: Let's Design a Better Ticketing System

Sat, 31 Aug 2024 00:00:00 GMT

Saturday’s Oasis ticket sales left thousands of fans disheartened after spending hours in virtual queues. As reported by the BBC and echoed by fans on X (formerly Twitter) and Reddit, the experience could be aptly described using a quote from Liam Gallagher himself:

Shite
— Liam Gallagher (@liamgallagher) April 20, 2023

Like many others, I have long been frustrated with platforms like Ticketmaster, See Tickets, and Gigs and Tours. This weekend, their reputations took another hit as fans faced inflated prices and technical glitches.

This made me think if I could design a better, more reliable ticketing system that prioritises fairness and user experience.

Let’s not make Sally wait any longer than she has to.

Some Might Say

This document outlines the design and technical requirements for an enhanced ticketing system aimed at reducing server load, improving user experience, and ensuring environmental sustainability.

The proposed system will achieve these goals through random selection, time-limited purchasing, and lightweight verification, with a focus on simplicity and excellent user experience.

Wonderwall

Traditional ticketing systems often crumble under the pressure of high-demand events, leading to poor user experiences, system crashes, and unfair ticket distribution.

graph TD
  A[User Joins Queue] --> B[Wait in Line]
  B --> C{Tickets Available?}
  C -->|Yes| D[Purchase Ticket]
  C -->|No| E[Event Sold Out]

The improved system seeks to address these issues more efficiently and user-friendly.

graph TD
A[User Signs Up 24 Hours Before Sale] --> B[Verify Email & Phone]
B --> C[Specify Preferences & Notification Method]
C --> D[Enter Selection Queue]
D --> E{Selected?}
E -->|Yes| F[Receive Notification]
E -->|No| G[Remain in Queue]
F --> H[Purchase Window Opens]
H --> I{User Clicks Link to Purchase}
I -->|Yes| J[10-Minute Purchase Window]
J --> K{Purchase Completed?}
K -->|Yes| L[Ticket Secured]
K -->|No| P[Exit Process]
G --> N{Tickets Remaining?}
N -->|Yes| E
N -->|No| O[Notify: Event Sold Out]

The Masterplan

🎲 Fair Selection System: A number of initial users (e.g., 1000) are selected randomly from the Queue, ensuring equal chances for all. The system takes user preferences into account, avoiding notifications to those who cannot fulfill their requirements.
⏱️ Time-Limited Purchase Window: Selected users have 10 minutes to click the link to start purchasing. After clicking the link, they have another 10 minutes to complete their purchase, keeping the process moving efficiently.
📅 Pre-Sale Deadline: Users must complete sign-up, specify preferences, and select their notification method at least 24 hours before the sale begins. This ensures they are in line for ticket selection.
📱 Multi-Channel Notifications: Users choose how they want to be notified: Push, SMS, Email, or any combination.
🔒 Simple Verification: Quick email and phone verification during sign-up ensures user authenticity.
🛡️ Anti-Scalping Measures: The system includes checks to prevent multiple accounts and bulk purchases.
🚶‍♂️ Stress-Free Waiting Room: Users can go about their day, knowing they’ll be notified if selected.
🔄 Controlled User Flow: If a selected user does not complete their purchase within the given time, they exit the process. Another user is then selected, maintaining a smooth and efficient flow of ticket sales.
🎯 Preference-Based Notifications: Users specify venue, number of tickets, price band, and accessibility needs in advance. The system only notifies those whose preferences match available tickets, ensuring a more efficient and satisfying purchase process.

Be Here Now

The system’s architecture is designed for simplicity and efficiency, allowing easy scaling without the need for expensive server infrastructure.

This design ensures that even with millions of users in the selection Queue, the system remains responsive and doesn’t require overpowered servers.

D’You Know What I Mean?

🔐 Efficient Sign-up Process: Utilises a CAPTCHA check to prevent bot attacks, reducing server load. Email and phone verification are processed asynchronously to ensure a smooth user experience.
⚙️ Event-Driven Architecture: Processes like user selection, notification, and purchase are event-driven, allowing for horizontal scaling and efficient resource use.
🚦 Controlled Concurrency: Limits the number of users making purchases simultaneously, naturally throttling demand and preventing server overload.
🗃️ Stateless Design: User state is managed in the database rather than server memory, enabling easy load balancing and server scaling.
📨 Queuing System: Utilises message queues to process notifications and selections, ensuring system stability during high-load periods.
🗄️ Caching Layer: Frequently accessed data, such as event details and available tickets, is cached, reducing database load and improving response times.

It’s Gettin’ Better (Man!!)

😊 User Satisfaction Rate: Targeting over 90%, measured through short post-process surveys.
🔼 System Uptime: Aiming for 99.99%, ensuring reliability during high-demand periods.
⏱️ Average Purchase Time: Less than 5 minutes from notification to completed transaction.
📉 Server Load Reduction: Targeting a 50% reduction compared to traditional queuing systems.

Little By Little

🚀 Efficient Sign-up: A quick form with only essential fields ensures minimal friction during registration.
🔄 Asynchronous Architecture: Handles traffic spikes smoothly, keeping the system responsive.
🔐 Data Protection: All user data is encrypted, ensuring privacy and security.
📱 Unique Accounts: Only one account is allowed per verified phone number, preventing fraud and scalping.

Talk Tonight

💬 Effective Communication: Clear and timely communication is essential for a successful ticketing system. Users should be kept informed using simple status screens like this:

+------------------------------------------+
|          Virtual Waiting Room            |
+------------------------------------------+
| Event: Oasis 2025                        |
|                                          |
| Status: You're in the queue              |
|                                          |
| We'll notify you if you're selected:     |
| • SMS: (44) 7970 123456                  |
| • Email: user@example.com                |
|                                          |
| [Update Contact Info]   [Exit]           |
+------------------------------------------+

🔔 Clear Notifications: Messages like “You’ve been selected! You have 10 minutes to start purchasing your tickets” keep users informed throughout the process.
📧 Helpful Emails: Include FAQs and next steps, guiding users through the purchasing process.
🆘 Support Access: An easy-to-find help section and support contact ensure users can resolve any issues quickly.

Stand by Me

This revised ticketing system prioritises user experience while maintaining simplicity and efficiency.

🧘 Stress-Free Waiting Process: Users can relax without anxiously watching a queue.
📣 Clear Communication: Users know precisely how and when they’ll be notified.
⚖️ Fair Chance for All: Random selection gives everyone an equal opportunity.
⏳ Efficient Purchase Process: Selected users can quickly secure their tickets.
🖥️ Reduced Server Load: Event-driven architecture handles high demand smoothly.
🚫 Scalper Prevention: Verification and purchase limits deter bulk buying.
💸 Cost-Effective Scaling: Simple architecture allows for efficient resource use.
🔄 Dynamic User Selection: As soon as one user completes or times out, another gets a chance, controlling the number of users purchasing at any time.
🎯 Preference-Based Notifications: Users specify their needs upfront, and the system only notifies those whose preferences match the available tickets, making the process more efficient and user-friendly.
📅 Pre-Sale Deadline: Ensures users complete all necessary steps at least 24 hours before the sale starts, confirming their eligibility and ensuring they are in line for selection.

Live Forever

This system creates a win-win scenario: users enjoy a more pleasant ticket-buying experience, and event organisers benefit from reduced costs and improved customer satisfaction.

The technical design ensures scalability and performance, using serverless architecture to keep costs low and manageable without requiring additional manual operational resources.

This approach should resolve the frustrations that Oasis fans encountered on Saturday, ensuring they won’t be looking back in anger.

The Leadership Gamble: Why Fighting for Your Team Is Essential for Success

Tue, 06 Aug 2024 00:00:00 GMT

I’ve been fortunate to work with some incredible leaders throughout my career.

These individuals have inspired and challenged their teams and driven them to achieve their full potential.

What truly sets a leader apart, however, is their ability to motivate and their willingness to fight for their team.

There’s something special about a leader who fights for their team.

In this post, I’ll explain what happens when a leader truly stands up for their team, sharing my firsthand experiences from a transformative project at Cambridge Assessment in 2007.

In the summer of 2007, Cambridge Assessment had just completed marking examination scripts for millions of students worldwide. The existing system, though functional, was slow, inefficient, and error-prone.

The team responsible had to work long hours to ensure accuracy and timeliness. It became clear that a more robust, scalable solution was needed.

As I detailed in my previous post, “How Agile Helped Cambridge Assessment to Scale,” I was hired to architect and build a new system as part of a team.

Leadership in Action

Our tech leader, who had been with Cambridge Assessment for a couple of years, fully understood the complexities of the existing system and the organization’s politics.

Some stakeholders were concerned that introducing a new team and system would alienate the existing staff, potentially leading to valuable team members leaving and taking with them the knowledge of running and supporting a key system.

However, our leader, who interviewed and hired us, believed in our ideas and the team’s capabilities. He fought to keep politics away from us, advocating for the new system’s benefits and value.

Navigating Challenges

Our leader faced a lot of pushback. People questioned our team’s methods, our tech choices, and our shift to agile practices like pair programming and test-driven development.

He backed our “strangler-fig” approach, where we’d gradually replace the old system. This step-by-step plan reassured everyone that the switch would be smooth and the new system would deliver.

Unwavering Support

Our leader’s support went beyond advocacy to active engagement. He stood firm against scepticism in countless meetings, making a case for our approach and ensuring we had space to experiment, especially with scalability — a critical aspect of the new system.

As an architect, I faced many criticisms and doubts. Without my leader’s trust and willingness to gamble on me, I would have been removed from the project. His support was a turning point in my career, and I’m grateful for his faith in me.

Taking an innovative approach meant experiencing and failing, which put my leader in challenging positions. Yet he never gave up on me.

His reputation was on the line, but he continued to fight for me and our team.

A Successful Transformation

By the summer of 2008, our solution was fully operational and easily handled the increased workload, significantly benefiting the entire organization with a smoother, more efficient process.

This experience was life-changing for me. Our leader’s support created an environment where we could grow, innovate, and feel psychologically safe.

His trust in us was critical to our success.

The Power of Leadership Dedication

A leader who fights for their team can make all the difference. It’s not just about making intelligent technical decisions; it’s also about navigating the complex human side of projects and building a culture of success.

Our leader’s tireless support enabled us to build a solution that raised the bar for Cambridge Assessment. His leadership showed us that believing in your team — and fighting for them — can lead to unprecedented outcomes.

The Fine Line Between Frustration and Desperation

Tue, 16 Jul 2024 00:00:00 GMT

As an England football fan, I know the feeling all too well. The heartbreak of watching your team come so close, only to fall short at the final hurdle once again. It’s now 58 years of hurt.

It’s a mixture of emotions – frustration, disappointment, and knowing the team could have done more.

I’ve experienced a similar feeling in my work as a consultant.

Reuters

You pour your heart and soul into a project, determined to deliver your best, but sometimes, even when the client is happy, you can’t shake the feeling that the outcome is good, but something is missing from making it great.

It’s a fine line between frustration and desperation, and it’s easy for others to misunderstand which side of that line you’re on. Frustration is a natural reaction when you know you have more to give and feel like you haven’t quite reached your full potential. Conversely, desperation can be perceived as a sign of weakness, a lack of confidence, or control.

The truth is, frustration is often a sign of passion, of a deep-seated desire to excel. It’s the driving force that pushes us to strive for greatness, to constantly seek ways to improve and refine our craft. On the other hand, desperation can be a paralysing emotion, leading to poor decision-making and a loss of focus.

The key is to recognise the difference and channel that frustration productively. Instead of dwelling on the what-ifs and missed opportunities, we need to take a step back, reflect on what we’ve learned, and use that knowledge to improve.

It’s not easy, and it’s a battle we all face at one point or another. But the true mark of a champion, whether on the football pitch or in the business world, is the ability to turn that frustration into fuel and use it as a driving force to reach new heights.

Even though we’ve talked about the golden generation before, this England team can turn frustration into motivation, learn from their failures, and strive for greatness. By the time the World Cup comes along in 2026, the players will have had more success with their clubs, spent more time playing together, and gained more experience knowing what it takes to win trophies and adapt their game.

They are not desperate to win the World Cup; they want to win it, and that desire will fuel what they do next.

Similarly, businesses should recognise the difference between frustration and desperation in their team members. When someone shows frustration, it shows a desire to improve, do their best work, and take ownership and responsibility.

Motivated people consistently deliver more valuable outcomes.

So, the next time you feel frustrated, remember it’s a sign of your commitment, passion, and desire to be the best. Embrace it, channel it, and let it drive you forward.

Empathy: The Underrated Leadership Skill That Boosts Productivity

Wed, 01 May 2024 00:00:00 GMT

As the English Football League nears the climax of another demanding season, leaders are confronted with the challenge of motivating weary players and handling the intense mental and physical pressures of chasing titles, securing promotions, or avoiding relegation.

As a Barcelona supporter, it might come as a surprise that I using Jose Mourinho, a figure often mired in controversy, who exemplified empathetic leadership during his reign at Inter Milan, leading to their historic treble win.

During the 2009-2010 season at Inter Milan, Mourinho noticed Wesley Sneijder’s exhaustion. Rather than ignoring the situation, he took an unconventional approach, deifying the usual win-at-all-costs mentality in sports by advising Sneijder to take some time off.

Mourinho noticed Sneijder’s exhaustion and said, “Wesley, you look tired. Take a few days off. Go get some sun with your wife and child.” This approach was unlike any other coach Sneijder had encountered, who typically focused only on training. Taking Mourinho’s advice, Sneijder went to Ibiza for three days.

This break not only rejuvenated him but also deepened his loyalty, significantly enhancing his performance for the remainder of the season. Upon his return, he felt so revitalized and committed that he was ready to go to battle for Mourinho. Sneijder later expressed

When I got back, I was prepared to kill and die for him.

This is a vital lesson for leaders everywhere: by actively listening to and addressing team concerns, you foster an environment of trust and respect.

In my own experience as a leader, I have witnessed similar transformations.

Leaders who prioritise empathy consistently prove more effective in revitalizing teams suffering from low morale and productivity. Their empathetic approach infuses these teams with new energy, strengthening loyalty and boosting performance. These experiences underscore that the ability to genuinely connect with and respond to team members not only enhances performance but also fosters a culture of mutual respect and dedication, leading to a more harmonious and productive work environment.

Empathetic leaders not only enhance team outcomes but also improve overall well-being, creating a supportive work environment that values both people and results.

Innovation Redefined: from 'Fail Fast' to 'Learn Fast'

Tue, 21 Nov 2023 00:00:00 GMT

Inspect and adapt loops are the heart of agile development. They enable continuous learning, improvement, and evolution.

In this post, I discuss the limitations of the “fail fast” approach and propose a more impactful alternative: “learn Fast”. This mindset embraces the inevitability of failure in innovation and transforms every challenge into a learning opportunity.

People are perpetual learners. Each product and feature is an opportunity for learning and growth. The more we learn, the more we grow. The more we grow, the more we learn. It’s a virtuous cycle.

While both principles encourage innovation and aim to have the same outcome, only one principle will make good things great, where every iteration is another lesson learned.

💔 Fail Fast

Fail fast minimises the time and resources spent on something that ultimately won’t work or deliver value. Detecting as soon as possible on when an assumption or bet isn’t working out requires judgment.

This judgement is critical to reducing waste, in this case, time and energy.

How the judgement is made should be decided up front and could include factors such as price, suitability, applicability, complexity, compatibility and adaptability.

🚀 Learn Fast

Learning fast is all about making it work, making it right, and making it fast.

It’s where successful teams have a continuous discovery process where they always identify new opportunities to deliver value.

It’s not a one-off but continuous, not reactive but proactive, not limited but infinite.

Learn Fast in Practice

In the next post, I’ll describe how learning fast is applied in practice.

Structured logging with Pino

Tue, 14 Nov 2023 00:00:00 GMT

Structured logging focuses on capturing data in a consistent and machine-readable format.

Unlike traditional text-based logs, structured logs are more straightforward to query and analyse, making extracting insights and debugging issues simpler.

In this post, we’ll take a look at an example of how structured logging with Pino.

In the examples below the logging approach is unstructured and lacks context, making it hard to parse or search through logs.

import pino from 'pino';

const logger = pino();

logger.info('User logged in', userId, new Date());
logger.info(`User: ${userId} logged in at: ${new Date()}`);

Instead you should switch to structured logging like this:

import pino from 'pino';

const logger = pino();

logger.info(
  {
    userId,
    date: new Date(),
  },
  'User logged in'
);

This output is structured in JSON format, making parsing and analyzing programmatically more accessible.

Adopting Pino for structured logging in Node.js applications significantly improves log management. It ensures logs are human-readable and machine-friendly, enhancing debugging and monitoring.

A Decade of Leadership: Celebrating Joe Parry's Legacy at the JavaScript & NodeJS Cambridge Meetup Group

Mon, 10 Jul 2023 00:00:00 GMT

This post celebrates and thanks Joe Parry as he takes a well-deserved bow and steps down from his role as the head of the JavaScript & NodeJS Cambridge Meetup Group.

He has been an extraordinary leader who has dedicated the past ten years to fostering a vibrant community of Javascriptors in Cambridge.

Photo from www.cambridgeindependent.co.uk

Way back on Tuesday, December 11, 2012, Joe began the group on the top floor of The Fountain in Cambridge. His vision, charisma, and passion were infectious, resulting in a group that has grown to include over 1,205 dedicated members, spanning beginners, professionals, and industry leaders.

Each meeting organised by Joe was a hub of innovation and learning. He worked tirelessly to bring in experts and industry influencers, ensuring every session was brilliant.

Joe always emphasised inclusivity, creating a safe space for beginners to ask questions, learn, and grow alongside seasoned veterans. The result was a symbiotic community where every member felt valued and welcome.

Under Joe’s stewardship, the JavaScript & NodeJS Cambridge Meetup Group went beyond the talks allowing members to build friendships, form valuable professional connections, and foster a supportive community.

It’s hard to imagine the JavaScript & NodeJS Cambridge Meetup Group without Joe at the helm. Hopefully, he will continue to participate and be involved somehow.

Thank you, Joe, for your ten years of service, guidance, and friendship. You have left an indelible mark on the Cambridge JavaScript community, and we will always be grateful for your invaluable contribution.

Photo of one of my many talks at the group in Hot Numbers, Cambridge.

I have been lucky enough to have been a regular speaker at the group, and while the group works out what happens next, here’s to Joe – a true champion of the JavaScript and NodeJS community.

Empowering Engineers: How Cambridge University Press Adopted a Shift Left Culture

Wed, 08 Mar 2023 00:00:00 GMT

Companies must be agile and respond quickly to changing customer needs in today’s fast-paced and constantly evolving technology landscape. That’s why DevOps practices that emphasise collaboration and communication between development and operations teams to deliver software rapidly, reliably, and at scale have become increasingly popular.

Shifting left, a core principle of DevOps can significantly benefit companies of all sizes. By empowering engineers to take on more operations responsibilities and promoting a culture of experimentation and innovation, companies can improve collaboration, increase reliability, and deliver high-quality software at scale.

In this post, I’ll discuss how and why Cambridge University Press adopted a shift left culture.

Photo by Madeleine Ragsdale on Unsplash

A New Culture

Cambridge University Press had a small operations team responsible for hosting Node.js applications used by millions of users daily on EC2.

They realised engineers needed more operations responsibilities to accommodate new and scale existing applications, so they adopted a shift left culture.

This led to the developing of a DevOps culture where engineers worked collaboratively with operations to build and deploy high-quality software at scale.

One of the most critical aspects of our culture was having a no “assholes” attitude, which made it possible for everyone to work together harmoniously towards a common goal.

There was no gatekeeping, and teams were encouraged to experiment and try new things without permission, creating an environment of trust and empowerment.

In addition, teams were encouraged to find solutions to their use cases and problems rather than adopting a one-size-fits-all approach, ultimately leading to better outcomes.

Migrating to Kubernetes

To foster this new culture, our engineers and operations teams worked closely to migrate the first application to run in Kubernetes.

While the operations team set constraints, such as structuring environment variables and capturing metrics in Prometheus and dashboards in Grafana, they did not dictate solutions to the engineering teams.

Keeping things simple, easy and consistent was crucial to our success.

For example:

Collaboration to create predefined templates for services, deployment, and ingress files.
A folder called’ infrastructure’ in the root directory was all that was required for a Node.js application to run in Kubernetes.
Infrastructure as code was in the same repository as the application source code, making continuous integration and delivery easy.

Experimentation and Innovation

Teams were encouraged to innovate, create, and explore. Our team was the first in the company to have feature branch previews, allowing product owners, testers and stakeholders to be agile and use features earlier in the development process.

In addition, we streamlined our deployment process to have zero downtime between deployments and automatic rollbacks if errors were detected.

Following the principle of “you build it, you own it”, teams were responsible for ensuring they had enough logging and instrumentation to diagnose bugs and performance issues.

Operations did not intrude on teams but made themselves available for guidance and support. This helped teams be more productive as they didn’t need an embedded ops person, and we were fortunate enough to have curious engineers who wanted to learn and tech.

Teams would get together to share their learning and adopt each other’s practices, which helped raise the bar in what we accomplished and our SRE skills and knowledge.

The outcome of adopting a shift left culture

Cambridge University Press’s adoption of a shift left culture had significant benefits.

Within a few months of starting the initiative, we migrated applications to serve traffic from production Kubernetes clusters.

By empowering engineers with more operations responsibilities and promoting a culture of experimentation and innovation, they could build and deploy high-quality software at scale collaboratively with the operations team.

The migration to Kubernetes was a critical milestone in this journey. Within a few months of starting the initiative, we migrated applications to serve traffic from production Kubernetes clusters.

The simplicity, consistency and ease of deployment made it possible for new applications to be Kubernetes-first.

The benefits of this approach were improved team collaboration, faster time-to-market, increased reliability, reduced downtime and streamlined deployment processes. This approach also helped save costs while maintaining high-quality services for its users.

Ultimately, Cambridge University Press’s shift left culture allowed them to stay ahead of the curve and respond more quickly to changing market needs.

Pino Redaction' A Simple Solution to Secure Logging in Node.js Applications

Wed, 22 Feb 2023 00:00:00 GMT

Pino is a popular and fast Node.js logging library that is designed for high-performance and low-overhead logging. It has many useful features, including support for structured logging, log levels, and log redaction.

Pino logging redaction allows you easily redact sensitive information logs, ensuring applications remain secure and compliant with regulations.

Photo by Newspaper Club

In this post, we’ll take a closer look at Pino logging redaction functionality, what it is, and how it can be used with examples.

What is Pino logging redaction?

Pino logging redaction is a feature that allows you to mask or remove sensitive data such as passwords, access tokens, or credit card numbers from logs. Redacting this data can help to protect it from unauthorized access, reduce the risk of data breaches and ensure compliance with things like GDPR.

Pino provides a simple way to configure redaction in your logs. You can use the redact option to define a list of properties that should be redacted from your logs. You can also specify a replacement value for the redacted properties, which is useful when you need to keep some information but want to mask certain parts.

Examples of Pino logging redaction

Let’s take a look at some examples of how to use Pino logging redaction.

Redacting a single property

Suppose you have a log entry that contains a password field. You want to redact the password from the log. Here’s how you can do it:

import pino from 'pino';

const logger = pino({
  transport: {
    target: 'pino-pretty',
  },
  redact: ['password'],
});

logger.info(
  {
    password: 'myPassword123',
    payment: {
      card: {
        number: '1234-5678-1234-5678',
        issuer: 'Visa',
      },
    },
  },
  'Log message'
);

which outputs:

[19:00:28.282] INFO (7132): Log message
    password: "[Redacted]"
    payment: {
      "card": {
        "number": "1234-5678-1234-5678",
        "issuer": "Visa"
      }
    }

Redacting nested properties

Sometimes, you may need to redact multiple properties from your logs. Here’s an example that redacts a credit card number property from the log output.

import pino from 'pino';

const logger = pino({
  transport: {
    target: 'pino-pretty',
  },
  redact: ['password', 'payment.card.number'],
});

logger.info(
  {
    password: 'myPassword123',
    payment: {
      card: {
        number: '1234-5678-1234-5678',
        issuer: 'Visa',
      },
    },
  },
  'Log message'
);

which outputs:

[19:00:28.282] INFO (7132): Log message
    password: "[Redacted]"
    payment: {
      "card": {
        "number": "[Redacted]",
        "issuer": "Visa"
      }
    }

Using the `censor` property

You can use the censor property to control how information is redacted.

import pino from 'pino';

const logger = pino({
  transport: {
    target: 'pino-pretty',
  },
  redact: {
    paths: ['password', 'payment.card.number'],
    censor: '🤫',
  },
});

logger.info(
  {
    password: 'myPassword123',
    payment: {
      card: {
        number: '1234-5678-1234-5678',
        issuer: 'Visa',
      },
    },
  },
  'Log message'
);

which outputs:

[19:00:28.282] INFO (7132): Log message
    password: "🤫"
    payment: {
      "card": {
        "number": "🤫",
        "issuer": "Visa"
      }
    }

Removing sensitive data

Sometimes you may want to remove sensitive data from your logs altogether. You can do this by setting the remove property to true.

import pino from 'pino';

const logger = pino({
  transport: {
    target: 'pino-pretty',
  },
  redact: {
    paths: ['password', 'payment.card.number'],
    remove: true,
  },
});

logger.info(
  {
    password: 'myPassword123',
    payment: {
      card: {
        number: '1234-5678-1234-5678',
        issuer: 'Visa',
      },
    },
  },
  'Log message'
);

which outputs:

[19:00:28.282] INFO (7132): Log message
    payment: {
      "card": {
        "issuer": "Visa"
      }
    }

Don't repeat yourself: Why you should use child loggers in Node.js

Mon, 20 Feb 2023 00:00:00 GMT

Logging is an essential part of any application, providing insight into the what’s happening behind the scenes. However, as your codebase grows, it can be challenging to keep track of all the different log statements and where they’re coming from. This is where child loggers come in.

Child loggers are a feature of many of the Node.js logging libraries such as Pino, Bunyan and winston have that allow you to create a new logger that inherits the configuration of its parent logger.

This means you can create child loggers that are pre-configured with specific options, making it easier to log messages without repeating the same configuration over and over again.

In this blog post, we’ll take a look at an example of how child loggers can help cut down on repetition in TypeScript and Pino.

In this example, we have a function handleRequest that handles an incoming HTTP request. We extract the x-request-id and x-user-id headers from the request and use them as context for our log statements.

import pino from 'pino';

const logger = pino({
  transport: {
    target: 'pino-pretty',
  },
});

function handleRequest(req, res) {
  const requestId = req.headers['x-request-id'];
  const userId = req.headers['x-user-id'];

  logger.info({ requestId, userId }, 'Handling request');

  logger.info({ requestId, userId }, 'Processing request');

  logger.info({ requestId, userId }, 'Finished handling request');

  res.send('Success');
}

const req = {
  headers: {
    'x-request-id': '123',
    'x-user-id': '456',
  },
};
const res = {
  send: (msg) => console.log(msg),
};
handleRequest(req, res);

As you can see the in the code above we’re repeating the same details on every log line. This can make the code harder to read and maintain, and it also means that we have to remember to include the same details on every log line.

The log output looks like this:

[19:40:01.825] INFO (73183): Handling request
    requestId: "123"
    userId: "456"
[19:40:01.825] INFO (73183): Processing request
    requestId: "123"
    userId: "456"
[19:40:01.825] INFO (73183): Finished handling request
    requestId: "123"
    userId: "456"

Using Child Loggers to Simplify Logging

Using a child logger, we can simplify this code and make it easier to read and maintain. Here’s an example of how we could refactor the code to use a child logger:

import pino from 'pino';

const logger = pino({
  prettyPrint: true,
});

function handleRequest(req, res) {
  const requestId = req.headers['x-request-id'];
  const userId = req.headers['x-user-id'];

  /*
  Next, we create a child logger using logger.child({ requestId, userId }).
  
  This child logger inherits all the settings of the parent logger,
  and also includes the requestId and userId as context.   
  */
  const childLogger = logger.child({ requestId, userId });

  /*
  We then use the child logger to log additional messages, which automatically
  include the context provided at creation time.
  */

  childLogger.info('Starting to handle request');

  childLogger.info('Processing request');

  childLogger.info('Finished handling request');

  res.send('Success');
}

The log output looks like this:

[19:40:01.263] INFO (73809): Starting to handle request
    requestId: "123"
    userId: "456"
[19:40:01.263] INFO (73809): Processing request
    requestId: "123"
    userId: "456"
[19:40:01.263] INFO (73809): Finished handling request
    requestId: "123"
    userId: "456"

As the code above shows using a child logger in this way provides several benefits over repeating the same details on every log line e.g.

logger.info({ requestId, userId }, 'Processing request');
// vs.
childLogger.info('Processing request');

First, it reduces the amount of code required to log context, making the code more concise and easier to read.

Second, it ensures that all log statements for a particular request include the same context, which can be helpful for debugging.

Finally, it allows you to easily add or remove context as needed without modifying all of your log statements.

Give them a try

If you’re not already using child loggers in your code, give them a try! They can make a big difference in the readability and maintainability of your logging code.

Child loggers are not only the future of efficient logging, but they also remind us that even in code, children are the future.

Data-Driven vs Assumptions : Why Engineers Should Take a Doctor's Approach

Fri, 10 Feb 2023 00:00:00 GMT

Engineers are responsible for designing and building complex systems that are expected to perform optimally under different conditions. However, without a deep understanding of how these systems behave, they risk making assumptions that can result in inefficiencies or even system failures.

In this post, we’ll discuss why understanding a system like a doctor helping a patient is essential for engineers and by taking a proactive and data-driven approach, engineers can ensure that their systems are performing optimally and can make informed decisions that result in improved performance and reliability.

Photo by Zach Vessels

Just like a doctor diagnosing a patient, engineers must take a proactive approach to understand the behaviour of their systems. This can involve various techniques such as logging, tracing, and metrics collection. By collecting this information, engineers can gain insight into the performance and behaviour of their systems, reducing the number of assumptions and unknowns.

For example, if an engineer notices that a system is running slow, they may assume that it is due to a lack of computing resources. However, upon further investigation through logging and tracing, they may discover that the slow performance is actually due to a bottleneck in the network or an issue with the database.

By taking a logging, tracing, and metrics approach, engineers can avoid making incorrect assumptions and make data-driven decisions that result in more effective and efficient systems. In addition, they can identify and resolve performance bottlenecks, detect and debug errors, and monitor system behaviour over time.

In the next post we’ll discuss why this is essential for engineers to be able to give accurate estimations.

Why Engineers Should Try to Reproduce Production Issues Locally

Mon, 23 Jan 2023 00:00:00 GMT

As engineers, one of our primary responsibilities is to ensure that the systems we build are stable and reliable.

However, despite our best efforts, issues and issues will inevitably arise in production environments. When this happens, it can be tempting to try and patch the problem and move on quickly.

However, recreating production issues locally is a critical step in the debugging and resolution process.

In this post, I’ll explain the benefits of reproducing production issues locally.

Photo by Aaron Huber on Unsplash

Controlled environment

One of the main benefits of recreating production issues locally is that it allows us to reproduce the issue in a controlled environment.

By doing so, we can more easily isolate and identify the root cause of the problem. Trying to do this is much more challenging when the issue is only present in a production environment where many other factors are at play.

Validate Resolutions

Recreating production issues locally allows us to test and validate any proposed fixes before deploying them. This is essential for preventing regressions and ensuring the fix resolves the issue. Recreating the issue locally can also help identify and resolve other problems that may have been overlooked.

Insights into the system

Another benefit of recreating production issues locally is that it can help improve our understanding of the system. By closely examining the issue, we can gain valuable insights into how the various components of the system interact, which can help us prevent similar problems from arising.

Improve communication and collaboration

Photo by thisisengineering

I’ve also found recreating production bugs locally has helped improve the engineering team’s communication and collaboration. By working together to reproduce and resolve the issue, we shared knowledge and learnt from one another, which helped improve the team’s morale and motivation.

Is it necessary to reproduce every defect before identifying and fixing it?

Of course, not every issue will be reproducible locally because of certain complex conditions and environments, but engineers should at least try to do so. An example of these situations could be the hardware the software is running on, extensions the users’ browser has installed or, as I once encountered, how users within China had a different experience of network conditions from those outside when connecting to an AWS instance within a Chinese region.

When teams cannot reproduce issues locally, then teams and businesses are put into a position where they have to decide if blind patches are a good idea and will not make a situation worse. One position you don’t want to find yourself in is looking for something similar with the same consequence but a different root cause. Adding more instrumentation can help to prevent this.

Then, of course, are the pressures on engineering teams to produce a fix.

These pressures could be internal, from management wanting to see action, or external, to calm down an angry customer.

Conclusion

Engineers should attempt to recreate production bugs to improve reliability and stability of software and not damage the company’s credibility or reputation.

One of the best ways of recreating production issues locally is by tests, which I’ll discuss in the next post.

Documentation isn't why your onboarding sucks

Tue, 29 Nov 2022 00:00:00 GMT

Some organisations have great onboarding processes, others not so much.

Having experienced onboarding with many clients over the years, I want to share what a good onboarding process is for an engineer.

Photo by Erwan Hesry on Unsplash

Why it matters?

Onboarding is the first impression a new employee has of your company. It’s the first time they get to see how you work, how you communicate, how you solve problems and how you treat your employees.

A new employee will form an opinion of your company based on their onboarding experience. If it’s a good experience, they will be more likely to stay with your company. If it’s a bad experience, they will be more likely to leave.

I know when starting with a new client, imposter syndrome kicks in. The faster I feel like I’m adding value, the more confident I can be about doing the job and if I’m going to be able to fit in.

Do it yourself

Being told there isn’t any documentation and that you should write it yourself as you learn doesn’t work. It’s a terrible idea and a waste of time, as you’ll be writing without knowing the rationale, or constraints the team had when making decisions.

More often than not, the fact you are writing the documentation yourself means you may struggle to help find people to answer the questions you will be asking.

Needle in a haystack

Reading outdated confluence pages can be very frustrating because, in my experience, the author didn’t write with the intent to take the reader on a journey. So if the pages don’t follow a structure or are organised in a way to understand what to do or where to go next, you may end up reading endless amounts of irrelevant information and feeling lost.

The Weeknd Super Bowl GIFfrom The Weeknd GIFs

Jira Tickets

Trying to learn by piecing together Jira tickets as an onboarding exercise can also be futile. With similar problems to reading confluence pages, there’s no way to know if a ticket is still relevant and have any confidence the actual implementation accurately reflects the acceptance criteria.

Self-documented code

No doubt you’ve heard the saying write once, read many times. Reading code might work in simple cases, and it can be an excellent way to understand some of the team’s standards and how to structure and organise code. However, it doesn’t help explain the why behind the implementation. In addition, it doesn’t give information on what role the code plays in the organisation’s architecture.

Unit tests

Unit tests ensure code works in a particular way under various conditions. So while they could help with learning, they won’t have written in an onboarding friendly way.

End-to-end tests

End-to-end tests can be helpful when written in collaboration with people who understand the business needs, such as product owners. Then especially in the form of specifications, they can be relied upon to act as a data source to understand product requirements. However, this won’t help in understanding the role this plays in the grand scheme of things or the product vision.

Pairing

Pairing is a great way to onboard a new team member.

Photo by ThisisEngineering RAEng on Unsplash

This doesn’t mean hand-holding through the codebase, but it does mean explaining the problem and the implemented solution.

It’s also a great way to get to know your new team member and for them to get to know you.

Depending on their experience, your ability to communicate, how fast they learn and the complexity of your code base and architecture, you may need to pair for a few days or a few weeks.

There should be breaks to work independently, so they can get a feel for the codebase and the problem they are solving.

Onboarding shouldn’t just be with engineers

New team members should spend time understanding the product and expected outcomes with product owners and stakeholders. Otherwise, they may find it difficult, if not impossible, to understand the vision, value and purpose of the work the team are doing.

Domain experts should be involved in the onboarding process as they can educate and provide guidance on domain boundaries, events, and business rules within the organisation using a ubiquitous (vocabulary shared by everyone involved in a project) language.

If anyone in your organisation knows rather than thinks how things work, it will be testers. Testers should be able to take you through various scenarios to show the rules and constraints of an application. So while product owners can describe a feature using a user story, testers should be able to verify and demonstrate the implementation of that feature.

So, what is a good onboarding process?

There isn’t a silver bullet of doing just one thing for a great onboarding process.

A good onboarding process will be a combination of things, but without a doubt, team culture and collaboration are crucial components of a great onboarding process.

While collaboration may slow down productivity in the short term, the team will be more productive in the long term, with all team members understanding the “why” and how they can make valuable contributions.

Teams should communicate like a team fielding in cricket

Tue, 08 Nov 2022 00:00:00 GMT

When working with clients, how teams communicate tells me more about the culture than anything else.

Successful teams should communicate like a team fielding in a game of cricket who work together towards the same goal giving each other support and encouragement.

By sharing information about the batters, pitch, conditions and strategy, a feilding team agree on a plan on how to bowl and position themselves.

Public vs Private communication

A team that trusts one another will use public channels more than private channels.

A team using public channels to communicate honestly and openly can share knowledge and learn from each other.

Communication in public channels helps a team’s agility by having a shorter feedback loop. They can inspect and adapt without waiting for a retrospective. As a result, some client teams found they didn’t need retrospectives.

All team members should be able to ensure psychological safety and call out others when they are not adhering to a code of conduct.

When unable to do so, teams will use private channels to communicate. The lack of transparency and knowledge sharing results in less coordination, efficiency and performance.

More meetings, More problems

Teams with private channels will have many more meetings than those who communicate over public channels. Ineffective meetings waste a lot of time, energy and effort and often lead to complaints about not having enough time to get work done.

Effective vs Ineffective communication

The way teams communicate is a reflection of the culture of the team. The culture of the team is a reflection of the leadership of the team. The team’s leadership is a reflection of the organisation’s culture.

When everyone can contribute to the conversation, there is a greater sense of ownership, responsibility and healthy team spirit.

Liz Fosslien perfectly illustrated this

pic.twitter.com/fVr5fPJlR5
— Liz Fosslien (@fosslien) September 19, 2022

When one person dominates a conversation, teams struggle to communicate when that person isn’t present. Therefore, it’s crucial that team members can facilitate discussions to have meetings with diverse views and outcomes.

Communication leads to success

Many great cricketing sides highlight how communication was crucial to their success.

Before 2003, the Indian Cricket team had many talented individuals who hadn’t fulfilled their potential.

During the World Cup in 2003, the team did something different.

India changed how they played and communicated by sharing knowledge of the conditions and the opposition to their advantage. “The Great Indian Huddle” symbolised the change in team spirit.

India went on to win their first World Cup.

Product teams can show the same level of success when they communicate effectively.

Organisations should provide training for them to be able to do so.

Facilitation is an art

Wed, 15 Jun 2022 00:00:00 GMT

A good facilitator seamlessly keeps the flow of conversations fluent and relevant, encouraging people to engage and have equal participation while remaining impartial and patient.

It’s a role similar to a conductor orchestrating a choir.

In the development world, a scrum master often performs the facilitation role in agile ceremonies.

Having worked with many clients over the years, I’ve realised that scrum masters who excelled in facilitation played a crucial role in how teams learnt to trust, build relationships, communicate, organise, coordinate and work together. They allowed product owners, developers, designers, and testers to do what they do best and deliver value.

These facilitators were passionate about their art. For them, it’s a lifelong journey to improve and adapt their skills by learning and working with different teams at start-ups and enterprises in various industries.

However, just like a lousy conductor could ruin a concert, a bad facilitator can have severe consequences for teams and companies.

I’ve sat through many ceremonies where people did not contribute because of fear or not having the opportunity to do so. The facilitator didn’t have the training or skills to intervene or, worst still, hasn’t even realised.

Facilitators know how to adapt. Instead of sticking to broken ideals and rituals, they work to resolve and improve by collaborating with others to do what’s best for them as a team and reach an outcome they would jointly like to achieve.

The simple fact is while anyone can facilitate, not everyone is, can or will be a great facilitator.

Creating An Animation Sequence Using Framer Motion

Tue, 01 Mar 2022 00:00:00 GMT

Framer Motion makes creating a sequence of animations easy.

In this example, we will create a React component that animates a circle in the center of three boxes.

This works by finding the bounding rectangle of each box. As we’re using React a ref. Which is created like this

const box1Ref = useRef<HTMLDivElement>(null);

and used as follows

<div
  className="w-20 h-20 border-2 border-purple-800 bg-purple-300"
  ref="{box1Ref}"
></div>

With a reference to the element the calling getBoundingClientRect provides us with position and size properties.

const domRect: DOMRect | undefined = box1Ref.current?.getBoundingClientRect();

With this information we can to calculate the position where the circle to should move to.

const calculateCenter = (box, circle) => {
  return {
    top: box.y + box.height / 2 - (circle.y + circle.height / 2),
    left: box.x + box.width / 2 - (circle.x + circle.width / 2),
  };
};

Now we can create an animation sequence that moves the circle between the three boxes, which is easy as Framer Motion returns a promise for animations which is resolved upon when the animation is complete.

const circleRect = circleRef.current?.getBoundingClientRect();
const box1 = calculateCenter(
  box1Ref.current?.getBoundingClientRect(),
  circleRect,
);
const box2 = calculateCenter(
  box2Ref.current?.getBoundingClientRect(),
  circleRect,
);
const box3 = calculateCenter(
  box3Ref.current?.getBoundingClientRect(),
  circleRect,
);
await animation.start({
  opacity: 1,
  transition: { duration: 1 },
});
await animation.start({
  top: box1.top,
  left: box1.left,
  transition: { duration: 1 },
});
await animation.start({
  top: box2.top,
  left: box2.left,
  transition: { duration: 1 },
});
await animation.start({
  top: box3.top,
  left: box3.left,
  transition: { duration: 1 },
});

The complete code for the example is available on CodePen

See the Pen Untitled by jagreehal (@jagreehal) on CodePen.

How to prevent deadlock for UI tickets

Wed, 18 Aug 2021 00:00:00 GMT

Should you build UI components and then add them to a page or build the page first then break it down into UI components is the chicken and egg of software development?

There are many approaches to develop UI components.

One of the aims of breaking a UI into components is to have reusability.

Reusable components enable companies to provide users with a consistent user experience and avoid the cost and time of solving the same problem over and over.

So how should a company break down the work of creating a user interface? In my opinion, the question should be how can we verify that independent components are working correctly.

The answer to that is, it depends :)… but in my experience, you’ll want to end up with a suite of UI components.

Can’t see the wood for the trees

I’ve seen so many teams getting into the trap of only verifying a component is working when used within a page. In that case, there will be a higher chance of ticket deadlock as neither the component ticket nor the page ticket can’t progress because it’s hard, if not impossible, what is or isn’t working.

Component driven design FTW

Component driven design allows teams to use composition to develop pages while allowing for variations for how the component would behave in various states, such as empty states.

Why great products need strong foundations

Wed, 12 May 2021 00:00:00 GMT

You’ve probably heard someone quote Norm Kerths’ Prime Directive during a retrospective

Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.

In my experience, engineering teams doing their best at that time inevitably end up having to shoehorn features as requirements change or new information becomes available.

In this post, I’ll discuss why great products need strong foundations.

A solid product foundation

A solid product foundation can:

influence an engineering teams morale

A poorly architected product can kill any motivation for innovation as there will be a significant amount of effort required to make improvements. Engineers may choose the easiest rather than the right thing to do.

impact extensibility, maintainability and stability of the product

When a team has to accommodate changes that don’t fit existing patterns, conventions and standards, the harder it will be to keep things simple, consistent or easy to follow.

allow for more accurate estimations

The more complex a product architecture, the less accurate the estimation becomes as it is harder to know the side effects of changes.

effect the organisations’ ability to adapt to the market in the long run

Once a team cannot release confidently, the business will find it hard to adapt to the market or user needs.

This is more than technical debt

I’ve used this shopping bag analogy with clients I’ve worked with.

To begin with, you take a shopping bag to the shops to pick up two orders placed by your partner. At the counter, a staff member gives you the first order containing three things you put into the bag.

Then you are given the second order containing another three items that you also put into the bag.

Not ideal, but you can live with that.

Maybe you can take the hit and file it under technical debt.

The staff member then hands you three additional items that were also in the second order.

They are awkward to fit into the bag.

If you’ve had known about the items at the start, you would have organised your bag differently.

You have two choices.

You could add the items. However, the cost of doing so means you can’t use the handle making everything you do next awkward and challenging.

Or you could take the time to empty your bag and organise things so you can carry them home.

What should teams do?

One is a long term win. The other is a short term win for which the consequences are unknown, and the precedent for ‘temporary fixes’ has been set.

Stakeholders, product managers etc., should trust engineers in what they think is the right thing to do before a complete rewrite is required or it becomes problematic if not impossible to add value to a product.

Introducing Code-First GraphQL APIs at Cambridge University Press

Mon, 22 Feb 2021 00:00:00 GMT

Having experienced the pain of schema-first GraphQL projects, I wanted to use a different approach for a project I led at Cambridge University Press.

In this post, I’ll describe what I used and how using code-first GraphQL worked out for us.

In my experience, having a single source of truth makes applications easier to understand, maintain and add features to.

A common approach when learning GraphQL is to adopt a schema-first approach. The difficulty and complexity of schema-first GraphQL APIs increases as the project and teams grow, as you have to ensure definitions and resolvers are in sync and have to maintain multiple sources of truth.

Code-first approach with Nexus

By using a code-first approach with Nexus, we were able to can have have a single source of truth as it will generate a GraphQL schema for you.

However Nexus does a lot more than saving from mistakes and time of generating a GraphQL scheme manually.

Thanks to excellent typescript integration Nexus provides a delightful developer experience with lightning-fast feedback.

The plugin system and extensibility points enabled us to create a GraphQL API that was easy to maintain and extend.

How how did Nexus and code-first GraphQL work out for us?

Having avoided the pitfalls of schema-first, the benefits of a code-first approach to building GraphQL APIs increased our productivity and enabled us to release features quickly knowing everything was in sync.

Over the next few months, I’ll post examples of how we used Nexus and share what we learned when creating GraphQL APIs.

Introducing Tailwind at Cambridge University Press

Mon, 22 Feb 2021 00:00:00 GMT

Having used Tailwind in my personal projects, I was already a convert of the utility first approach. When I was given the opportunity to lead a new project at Cambridge University Press, Tailwind was the first tool I reached for.

In this post, I’ll discuss why and how using Tailwind worked out for us.

The failed promise of CSS Frameworks

Throughout my career, I witnessed teams struggling to maintain web applications using UI frameworks such as Bootstrap, Foundation and Material UI.

The initial benefits didn’t last long, and their web applications eventually ended up the same sorry way.

Productivity slowed as teams struggled to try to make sense of spaghetti codebases unable to deal with yet another specificity war or struggle to update the UI kit they had built upon.

Teaching the team about Tailwind

As this would be the first project in the organisation to use Tailwind, I wanted to gain my colleagues’ support. Having experienced the pain of using existing solutions, they were open to an alternative approach.

So I used our developer meetings to explain how and why I planned to use Tailwind.

I’m currently writing up what I presented. In the meantime, this is the link to the repository I created for my Tailwind Talk

So how did choosing Tailwind work out for us?

The developer experience and low learning curve of Tailwind, made it easy for people to learn it’s just surgery syntax on top of CSS with well thought out conventions.

By being able to specify constraints in the Tailwind configuration file, it was straightforward for the team to build components that followed brand guidelines.

It was simpler to understand what was going on in pull requests as we no longer had to match class names in markup and style in separate files, especially for responsive design.

Using utility classes in template files made collaboration easier with everyone from design to accessibility experts during pairing sessions.

The site looked incredible and knowing we only shipped the CSS that was actually being used helped us build a performant web application.

Using Tailwind was a success for the project, and team members loved using it.

How do I know this? They went on to use it on projects they led.

Introducing Typescript at Cambridge University Press

Mon, 22 Feb 2021 00:00:00 GMT

Having worked with many clients who struggled to maintain Javascript projects, Typescript was a no brainer when choosing what to use for the project I was leading.

In this post, I’ll share what and how I managed the challenges of introducing and using Typescript.

While Typescript adoption grew, I was always surprised how many of my clients chose not to use it.

Some firmly held the view Javascript “wasn’t Java” and shouldn’t be typed, others said their projects were simple and wouldn’t benefit from strong typing.

Those who understood the benefits didn’t know they could adopt Typescript incrementally without rewriting everything in one go.

To convince my clients the benefits of Typescript was more than type safety, I presented a talk based on “Making Impossible States Impossible” by Richard Feldman to clients and at the local Javascript meetup group in Cambridge.

During the talk, I showed how companies could adopt typed Javascript using both Flow and Typescript incrementally.

Evangelising Typescript

Having taught developers at Arm how to use Stencil, a library that required UI components to be written in Typescript, I had learnt how to train colleagues without them feeling overwhelmed.

My aim was for anyone working on my project to contribute, even if they had little or no experience using Typescript.

We did many pairing sessions and shared ideas, concepts and patterns while avoiding pitfalls and keeping things simple and easy to understand.

It was fine for a developer to add ts-ignore comment at the top of a file and exclude a file from being typed checked during a feature’s development.

This meant developers could focus on the feature they were building and share preview URLs with stakeholders, product owners and testers earlier during the development process. Meanwhile, we worked collaboratively behind the scenes to ensure everything was type-checked before merging the feature branch.

For the adoption of Typescript to be successful, we needed to be able to demonstrate the value.

Compared to similar projects at Cambridge University Press, which displayed information for books and articles, our codebase was cleaner as we were able to differentiate between types.

Over the next few months, I’ll post more examples about how Typescript enabled us to have an awesome developer experience.

How did using TypeScript workout?

I’m confident Typescript will be adopted in both existing and future projects at Cambridge University Press.

Developers enjoyed using Typescript as it enabled them to focus on building features benefitting from better IntelliSense and faster feedback in their IDE than executing code within a browser. The organisation benefited from increased productivity, fewer bugs and having a product that was easier to maintain and extend.

How To Use Tailwind Scoped Css Styles With Next.js

Thu, 21 Jan 2021 00:00:00 GMT

In this post I’ll show why and how you can use scoped styles with Tailwind and Next.js.

Why

Using scoped CSS could save you from styling conflicts such as a naming clash where there are multiple classes with the same name, as shown in the example below.

How

CSS modules provide the functionality to have scoped styles.

And the good news is Next.js supports css modules out of the box!

All you need to do is create a file that ends with ‘.module.css’ such as ‘TailwindScopedComponent.module.css’

.button {
  @apply border border-green-500 bg-green-500 text-white rounded-md px-4 py-2 m-2;
}

and then import it within your component

import styles from './TailwindScopedComponent.module.css';

export default function Button() {
  return (
    <button className={styles.button}>I use scoped Tailwind classes</button>
  );
}

The repository for the demo can be found here.

Alternatives

Another approach worth considering is to use a prefix with Tailwind

How To Use Tailwind Scoped Css Styles With Svelte

Thu, 21 Jan 2021 00:00:00 GMT

In this post I’ll show why and how you can use scoped styles with Tailwind and Svelte.

Why

Using scoped CSS could save you from styling conflicts such as a naming clash where there are multiple classes with the same name, as shown in the example below.

How

Svelte makes using converting a component to use scoped styles with Tailwind easy thanks to single file components!

Before

<button
  class="border border-green-500 bg-green-500 text-white rounded-md px-4 py-2 m-2"
>
  I use Tailwind classes
</button>

After

and this is what it looks like using scoped styles

<button class="button">I use scoped Tailwind classes</button>

<style lang="postcss">
  .button {
    @apply border border-green-500 bg-green-500 text-white rounded-md px-4 py-2 m-2;
  }
</style>

The repository for the demo can be found here.

Alternatives

Another approach worth considering is to use a prefix with Tailwind

How To Use Tailwind Scoped Css Styles With Vue

Tue, 19 Jan 2021 00:00:00 GMT

In this post I’ll show why and how you can use scoped styles with Tailwind and Vue.

Why

Using scoped CSS could save you from styling conflicts such as a naming clash where there are multiple classes with the same name, as shown in the example below.

How

Vue makes using converting a component to use scoped styles with Tailwind easy thanks to single file components!

Before

<template>
  <button
    class="border border-green-500 bg-green-500 text-white rounded-md px-4 py-2 m-2"
  >
    I use Tailwind classes
  </button>
</template>

After

and this is what it looks like using scoped styles

<template>
  <button class="button">I use scoped Tailwind classes</button>
</template>

<style lang="postcss" scoped>
.button {
  @apply border border-green-500 bg-green-500 text-white rounded-md px-4 py-2 m-2;
}
</style>

The repository for the demo can be found here.

Alternatives

Another approach worth considering is to use a prefix with Tailwind

Using Tailwind? Consider adding a prefix

Tue, 19 Jan 2021 00:00:00 GMT

Tailwind is rightly receiving all the attention it deserves. Every developer I’ve introduced it to falls in love with working with CSS again!

On a recent Nuxt project, I noticed a potential issue when using Tailwind class names.

Suddenly ‘bg-green-500’ didn’t behave as expected, as an external component also using the same class had set the color to something else.

Workarounds such as adding !important unnecessarily will get into specificity and wars that inevitably end in tears.

Another workaround could be to use @apply and scoped styles, CSS modules or CSS-in-JS, but you lose some of the developer experience Tailwind offers.

Using scoped styles with Tailwind is easy and you can read how to do that with Vue, Next.js and Svelte.

Adding a prefix to Tailwind

However the simplest thing could be is to set a prefix in your Tailwind configuration file.

module.exports = {
  prefix: 'tw-',
};

and then use the prefix for your tailwind classes so ‘bg-green-500’ becomes ‘tw-bg-green-500’

<button
  type="button"
  class="tw-border tw-border-green-500 tw-bg-green-500 tw-text-white tw-rounded-md tw-px-4 tw-py-2 tw-m-2"
>
  Success
</button>

You can see how to add a prefix below

Yes, there’s still a chance of naming conflicts, but this is the best solution I can think of that keeps the awesome developer experience Tailwind with some level of safety without using @apply and scoped styles.

In my opinion, if you’re building a component library in Tailwind, adding a prefix will save you and your users from styling conflicts.

If you’re building a large app the earlier you start using a prefix the less headache you’ll have of hunting down and changing Tailwind CSS classes later.

Creating Stencil Apps with Tailwind

Wed, 13 Jan 2021 00:00:00 GMT

Stencil is a compiler for creating web components that work alone or with web applications written in React, Vue, Angular, Svelte, Ember or anything else you can think of.

Tailwind CSS deserves all praise love it’s getting right now as the developer experience is incredible.

In this post I’ll show how to create a Stencil web app using Tailwind CSS.

You can create a Stencil web application by running

npm init stencil

and choosing ‘app’ when asked to pick a starter

next add the following dev dependencies

npm install -D tailwindcss@latest @stencil/postcss@latest autoprefixer@latest

and run the following command to create a Tailwind configuration file ‘tailwind.config.js’.

npx tailwindcss init

In the Tailwind configuration file set the array of paths for where you’ll be using Tailwind classes. This is required for removing unused CSS when creating a production build

module.exports = {
  purge: [],
  purge: ['./src/**/*.html', './src/**/*.tsx', './src/**/*.css'],
  darkMode: false, // or 'media' or 'class'
  theme: {
    extend: {},
  },
  variants: {},
  plugins: [],
};

You can configure Stencil to use PostCSS as a plugin in ‘stencil.config.ts’

import { Config } from '@stencil/core';
import { postcss } from '@stencil/postcss';
import tailwindcss from 'tailwindcss';
import autoprefixer from 'autoprefixer';

// https://stenciljs.com/docs/config

export const config: Config = {
  globalStyle: 'src/global/app.css',
  globalScript: 'src/global/app.ts',
  taskQueue: 'async',
  outputTargets: [
    {
      type: 'www',
      // comment the following line to disable service workers in production
      serviceWorker: null,
      baseUrl: 'https://myapp.local/',
    },
  ],
  plugins: [
    postcss({
      plugins: [tailwindcss(), autoprefixer],
    }),
  ],
};

The next step is to inject Tailwind’s ‘base’, ‘components’, and ‘utilities’ styles in ‘src/global/app.css’.

@tailwind base;
@tailwind components;
@tailwind utilities;

That’s all the configuration done! Now you can start the Stencil development server by running

npm start

Using Tailwind in Stencil Components

It’s possible to use Stencil with the ShadowDOM which will be covered in a later post, but lets keep things simple for now

Open the file ‘src/components/app-root/app-root.tsx’ and remove the styleUrl and shadow properties from the component, and replace the header element to use Tailwind CSS classes.

import { Component, h } from '@stencil/core';

@Component({
  tag: 'app-root',
  styleUrl: 'app-root.css',
  shadow: true,
})
export class AppRoot {
  render() {
    return (
      <div>
        <header>
          <h1>Stencil App Starter</h1>
        </header>
        <header class="bg-purple-700 h-16 flex items-center shadow-md">
          <h1 class="text-white text-xl font-bold px-3">Stencil App Starter</h1>
        </header>

        <main>
          <stencil-router>
            <stencil-route-switch scrollTopOffset={0}>
              <stencil-route url="/" component="app-home" exact={true} />
              <stencil-route url="/profile/:name" component="app-profile" />
            </stencil-route-switch>
          </stencil-router>
        </main>
      </div>
    );
  }
}

The output for using Tailwind CSS with Stencil is shown below.

The code for how to use Tailwind CSS with Stencil can be found on GitHub.

In upcoming posts I’ll show how to create component libraries using Stencil and Tailwind covering topics such as scoped CSS styles.

How to fix: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission

Tue, 05 Jan 2021 00:00:00 GMT

It’s a good thing browsers don’t allow sites to autoplay audio or videos without user interaction.

However, on iOS in particular, you might encounter the error

The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

when there is an async operation between the user interaction and playing of audio.

The good news is there’s a workaround for it.

Playing audio in web apps

Before a site can play audio, a user needs to have interacted with it for example, clicking on a play button.

The code for this is simple

const App = () => {
  const [audioError, setAudioError] = useState<Error>();
  const handleClick = () => {
    setAudioError(undefined);
    new Audio(
      'https://www.videomaker.com/sites/videomaker.com/files/downloads/free-sound-effects/Free_ExplosionSharp_6048_97_1.wav'
    )
      .play()
      .catch((e) => {
        setAudioError(e);
      });
  };

  return (
    <div>
      <p className="mb-4">Clicking play works everywhere!</p>
      <button
        className="border-2 border-green-700 bg-green-500 text-white rounded-md px-4 py-2 text-center"
        onClick={handleClick}
      >
        Play ▶
      </button>
      {audioError && <div className="mt-4 text-red-600">AUDIO ERROR: {audioError.message}</div>}
    </div>
  );
};

However, if in between clicking the button and playing audio, an async operation occurs then the behaviour of browsers differ.

const handleClick = async () => {
  setAudioError(undefined);
  await new Promise((r) => setTimeout(r, 2000));
  new Audio(
    'https://www.videomaker.com/sites/videomaker.com/files/downloads/free-sound-effects/Free_ExplosionSharp_6048_97_1.wav'
  )
    .play()
    .catch((e) => {
      setAudioError(e);
    });
};

For example, clicking on the ‘play’ button after a promise is resolved in Chrome plays the audio, whereas, in Safari, you’ll see the error below:

The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

Fixing the ‘The request is not allowed’ bug

The workaround is to load the audio in one interaction, and play the audio in another interaction. Now you can run as many promises as you want, the browser won’t error.

The downside of this approach is you will have to change the UX of your web app.

import { useState } from 'react';

const App = () => {
  const [audioError, setAudioError] = useState<Error>();
  const [audio, setAudio] = useState<HTMLAudioElement>();

  // 1. load the audio in a user interaction
  const handleLoadClick = () => {
    const _audio = new Audio(
      'https://www.videomaker.com/sites/videomaker.com/files/downloads/free-sound-effects/Free_ExplosionSharp_6048_97_1.wav'
    );
    _audio.load();
    _audio.addEventListener('canplaythrough', () => {
      console.log('loaded audio');
      setAudio(_audio);
    });
  };

  // 2. now you can play the audio on all subsequent events
  const handleClick = async () => {
    setAudioError(undefined);
    await new Promise((r) => setTimeout(r, 2000));
    audio &&
      audio.play().catch((e) => {
        setAudioError(e);
      });
  };

  return (
    <div className="flex flex-col space-y-4">
      <p className="mb-4">
        The work around is to load the audio after a user interaction. Now you can play audio after
        async operations on iOS.
      </p>
      <button
        className="border-2 border-blue-700 bg-blue-500 text-white rounded-md px-4 py-2 text-center"
        onClick={handleLoadClick}
        disabled={audio !== undefined}
      >
        {audio ? '▼ Click the button below to play the audio' : 'Load Audio!'}
      </button>

      <button
        className={`border-2 border-green-700 bg-green-500 ${
          audio ? '' : 'cursor-not-allowed'
        } text-white rounded-md px-4 py-2 mt-4 text-center`}
        onClick={handleClick}
        disabled={audio === undefined}
      >
        {audio ? 'Play after resolving a promise ▶' : '▲ Click the button above to load audio'}
      </button>
      {audioError && <div className="mt-4 text-red-600">AUDIO ERROR: {audioError.message}</div>}
    </div>
  );
};

export default App;

You can try it for yourself here by loading the audio before clicking the play button

The full code for the examples above can be found here

Getting people to believe in your vision

Mon, 14 Dec 2020 00:00:00 GMT

It can be a struggle, endlessly trying to convince your organisation to believe in your ideas and vision.

In this post, I’ll share how I managed to do this.

Having been part of a team that successfully launched Cambridge Core, a web application providing digital access to books and journals, I left Cambridge University Press to join Compare The Market.

The tech first attitude at Compare The Market was amazing. The experience taught me better ways to build and architect web applications.

In 2018 I returned to Cambridge University Press.

Having had time to reflect on technical choices made when building Cambridge Core, I now knew how we could improve the product.

Building reputation and trust

I already had trust within my team, but for management to believe in my vision, I needed to demonstrate my ability, commitment and knowledge of the existing product. I did this by:

Getting involved when we had production issues. Using the same principles as Compare The Market, I helped the team get a better insight into how the application was performing and how we could improve performance.
Training, pairing and helping teammates develop features and resolve bugs.
Volunteering for things that need doing, but knew nothing about. I took on the lead role of moving an application onto a new Kubernetes platform. By learning on the job and in my own time, I gave the business confidence they could leave it to me to get the job done.
Collaborating with product owners to write tickets and develop features to break down silos.
Working with our scrum master on improving our workflow, and trying things to make us work even better as a team.
Sharing ideas with developers, architects, operations and management about technical improvements we could make.
Presenting talks about my ideas inside and outside of the organisation at user groups and conferences. The feedback from the tasks enabled me to inspect, adapt and improve my ideas.

Take a chance on me

By doing the above, Cambridge University Press gave me the opportunity to lead a new multi-million-pound project, Higher Education.

An opportunity that would change my life forever.

Hopefully, this helps or inspires you to persevere and continue carving out the chance to lead and implement your vision.

How agile helped Cambridge Assessment to scale

Mon, 07 Dec 2020 00:00:00 GMT

In 2008 the media were reporting on mistakes and delays in marking examination scripts. To ensure they never made the front page of newspapers for the wrong reasons, Cambridge Assessment decided to rethink how they electronically marked examination scripts for millions of students worldwide.

Example of a generic marking sheet

Their existing solution, developed using BizTalk, processed tasks in a single long-running transaction. While it was able to process the current number of scripts, Cambridge Assessment wasn’t confident it would be able to handle the ever-growing increase in demand as it had not been designed to work at such a scale.

I was hired along with two other engineers, Richard Miller and Ian Johnson, to create a solution that would scale. We joined a team alongside a project manager, test team and a member of the existing development team, whose experience and knowledge of the business domain gave us invaluable insight.

Our Engineering Manager

Adam Straughan was a true engineers manager.

He was open, honest and had the courage to voice his opinions which some of the management team didn’t like.

He trusted and empowered us to get on with it. He was passionate about creating a development culture to create innovative solutions using practices such as test-driven development and continuous deployment, a first for a project at Cambridge Assessment.

It wouldn’t be an understatement to say he played a significant role in revolutionising software development at Cambridge Assessment.

The agile approach

As far as I know, this was one of Cambridge Assessments first agile projects at this scale.

I had already worked on so many “agile” projects, but this was the best I had experienced.

All thanks to Richard, who did both a scrum master and engineer role at the highest level.

He was able to explain why we were doing what we did and made everything ridiculously simple. I still use his ideas in the teams I lead today.

Stand-ups were short, and to the point, planning sessions focused, and our retrospectives made a difference as we improved as a team throughout the project.

Gaining trust and confidence from the business

Our agile approach was perfect for gaining trust and confidence from the business. We needed to show outcomes rather than just output.

We had access to a stakeholder who had in-depth knowledge and experience of the complexities and rules in marking examinations.

After every sprint, we did demos to the business to show we were applying the same marking algorithms and able to compute the same result as the existing solution. We started with simple user stories at the start of the project before moving on more complex scenarios such as remarks.

Forming, Storming, Norming, and Performing

The four of us had never worked together before. Some of our skills and approaches overlapped others didn’t. We were all passionate about software development and went to the same user groups outside of work.

Our team grew over time with more engineers joining.

We learnt a lot from each other, probably no more than when we had conflicting views. It was my first time working with people who were passionate about the product and had to learn how to present my ideas, so they were simple to understand.

It was a psychologically safe environment to disagree. We trusted each other and ensured discussions remained constructive and not personal.

We knew how vital collaboration was. All members of the team put forward ideas and paired together when we needed to. We used whiteboards for discussions which helped to ensure all team members had a shared understanding.

To this day, we have no doubt the application would never have been as good as it was without us being passionate, empowered and having a feeling of ownership which gave us the energy to voice our opinions.

The architecture

As well as being able to process exam scripts correctly, the scalability of our solution would ultimately determine the success of our solution.

It didn’t take long for us to work out our approach. As we attended the same talks and read the same resources from blog posts to books, we had similar ideas about what to do.

We split the application into microservices each having a single responsibility. There was a service for:

storing examination scripts sent from a third party
for processing examination scripts
uploading results to an internal service

Transactions boundaries were small using queues to communicate between the services, which enabled additional benefits such as replayability.

So now we could deploy and scale components independently as required. Not bad in the days before auto-scaling!

Developing the solution

We worked closely with the test team to learn about the business domain, which helped us develop a ubiquitous language.

We added instrumentation which enabled us to develop dashboards and build tools to replay scripts which helped to debug and troubleshoot.

We were able to call on architects and database experts to ensure we were on the right path and give advice on best practices and what to look out for.

Our shared philosophy enabled us to build a culture in which engineers could easily understand the vision, collaborate, share ideas, innovate and contribute.

Small details such as being able to reproduce a bug with a unit test before fixing ensured we built a high-quality application.

As our team grew, we learned and adapted to ensure people were happy and enjoying working on the project. For example, we dedicated a different engineer every sprint to take on a ‘Batman’ role to answer questions from the business and the test team, allowing other engineers to focus without being distracted and having to switch context.

Being in a support role helped engineers learn how crucial logging was for them to understand why something was happening. Now when the engineer returned to regular duties, they understood how critical it was to write tests, handle errors and instrument their code.

Making the grade

Our agile approach of having short feedback loops and incrementally adding features helped us keep focus.

The application was launched on time and successfully processed millions of examination scripts for years to come allowing Cambridge Assessment to build upon its’ excellent reputation as a world-class qualification provider.

Creating a product culture at Cambridge University Press

Tue, 01 Dec 2020 00:00:00 GMT

In 2015 I was hired by Cambridge University Press to develop a web application providing digital access to over 35,000 books and 1.5 million journal articles, consolidating several smaller sites. The application would go onto have over 2 million users a day and generate £65 million in revenue per year.

It was a fantastic technical learning opportunity, but it was our culture and approach to product development that would teach me the most important lesson of all.

Our culture

I realised on my very first day that this contract would be different. Unlike every other place I had worked, everyone from the product leader, product owners and scrum master, to testers and developers genuinely wanted to help each other.

Our team was made up of people from various backgrounds at different stages in their careers. Nobody thought they were better than anyone else, with egos checked in at the door. Team members were happy to get up from their desks to collaborate together. Company politics were kept away from the team, and I was never put in any position where I had to choose a side or get involved in any kind of way.

All this made it easy for me to settle in.

Product Leadership

Up to this point in my career, I had worked with some outstanding people but not an exceptional product leader.

Vicky Drummond is what authors such as Marty Cagan describe as a great product leader. She had a passion for the product, understood the market and had a clear vision.

As well as having the skills, knowledge and experience, Vicky was trusted and loved by the team. She encouraged and supported the team to innovate and try things. Her enthusiasm was infectious. Everyone on the team wanted the product to be successful.

I would have run through a brick wall for Vicky.

Research into the market and storyboards helped us understand user journeys. Four personas were created, and every user story started with one of them, and so during planning, it was clear to the team why we were building a feature and what the expected outcome was.

Inspect and adapt

We adapted how we did things throughout the development of the product, ensuring agile was still working for us rather than being a checklist of things we had to do. This happened because the team trusted one another.

Our retrospectives were a safe place to voice concerns and issues, always starting by reading the prime directive. Paul Telford, our excellent scrum master, ensured everyone contributed and helped the team decide how to resolve problems and held us accountable for actions. This was different from many of the places where little was actually done.

Our stand-ups were more like catch-ups. We shared jokes, and it was something we wanted to do rather than had to do. As some of the team were based remotely, we organised events such as board game evenings when they came to the office in Cambridge, which created a bond between us.

Instead of the scrum master telling us what to do, he encouraged us to use our initiative and self organise. He wanted to talk less so the team could talk more.

If anybody was stuck or needed help, they used Slack to raise any things straight away rather than wait until the next morning.

We had empowerment and ownership of the product we were building.

People create a product culture, not processes

Cambridge Core was successfully launched in 2016, and I left the year after.

No other workplace has given me the same joyful experience developing a product.

I’ve read many books, watched talks and listened to podcasts about building a good product team. Many companies think they are doing the right thing by following a process rather than investing in current staff, hiring the right people and having a talented product leader.