# Test WebMCP tools like an API (with Playwright)

By Jag Reehal, 2026-09-07. Canonical: https://arrangeactassert.com/posts/webmcp-tools-are-an-api-test-them-like-one/

<!-- Excerpt Start -->

The Chrome team's [best practices](https://developer.chrome.com/docs/ai/webmcp/best-practices) and [use cases](https://developer.chrome.com/docs/ai/webmcp/use-cases) guides end on evaluation tests.

Evals cover the agent's judgement. Did it pick the right tool, did it fill the form the way the user meant, did it stop when it should have asked. You can't hard-code those outcomes, which is why the guides send you to [evaluation-driven development](https://web.dev/learn/ai/evaluation-driven-development).

Underneath that sits a layer that is deterministic and dull. Once your page calls `registerTool`, it advertises callable functions with typed schemas to whatever agent is driving. That's an API. Get an `enum` wrong and an agent walks a path your buttons never allow, and the eval that catches it will report that the model got confused. Your UI tests will never see it.

I measured what Chrome 152 does when an agent calls these methods.

![The default lane on the command line: the six WebMCP tests, plus the two auth setup tests the chromium project depends on, passing in under three seconds](/static/gifs/webmcp-tests.gif)

<!-- Excerpt End -->

## The API you didn't know you shipped

This is the imperative API: `registerTool` in script. The same contract test applies when Chrome synthesises the schema from `toolname` on a form.

The cart page registers two, `add_to_cart` and `get_cart`. This is the one with a schema worth testing:

```typescript
await document.modelContext.registerTool({
  name: 'add_to_cart',
  description: 'Add a quantity of one catalogue item to the cart',
  inputSchema: {
    type: 'object',
    properties: {
      sku: { type: 'string', enum: ['espresso', 'cold-brew'] },
      qty: { type: 'number' },
    },
    required: ['sku'],
  },
  execute: (input) => JSON.stringify(addToCart(input.sku, input.qty ?? 1)),
});
```

Read that as a developer and it's a function. Read it as an agent and it's a contract: these are the arguments, this one is mandatory, and the only two SKUs in the world are espresso and cold brew. The agent builds its call from the schema before it ever touches your handler.

A typo in that `enum` is a production bug your UI tests will never see. Your buttons don't read the schema. The agent reads nothing else.

## Two problems before you write a single test

**Bundled Chromium has no WebMCP.** Playwright ships its own Chromium, and `document.modelContext` isn't in it. Your everyday suite can't see the API at all.

**Chrome ships it behind flags, on a moving draft.** The surface has already moved once, from `navigator` to `document`. A suite pinned to today's behaviour goes stale without telling you.

Solve the first by faking the browser and you inherit the second: your fake drifts away from Chrome and your tests keep passing. So [card 42](https://github.com/jagreehal/playwright-cookbook/tree/main/src/42-webmcp-tools) runs two lanes.

| Lane | Browser | Runs | Proves |
|------|---------|------|--------|
| default | bundled Chromium, `document.modelContext` double | every `pnpm test` | your page's tools: schemas, handlers, and the state they share with the UI |
| native | real Chrome with the flags on | on demand | the double still matches the browser |

You'll spend your time in the default lane. It runs on any machine, needs no flagged browser, and tests the code you wrote. The native lane exists to stop the double turning into fiction.

## The everyday lane

The double goes in through `addInitScript`, which runs before any page script, so the page finds `document.modelContext` exactly where it expects it. Reverse that order and the page registers nothing.

Then you test the tools the way an agent would use them.

**The schema is the contract, so assert on it.**

```typescript
test('a tool schema names the arguments an agent must send', async ({ page }) => {
  const [tool] = await page.evaluate(() => document.modelContext!.getTools());

  const schema = JSON.parse(tool!.inputSchema);

  expect(schema.required).toEqual(['sku']);
  expect(schema.properties.sku?.enum).toEqual(['espresso', 'cold-brew']);
});
```

Note the `JSON.parse`. The spec stringifies `inputSchema` on the descriptor for the imperative API, so `tool.inputSchema.properties` is `undefined` and a test that forgets this passes for the wrong reason.

**One cart, two doors.** An agent calling a tool and a user clicking a button have to land in the same state. This is the assertion I'd fight to keep if I could only have one:

```typescript
test('a tool call and a click reach the same cart', async ({ page }) => {
  await page.getByTestId('add-cold-brew').click();

  await page.evaluate(async () => {
    const tools = await document.modelContext!.getTools();
    const add = tools.find((tool) => tool.name === 'add_to_cart')!;
    await document.modelContext!.executeTool!(add, JSON.stringify({ sku: 'espresso' }));
  });

  await expect(page.getByTestId('cart-count')).toHaveText('2');
  await expect(page.getByTestId('cart')).toContainText('1 x cold-brew');
  await expect(page.getByTestId('cart')).toContainText('1 x espresso');
});
```

A tool that reports success while the cart stays empty has lied to the agent, and only the UI assertion catches it. The best practices guide asks you to update the interface once a function completes, because agents read the interface to plan their next step. This is that bullet, written as a test.

![The same six WebMCP tests in Playwright UI mode, scoped to the card so the setup tests are out of shot, with the cart test selected. The trace shows the click, the executeTool call, and a DOM snapshot holding one cold brew and one espresso](/static/gifs/webmcp-ui-mode.gif)

**The browser that never shipped the API.** Most of your users are on one. Drop the double and check the page still sells coffee:

```typescript
await page.addInitScript(() => {
  Object.defineProperty(document, 'modelContext', { configurable: true, value: undefined });
});
```

The best practices guide also asks you to register tools when a page state makes them useful and unregister them after. An `AbortSignal` owns that lifetime, which is how a component drops its tools on unmount. Assert both halves, because a tool that outlives the screen it belongs to is an agent calling into a page state you've already torn down:

```typescript
test('an AbortSignal drops the registration it owns', async ({ page }) => {
  const names = await page.evaluate(async () => {
    const controller = new AbortController();
    await document.modelContext!.registerTool(
      {
        name: 'temporary_probe',
        description: 'Registered for the length of one controller',
        execute: () => 'ok',
      },
      { signal: controller.signal },
    );
    const before = (await document.modelContext!.getTools()).map((t) => t.name);
    controller.abort();
    const after = (await document.modelContext!.getTools()).map((t) => t.name);
    return { after, before };
  });

  expect(names.before).toContain('temporary_probe');
  expect(names.after).not.toContain('temporary_probe');
});
```

That one runs in the native lane. The everyday lane could run it against the double, but I am the one who taught the double to drop a tool on abort, so the test would only ever agree with itself. A claim about the browser has to be checked against the browser.

## What Chrome actually does

Measured against Chrome 152.0.7977.65, not read off the explainer:

| Behaviour | What Chrome does |
|-----------|------------------|
| `getTools()` order | sorted by name, ignoring registration order |
| `inputSchema` on a descriptor | a JSON **string**, so parse before asserting |
| `executeTool(tool, input)` | the [draft](https://webmachinelearning.github.io/webmcp) takes an object; Chrome 152 wants a JSON string, and an object rejects with `UnknownError` |
| handler returns an object | serialised, so `{a:1}` arrives as `'{"a":1}'` |
| handler returns `undefined` | arrives as the string `"undefined"` |
| handler throws | the call **rejects** with `UnknownError`, and your message is replaced with a generic one |
| duplicate tool name | `InvalidStateError` |
| `AbortSignal` on registration | aborting withdraws the tool |

The throwing case is the one that will cost you an afternoon. Your handler raises `Unknown SKU: tea`, and the agent receives none of that. It gets a rejection and a sentence about a script function that failed.

The guides tell you to validate strictly in code and loosely in schema, and to write errors the model can self-correct from. That advice only works if you return the error rather than throw it:

```typescript
execute: (input) => {
  try {
    return JSON.stringify(addToCart(input.sku, input.qty ?? 1));
  } catch (error) {
    return `Could not add that item: ${error.message}`;
  }
}
```

## The double is a claim

A test double says "this is what the browser does". Nothing keeps that claim true except reading the original again, which is the native lane's only job. It drives real Chrome with the flags on, and the files end in `.contract.ts` so the everyday run skips them.

It earned its place while I was building the card. My double resolved a throwing handler with the error message as text, which is what the WebMCP libraries I'd read do: they catch, and hand the agent something readable. I credited the browser for library behaviour. The first native run rejected instead, and both lanes now assert what Chrome does.

When one of these fails, suspect the double before the browser.

```typescript
test('the browser owns the context, not a page script', async ({ page }) => {
  const shape = await page.evaluate(() => ({
    registerTool: document.modelContext!.registerTool.toString(),
    onPrototype: Object.getOwnPropertyNames(
      Object.getPrototypeOf(document.modelContext!),
    ).sort(),
  }));

  expect(shape.registerTool).toContain('[native code]');
  expect(shape.onPrototype).toContain('registerTool');
});
```

That test fails if the double leaks into the native lane, which is the failure mode that would quietly invalidate everything else.

## Two traps

**A `data:` URL has no WebMCP.** Its origin is opaque, and `document.modelContext` is absent there in a browser that exposes it on every real page. Probe on `http://localhost`. I lost an hour concluding the feature was missing from a Chrome that had it.

**Version checks lie.** `>= 152` passes on a build with the flags off and fails on a future build that renames nothing. Ask the browser, on the page you're about to test:

```typescript
const available = await page.evaluate(
  () => typeof document.modelContext?.registerTool === 'function',
);
test.skip(!available, 'This Chrome exposes no document.modelContext');
```

Then feature-detect `executeTool` on its own. The draft puts it on `ModelContext`, but Chrome's testing surface is behind `WebMCPTesting`. A browser can expose registration without the hook, or the hook can disagree with the IDL, which is why the native lane exists.

## Where the evals start

None of this tells you whether an agent picks `add_to_cart` over `search_products`, whether your description reads as an execution or an initiation, or whether an agent can complete the shopping list on the [use-cases page](https://developer.chrome.com/docs/ai/webmcp/use-cases) without adding the same item to two carts. Those are the questions the Chrome guides point at, and they need an agent in the loop.

They also assume the layer underneath holds. Every eval you run against a wrong `enum` measures your model's patience with your schema.

---

```bash
pnpm test:webmcp            # default lane, no special browser
pnpm test:webmcp:native     # native lane, needs a Chrome with WebMCP
```

The native lane wants a Chrome that ships WebMCP, launched with `--enable-experimental-web-platform-features` and `--enable-features=WebMCPTesting,DevToolsWebMCPSupport`. Point `CHROME_BIN` at it if the default search misses yours.

The pattern is also an agent skill, [`playwright-webmcp`](https://github.com/jagreehal/playwright-cookbook/tree/main/skills/playwright-webmcp). It wires the two lanes into a repo and carries the Chrome behaviours as a reference file the agent has to read before it writes the double. That's the part I'd otherwise expect an agent to get wrong, because the libraries it has seen catch handler errors and return the message as text, and it will write that down as browser behaviour. Mine did.

WebMCP is a draft under active discussion, so some of the table above will age. The native lane is how I'll find out when it does.

- The card: [`src/42-webmcp-tools`](https://github.com/jagreehal/playwright-cookbook/tree/main/src/42-webmcp-tools)
- Live card docs: [Card 42](https://jagreehal.github.io/playwright-cookbook/cards/42)
- Live harness: [/webmcp](https://jagreehal.github.io/playwright-cookbook/webmcp)
- The skill: [`skills/playwright-webmcp`](https://github.com/jagreehal/playwright-cookbook/tree/main/skills/playwright-webmcp)
- The explainer: [webmachinelearning/webmcp](https://github.com/webmachinelearning/webmcp)
- Chrome's guides: [best practices](https://developer.chrome.com/docs/ai/webmcp/best-practices), [use cases](https://developer.chrome.com/docs/ai/webmcp/use-cases), [build tools](https://developer.chrome.com/docs/ai/webmcp/build-tools), [secure tools](https://developer.chrome.com/docs/ai/webmcp/secure-tools)