Test WebMCP tools like an API (with Playwright)

The Chrome team's best practices and use cases guides end on evaluation tests.

Evals cover the agent's judgement. Did it pick the right tool, did it fill the form the way the user meant, did it stop when it should have asked. You can't hard-code those outcomes, which is why the guides send you to evaluation-driven development.

Underneath that sits a layer that is deterministic and dull. Once your page calls registerTool, it advertises callable functions with typed schemas to whatever agent is driving. That's an API. Get an enum wrong and an agent walks a path your buttons never allow, and the eval that catches it will report that the model got confused. Your UI tests will never see it.

I measured what Chrome 152 does when an agent calls these methods.

The API you didn't know you shipped

This is the imperative API: registerTool in script. The same contract test applies when Chrome synthesises the schema from toolname on a form.

The cart page registers two, add_to_cart and get_cart. This is the one with a schema worth testing:

await document.modelContext.registerTool({
  name: 'add_to_cart',
  description: 'Add a quantity of one catalogue item to the cart',
  inputSchema: {
    type: 'object',
    properties: {
      sku: { type: 'string', enum: ['espresso', 'cold-brew'] },
      qty: { type: 'number' },
    },
    required: ['sku'],
  },
  execute: (input) => JSON.stringify(addToCart(input.sku, input.qty ?? 1)),
});

Read that as a developer and it's a function. Read it as an agent and it's a contract: these are the arguments, this one is mandatory, and the only two SKUs in the world are espresso and cold brew. The agent builds its call from the schema before it ever touches your handler.

A typo in that enum is a production bug your UI tests will never see. Your buttons don't read the schema. The agent reads nothing else.

Two problems before you write a single test

Bundled Chromium has no WebMCP. Playwright ships its own Chromium, and document.modelContext isn't in it. Your everyday suite can't see the API at all.

Chrome ships it behind flags, on a moving draft. The surface has already moved once, from navigator to document. A suite pinned to today's behaviour goes stale without telling you.

Solve the first by faking the browser and you inherit the second: your fake drifts away from Chrome and your tests keep passing. So card 42 runs two lanes.

You'll spend your time in the default lane. It runs on any machine, needs no flagged browser, and tests the code you wrote. The native lane exists to stop the double turning into fiction.

The everyday lane

The double goes in through addInitScript, which runs before any page script, so the page finds document.modelContext exactly where it expects it. Reverse that order and the page registers nothing.

Then you test the tools the way an agent would use them.

The schema is the contract, so assert on it.

test('a tool schema names the arguments an agent must send', async ({ page }) => {
  const [tool] = await page.evaluate(() => document.modelContext!.getTools());

  const schema = JSON.parse(tool!.inputSchema);

  expect(schema.required).toEqual(['sku']);
  expect(schema.properties.sku?.enum).toEqual(['espresso', 'cold-brew']);
});

Note the JSON.parse. The spec stringifies inputSchema on the descriptor for the imperative API, so tool.inputSchema.properties is undefined and a test that forgets this passes for the wrong reason.

One cart, two doors. An agent calling a tool and a user clicking a button have to land in the same state. This is the assertion I'd fight to keep if I could only have one:

test('a tool call and a click reach the same cart', async ({ page }) => {
  await page.getByTestId('add-cold-brew').click();

  await page.evaluate(async () => {
    const tools = await document.modelContext!.getTools();
    const add = tools.find((tool) => tool.name === 'add_to_cart')!;
    await document.modelContext!.executeTool!(add, JSON.stringify({ sku: 'espresso' }));
  });

  await expect(page.getByTestId('cart-count')).toHaveText('2');
  await expect(page.getByTestId('cart')).toContainText('1 x cold-brew');
  await expect(page.getByTestId('cart')).toContainText('1 x espresso');
});

A tool that reports success while the cart stays empty has lied to the agent, and only the UI assertion catches it. The best practices guide asks you to update the interface once a function completes, because agents read the interface to plan their next step. This is that bullet, written as a test.

The browser that never shipped the API. Most of your users are on one. Drop the double and check the page still sells coffee:

await page.addInitScript(() => {
  Object.defineProperty(document, 'modelContext', { configurable: true, value: undefined });
});

The best practices guide also asks you to register tools when a page state makes them useful and unregister them after. An AbortSignal owns that lifetime, which is how a component drops its tools on unmount. Assert both halves, because a tool that outlives the screen it belongs to is an agent calling into a page state you've already torn down:

test('an AbortSignal drops the registration it owns', async ({ page }) => {
  const names = await page.evaluate(async () => {
    const controller = new AbortController();
    await document.modelContext!.registerTool(
      {
        name: 'temporary_probe',
        description: 'Registered for the length of one controller',
        execute: () => 'ok',
      },
      { signal: controller.signal },
    );
    const before = (await document.modelContext!.getTools()).map((t) => t.name);
    controller.abort();
    const after = (await document.modelContext!.getTools()).map((t) => t.name);
    return { after, before };
  });

  expect(names.before).toContain('temporary_probe');
  expect(names.after).not.toContain('temporary_probe');
});

That one runs in the native lane. The everyday lane could run it against the double, but I am the one who taught the double to drop a tool on abort, so the test would only ever agree with itself. A claim about the browser has to be checked against the browser.

What Chrome actually does

Measured against Chrome 152.0.7977.65, not read off the explainer:

The throwing case is the one that will cost you an afternoon. Your handler raises Unknown SKU: tea, and the agent receives none of that. It gets a rejection and a sentence about a script function that failed.

The guides tell you to validate strictly in code and loosely in schema, and to write errors the model can self-correct from. That advice only works if you return the error rather than throw it:

execute: (input) => {
  try {
    return JSON.stringify(addToCart(input.sku, input.qty ?? 1));
  } catch (error) {
    return `Could not add that item: ${error.message}`;
  }
}

The double is a claim

A test double says "this is what the browser does". Nothing keeps that claim true except reading the original again, which is the native lane's only job. It drives real Chrome with the flags on, and the files end in .contract.ts so the everyday run skips them.

It earned its place while I was building the card. My double resolved a throwing handler with the error message as text, which is what the WebMCP libraries I'd read do: they catch, and hand the agent something readable. I credited the browser for library behaviour. The first native run rejected instead, and both lanes now assert what Chrome does.

When one of these fails, suspect the double before the browser.

test('the browser owns the context, not a page script', async ({ page }) => {
  const shape = await page.evaluate(() => ({
    registerTool: document.modelContext!.registerTool.toString(),
    onPrototype: Object.getOwnPropertyNames(
      Object.getPrototypeOf(document.modelContext!),
    ).sort(),
  }));

  expect(shape.registerTool).toContain('[native code]');
  expect(shape.onPrototype).toContain('registerTool');
});

That test fails if the double leaks into the native lane, which is the failure mode that would quietly invalidate everything else.

Two traps

A data: URL has no WebMCP. Its origin is opaque, and document.modelContext is absent there in a browser that exposes it on every real page. Probe on http://localhost. I lost an hour concluding the feature was missing from a Chrome that had it.

Version checks lie. >= 152 passes on a build with the flags off and fails on a future build that renames nothing. Ask the browser, on the page you're about to test:

const available = await page.evaluate(
  () => typeof document.modelContext?.registerTool === 'function',
);
test.skip(!available, 'This Chrome exposes no document.modelContext');

Then feature-detect executeTool on its own. The draft puts it on ModelContext, but Chrome's testing surface is behind WebMCPTesting. A browser can expose registration without the hook, or the hook can disagree with the IDL, which is why the native lane exists.

Where the evals start

None of this tells you whether an agent picks add_to_cart over search_products, whether your description reads as an execution or an initiation, or whether an agent can complete the shopping list on the use-cases page without adding the same item to two carts. Those are the questions the Chrome guides point at, and they need an agent in the loop.

They also assume the layer underneath holds. Every eval you run against a wrong enum measures your model's patience with your schema.

pnpm test:webmcp            # default lane, no special browser
pnpm test:webmcp:native     # native lane, needs a Chrome with WebMCP

The native lane wants a Chrome that ships WebMCP, launched with --enable-experimental-web-platform-features and --enable-features=WebMCPTesting,DevToolsWebMCPSupport. Point CHROME_BIN at it if the default search misses yours.

The pattern is also an agent skill, playwright-webmcp. It wires the two lanes into a repo and carries the Chrome behaviours as a reference file the agent has to read before it writes the double. That's the part I'd otherwise expect an agent to get wrong, because the libraries it has seen catch handler errors and return the message as text, and it will write that down as browser behaviour. Mine did.

WebMCP is a draft under active discussion, so some of the table above will age. The native lane is how I'll find out when it does.

• The card: src/42-webmcp-tools
• Live card docs: Card 42
• Live harness: /webmcp
• The skill: skills/playwright-webmcp
• The explainer: webmachinelearning/webmcp
• Chrome's guides: best practices, use cases, build tools, secure tools