Let AI judge. Let code decide.
23 Nov 2026
Ask a local decision model four yes-or-no questions about a support message, then let TypeScript pick the action. On my laptop, Ollama’s tev1 made the right call on five sample conversations, and its trace showed one wrong answer the rules had already made harmless.
A test on 18 labelled messages found the rest: a refund status question read as a refund request, and a cautiously worded fraud report that nobody would see.
The tempting pattern
A common way to build an AI support flow is to give a model the conversation, the account and the refund policy, then ask it what to do:
const { text: decision } = await generateText({
model,
prompt: `Here is the customer conversation. Here is their account.
Here are our refund policies. Decide what we should do.`,
});
That prompt asks the model to do two jobs:
- Work out what the customer means.
- Decide what the application is allowed to do.
Only the first job needs a model. The second is business logic, and you already have a good tool for that: code.
Think about how a bank handles the post. A clerk opens each letter and notes what the customer is asking for: a refund, a complaint, a change of address. The clerk does not decide whether the refund gets paid. A rulebook does, and the bank changes the rulebook without retraining the clerk. A language model is good at the clerk’s job. Asking it to do both jobs means writing the rulebook into the prompt and hoping the model follows it the same way every time.
Code behaves the same way every time. Run amount <= 25 a million times with the same amount and you get the same answer a million times. Engineers call that deterministic. A language model is probabilistic: it picks its answer by weighing likelihoods, so the same message can come back with a different answer on another run. That is fine for understanding what someone means. It is a problem for deciding whether to send them £240.
When the rules live in the prompt, every policy change becomes a prompt edit and every edge case adds another paragraph. You end up with a rulebook written in English that a model reinterprets on every request. The Determinism Dilemma covers why that sits badly with how most of us test software.
Split the decision into questions
A customer writes:
I’m not saying someone stole my card, but there’s a £240 charge from a shop I’ve never visited. Can you get my money back?
List the questions behind the decision and mark which ones need someone to read the language:
| Question | Kind |
|---|---|
| Does the customer want money back? | Judgment |
| Are they reporting an unauthorised charge? | Judgment |
| Are they asking for a human? | Judgment |
| Are they asking for a fee to be waived? | Judgment |
| Is the refund claim eligible? | Fact from the account |
| Is the amount £25 or less? | Fact from the account |
| Does fraud take priority over refunds? | Rule |
The model answers the four judgments. The account supplies the facts. Code holds the rules.
---
alt: "A customer message goes to tev1, which returns four probabilities: wants refund, reports fraud, requests human, requests fee waiver. Those combine with account facts, eligibility and amount, in a decide function written in TypeScript, which returns the action and the reason."
---
flowchart LR
M["Customer message"] --> J["tev1<br/>4 yes/no questions"]
J --> P["wantsRefund 0.99<br/>reportsFraud 0.90<br/>requestsHuman 0.47<br/>requestsFeeWaiver 0.92"]
F["Account facts<br/>eligible, amount"] --> D{"decide()<br/>in TypeScript"}
P --> D
D --> A["Action + reason"]
How to read the numbers
The model answers each yes-or-no question with a number between 0 and 1, the probability that the answer is yes:
| Number | Read it as |
|---|---|
| 0.99 | Almost certainly yes |
| 0.90 | Very likely yes |
| 0.50 | Can’t tell |
| 0.10 | Very likely no |
| 0.01 | Almost certainly no |
Treat those readings as the model’s claim. A 0.9 means right nine times in ten only if the model is calibrated, and the labelled test further down checks how far tev1 lives up to its numbers.
A threshold is the line you draw. “Treat it as fraud if reportsFraud is 0.6 or higher” is a threshold. You choose it in code. A low line catches more real cases and raises more false alarms. A high line raises fewer false alarms and misses more real cases. Where you put it depends on what each mistake costs you: a missed fraud report costs more than a fraud analyst reading a message that turns out to be fine.
Asking the questions
The model here is a decision model, a kind of model that writes no text. You give it some input and a list of questions, and it returns a number for each answer. That makes it fast and easy to plug into code, and it cannot ramble or invent an action you never offered. Ollama, which runs models on your own computer, serves decision models such as tev1 on /v1/systemone. Local decision models with AI SDK Ollama covers the API and how three of them compare on ticket routing. Through ai-sdk-ollama, a boolean question goes to Ollama as its noul type and comes back as the probability of yes:
import { experimental_evaluate as evaluate } from 'ai';
import { ollama } from 'ai-sdk-ollama';
export const QUESTIONS = {
wantsRefund: { type: 'boolean', instructions: 'Does the customer want money back?' },
reportsFraud: { type: 'boolean', instructions: 'Does the customer report a transaction they did not authorise?' },
requestsHuman: { type: 'boolean', instructions: 'Does the customer explicitly request a human?' },
requestsFeeWaiver: { type: 'boolean', instructions: 'Is the customer asking for a fee to be removed or waived?' },
} as const;
export async function judge(message: string, modelId = 'tev1') {
const { answers } = await evaluate({
model: ollama.evaluationModel(modelId),
state: { message },
questions: QUESTIONS,
});
return {
wantsRefund: answers.wantsRefund.probability,
reportsFraud: answers.reportsFraud.probability,
requestsHuman: answers.requestsHuman.probability,
requestsFeeWaiver: answers.requestsFeeWaiver.probability,
};
}
None of these questions asks whether the customer should get a refund, or which team should take the case. The model never sees the £25 limit or the account.
The repo pins ai 7.0.124 and ai-sdk-ollama 4.4.0. Later releases rename the call: from ai 7.0.137 and ai-sdk-ollama 4.5.0 you write experimental_decide and ollama.decisionModel(), and evaluationModel() stays as a deprecated alias.
Deciding in code
The policy is a plain function:
export const RULES = { fraudThreshold: 0.6, intentThreshold: 0.8, uncertainBelow: 0.4, autoRefundLimit: 25 };
export function decide(j: Judgments, a: Account, r = RULES) {
if (j.reportsFraud >= r.fraudThreshold) return { decision: 'FRAUD_REVIEW', reason: `reportsFraud ${j.reportsFraud} >= ${r.fraudThreshold}` };
if (j.requestsHuman >= r.intentThreshold) return { decision: 'HUMAN_HANDOFF', reason: '...' };
if (j.wantsRefund >= r.intentThreshold) {
if (!a.refundEligible) return { decision: 'REFUND_REVIEW', reason: 'refund wanted, account not eligible' };
return a.amount <= r.autoRefundLimit
? { decision: 'REFUND', reason: '...' }
: { decision: 'REFUND_REVIEW', reason: '...' };
}
if (j.requestsFeeWaiver >= r.intentThreshold) {
return a.feeWaiverEligible ? { decision: 'WAIVE_FEE', reason: '...' } : { decision: 'FEE_REVIEW', reason: '...' };
}
// Nothing cleared its bar, but something sits in the grey zone: a person looks.
if (Object.values(j).some((p) => p >= r.uncertainBelow)) return { decision: 'HUMAN_HANDOFF', reason: 'uncertain: ...' };
return { decision: 'ANSWER', reason: 'no judgment above 0.4' };
}
The order of the checks sets the priority. I set the fraud threshold lower than the others because a missed fraud report costs more than a false alarm. Anything between 0.4 and its threshold goes to a person instead of to code that moves money. I picked these numbers before measuring anything, and the test results below show why you should not keep them unchanged.
Because the rules are code, you test them like code. Nine node:test cases run without a model in about half a second:
test('fraud takes priority over refund', () => {
assert.equal(decide({ ...none, wantsRefund: 0.99, reportsFraud: 0.94 }, eligible).decision, 'FRAUD_REVIEW');
});
test('the £25 limit is inclusive', () => {
assert.equal(decide({ ...none, wantsRefund: 0.95 }, { ...eligible, amount: 25 }).decision, 'REFUND');
assert.equal(decide({ ...none, wantsRefund: 0.95 }, { ...eligible, amount: 25.01 }).decision, 'REFUND_REVIEW');
});
You can change the refund limit, reorder the priorities or add a rule without touching a prompt, and without paying for a single model call to check it.
What tev1 did with five conversations
I ran five conversations through tev1 on an M1 Pro. For each one the script prints a trace: every number the model returned, every account fact, every rule and the final decision with its reason. Here is the trace for the £240 message:
Judgments (tev1, 1263ms):
wantsRefund 0.99
reportsFraud 0.90
requestsHuman 0.47
requestsFeeWaiver 0.92
Facts:
refundEligible false
amount 240
Rules:
fraudThreshold 0.6
Decision: FRAUD_REVIEW
Reason: reportsFraud 0.90 >= 0.6
The customer avoided the word fraud, and the model still scored an unauthorised charge at 0.90. The decision is right.
The trace also shows a wrong answer. requestsFeeWaiver came back at 0.92, and the message mentions no fee. It changed nothing because decide() checks fraud first. A log that recorded only FRAUD_REVIEW would look perfect and hide the bad judgment until a message arrived where it mattered.
The other four conversations went where a person would send them:
| Message | Decision | Why |
|---|---|---|
| £35 late fee after a bank holiday, “can you remove it?” | WAIVE_FEE |
requestsFeeWaiver 0.99, account eligible |
| Charged £12 after cancelling, “please refund it” | REFUND |
wantsRefund 0.99, eligible, £12 ≤ £25 |
| “Going round in circles with the chatbot. Put me through to a person.” | HUMAN_HANDOFF |
requestsHuman 0.97 |
| “What’s my current balance?” | ANSWER |
every judgment below 0.4 |
Each message took about a second for four questions.
The same five conversations, with the model deciding
To check the tempting pattern against this one, I wrote the same policy as numbered rules in a prompt, added the account and the message, and asked two local chat models to reply with one action. Each model saw each conversation three times.
| Approach | Right | Same answer every run | Time per decision |
|---|---|---|---|
tev1 answers 4 questions, decide() picks |
15/15 | 5 of 5 conversations | about 1 s |
granite4.2:3b reads the policy and picks |
13/15 | 3 of 5 conversations | 5 to 60 s |
qwen3.5:4b reads the policy and picks |
15/15 | 5 of 5 conversations | 17 to 205 s |
Both chat models ran on Ollama 0.40.17 with the settings their model files ship with: Q4_K_M quantisation, temperature 1, and thinking on for qwen, which is its default. The script sets no temperature and does not warm the models up first. tev1 runs in mxfp8 at temperature 0. Turning qwen’s thinking off would cut its time, and I have not measured what it would do to its accuracy.
granite sent “What’s my current balance?” to a human on one run and answered it on the other two. On another run it flagged the cancelled £12 subscription as fraud, and took 60 seconds to do it. Same message, same prompt, different action. That is what probabilistic looks like when the model owns the decision.
qwen got every decision right, which is worth saying plainly: a capable model can follow a short written policy. It spent up to three and a half minutes per message to do it, because it writes out its reasoning before answering. And when it is right, you still cannot see why. Its log says FRAUD_REVIEW. It does not say whether it saw a fraud report, or whether it read the £25 rule correctly, and the next policy change goes back into a prompt you can only test by running the model.
With tev1 and decide(), the policy lives in code that gives the same answer every time and can be tested in half a second, and the model only does the part that needs reading.
Two kinds of failure
With the judgments in the trace, you can tell a wrong reading from a wrong rule:
- The judgment was wrong. Change the question, the model or the threshold. Leave the rules alone.
- The judgment was right, and the routing was wrong. Fix
decide()and add a unit test, a small automated check that runs the function with fixed inputs. Leave the model alone.
The fix for one is in a different file from the fix for the other, and with a single “decide what to do” prompt you cannot tell which one you need. The same idea runs through tracking every decision in payments: record the inputs to a decision next to the decision, so you can tell later which input was wrong.
Testing the judgments on labelled messages
Five correct conversations is too few to trust. The judgments need their own test. A labelled message is one where you have written down the right answers in advance, so a script can compare them with what the model said. I wrote 18 messages: four fraud reports, three refund requests, three requests for a person, two fee waivers and six that should trigger nothing, including near misses such as “When will my refund from last week arrive?”.
tev1, Ollama 0.40.17, Apple M1 Pro 32 GB, median 1,047 ms per message:
| Question | Should be yes | Threshold | Missed | False alarms | Missed at 0.5 | False alarms at 0.5 |
|---|---|---|---|---|---|---|
| wantsRefund | 4 | 0.8 | 0 | 1 | 0 | 4 |
| reportsFraud | 4 | 0.6 | 1 | 0 | 1 | 0 |
| requestsHuman | 3 | 0.8 | 2 | 0 | 0 | 0 |
| requestsFeeWaiver | 2 | 0.8 | 0 | 2 | 0 | 3 |
wantsRefund scored every refund request at 0.96 or higher. It also scored “When will my refund from last week arrive?” at 0.95. That customer wants a status update, and this flow would open a refund review. At a 0.5 threshold the false alarms rise to four, because both fee-waiver messages scored 0.57 to 0.59.
reportsFraud caught three of four. It scored “I’m not saying it’s fraud, but I’ve never used that shop.” at 0.19, so the message lands on ANSWER and nobody looks at a possible fraud report. Of all 18 messages, that is the miss a person would call a real mistake.
requestsHuman scored “put me through to a real person” at 0.95, “I’d like to speak to a manager” at 0.71 and “can someone call me?” at 0.52. The last two fall below 0.8, and both still reach a person, because the grey-zone rule sends anything above 0.4 to HUMAN_HANDOFF. I wrote that rule before I had seen these numbers.
requestsFeeWaiver mixes up “refund this charge” with “remove this fee”. It scored 0.90 on the £240 message and 0.81 on the cancelled subscription. Both messages also scored 0.99 for wantsRefund, and decide() checks refunds before waivers, so the overlap never changed an action. I added a unit test that pins that order, because a future reorder would turn those messages into fee waivers.
All nine rules tests pass, and on their own they say nothing about these problems. The test on labelled messages shows where the inputs to the rules are weak. Every fix it suggests sits outside decide():
- tighten a question, for example by asking whether the customer wants money back that has not already been agreed, then rerun the test,
- set each threshold from a table like this one and from the cost of each kind of mistake,
- add deterministic signals next to the judgments, such as “this account has never paid this merchant” for fraud.
A score of 0.9 does not promise the model is right nine times in ten. A model whose scores keep that promise is called calibrated, and you only find out whether yours is by testing it on your own messages. Eighteen messages show the shape of each question’s errors. They are too few to set a production threshold.
Three kinds of test
The split gives each layer its own kind of test:
| Layer | Test | Needs a model |
|---|---|---|
| Business rules | Unit tests on decide() |
No |
| Judgments | Labelled messages, missed and false alarm counts per question | Yes |
| The whole flow | Conversations through judge() and decide() |
Yes |
When you upgrade the model, rerun the labelled messages and compare the table. When you change the policy, run the unit tests.
Extending the policy without rewriting prompts
The fee waiver went in as one question in QUESTIONS, one if in decide() and one unit test with two cases. The existing questions stayed the same, and no prompt grew a paragraph of exceptions.
Where this fits
Sorting input into categories like this is called classification. The design choice here is to ask several narrow questions instead of having one model choose between seven outcomes. If your app already moves a case through fixed steps (received, under review, refunded), each step has conditions that must hold before the case can move on. Engineers call those conditions guards. The model’s answers become inputs to those guards, the same as an amount or an account flag:
--- alt: "A message becomes semantic judgments, which combine with facts into a validated event. State machine guards check the event, a deterministic transition runs, and only then does an authorised side effect happen." --- flowchart LR M["Message"] --> J["Judgments"] --> E["Validated event<br/>+ facts"] --> G["Guards"] --> T["Transition"] --> S["Side effect"]
The model can tell you a customer sounds urgent. It cannot approve a payment, skip a compliance check or override an insufficient-funds rule, because your code never asks it to.
A wrong or manipulated score still reaches decide(). If a customer writes a message that pushes wantsRefund to 0.99 on an eligible account, decide() refunds up to £25 and no person looks. With the rules in code you can see that limit, test it and lower it.
The approach has costs. Every question adds time while the model runs, about a second for four on my laptop. Small models make mistakes, and the table above shows four kinds. Badly split questions overlap, as the refund and fee-waiver pair did. When the input is already structured, plain code wins. When the task is open-ended writing or planning, a general-purpose model fits better. This pattern suits the space between those two: unstructured human messages feeding a workflow whose outcomes you already know.
Run it yourself
The code, the labelled messages and the logs are in ai-sdk-ollama-decision-models:
ollama pull tev1
pnpm install
pnpm test # rules, no model
pnpm support # five conversations with full traces
pnpm eval-judgments # 18 labelled messages, misses and false alarms per question
pnpm llm-decides # the same five conversations with a chat model choosing the action
Pass a model name to compare, for example pnpm eval-judgments nimble. Swap in your own messages before you trust any threshold in this post.