Test an AI model before you rely on it: six mistakes I made testing Jev
Jamie Watters
Operational resilience and AI delivery practitioner.
Who this is for. You're choosing an AI tool, model or prompt for work that matters, and someone has shown you a comparison where one beats another. It might be the vendor's, a colleague's, or one you ran yourself. By the end you'll have a checklist and know which results to believe. Checking someone else's comparison with it takes an afternoon. Running your own test properly takes days.
Skip it if you only want to know whether Jev can replace GPT-5.4. Short answer: on my hardest fact-checks I can't tell them apart, Jev costs about a fiftieth as much, and it caught fewer of the wrong claims, though that gap isn't significant either. The full re-test is here.
On the length. About twenty-two minutes. The checklist is near the end, under "The checklist". The rest is where each line of it came from.
On 20 September I published that Jev, a new AI model from TypeSafe, beat GPT-5.4 by 11.1 points on the hardest fact-checks I could give it, at a fiftieth of the cost. The result came from my own benchmark, JevBench.
It didn't hold. On 22 September I ran every model five times on the same claims. On JevBench's hard claims Jev's lead over GPT-5.4 averaged 3.3 points, which is 0.6 of one claim, and none of the tests I ran could tell the two apart. Jev also caught fewer of the wrong claims: 15 of the 19 in the set, against GPT-5.4's 18. That gap isn't significant either, but it points the other way.
Same claims, same prompt, same model names. What changed was my testing.
Everything that went wrong is something anyone can do when they compare two AI tools. So this is the whole arc, from headline to correction, with every raw run open, including the runs that proved me wrong. Six mistakes, what each one did to the result, and the checklist I'd now run before relying on any AI comparison, mine included.
What I was testing
Jev isn't a chatbot. It doesn't write anything. You give it a piece of text and a question with a fixed set of answers, and it hands back one of those answers and a probability. TypeSafe charges $0.042 per million input tokens for it, with output free (TypeSafe's model docs).
The job I gave it is one I actually need done. Before anything publishes under my name, a script checks it. That script can tell whether a number has a source attached. It can't tell whether the source says the number. So the question for each claim was: here is a sentence, here is the passage it cites, does the passage support the sentence as written? Three possible answers: supported, not supported, or the passage doesn't cover it.
JevBench has 42 claims. 18 are hard: real sentences from an article of mine that had to be rewritten, each paired with the passage it cites (for four of them, a passage pieced together from an audit of the paper). Nine of the 18 are wrong, six are right, and three cite a passage that doesn't cover them. 24 are easy: controls I built by restating a passage, altering it, or making a claim it doesn't mention, so the right answer is plain. Four systems answered all of them: Jev, and three frontier models, GPT-5.4, Claude Sonnet 5 and Gemini 3.1 Pro.
Here's how the headline moved, piece by piece, all on the same JevBench claims:
| run | what I published about the hard 18 | what it turned out to be |
|---|---|---|
| 20 September, one run each | Jev beats every frontier model by 11.1 points | a Jev run at the top of its range, against a GPT-5.4 run at the bottom of its |
| 21 September, frontier models allowed to think first, one run each | Jev's lead over GPT-5.4 is 16.7 points | measured from that same top-of-range Jev run |
| 22 September, five runs each | Jev's lead averages 3.3 points, not significant | the test can't tell them apart |
Most of what follows is what the five reruns, and six outside AI reviews of the correction, made me see.
Mistake 1: I ran it once
AI models don't always give the same answer when you ask the same question twice. I knew that, and I tested as though I didn't.
In five reruns on 22 September, Jev scored 77.8% on the 18 hard JevBench claims once, and 72.2% in the other four. The run I'd published on 20 September scored 77.8%, the top of that range. GPT-5.4's five reruns sat between 66.7% and 72.2%, and its published run scored 66.7%, the bottom. Put Jev's best beside GPT-5.4's worst and you get 11.1 points. Put the averages side by side and you get 3.3.
I also ran a pinned version of Jev five times, and it turned out to be the same model. Counting those JevBench runs too, 77.8% came up in four runs out of ten, and the ten-run average is 74.4%. Either way, the run I published was at the top of Jev's range.
On a small test, a gap is counted in items, not percentages. With 18 claims, each one is worth 5.6 points. My 11.1-point lead was two claims. In that one JevBench run, Jev got one more claim right than it usually does, and GPT-5.4 got one fewer. That's the whole headline.
I didn't even need the reruns to see it. In that first JevBench run the two models disagreed on only four hard claims: three went Jev's way and one went GPT-5.4's. An exact test on four disagreements gives p = 0.625, nowhere near significant. Counting the disagreements, step 10 of the checklist below, would have stopped the headline on day one.
What to do instead. Count the items the two options disagree on before anything else. Then run each option at least five times and write down every score, not just the average. If the winner's worst run doesn't beat the loser's best run, don't call a winner. That's a quick sanity check, not a test. And spread the runs over more than one day: my five-run JevBench test ran in one 19-minute window, so it can't show whether a model behaves differently on another day.
Mistake 2: I headlined a score that wasn't the job
Before the first run, I wrote a pass mark into the JevBench README. Jev would replace the check in my publishing gate only if it caught at least 80% of the wrong claims and at least as many as the frontier models. At least 60% of the sentences it flagged had to be really wrong, and it had to be cheaper and faster. I wrote it down in advance so the result couldn't become a story about whichever way it landed. I committed it to my private working repo at 11:10 on 20 September, 26 minutes before the first run. The public repo only starts that afternoon, so on the timing you'll have to take my word.
On day one, in the first JevBench run, Jev caught 15 of the 19 wrong claims, 78.9%. GPT-5.4 caught 17. By my own rule the answer was a first filter, not a replacement. That answer was in the body of the article. The headline used a different number, accuracy on the hard 18, where Jev was ahead.
Five runs made the gap between those two numbers plain. Counting a claim as caught when a model caught it in at least three of its five JevBench runs, Jev caught 15 of the 19 wrong claims, GPT-5.4 18, and Sonnet 5 and Gemini 16 each. Jev's lead on the hard claims had come from the other direction: GPT-5.4 failing three claims that were actually right. The other half of a checker's job is not flagging good claims as wrong, and there they were close. Across the five JevBench runs, Jev did it 2 or 3 times a run, GPT-5.4 4 times every run, Sonnet 5 2 to 4 and Gemini 2 to 5.
Neither gap is significant on this few claims. But for a checker the two kinds of error don't cost the same. If it wrongly flags a good sentence, I look at it, find it's fine, and lose five minutes. If it wrongly passes a bad one, the bad sentence goes out under my name. Accuracy rolls those two cases into one number, so the expensive mistakes disappear into the cheap ones.
What to do instead. Before running anything, write down the job and pick the score that matches it. For a checker, that's the share of wrong answers it catches, with false alarms reported beside it. For something that drafts text, it might be the share of drafts you'd send without editing. Headline that score. Put overall accuracy underneath, never on top.
Mistake 3: I tested it on my own writing, labelled by me
All 18 hard JevBench claims are sentences from one article of mine. The passages come from the papers that article cited. The right answers were set by one person, me, from audits of those papers that another AI model produced inside my publishing process, and that I checked by hand against the papers.
That has two costs, and the five runs showed me both.
The first is that one of my labels is probably wrong. Claim sb-06 says part of a paper's reported
gain "is unattributed". The paper doesn't use that word. I'd labelled the sentence supported, as a fair
paraphrase. When I gave the models an extra answer for "roughly right but altered", every system chose
it for that claim: Jev on all five runs, and the three frontier models on the one run each got. And it isn't any claim: Jev got sb-06 right three times in five and
GPT-5.4 never did, so it's one of the claims Jev's lead rested on. Take it out and, on the other 17
JevBench claims, Jev and GPT-5.4 both average 74.1%.
It goes further than one label. All three hard claims where Jev was right and GPT-5.4 wrong, rz-05,
rz-06 and sb-06, are sentences I labelled supported as fair shortenings of the source. Take all
three out and, on the other 15 JevBench claims, GPT-5.4 averages 84.0% and Jev 70.7%. So which model
comes out ahead on my hard set depends on how generous I was about paraphrase when I labelled it.
That's a fact about the test, not the models.
The second is quieter. The nine wrong sentences in my hard set are mistakes that had already got past me: past my writing, my automated check and a first round of review. So the hard set leans towards what I miss, not what a checker sees in normal work. That's the right set for asking whether a tool catches my blind spots. It's the wrong set for predicting how it does on anyone else's.
What to do instead. Test on items that look like your real work, in roughly the proportions you'll meet them: written by someone else, or real failures you didn't pick for the test. Have a second person label them without seeing your answers, and report how often you agreed. Settle disagreements by a rule you wrote down first, or drop the item. Don't let the models settle it, and don't change an item or a label after you've seen how a model answered it.
Mistake 4: I published before testing the setup
My first run made three setup choices without testing any of them. I called Jev by a name that moves when a new version ships, and didn't record which version answered. I asked it one question per claim, when TypeSafe's own guidance recommends several narrow questions about the same text in one call. I made the frontier models answer straight away, with no room to reason first, which every reviewer of that piece raised. And I set no temperature or seed for any of them.
Every one of those could have moved the result, and I published the result anyway. When I tested them afterwards, none of the three helped. Pinning made no detectable difference, and the other two scored lower on average. None of the three changes is statistically significant.
Letting GPT-5.4, Sonnet 5 and Gemini think before answering improved none of them on the hard 18, at 2.4 to 4.2 times the cost per claim. That was one run per model, and none of the changes is significant, so the honest reading of that JevBench run is that it showed no improvement, not that it made things worse.
TypeSafe recommends asking several narrow questions at once. On JevBench that cost the same as one question: $0.000026 per claim against $0.000025. It scored 6.2 points lower on all 42. That isn't significant either (p = 0.219), and it's one way of splitting the question, not every way.
Pinning the version made no detectable difference, because every call on 22 September was answered by the same version anyway. But my first run didn't record which version answered it, so I can't prove what I tested on 20 September. The evidence says it was very probably the same model. The file can't show it.
What to do instead. Treat every recommended setup as a guess until you've tested it against the plain one, whether it comes from the vendor's docs, a colleague, or the common advice to "turn on reasoning". Pin the model version, write down every setting, and record which version actually answered. A vendor's advice may be right for their examples. Whether it's right for your job is something only your test can tell you.
Mistake 5: I misread the confidence score, twice
Both Jev and the frontier models return a confidence number with each answer. It's tempting to treat that number as permission: trust the confident answers, check the rest. I got it wrong in both directions.
The first time, I tested GPT-5.4's confidence at six round thresholds, 0.60 up to 0.95, chosen before I'd looked at its answers. All 42 of its JevBench answers sat between 0.94 and 0.99. Five of my six thresholds were below every one of them, so the results came out flat, and I published that its confidence told you nothing. It does. Tested at the values it actually returns, keeping only its 0.99 answers gave 64.3% coverage at a 3.7% error rate. That correction is its own piece.
The second time was about Jev. Asked the identical question five times, Jev changed its answer on 2 of the 42 JevBench claims, and every answer that changed had a confidence below 0.47 on every run. Across 420 calls, no claim that reached 0.5 on any run ever changed its answer. My first draft of the correction presented 0.5 as a line above which you could trust Jev. Several of the outside reviewers said I'd turned a rule about answers staying the same into a rule about answers being right.
They were right, and the data shows why. Claim sb-03 is a quotation in my article that is really a
section heading joined to a sentence from further down the paper. On JevBench's plain question, Jev
called it supported in all twelve of its runs, at a confidence of 0.98 or 0.99 every time. It's
wrong. All four models got it wrong in every plain-question run. Sonnet 5 and Gemini only got it
right when I offered a separate answer for "altered". A confident answer that stays the same can
still be wrong.
What to do instead. Test a confidence score at the values the model actually returns, not at round numbers you picked. Then test it against whether answers were right, not just whether they stayed the same. If it passes both, you can use it to decide what to check first. It still doesn't tell you what you can stop checking.
Mistake 6: I called "can't tell" a tie
The correction first went live with the word "tie" in its title. All six outside reviewers objected, and they were right.
Here's what the five JevBench runs support. The test works on each model's majority answer, counting a claim as right when a model got it right in at least three of five runs. On that measure Jev is one claim ahead, not 3.3 points. Jev was right on 3 hard claims where GPT-5.4 was wrong, and GPT-5.4 on 2 where Jev was wrong. An exact McNemar test, which looks only at those disagreements, gives p = 1.000. A rough 95% interval on the gap (a Wald interval, the simple normal approximation) runs from about 19 points worse for Jev to 30 points better. With only five disagreements, even that rough interval is too kind.
That doesn't say the two are equal. It says that on 18 claims the test can't rule out Jev being about 19 points worse or 30 points better. "Tie" claims something the test can't measure. The word I'd written down before the runs for this outcome was "shrinks": Jev's average still leads, but the ranges overlap. I should have used it.
What to do instead. When a test finds no significant gap, say "can't tell them apart", and say how big a gap it could have missed. If you need to show two options are as good as each other, decide before you run how close counts as "as good", and use enough items to show it. As a rough guide from the JevBench data: if two models disagree on the same share of items as Jev and GPT-5.4 did, narrowing that interval to 5 points either way takes something like 430 items, not 18. Actually showing "as good as" within 5 points, with a standard equivalence test (5% error, 80% power), takes about 950. Both rest on a disagreement rate measured on 18 hard items, so treat them as sizes, not targets.
What a fair test needs
Put the six fixes together and you have a test worth relying on. None of it is advanced. Most of it is writing things down before you look.
JevBench, as it stands, does about half of it: five runs, a pass mark set in advance, the score that matches the job reported, the setup tested, and exact tests on the disagreements. It doesn't have items that look like anyone else's work, a second labeller, a margin for "as good as", or runs on different days. So it can tell you that my first headline was wrong. It can't yet tell you whether Jev is as good as GPT-5.4 at this job.
The checklist
This is what I'd now run before relying on any "A beats B" result for an AI tool, model or prompt. That includes a vendor's result, a colleague's, and mine. If you're checking someone else's result, ask for the answers to each step. If they can't give them, you have your answer. Copy it.
BEFORE YOU RELY ON "A BEATS B"
Before you run anything
1. Write down the job, and the score that matches it.
For a checker: of the wrong answers, how many did it catch,
and how many good ones did it wrongly flag?
Overall accuracy goes beside those, never above them.
2. Write down the pass mark, and how close counts as "as good as".
Put it somewhere you can't quietly edit: a commit, a sent email.
3. Use items that look like your real work, in roughly the
proportions you'll meet them. Ones you didn't write, or real
failures you didn't pick for the test.
4. Have a second person label them without seeing your labels.
Write down how often you agreed. Settle disagreements by a rule
you wrote first, or drop the item. Never let a model settle it.
5. Work out what one item is worth: 100 divided by the number of
items. A gap of one or two items is not a result.
6. Pin the model version. Write down every setting, including
temperature, and the prompt word for word.
While you run
7. Run each option an odd number of times, at least five, on more
than one day. Record which version actually answered.
8. Try the recommended setup (the vendor's, or "turn on reasoning")
against the plain one. Pick your setup on some items, then score
it on others you haven't looked at. Never change an item or a
label after seeing how a model answered it.
When you read the results
9. Write down every run's score, not just the average.
If the winner's worst run doesn't beat the loser's best, don't
call a winner: get more runs or more items.
(A quick sanity check, not a test.)
10. Count the items where one option was right and the other wrong.
Fewer than six, and the usual exact McNemar test (two-sided, at
5%) can't call the gap significant, however lopsided it is.
11. Run that test on those items. Reduce each option's runs to one
answer per item first (right if right in most runs), or compare
one run against one run. Both throw information away: this is
the floor, not the best analysis.
12. Comparing against more than one option? Name the main comparison
before you run. Treat the others as a first look, not a result.
13. Not significant means "can't tell them apart". Never "tie".
Showing "as good as" needs the margin from step 2, an
equivalence test, and far more items than feels necessary.
14. Break the errors down by kind. Put a rough cost on each kind,
and look hardest at the expensive ones.
15. If there's a confidence score, sort the answers by it, using the
values the model actually returns. Among the most confident,
the error rate should be clearly lower than overall. Check that
on items you didn't use to pick the cut-off.
Step 10 is arithmetic, not opinion. With five disagreements all going one way, the exact test's two-sided p-value is 0.0625, which misses the usual 0.05 line. With six, it's 0.031. If your comparison has fewer than six disagreements, that test can't make the gap significant. One catch: the test assumes the items are independent. My 18 hard sentences come from one article citing three papers, so they aren't fully independent, and the real uncertainty is wider than the test says.
For step 11, mcnemar.py in the JevBench repo compares
two files, one answer per item each, with nothing beyond standard Python:
python3 mcnemar.py before.jsonl after.jsonl
Each file is one JSON object per line, with an id, the right answer as label, and the model's
answer as predicted. To use five runs, write one file per option holding each item's majority
answer first; tests34.py in the same repo does that reduction for the JevBench runs. For step 15,
risk_coverage.py --sweep-own in the same repo sorts answers by confidence at the model's own values.
Where this checklist stops
The strongest objection comes from people who evaluate models for a living, and it's fair. Five runs and an exact test on 18 items is still a weak design. A proper analysis would use every run for every item rather than a majority vote, test for equivalence against a margin set in advance, and run on hundreds of items drawn from the work the tool will actually do.
It would also deal with running several comparisons at once. I compared Jev with three frontier models and tested each gap on its own, and the more gaps you test, the likelier one of them looks real by chance. One of mine did: in the reasoning test, Jev against Sonnet 5 came out at p = 0.031, which doesn't survive allowing for three comparisons. And choosing which models to report after seeing the results is the same problem in another form. On 20 September I also ran Claude Sonnet 4.5 and Gemini 3.8 Flash. The Sonnet 4.5 run is mentioned in the JevBench README; the Gemini 3.8 Flash run sits in the repo and never made it into any of the articles.
By that standard, even a fully corrected JevBench is a pilot, not a verdict. I agree with all of it. The checklist isn't a substitute for that work. It's the floor underneath it: the checks that would have stopped my headline. They're cheap enough to run on any comparison you're shown, and a comparison that fails them isn't worth the more careful analysis until it has more items.
The other fair objection is that I've built a checklist from one small benchmark on one kind of task. That's true too. Every mistake above happened to me on fact-checking. Whether they're the mistakes that matter most for, say, choosing a coding assistant is something I can't show from this data.
Send me your published mistakes
The fix for mistake 3 is claims I didn't write. The next version of JevBench will be built from other people's published mistakes: sentences that went out, cited a source, and turned out not to say what the source said, then were corrected.
If you have one, I'd like it. What's useful is the published sentence, the source it cites, the passage, what was wrong with it, and a link to the correction. Open an issue on github.com/TheWayWithin/jev-bench with those five things. The new test will be designed and written down before anything runs, and a result that goes against Jev gets written up the same week and with the same prominence as one that goes for it.
One run said 11.1 points. Five runs said 0.6 of one claim. Same claims, same prompt, same model names.
If it saved you an afternoon, buy me a coffee.
Sources
The benchmark: JevBench, at github.com/TheWayWithin/jev-bench.
First run 20 September 2026, one run per model. Reasoning-allowed run 21 September, one run per
frontier model. Five runs per system on 22 September, 18:29 to 18:48 EDT, scoring rules committed
before the first call. Jev via TypeSafe; openai/gpt-5.4, anthropic/claude-sonnet-5 and
google/gemini-3.1-pro-preview via OpenRouter. Verdicts: results/2026-09-22-test2-verdict.md and
results/2026-09-23-tests-3-4-verdict.md. python3 tests34.py analyse reproduces the five-run
figures without an API key. The outside reviews of the correction and of this piece are in my working files, not the
repo, so the reviewer counts above are my account.
The labelled set is built from source audits of three papers: SkillsBench (arXiv:2602.12670), ReasoningBank (arXiv:2509.25140) and R-Zero (arXiv:2508.05004).
Jev's price: TypeSafe's model docs, read 20 September 2026.
The four pieces this one draws on, in the order they were published: Jev is not an LLM, the single-run comparison; GPT-5.4 beats Jev on a confidence threshold, the confidence correction; Paying your AI to think longer didn't fix its mistakes, the reasoning test; and I said Jev beat GPT-5.4. Five reruns can't tell them apart, the five-run correction.