GPT-5.4 beats Jev on a confidence threshold, once tested at its own values.
Jamie Watters
Operational resilience and AI delivery practitioner.
I published the wrong instrument last week and called it a result. On the JevBench set, I said GPT-5.4's confidence score gives you no usable threshold: flat across six settings, wrong 14.3% of the time no matter where I drew the line. That's false. Swept over its own values instead of the six I chose, GPT-5.4 runs from 64.3% coverage at a 3.7% error rate up to 100% coverage at 14.3%. At every coverage point the two curves share, it beats Jev: 3.7% against 7.4% at 64.3% coverage, 5.9% against 11.8% at 81.0%, 8.3% against 13.9% at 85.7%. My six thresholds ran from 0.60 to 0.95. Every one of GPT-5.4's 42 answers came back between 0.94 and 0.99. I tested a model with a ruler that didn't reach into its range, read the flat line that produced, and published the ruler's limit as the model's.
Two smaller JevBench figures were wrong for the same reason. I said Gemini drops one claim at 0.90; swept over its own three values it drops ten above 0.95, landing at 76.2% coverage and a 9.4% error rate rather than 19.0%. And I priced a cascade at $0.000424 per claim when it actually costs $0.000439, because the claims Jev hands off are longer and dearer to re-check than an average claim, and my arithmetic charged the average anyway.
All three were caught by sweeping the raw data against its own values rather than a grid chosen in advance, which is the exact check this piece tells you to run on your own model. I didn't run it on mine before I published. This is that check, run properly, on the same 42 claims as last time. The finding changes: GPT-5.4 has a working threshold, just not at the numbers I tried first, and at those numbers it beats Jev.
Who this is for. You have built something where a model gives you an answer and sometimes that answer is wrong. You cannot check every one by hand and you do not want to. The model hands back a confidence number with each answer, so the obvious move is to trust the confident ones and look at the rest yourself. It took me until this week to check whether that number is worth anything. By the end you will know how to run the same check on your own setup in about five minutes, what it buys when it works, and the case where it does not work at all.
Skip it if you want a Jev tutorial, or the theory behind calibration. There is no maths in here past dividing one count by another.
On the length. About twenty-six minutes, and most of that is tables. If you only want the method, skip to "Run this on your own setup": it is a table you fill in and a pass rule. Everything before it is how I got the numbers and everything after is where they are thin, which is here because a benchmark you cannot argue with is a benchmark you cannot check.
TypeSafe came out of stealth with Jev on 15 September. It returns a typed answer and a probability instead of text, and it costs $0.042 per million input tokens. If you are looking at it this week, you are probably looking for the same thing I was: a way to stop paying for a checking step.
The obvious build is the one TypeSafe's own citation-check cookbook points at. Run every claim through Jev, auto-accept anything at 0.8 or above, look at the rest yourself. It is cheap, it is fast, and its confidence score moves in a way you can act on, which I measured on Sunday and still stand by. Calibration is a property of a population, not of an answer, and one run over 42 claims is not a general proof of it. What that run shows is that Jev's score ranks its own answers usefully on this task. It turns out the frontier models have a usable score too, once you test them at the numbers they actually return rather than the numbers I first guessed.
This piece is about what that threshold actually buys, because I had not measured it and neither, as far as I can find, has anyone else.
The short version, from the JevBench runs: on 42 claims it skips 69% of the checking and is wrong on 6.9% of what it skips, which is a real saving on a real job. On the 18 of those that are sentences from something I actually published, rather than controls I built myself, no threshold in the tested grid gets any of the four models under a 10% error rate. And the one claim Jev accepts at every setting from 0.60 to 0.95, at a stated confidence of 0.98, is the hardest error in the set to catch by eye.
So the misreading to avoid is the one I nearly made. A calibrated confidence score does not tell you which checks you can stop doing. It tells you which ones to do first.
No new model calls here: same 42 claims, same four models, same runs as Sunday, read a different way. The script is in the repo.
The two numbers that decide it
You have a checking step. A model looks at each claim, returns a verdict and a confidence. You pick a threshold. At or above it you take the verdict and move on. Below it, a person looks.
Two numbers describe every threshold. Coverage is the share of claims you take without looking at them. Error rate is the share of those you took that were wrong. Move the threshold and both move together: that is the risk and coverage curve, and it is the thing to look at before wiring any of this into a pipeline.
The question underneath it is blunter. Does changing the threshold change anything?
On the full set, the grid I chose was the wrong instrument for two of the four
Jev, all 42 JevBench claims:
| threshold | coverage | error rate among accepted |
|---|---|---|
| 0.95 | 57.1% | 4.2% |
| 0.90 | 61.9% | 3.8% |
| 0.85 | 64.3% | 7.4% |
| 0.80 | 69.0% | 6.9% |
| 0.70 | 76.2% | 9.4% |
| 0.60 | 76.2% | 9.4% |
That is a curve. Accept less, be wrong less. At 0.80, which is the auto-accept threshold TypeSafe's own citation-check cookbook proposes, you take 29 of the 42 verdicts unchecked and two of them are wrong.
One detail in there is worth more than the headline. Coverage stops climbing at 76.2%. Ten of the 42 JevBench claims came back under 0.6 and no threshold in the grid picks them up. Jev refuses to commit on a quarter of the set, which is the behaviour you want from something you are about to trust with the rest.
Now GPT-5.4, tested on the same six numbers:
| threshold | coverage | error rate among accepted |
|---|---|---|
| 0.95 | 97.6% | 12.2% |
| 0.90 | 100% | 14.3% |
| 0.85 | 100% | 14.3% |
| 0.80 | 100% | 14.3% |
| 0.70 | 100% | 14.3% |
| 0.60 | 100% | 14.3% |
Five of these six rows land on 100% coverage and 14.3% error on the JevBench set. That's what I reported last week, and it's a true reading of these six numbers. Five of them, 0.90 down to 0.60, sit below GPT-5.4's lowest confidence of 0.94 and tell you nothing: they accept all 42 answers regardless. Only the sixth, 0.95, sits inside GPT-5.4's actual range, which is why that one row alone differs from the other five.
Swept over GPT-5.4's own six distinct values instead of mine:
| threshold | coverage | error rate among accepted |
|---|---|---|
| 0.99 | 64.3% | 3.7% |
| 0.98 | 81.0% | 5.9% |
| 0.97 | 85.7% | 8.3% |
| 0.96 | 90.5% | 10.5% |
| 0.95 | 97.6% | 12.2% |
| 0.94 | 100% | 14.3% |
That is a curve, and on the JevBench set it beats Jev's at every coverage level the two share: 3.7% against 7.4% at 64.3%, 5.9% against 11.8% at 81.0%, 8.3% against 13.9% at 85.7%. Only at 100% coverage, where neither model is choosing anything, do the two land on the same 14.3% error. But Jev's curve keeps going where GPT-5.4's cannot: below 64.3% coverage, down to 28.6%, Jev is right on everything it accepts, 0.0% error, a setting GPT-5.4 has no confidence value low enough to reach. Nothing about GPT-5.4 as a model was flat. The six numbers I tested it at were flat, because I picked them before I looked at what GPT-5.4 actually returns.
Gemini 3.1 Pro, same six numbers:
| threshold | coverage | error rate among accepted |
|---|---|---|
| 0.95 | 97.6% | 17.1% |
| 0.90 | 97.6% | 17.1% |
| 0.85 | 100% | 19.0% |
| 0.80 | 100% | 19.0% |
| 0.70 | 100% | 19.0% |
| 0.60 | 100% | 19.0% |
I reported that one claim drops out at 0.90, which is also a true reading of these six numbers: coverage does fall by one claim between the 0.90 and 0.85 rows. Gemini returns exactly three distinct confidence values on this run: 1.00, 0.95 and 0.85. My grid caught that boundary and missed the one above it entirely, because I never tested anything higher than 0.95.
Swept over Gemini's own three values:
| threshold | coverage | error rate among accepted |
|---|---|---|
| 1.00 | 76.2% | 9.4% |
| 0.95 | 97.6% | 17.1% |
| 0.85 | 100% | 19.0% |
Ten of the 42 JevBench claims sit below a perfect 1.00, not one. Requiring it cuts coverage to 76.2% and error to 9.4%, the best number Gemini reaches anywhere in this piece. That's one usable setting rather than six, and it only shows up once you test the model at a value it actually returns.
The lesson isn't really about these two models. It's about the six numbers 0.95, 0.90, 0.85, 0.80, 0.70, 0.60. They're round, they look sensible, and they came from nowhere in the data. For Jev, whose 42 answers spread from 0.24 to 1.00, that grid happens to land inside the range and shows real structure. For GPT-5.4, whose answers spread from 0.94 to 0.99, the same six numbers sit almost entirely below where anything happens. For Gemini, three distinct values, the grid caught one boundary and missed another. A grid chosen before you look at the data can make a model look like it has no usable threshold. What it actually proves is that the grid missed the model's range, and those are not the same finding.
The same mistake, in a coarser form, from the piece before this one
I said all three frontier models put nearly everything in their top confidence bucket and that their confidence therefore carried no information. That was also an instrument problem rather than a model problem, and it's now wrong for all three: GPT-5.4 and Gemini above, and Claude Sonnet 5, which is what this section corrects.
| threshold | Sonnet 5 coverage | error rate among accepted |
|---|---|---|
| 0.95 | 57.1% | 8.3% |
| 0.90 | 73.8% | 9.7% |
| 0.85 | 88.1% | 13.5% |
| 0.80 | 92.9% | 15.4% |
| 0.70 | 100% | 14.3% |
That moves. Sonnet's stated confidence spans 0.70 to 0.99 and the threshold buys you something real.
I missed it because my reliability table used five buckets of 0.2 each. That put 39 of Sonnet's 42 answers in one box labelled 0.8 to 1.0, and threw away the structure inside the box before I ever looked at it. The bucket was too wide, so I read a flat table as a flat model. Same run, same file, different grain.
On the JevBench set, Jev still matches or beats Sonnet at every coverage level they share, swept over each one's own values: it wins outright at 57.1% coverage (4.2% error against 8.3%) and at 73.8% (6.5% against 9.7%), and ties everywhere else they overlap. But "frontier models have no usable confidence" was too broad a sentence, and one of the three has one.
On the claims that actually matter, the picture reverses
Everything above is all 42 JevBench claims, and 24 of those are constructed controls I built myself. The eighteen that carry the headline are real sentences from a published article, paired with the source passage they cite. GPT-5.4's win over Jev, above, is an all-42 result. It does not survive splitting the set: GPT-5.4 gets 100% of the 24 controls right and only 66.7% of the 18 real ones, where Jev gets 77.8%. On the claims that actually matter, Jev is still ahead.
Here is every model on those eighteen, at the best threshold inside my original six-point grid:
| model | best threshold | coverage | error rate among accepted |
|---|---|---|---|
| Jev | 0.90 | 38.9% | 14.3% |
| GPT-5.4 | 0.95 | 94.4% | 29.4% |
| Claude Sonnet 5 | 0.90 | 61.1% | 27.3% |
| Gemini 3.1 Pro | 0.95 | 94.4% | 41.2% |
Nothing in that grid gets the error rate under 10% on the JevBench real claims. Jev's best is 14.3%, and to get it you accept 7 of the 18 instead of 11, so you give up a third of what little saving there was and you are still wrong one time in seven.
Swept over each model's own JevBench values instead, two of the four touch 0% error. But only by accepting almost nothing. Jev hits it at 0.99: 3 of the 18 claims, none wrong. Sonnet hits it at 0.99 too: 1 of 18, none wrong. GPT-5.4 comes close but doesn't reach it, at its own top value, also 0.99: 8 of 18, 1 wrong, 12.5%. None of that is a threshold worth building a pipeline on. It is answering the one, three or eight claims each model is most certain of, and sending the other 17, 15 or 10 to a person regardless. A result on 1 or 3 claims flips to double digits the moment a single verdict changes: a few steps down for Jev, at 0.91, it is already back to 14.3% error at 38.9% coverage, the same number the six-point grid found. Gemini does not get close even at its best: its own top cut, at 1.00, is 55.6% coverage and 30.0% error, better than the grid's 41.2% but nowhere near 10%.
So on real published sentences, the honest answer to how much checking you can safely stop doing is: none of it, at any threshold that accepts enough claims to be worth having. There is no threshold on any of these four models at which I would let a sentence go out under my name unchecked. The threshold works on the controls I built. It does not work on the sentences the controls were built to stand in for, which is the only reason I built them.
That is the same inversion the first piece found, showing up in a different measurement, and it is why GPT-5.4's win above does not survive the split. Easy material says the problem is solved. Real material says it is not.
Error rate is the wrong risk number anyway
There is a subtler mistake buried in everything above, and it took me a while to see it.
A gate does not treat all errors the same. If the model says a sentence is unsupported and it is wrong, I go and look at the sentence, find it is fine, and publish it. That costs me five minutes. If the model says a sentence is supported and it is wrong, the sentence goes live under my name. Averaging those two into one error rate hides the expensive half inside the cheap half.
So the honest number is narrower: of the claims auto-accepted as supported, how many should not have been published?
On the eighteen real claims, twelve should not be published. At 0.80:
| model | published unchecked | of those, bad | share of the bad ones that escaped |
|---|---|---|---|
| Jev | 4 | 1 | 8.3% |
| GPT-5.4 | 5 | 2 | 16.7% |
| Claude Sonnet 5 | 8 | 3 | 25.0% |
| Gemini 3.1 Pro | 6 | 3 | 25.0% |
Jev publishes the fewest sentences unchecked and the fewest bad ones. It also publishes four and gets one of them wrong, so read the right-hand column as a direction and not a rate.
The claim Jev waves through is the same one at every threshold from 0.60 to 0.95, and it is the same one all four models miss. It's a quotation in my article presented as one sentence from the SkillsBench paper. The paper has the first half as a section heading and the second half as a sentence further down. Every word is in the document, in that order. Nothing was checking whether they had ever been one sentence, and Jev returned that verdict at 0.98.
No threshold can catch a claim the model is confidently wrong about, because the threshold reads the confidence. That is the floor under this whole method and it does not move.
Escalating to a bigger model makes the hard half worse
The obvious build is a cascade. Take Jev's verdict when it is confident, hand the rest to GPT-5.4, pay a fraction of the price. So I ran it.
At a 0.80 threshold it escalates 31.0% of the JevBench claims, 13 of the 42, and costs $0.000439 per claim, against $0.001289 for GPT-5.4 alone. Roughly a third of the price. My first figure here, $0.000424, priced those 13 escalated claims at GPT-5.4's average cost across all 42. They aren't average: they're the claims Jev found hardest to read, which on this run means longer ones, and GPT-5.4 charges by the token. Pricing what those 13 claims actually cost, rather than the model's average, moves the figure from $0.000424 to $0.000439. Accuracy across all 42 comes out at 88.1%, which beats both Jev alone and GPT-5.4 alone, both of which scored 85.7%.
Then look at the eighteen real ones in JevBench. The cascade gets 72.2% of those right. Jev alone gets 77.8%. The cascade also publishes two bad sentences where Jev alone publishes one.
Paying more made it worse. Here is why, and I checked it rather than assuming it:
| the 8 real claims Jev escalated at 0.80 | accuracy |
|---|---|
| Jev's own escalated verdict | 75.0% |
| GPT-5.4 on the same eight | 62.5% |
Across all 42 JevBench claims the escalation does help: GPT-5.4 gets 76.9% of the escalated ones right against Jev's 69.2%. On the real eighteen it reverses. The claims Jev is unsure about are hard for a reason, and the expensive model is not better at them, only more certain.
So the escalation target is the decision, not an implementation detail. Sending uncertain claims to a bigger model is a cost optimisation dressed as a safety measure. For real prose, the thing on the other end of the escalation has to be a person.
What I am doing with it
Jev goes into my publishing gate at 0.80. Anything it flags gets blocked and I look at it. Anything it clears below the line gets escalated to me, not to another model. Anything it clears above the line still gets read, because of the spliced quotation.
Which means the honest accounting of what this buys me is: not less checking, but better-ordered checking. Four of Jev's six errors are in the thirteen claims it hands back, and knowing which thirteen is worth something even though it does not let me skip the other twenty-nine.
Your task might be the easy kind, and that changes the answer
Here is the strongest objection to everything above, and I think it is largely right.
My eighteen hard claims are sentences from one article I wrote, and the errors in them are a particular sort: a quotation spliced from two places, a true number attached to the wrong quantity, a result from the wrong version of a paper. If you are classifying support tickets, extracting invoice fields, or checking whether an answer used the retrieved document, your failures look nothing like that. They look far more like my constructed controls, where one thing is changed and there is a clean right answer.
On the controls the threshold works. Jev missed only two of those 24, and both came back at 0.37 and 0.71, below its own 0.8 line, so a threshold would have caught them both.
So the finding is not "thresholds never work". It is that a threshold's usefulness depends on whether your errors are clean or subtle, and you cannot tell which kind you have from an accuracy score. The run below tells you, on your own data, in about five minutes.
Run this on your own setup before you build anything
There is no maths in it past dividing one count by another.
Get a set of answers where you already know whether the model was right. A spreadsheet, a log, a folder of bug reports, whatever you have. I used 42 and that was enough to see what was going on. Each one needs two things beside it: the confidence the model gave, and whether it turned out to be correct.
Then fill in this table. For each threshold, count how many answers came in at or above it, and how many of those were wrong.
| threshold | how many at or above | how many of those wrong | error rate |
|---|---|---|---|
| 0.95 | |||
| 0.90 | |||
| 0.85 | |||
| 0.80 | |||
| 0.70 | |||
| 0.60 |
Two counts, six times. That is the whole method, and it is why risk_coverage.py is counting rather than doing anything clever.
Before any of that, check where your six numbers actually sit against the model's own values. This is the step I skipped, and it is the reason the first version of this piece was wrong about two of the four models. Find the lowest and highest confidence the model actually returned in your set. If your six thresholds don't span that range, they cannot tell you anything: pick threshold values from your own data's distinct scores instead of a round-number grid, the way risk_coverage.py --sweep-own does, and only then move to the pass rule below. A model that looks flat across a chosen grid may simply have a range the grid never reaches. GPT-5.4's 42 answers on this run sat between 0.94 and 0.99; my grid stopped at 0.95 and never went higher, so five of my six thresholds told me nothing about it.
The pass rule. Compare your lowest threshold to your highest. If the error rate at the top is not at least a third lower than at the bottom, and the coverage at the top is not at least ten points lower, your model's confidence is not telling you which answers to check, and a threshold built on it will change nothing. That bar is a judgement, not a standard, and I am writing it down so it can be argued with rather than left in my head. On the JevBench set, swept over its own full range, Jev's error rate goes from 0.0% at its top two settings to 14.3% at accepting everything. GPT-5.4's, swept over its own six values, went 3.7% to 14.3% while coverage moved from 64.3% to 100%. Both carry real information. Gemini's sits at one end of its range: 9.4% at its top setting against 19.0% at accepting everything, which clears the bar but leaves you only one usable setting rather than several.
Then check the second column too. If nothing ever falls below your threshold, the model is not declining to answer, it is just always confident. Jev left ten of 42 below 0.6 whatever I set. That refusal is the part you are actually buying. One trap in that check: it is relative to where you put the threshold, and it is the same trap as the range check above. Set it at 0.90 on GPT-5.4's own values and you will see nothing below it either, because 0.90 sits under all six of its own distinct scores. Sweep your own values before concluding anything.
Now fill in the second table, which is the one that decides it. The first table pools two errors that cost completely different amounts. This one counts only what goes live unchecked.
| threshold | accepted AND the model said it was fine | how many of those were not fine |
|---|---|---|
| 0.95 | ||
| 0.90 | ||
| 0.85 | ||
| 0.80 | ||
| 0.70 | ||
| 0.60 |
Mine at 0.80, on the 18 real claims: four published unchecked, one of them wrong. That is the number I could not live with, and it is invisible in the first table.
Three things I got wrong and you should not repeat. Run both tables twice, once on clean test cases and once on real material you got wrong in production, because the answer differs and the clean run is the flattering one. Pick your threshold on one half of your data and report the numbers on the other half. And sweep your own model's actual values before you pick a grid at all. I did none of the three. My figures are in-sample, which makes them a reason to test 0.8, not evidence that 0.8 is safe, and the range mistake is the one that put a wrong central claim on a live page under my name.
You will not find any of this out from the accuracy figure, because the accuracy figure is fine.
What this is not
One run per model, no repeats, so none of these numbers has an error bar. The eighteen real claims carry the headline and several of the cells above have single-figure denominators: four claims published unchecked, one of them wrong. That is a direction, not a percentage, and I have written it as a count wherever the count is small enough to matter.
The labels come from source audits of my own published article, checked by hand against the primaries rather than by an independent panel. One of the eighteen is marked contested in the data. Every claim in the hard set comes out of one person's publishing pipeline, which is the weakness I cannot fix on my own and the reason the repo asks for other people's corrections.
The eighteen are not a sample of incoming claims. They are a sample of what already got past me, and a peer reviewer put that more sharply than I had. These are sentences that survived writing, an automated gate and a first round of review, and were then labelled from audits of that same article. So the set is selected for looking supported when it is not, which is precisely the failure I am measuring. That makes "on the real claims, nothing works" a finding about the errors that already escape my checking, not about everything a checking step sees. It is still the right set for asking whether a threshold catches what I miss. It is the wrong set for estimating what a threshold would do to a live stream, and I had been sliding between the two.
There is also no held-out set. I looked at the confidence-versus-error relationship, took 0.8 because TypeSafe proposes it, and then reported the coverage and risk on the same 42 observations. Those numbers are in-sample. They say the threshold is worth testing as a routing rule; they do not establish 0.8 as a safe operating point for claims I have not seen.
And my unsupported label merges two different things, a flat contradiction and a claim that is correct in substance but wrong as written. TypeSafe's own guidance is that Jev answers the question you were asking rather than the one you meant, so a compound category is the wrong shape to hand it. That is listed in the repo as a known weakness of the first run, and it matters more here than it did there: this article's whole contribution is separating supported from everything else, and the confound sits inside the category I built the separation on.
The two confidence numbers may not be the same kind of object, and this piece leans on comparing them harder than the first one did. Jev's is a probability the model is built to produce. The frontier figure is a number I asked for in a JSON field, so it is self-report rather than anything the model measured. That is still the right comparison for a pipeline, because the self-reported number is what you would actually receive and act on. It is not a like-for-like test of calibration, and GPT-5.4's answers clustering into a narrow 0.94-to-0.99 band is itself possibly a fact about asking a model to rate itself in a field, rather than about the claims it was rating.
And the biggest one is unchanged from Sunday: the frontier models answered straight into a JSON verdict with no room to reason first. Every number here about them is a number about schema-constrained models, and a model allowed to think before it answers may well spread its confidence differently. Gemini 3.1 Pro is also a preview build.
Running them again with a reasoning step is the next test and it goes out on Tuesday. If Jev loses it, that article gets written with the same prominence as this one.
The script that produced every table above is risk_coverage.py at github.com/TheWayWithin/jev-bench. It needs no API keys and no venv: the raw runs are already in the repo, so you can check any figure here in about a minute.
If it saved you an afternoon, buy me a coffee.
Sources
The benchmark: JevBench, run 20 September 2026, analysed 21 September. Jev via TypeSafe, called through the moving jev-latest alias rather than a pinned version, which is recorded in every run file and is one of the repo's listed weaknesses. And openai/gpt-5.4, anthropic/claude-sonnet-5 and google/gemini-3.1-pro-preview via OpenRouter. 42 claims, one run per model. Every coverage, error rate, count and price above is computed from those runs by risk_coverage.py, and the per-claim results are in the repo alongside it.
The labelled set is built from source audits of three papers: SkillsBench (arXiv:2602.12670), ReasoningBank (arXiv:2509.25140) and R-Zero (arXiv:2508.05004).
The 0.8 auto-accept threshold is TypeSafe's own, from their citation-check cookbook.
The first piece in this series, with the accuracy, cost and latency numbers this one rests on: Jev is not an LLM.