Jev at TypeSafe's own 0.8 threshold: 69% of checks skipped, and the worst errors still get through.
Jamie Watters
Operational resilience and AI delivery practitioner. Technology since 1985.
Correction in progress, added 21 September. The central claim below, that GPT-5.4's confidence score gives you no usable threshold, is wrong, and it is wrong because of a mistake this article warns you about. My six thresholds stopped at 0.95. Every one of GPT-5.4's 42 answers sits between 0.94 and 0.99, so the entire range I was testing was below where its variation lives. Swept over its own values, GPT-5.4 goes from 64.3% coverage at a 3.7% error rate to 100% coverage at 14.3%, which is a working control, and at every matched coverage level it beats Jev rather than losing to it. Two other figures are also wrong: Gemini drops ten claims above 0.95, not one, and the cascade costs $0.000439 per claim rather than $0.000424, because the escalated claims are dearer than GPT-5.4's average. An independent adversarial check found all three by sweeping the raw data, which is precisely the instruction I gave readers further down and did not follow myself. The article is being rewritten around what the data says. Nothing below this line has been edited yet, so read it knowing the frontier-model comparison does not hold.
Who this is for. You have built something where a model gives you an answer and sometimes that answer is wrong. You cannot check every one by hand and you do not want to. The model hands back a confidence number with each answer, so the obvious move is to trust the confident ones and look at the rest yourself. It took me until this week to check whether that number is worth anything. By the end you will know how to run the same check on your own setup in about five minutes, what it buys when it works, and the case where it does not work at all.
Skip it if you want a Jev tutorial, or the theory behind calibration. There is no maths in here past dividing one count by another.
On the length. About nineteen minutes, and most of that is tables. If you only want the method, skip to "Run this on your own setup": it is a table you fill in and a pass rule. Everything before it is how I got the numbers and everything after is where they are thin, which is here because a benchmark you cannot argue with is a benchmark you cannot check.
TypeSafe came out of stealth with Jev on 15 September. It returns a typed answer and a probability instead of text, and it costs $0.042 per million input tokens. If you are looking at it this week, you are probably looking for the same thing I was: a way to stop paying for a checking step.
The obvious build is the one TypeSafe's own citation-check cookbook points at. Run every claim through Jev, auto-accept anything at 0.8 or above, look at the rest yourself. It is cheap, it is fast, and unlike two of the three frontier models its confidence score moves in a way you can act on, which I measured on Sunday and still stand by. Calibration is a property of a population, not of an answer, and one run over 42 claims is not a general proof of it. What that run shows is that Jev's score ranks its own answers usefully on this task.
This piece is about what that threshold actually buys, because I had not measured it and neither, as far as I can find, has anyone else.
The short version, from the JevBench runs: on 42 claims it skips 69% of the checking and is wrong on 6.9% of what it skips, which is a real saving on a real job. On the 18 of those that are sentences from something I actually published, rather than controls I built myself, no threshold in the tested grid gets any of the four models under a 10% error rate. And the one claim Jev accepts at every setting from 0.60 to 0.95, at a stated confidence of 0.98, is the hardest error in the set to catch by eye.
So the misreading to avoid is the one I nearly made. A calibrated confidence score does not tell you which checks you can stop doing. It tells you which ones to do first.
No new model calls here: same 42 claims, same four models, same runs as Sunday, read a different way. The script is in the repo.
The two numbers that decide it
You have a checking step. A model looks at each claim, returns a verdict and a confidence. You pick a threshold. At or above it you take the verdict and move on. Below it, a person looks.
Two numbers describe every threshold. Coverage is the share of claims you take without looking at them. Error rate is the share of those you took that were wrong. Move the threshold and both move together: that is the risk and coverage curve, and it is the thing to look at before wiring any of this into a pipeline.
The question underneath it is blunter. Does changing the threshold change anything?
On the full set, one of the four responds to the setting
Jev, all 42 JevBench claims:
| threshold | coverage | error rate among accepted |
|---|---|---|
| 0.95 | 57.1% | 4.2% |
| 0.90 | 61.9% | 3.8% |
| 0.85 | 64.3% | 7.4% |
| 0.80 | 69.0% | 6.9% |
| 0.70 | 76.2% | 9.4% |
| 0.60 | 76.2% | 9.4% |
That is a curve. Accept less, be wrong less. At 0.80, which is the auto-accept threshold TypeSafe's own citation-check cookbook proposes, you take 29 of the 42 verdicts unchecked and two of them are wrong.
One detail in there is worth more than the headline. Coverage stops climbing at 76.2%. Ten of the 42 JevBench claims came back under 0.6 and no threshold in the grid picks them up. Jev refuses to commit on a quarter of the set, which is the behaviour you want from something you are about to trust with the rest.
Now GPT-5.4, same 42 claims:
| threshold | coverage | error rate among accepted |
|---|---|---|
| 0.95 | 97.6% | 12.2% |
| 0.90 | 100% | 14.3% |
| 0.85 | 100% | 14.3% |
| 0.80 | 100% | 14.3% |
| 0.70 | 100% | 14.3% |
| 0.60 | 100% | 14.3% |
That is not a curve. It is one point printed six times. Every threshold from 0.60 to 0.90 gives you the same deal on the JevBench set: accept everything, be wrong 14.3% of the time, and nothing you can set changes it.
Gemini 3.1 Pro is nearly as flat and not quite. Across the same 42 JevBench claims it accepts all of them at every setting from 0.60 to 0.85, wrong on 19.0% of them. At 0.90 exactly one claim drops out, giving 97.6% coverage and 17.1% error, and 0.95 is identical to 0.90. One claim of movement across the whole range is not a control you can use.
Put a threshold on top of either of those and it changes nothing about what gets checked. It will still look like it is working, because the setting produces a number either way.
A correction to what I wrote on Sunday
I said all three frontier models put nearly everything in their top confidence bucket and that their confidence therefore carried no information. That is right for GPT-5.4 and Gemini. It is wrong for Claude Sonnet 5.
| threshold | Sonnet 5 coverage | error rate among accepted |
|---|---|---|
| 0.95 | 57.1% | 8.3% |
| 0.90 | 73.8% | 9.7% |
| 0.85 | 88.1% | 13.5% |
| 0.80 | 92.9% | 15.4% |
| 0.70 | 100% | 14.3% |
That moves. Sonnet's stated confidence spans 0.70 to 0.99 and the threshold buys you something real.
I missed it because my reliability table used five buckets of 0.2 each. That put 39 of Sonnet's 42 answers in one box labelled 0.8 to 1.0, and threw away the structure inside the box before I ever looked at it. The bucket was too wide, so I read a flat table as a flat model. Same run, same file, different grain.
Jev still wins where the two do the same amount of work. At 57.1% coverage on the JevBench set, Jev is wrong on 4.2% of what it accepts and Sonnet on 8.3%. But "frontier models have no usable confidence" was too broad a sentence, and one of the three has one.
On the claims that actually matter, nothing works
Everything above is all 42, and 24 of those are constructed controls I built myself. The eighteen that carry the headline are real sentences from a published article, paired with the source passage they cite. Here is every model on those eighteen, at their own best threshold:
| model | best threshold | coverage | error rate among accepted |
|---|---|---|---|
| Jev | 0.90 | 38.9% | 14.3% |
| GPT-5.4 | 0.95 | 94.4% | 29.4% |
| Claude Sonnet 5 | 0.90 | 61.1% | 27.3% |
| Gemini 3.1 Pro | 0.95 | 94.4% | 41.2% |
Nothing in the whole JevBench grid gets the error rate under 10%. Jev's best is 14.3%, and to get it you accept 7 of the 18 instead of 11, so you give up a third of what little saving there was and you are still wrong one time in seven.
So on real published sentences, the honest answer to how much checking you can safely stop doing is: none of it. There is no threshold on any of these four models at which I would let a sentence go out under my name unchecked. The threshold works on the controls I built. It does not work on the sentences the controls were built to stand in for, which is the only reason I built them.
That is the same inversion the first piece found, showing up in a different measurement. Easy material says the problem is solved. Real material says it is not.
Error rate is the wrong risk number anyway
There is a subtler mistake buried in everything above, and it took me a while to see it.
A gate does not treat all errors the same. If the model says a sentence is unsupported and it is wrong, I go and look at the sentence, find it is fine, and publish it. That costs me five minutes. If the model says a sentence is supported and it is wrong, the sentence goes live under my name. Averaging those two into one error rate hides the expensive half inside the cheap half.
So the honest number is narrower: of the claims auto-accepted as supported, how many should not have been published?
On the eighteen real claims, twelve should not be published. At 0.80:
| model | published unchecked | of those, bad | share of the bad ones that escaped |
|---|---|---|---|
| Jev | 4 | 1 | 8.3% |
| GPT-5.4 | 5 | 2 | 16.7% |
| Claude Sonnet 5 | 8 | 3 | 25.0% |
| Gemini 3.1 Pro | 6 | 3 | 25.0% |
Jev publishes the fewest sentences unchecked and the fewest bad ones. It also publishes four and gets one of them wrong, so read the right-hand column as a direction and not a rate.
The claim Jev waves through is the same one at every threshold from 0.60 to 0.95, and it is the same one all four models miss. It's a quotation in my article presented as one sentence from the SkillsBench paper. The paper has the first half as a section heading and the second half as a sentence further down. Every word is in the document, in that order. Nothing was checking whether they had ever been one sentence, and Jev returned that verdict at 0.98.
No threshold can catch a claim the model is confidently wrong about, because the threshold reads the confidence. That is the floor under this whole method and it does not move.
Escalating to a bigger model makes the hard half worse
The obvious build is a cascade. Take Jev's verdict when it is confident, hand the rest to GPT-5.4, pay a fraction of the price. So I ran it.
At a 0.80 threshold it escalates 31.0% of the JevBench claims and costs $0.000424 per claim, against $0.001289 for GPT-5.4 alone. A third of the price. Accuracy across all 42 comes out at 88.1%, which beats both Jev alone and GPT-5.4 alone, both of which scored 85.7%.
Then look at the eighteen real ones in JevBench. The cascade gets 72.2% of those right. Jev alone gets 77.8%. The cascade also publishes two bad sentences where Jev alone publishes one.
Paying more made it worse. Here is why, and I checked it rather than assuming it:
| the 8 real claims Jev escalated at 0.80 | accuracy |
|---|---|
| Jev's own escalated verdict | 75.0% |
| GPT-5.4 on the same eight | 62.5% |
Across all 42 JevBench claims the escalation does help: GPT-5.4 gets 76.9% of the escalated ones right against Jev's 69.2%. On the real eighteen it reverses. The claims Jev is unsure about are hard for a reason, and the expensive model is not better at them, only more certain.
So the escalation target is the decision, not an implementation detail. Sending uncertain claims to a bigger model is a cost optimisation dressed as a safety measure. For real prose, the thing on the other end of the escalation has to be a person.
What I am doing with it
Jev goes into my publishing gate at 0.80. Anything it flags gets blocked and I look at it. Anything it clears below the line gets escalated to me, not to another model. Anything it clears above the line still gets read, because of the spliced quotation.
Which means the honest accounting of what this buys me is: not less checking, but better-ordered checking. Four of Jev's six errors are in the thirteen claims it hands back, and knowing which thirteen is worth something even though it does not let me skip the other twenty-nine.
Your task might be the easy kind, and that changes the answer
Here is the strongest objection to everything above, and I think it is largely right.
My eighteen hard claims are sentences from one article I wrote, and the errors in them are a particular sort: a quotation spliced from two places, a true number attached to the wrong quantity, a result from the wrong version of a paper. If you are classifying support tickets, extracting invoice fields, or checking whether an answer used the retrieved document, your failures look nothing like that. They look far more like my constructed controls, where one thing is changed and there is a clean right answer.
On the controls the threshold works. Jev missed only two of those 24, and both came back at 0.37 and 0.71, below its own 0.8 line, so a threshold would have caught them both.
So the finding is not "thresholds never work". It is that a threshold's usefulness depends on whether your errors are clean or subtle, and you cannot tell which kind you have from an accuracy score. The run below tells you, on your own data, in about five minutes.
Run this on your own setup before you build anything
There is no maths in it past dividing one count by another.
Get a set of answers where you already know whether the model was right. A spreadsheet, a log, a folder of bug reports, whatever you have. I used 42 and that was enough to see what was going on. Each one needs two things beside it: the confidence the model gave, and whether it turned out to be correct.
Then fill in this table. For each threshold, count how many answers came in at or above it, and how many of those were wrong.
| threshold | how many at or above | how many of those wrong | error rate |
|---|---|---|---|
| 0.95 | |||
| 0.90 | |||
| 0.85 | |||
| 0.80 | |||
| 0.70 | |||
| 0.60 |
Two counts, six times. That is the whole method, and it is why risk_coverage.py is counting rather than doing anything clever.
The pass rule. Compare the top row to the bottom row. If the error rate at 0.95 is not at least a third lower than the error rate at 0.60, and the coverage at 0.95 is not at least ten points lower, your model's confidence is not telling you which answers to check, and a threshold built on it will change nothing. That bar is a judgement, not a standard, and I am writing it down so it can be argued with rather than left in my head. Jev's went 4.2% to 9.4% across the JevBench set, so the score is carrying information. GPT-5.4's went 12.2% to 14.3% while its coverage went 97.6% to 100%, which is no choice at all. It kept every one of its 42 answers at or above 0.94, so every setting from 0.60 to 0.90 accepts precisely the same claims. Gemini's lowest was exactly 0.85, which buys one claim of movement in the whole range and nothing else.
Then check the second column too. If nothing ever falls below your threshold, the model is not declining to answer, it is just always confident. Jev left ten of 42 below 0.6 whatever I set. That refusal is the part you are actually buying. One trap in that check: it is relative to where you put the threshold. Set it at 0.90 and see nothing below it, and you have learned nothing: GPT-5.4 still accepts 97.6% of the JevBench set even at 0.95. Sweep the whole range before concluding anything.
Now fill in the second table, which is the one that decides it. The first table pools two errors that cost completely different amounts. This one counts only what goes live unchecked.
| threshold | accepted AND the model said it was fine | how many of those were not fine |
|---|---|---|
| 0.95 | ||
| 0.90 | ||
| 0.85 | ||
| 0.80 | ||
| 0.70 | ||
| 0.60 |
Mine at 0.80, on the 18 real claims: four published unchecked, one of them wrong. That is the number I could not live with, and it is invisible in the first table.
Two more things I got wrong and you should not repeat. Run both tables twice, once on clean test cases and once on real material you got wrong in production, because the answer differs and the clean run is the flattering one. And pick your threshold on one half of your data and report the numbers on the other half. I did neither. My figures are in-sample, which makes them a reason to test 0.8, not evidence that 0.8 is safe.
You will not find any of this out from the accuracy figure, because the accuracy figure is fine.
What this is not
One run per model, no repeats, so none of these numbers has an error bar. The eighteen real claims carry the headline and several of the cells above have single-figure denominators: four claims published unchecked, one of them wrong. That is a direction, not a percentage, and I have written it as a count wherever the count is small enough to matter.
The labels come from source audits of my own published article, checked by hand against the primaries rather than by an independent panel. One of the eighteen is marked contested in the data. Every claim in the hard set comes out of one person's publishing pipeline, which is the weakness I cannot fix on my own and the reason the repo asks for other people's corrections.
The eighteen are not a sample of incoming claims. They are a sample of what already got past me, and a peer reviewer put that more sharply than I had. These are sentences that survived writing, an automated gate and a first round of review, and were then labelled from audits of that same article. So the set is selected for looking supported when it is not, which is precisely the failure I am measuring. That makes "on the real claims, nothing works" a finding about the errors that already escape my checking, not about everything a checking step sees. It is still the right set for asking whether a threshold catches what I miss. It is the wrong set for estimating what a threshold would do to a live stream, and I had been sliding between the two.
There is also no held-out set. I looked at the confidence-versus-error relationship, took 0.8 because TypeSafe proposes it, and then reported the coverage and risk on the same 42 observations. Those numbers are in-sample. They say the threshold is worth testing as a routing rule; they do not establish 0.8 as a safe operating point for claims I have not seen.
And my unsupported label merges two different things, a flat contradiction and a claim that is correct in substance but wrong as written. TypeSafe's own guidance is that Jev answers the question you were asking rather than the one you meant, so a compound category is the wrong shape to hand it. That is listed in the repo as a known weakness of the first run, and it matters more here than it did there: this article's whole contribution is separating supported from everything else, and the confound sits inside the category I built the separation on.
The two confidence numbers may not be the same kind of object, and this piece leans on comparing them harder than the first one did. Jev's is a probability the model is built to produce. The frontier figure is a number I asked for in a JSON field, so it is self-report rather than anything the model measured. That is still the right comparison for a pipeline, because the self-reported number is what you would actually receive and act on. It is not a like-for-like test of calibration, and "GPT-5.4's threshold does nothing" is partly a statement about what happens when you ask a model to rate itself in a field.
And the biggest one is unchanged from Sunday: the frontier models answered straight into a JSON verdict with no room to reason first. Every number here about them is a number about schema-constrained models, and a model allowed to think before it answers may well spread its confidence differently. Gemini 3.1 Pro is also a preview build.
Running them again with a reasoning step is the next test and it goes out on Tuesday. If Jev loses it, that article gets written with the same prominence as this one.
The script that produced every table above is risk_coverage.py at github.com/TheWayWithin/jev-bench. It needs no API keys and no venv: the raw runs are already in the repo, so you can check any figure here in about a minute.
If it saved you an afternoon, buy me a coffee.
Sources
The benchmark: JevBench, run 20 September 2026, analysed 21 September. Jev via TypeSafe, called through the moving jev-latest alias rather than a pinned version, which is recorded in every run file and is one of the repo's listed weaknesses. And openai/gpt-5.4, anthropic/claude-sonnet-5 and google/gemini-3.1-pro-preview via OpenRouter. 42 claims, one run per model. Every coverage, error rate, count and price above is computed from those runs by risk_coverage.py, and the per-claim results are in the repo alongside it.
The labelled set is built from source audits of three papers: SkillsBench (arXiv:2602.12670), ReasoningBank (arXiv:2509.25140) and R-Zero (arXiv:2508.05004).
The 0.8 auto-accept threshold is TypeSafe's own, from their citation-check cookbook.
The first piece in this series, with the accuracy, cost and latency numbers this one rests on: Jev is not an LLM.