Jev is not an LLM. On the hard half it beat GPT-5.4, Gemini 3.1 Pro and Sonnet 5, at a fiftieth of the cost.
Jamie Watters
Operational resilience and AI delivery practitioner.
Jev does not write anything. That is the first thing to understand about it and the reason it is interesting.
You hand it some text and a question with a fixed set of answers, and it hands back one of those answers and a probability. No sentence, no explanation, nothing to parse. TypeSafe's documentation says jev-1.13 "is not trained to generate text", which is a claim about the output and not about what happens inside it. TypeSafe, the company behind it, calls this a System One model. Whatever you call it, it is not a small language model and it is not a chatbot with the personality filed off. It is a different shape of thing, and it costs $0.042 per million input tokens with output free.
TypeSafe came out of stealth on 15 September 2026 with a $40M seed round led by DCVC. That is five days ago. The company was founded in 2024, so the model is older than its announcement; what is five days old is anybody outside being able to look at it.
I gave it a job I actually need doing, on 42 checks where I already knew the right answer, against GPT-5.4, Claude Sonnet 5 and Gemini 3.1 Pro.
Across all 42 it tied the best of them. On the 24 easy ones it lost, and the frontier models were perfect. On the 18 hard ones it beat every one of them by eleven points, at a fiftieth of the price of the cheapest and four times the speed.
That split is the whole article.
Who this is for. Anyone whose pipeline has a checking step in it: verifying claims against sources, grading outputs, deciding what a human needs to look at. By the end you will know whether this class of model is worth an afternoon, and what none of these models can do yet.
Skip it if you want a tutorial for Jev. This is a measurement and an argument, not a walkthrough.
Everything is open. The 42 claims, the code, every raw run and the scored output are at github.com/TheWayWithin/jev-bench, MIT licensed. You do not have to take my word for any number below. A full four-model run is about 35 cents.
On the length. About eleven minutes. The method is in here on purpose, because a benchmark you cannot argue with is a benchmark you cannot check.
Here is why I was testing anything at all.
Two days ago I published an article that had to be rewritten rather than patched, and every error in it was the same one: a figure or a quotation that traced to a real source, where the source did not say what my sentence said it said.
Three of them, from its published corrections log. The SkillsBench numbers came from version one of the paper and the metadata around them from version four, and I had not noticed the paper had been revised. I attributed four acknowledgements to ReasoningBank's limitations appendix; it contains three, and none of them is any of my four. I quoted a sentence about performance degradation and cited version one of R-Zero, where that sentence appears only in version four.
Four outside reviews and a set of source audits found them. My own pre-publish gate found none of them, and it never could have, because of what it checks.
The gate checks the wrong half
I have a script that runs before anything publishes. It catches banned words, em dashes, American spellings, closing rhetorical questions, and any figure that appears in a paragraph without a named source attached. That last check is the one that matters here and it is the one that fails.
It checks that a number has a source. It has no way to check that the source says the number.
There is no cheap way to close that gap. Reading the primary source is the only thing that has ever worked, and reading eight papers properly is most of a day. So the question was whether a machine could do the narrow version of it: here is a sentence, here is the passage it cites, does the passage support the sentence as written.
What I actually tested
I had the labels sitting there already. The source audits on that article go claim by claim through eight papers, quote the passage, and give a verdict. That is a labelled set, built by someone checking the primaries, including every case where a confident model had got it wrong.
So I built one. 42 claims, split in two.
Eighteen of them are real: verbatim sentences from the published article, paired with the source text they cite, labelled from the audits. These are hard on purpose. They are sentences that survived writing, a gate, and a first round of review.
The other 24 are constructed controls, built from passages quoted verbatim in the SkillsBench, ReasoningBank and R-Zero audits. A sentence that restates the passage. The same sentence with one digit changed. A claim about something the passage does not mention. They exist so that the confidence curve has enough points to say anything.
Three possible answers for each: the source supports the claim, the source addresses it and does not support it as written, or the source does not speak to it at all. That middle category is doing real work. It covers flat contradiction and it also covers the much commoner case of a sentence that is roughly right and wrong in its detail, because from the gate's point of view those are the same event. Do not publish this sentence.
Then the systems. GPT-5.4, Claude Sonnet 5 and Gemini 3.1 Pro, all through OpenRouter, each given a JSON schema to answer into so they could not wander off into prose. And Jev, which needs no schema, because a fixed set of answers is the only thing it can produce.
Every model saw identical state and identical wording. None was tuned for the task.
I wrote the pass mark before I ran it
This is the part I would skip if I were being honest about my instincts, and it is the part that stops the write-up becoming a story about whichever way it landed.
Before the first call, I wrote this into the JevBench README. Jev replaces the check in my gate only if it catches at least 80% of the bad sentences. It must catch at least as many as the frontier models do, with fewer than two false alarms in five, and be cheaper and faster. Below 60% caught, it is a no. Between the two, it is a pre-filter rather than a replacement.
The numbers
| all 42 | the easy 24 | the hard 18 | cost per claim | seconds | |
|---|---|---|---|---|---|
| Jev | 85.7% | 91.7% | 77.8% | $0.000025 | 0.23 |
| GPT-5.4 | 85.7% | 100% | 66.7% | $0.001289 | 1.01 |
| Claude Sonnet 5 | 85.7% | 100% | 66.7% | $0.002228 | 2.76 |
| Gemini 3.1 Pro | 81.0% | 100% | 55.6% | $0.004824 | 3.78 |
Read the easy column and Jev is the worst of the four. Read the hard column and it is eleven points clear of the best of them. Those are the same 42 claims and the same four models.
Three of them landing on 85.7% looks too neat, so I checked the JevBench files claim by claim. It is 36 out of 42, three times over, and it is a coincidence of totals rather than a copied file. The models fail on different claims. Jev's six misses are ctl-03, ctl-17, rb-04, rz-03, rz-04 and sb-03. GPT-5.4's are rb-04, rz-03, rz-05, rz-06, sb-03 and sb-06. They share three. No two models in the run produced identical answers; the closest pair agreed on 40 of 42 and the rest on 36 to 38.
There is a structural difference hiding inside that tie, and it is the reason the first column is the least useful one in the table. Two of Jev's six errors are in the easy set. Every single error the three frontier models made is in the hard eighteen. Same total, opposite shape.
Cost is measured rather than estimated. The frontier figures are what OpenRouter actually charged. Jev's is computed from the tokens it returned at TypeSafe's published rate.
One thing to be straight about, because the marketing is not. The price gap is enormous against frontier models and small against cheap ones. Jev at TypeSafe's published $0.042 per million input tokens sits beside Groq's Llama 3.1 8B at around $0.05, which is a difference of about a fifth, not a factor of fifty. Every multiple in this article is measured against the three frontier models above, because those are what people actually put in a checking step. Against a small open model the argument has to be won on the result, not the invoice.
The inversion is the finding
Change a digit in a sentence and ask whether the source still supports it, and a frontier model gets it right every time. All three were perfect on the constructed controls. Jev was not: it missed two of 24.
Give the same models a real published sentence and they fall over. GPT-5.4 and Sonnet 5 drop to two in three. Gemini drops to just over half. Jev drops least.
The two sets differ in more than difficulty, and I cannot isolate which difference is doing the work. The hard set is real prose, where the mistake is almost never a flat contradiction. It is a true number pointed at the wrong quantity, a genuine quote assembled from two places, a correct result from a different version of the paper. Nothing in those sentences is false on its face. They are wrong in a way you only see by holding the sentence against the passage and noticing that the two are not about quite the same thing.
That is the class of error that survives every check. It is also, I think, why synthetic benchmarks keep reporting that this problem is solved.
Every model is sure, and one of them isn't
Correction, added 21 September. Two sentences in this section originally said the frontier models' confidence "carries no information" and that you cannot use it to decide what to check. That's wrong. GPT-5.4's 42 answers cluster between 0.94 and 0.99, a narrow band, and I tested it against six round thresholds that stopped at 0.95. Tested at its own values instead, GPT-5.4's confidence does sort its answers, and on the full 42 claims it beats Jev at every coverage level the two share but the top, where both accept everything and land on the same 14.3% error. On the 18 real claims that carry this article's headline it is closer than "Jev wins": GPT-5.4 is ahead at low coverage (12.5% error at 44.4% coverage, against Jev's 25.0%), the two tie through most of the middle of the range, and Jev only pulls ahead above 90% coverage, where both are already wrong on more than one claim in five. What holds: GPT-5.4 said 0.98 on average and was right 85.7% of the time, which is genuine miscalibration. Calibration and the ability to sort your own right answers from wrong ones are different properties, and I'd collapsed them into one claim. The full re-run, all four models, is in a follow-up piece.
GPT-5.4 put all 42 of its JevBench answers in the top confidence bucket, at a mean stated confidence of 0.98, and was right 85.7% of the time. Claude Sonnet 5 put 39 of 42 there. Gemini put all of them.
Jev's spread across four buckets. Its top bucket, 29 of the 42 JevBench items, was right 93.1% of the time. Everything below that was right 60 to 80%.
This is the part where I should say what I found out afterwards. TypeSafe has not published a calibration result. No expected calibration error, no reliability diagram, no third-party audit of the claim their whole product rests on. Their own worked example of citation checking uses eight citations from a single internet standards document. So a Sunday afternoon and 42 claims is not a small contribution to the public record on this model. It is close to all of it, which says more about how new this is than about the benchmark.
The practical difference: GPT-5.4's answers were not actually all identical at 0.98, that was only the mean. On the JevBench set they span 0.94 to 0.99, narrow enough that the round thresholds I tested first, 0.60 to 0.95, mostly sat below the range and showed nothing moving. Tested at its own values, the number does sort its answers: keep only its 0.99s and you get 64.3% coverage at 3.7% error; keep everything down to 0.94 and error rises to 14.3%. What it does not do is match its own accuracy: it said 0.98 on average and was right 85.7% of the time.
With Jev you can draw a line. The buckets here are fifths, so the top one starts at 0.8, which turns out to be the same threshold TypeSafe's own citation-check cookbook proposes for auto-accept. I did not know that when I picked the bins. Take everything at 0.8 and above, send the other thirteen to a reasoning model or to a person. And the line works: both of the easy claims Jev got wrong came back at 0.37 and 0.71, below its own threshold. It did not quietly fail. It said it was unsure, and it was right to be.
What nothing caught
Three of the eighteen hard claims were missed by all four models.
One is the spliced quotation, the phrase in quotation marks that is a heading joined to a sentence from later in the paper. Every model called it supported. The words are all there, in that order, in that document. Nothing was checking whether they were ever one sentence.
The other is "sixteen points of decay", where the arithmetic is right and the thing that decayed is the accuracy of self-generated training labels rather than model capability. Every model called it supported, because the number matches and the sentence reads as though it follows.
TypeSafe's own documentation predicts that second one. Their page on where Jev is unreliable says plainly that it "is not a calculator", that it "recognizes the shape of an answer rather than tallying", and that it "reads dates as text, not as ordered quantities". A claim whose only fault is that a correct number is pointed at the wrong quantity is precisely the shape that slips through. They published that before I ran anything, and it is worth more than most benchmark results.
The third is a sentence of mine that claimed a paper's caveats are "published in the place almost nobody reads". The correct answer is that the source does not address it: a paper reports its limitations, it does not report who reads them. All four models called it unsupported, which is a reasonable-sounding verdict on a question the passage never touches.
All three are sentences that got published under my name and were changed in the rewrite. The first two are flagged in the source audits rather than itemised in the corrections log, which lists the framing and the versioning errors: the audits are where the sentence-level verdicts live, and they are what the labels here come from. Nine of the eighteen were solved by every model. Three were solved by none. The interesting work is in the six in between, and Jev won most of those.
What I am doing with it
Jev goes into my gate as a pre-filter, not a replacement. Anything it flags at high confidence gets blocked. Anything below the line gets escalated. It does not pass a sentence on its own authority, because of those three.
By the rule I wrote into JevBench beforehand, it does not replace anything: it caught 78.9% of the bad sentences against the 80% bar, and GPT-5.4 caught 89.5%. It earns its place on precision and on price. When Jev says a sentence is unsupported it is right 88.2% of the time, against 84.2% for Sonnet 5, 81.0% for GPT-5.4 and 76.2% for Gemini, and it costs a fiftieth as much.
The wider thing I take from it is not about any of these products. A model's calibration and its ability to sort its own right answers from its wrong ones are different properties, and they do not arrive together. I had been treating a confidence score as a free extra that comes with an answer, and I collapsed those two properties into one when GPT-5.4's badly calibrated 0.98 looked, at the six thresholds I happened to pick, like it wasn't sorting anything either. It was. On the 18 real claims that carry this article's headline, no model's confidence gets you below roughly one error in ten at a coverage worth having, and Jev is not simply the best of the four there: GPT-5.4 is actually ahead at the lower coverage levels, the two tie through most of the range, and Jev only leads once both are already wrong on more than a fifth of what they accept. The full four-model comparison, done properly, is the follow-up piece.
Turn the accuracy numbers round and that would still hold. A system that is slightly less accurate but tells you when to stop trusting it is the more useful of the two, because you can route the uncertain cases somewhere else.
What this is not
One run per model, no repeats, so none of these numbers has an error bar. Eighteen items carry the headline, which is enough to show a direction and not enough to put a decimal place on it. The labels come from source audits produced by another model and checked by hand against the primaries. They are not the verdict of an independent panel. One of the eighteen is marked as contested in the data, because a careful person could argue it either way.
The biggest open question is whether the frontier models were handicapped. They answered straight into a JSON verdict with no room to work through the comparison first, which is not how anyone sensible uses them on a task like this. Jev pays no equivalent cost, because it was never going to reason out loud. So the honest reading of the hard-set gap is that it is a gap against schema-constrained frontier models, and this piece cannot yet tell you whether it survives against the same models allowed to think before they answer. That run is next, and I will publish it either way.
A second thing worth naming: the two confidence numbers may not be the same kind of object. Jev's is a probability the model is built to produce. The frontier figure is a number I asked for in a JSON field, which is self-report rather than anything the model measured. That comparison is still the right one for a pipeline, because the self-reported number is what you would actually receive, but it is not a like-for-like calibration test and I should not call it one.
Three further design flaws I can name, now that I have read the documentation properly. I called the model by its moving alias rather than pinning a version, so a future run may not reproduce this one. My "unsupported" label merges flat contradiction with correct-but-altered, and TypeSafe's own guidance is that Jev answers the question you wrote rather than the one you meant, so a compound category is the wrong shape to hand it. And I asked one Choice question where the thing they actually sell is several questions fanned out over the same text in one pass, billed once. Any of the three could move the result, and I would not call the gap settled until they are fixed.
Gemini 3.1 Pro is a preview build. Latency for all three frontier models is end to end through OpenRouter and includes its routing hop, so it is not the providers' own. The frontier models answer into a JSON schema, which is a design choice worth knowing about: without it they spend the token budget reasoning in prose and often never reach a verdict at all.
Forty-two claims against four models took under six minutes of model time and cost 35 cents, and almost all of both was the frontier models.
The whole thing is at github.com/TheWayWithin/jev-bench: the claims, the labels, the harness, every raw run, and the known weaknesses written out rather than left for you to find. Two contributions would genuinely help. More real claims from someone else's corrections, because the labels here all come from one person's publishing pipeline and that is the one weakness I cannot fix on my own. And a rerun with the frontier models allowed to reason first, which is the experiment that decides whether any of this holds.
If it saved you an afternoon, buy me a coffee.
Sources
The benchmark: JevBench, run 20 September 2026. Jev via TypeSafe, called through the moving jev-latest alias rather than a pinned version, which is what every run file records, and openai/gpt-5.4, anthropic/claude-sonnet-5 and google/gemini-3.1-pro-preview via OpenRouter. 42 claims, one run per model. Every number above comes from those runs and the per-claim results are recorded alongside them.
The labelled set is built from source audits of three papers: SkillsBench (arXiv:2602.12670), ReasoningBank (arXiv:2509.25140) and R-Zero (arXiv:2508.05004).
TypeSafe's own documentation, for the pricing and the model identifier: https://docs.typesafe.ai/models
Correction, 21 September: the confidence-threshold section originally said the frontier models' stated confidence carried no information. It doesn't hold up; the full re-run is in GPT-5.4 beats Jev on a confidence threshold, once tested at its own values.