Jev is not an LLM. On the hard half it beat GPT-5.4, Gemini 3.1 Pro and Sonnet 5, at a fiftieth of the cost.
Jamie Watters
Operational resilience and AI delivery practitioner. Technology since 1985.
Jev does not write anything. That is the first thing to understand about it and the reason it is interesting.
You hand it some text and a question with a fixed set of answers, and it hands back one of those answers and a probability. No sentence, no explanation, no reasoning to read, because there is none. TypeSafe, the company behind it, calls this a System One model. Whatever you call it, it is not a small language model and it is not a chatbot with the personality filed off. It is a different shape of thing, and it costs $0.042 per million input tokens with output free.
TypeSafe came out of stealth on 15 September 2026 with a $40M seed round led by DCVC. The product is five days old.
I gave it a job I actually need doing, on 42 checks where I already knew the right answer, against GPT-5.4, Claude Sonnet 5 and Gemini 3.1 Pro.
Across all 42 it tied the best of them. On the 24 easy ones it lost, and the frontier models were perfect. On the 18 hard ones it beat every one of them by eleven points, at a fiftieth of the price of the cheapest and four times the speed.
That split is the whole article.
Who this is for. Anyone whose pipeline has a checking step in it: verifying claims against sources, grading outputs, deciding what a human needs to look at. By the end you will know whether this class of model is worth an afternoon, and what none of these models can do yet.
Skip it if you want a tutorial for Jev. This is a measurement and an argument, not a walkthrough. The code is boring and the docs cover it.
On the length. About eleven minutes. The method is in here on purpose, because a benchmark you cannot argue with is a benchmark you cannot check.
Here is why I was testing anything at all.
Two weeks ago I published an article that was wrong in three places, and every one of them was a number that traced to a real source. The source just did not say what my sentence said it said.
One was a quotation. I had written a phrase inside quotation marks and attributed it to a paper. The words exist in that paper. They exist as a section heading spliced onto a sentence from three paragraphs later. Another was a figure that fell from 79% to 63%. That is exactly what the R-Zero paper reports. The thing that fell, though, was the accuracy of the model's own self-generated training labels, and I had written it as though the model's capability had fallen. The third was a result that only appears in version four of a paper I had cited as version one.
Four peer reviewers and a separate source audit found them. My own pre-publish gate found none of them, and it never could have, because of what it checks.
The gate checks the wrong half
I have a script that runs before anything publishes. It catches banned words, em dashes, American spellings, closing rhetorical questions, and any figure that appears in a paragraph without a named source attached. That last check is the one that matters here and it is the one that fails.
It checks that a number has a source. It has no way to check that the source says the number.
There is no cheap way to close that gap. Reading the primary source is the only thing that has ever worked, and reading eight papers properly is most of a day. So the question was whether a machine could do the narrow version of it: here is a sentence, here is the passage it cites, does the passage support the sentence as written.
What I actually tested
I had the labels sitting there already. The source audits on that article go claim by claim through eight papers, quote the passage, and give a verdict. That is a labelled set, built by someone checking the primaries, including every case where a confident model had got it wrong.
So I built one. 42 claims, split in two.
Eighteen of them are real: verbatim sentences from the published article, paired with the source text they cite, labelled from the audits. These are hard on purpose. They are sentences that survived writing, a gate, and a first round of review.
The other 24 are constructed controls, built from passages quoted verbatim in the SkillsBench, ReasoningBank and R-Zero audits. A sentence that restates the passage. The same sentence with one digit changed. A claim about something the passage does not mention. They exist so that the confidence curve has enough points to say anything.
Three possible answers for each: the source supports the claim, the source addresses it and does not support it as written, or the source does not speak to it at all. That middle category is doing real work. It covers flat contradiction and it also covers the much commoner case of a sentence that is roughly right and wrong in its detail, because from the gate's point of view those are the same event. Do not publish this sentence.
Then the systems. GPT-5.4, Claude Sonnet 5 and Gemini 3.1 Pro, all through OpenRouter, each given a JSON schema to answer into so they could not wander off into prose. And Jev, which needs no schema, because a fixed set of answers is the only thing it can produce.
Every model saw identical state and identical wording. None was tuned for the task.
I wrote the pass mark before I ran it
This is the part I would skip if I were being honest about my instincts, and it is the part that stops the write-up becoming a story about whichever way it landed.
Before the first call, I wrote this into the JevBench README. Jev replaces the check in my gate only if it catches at least 80% of the bad sentences. It must catch at least as many as the frontier models do, with fewer than two false alarms in five, and be cheaper and faster. Below 60% caught, it is a no. Between the two, it is a pre-filter rather than a replacement.
The numbers
| all 42 | the easy 24 | the hard 18 | cost per claim | seconds | |
|---|---|---|---|---|---|
| Jev | 85.7% | 91.7% | 77.8% | $0.000025 | 0.23 |
| GPT-5.4 | 85.7% | 100% | 66.7% | $0.001289 | 1.01 |
| Claude Sonnet 5 | 85.7% | 100% | 66.7% | $0.002228 | 2.76 |
| Gemini 3.1 Pro | 81.0% | 100% | 55.6% | $0.004824 | 3.78 |
Read the easy column and Jev is the worst of the four. Read the hard column and it is eleven points clear of the best of them. Those are the same 42 claims and the same four models.
Cost is measured rather than estimated. The frontier figures are what OpenRouter actually charged. Jev's is computed from the tokens it returned at TypeSafe's published rate.
One thing to be straight about, because the marketing is not. The price gap is enormous against frontier models and small against cheap ones. Jev at TypeSafe's published $0.042 per million input tokens sits beside Groq's Llama 3.1 8B at around $0.05, which is a difference of about a fifth, not a factor of fifty. Every multiple in this article is measured against the three frontier models above, because those are what people actually put in a checking step. Against a small open model the argument has to be won on the result, not the invoice.
The inversion is the finding
Change a digit in a sentence and ask whether the source still supports it, and a frontier model gets it right every time. All three were perfect on the constructed controls. Jev was not: it missed two of 24.
Give the same models a real published sentence and they fall over. GPT-5.4 and Sonnet 5 drop to two in three. Gemini drops to just over half. Jev drops least.
The difference between the two sets is not difficulty in any general sense. The hard set is real prose, where the mistake is almost never a flat contradiction. It is a true number pointed at the wrong quantity, a genuine quote assembled from two places, a correct result from a different version of the paper. Nothing in those sentences is false on its face. They are wrong in a way you only see by holding the sentence against the passage and noticing that the two are not about quite the same thing.
That is the class of error that survives every check. It is also, I think, why synthetic benchmarks keep reporting that this problem is solved.
Every model is sure, and one of them isn't
GPT-5.4 put all 42 of its JevBench answers in the top confidence bucket, at a mean stated confidence of 0.98, and was right 85.7% of the time. Claude Sonnet 5 put 39 of 42 there. Gemini put all of them.
Jev's spread across four buckets. Its top bucket, 29 of the 42 JevBench items, was right 93.1% of the time. Everything below that was right 60 to 80%.
This is the part where I should say what I found out afterwards. TypeSafe has not published a calibration result. No expected calibration error, no reliability diagram, no third-party audit of the claim their whole product rests on. Their own worked example of citation checking uses eight citations from a single internet standards document. So a Sunday afternoon and 42 claims is not a small contribution to the public record on this model. It is close to all of it, which says more about how new this is than about the benchmark.
The practical difference: if every answer comes back at 0.98, the number cannot tell you which answers to check. You check all of them or you check none, and if you were going to check all of them you did not need the model.
With Jev you can draw a line. Take the top bucket, send the other thirteen to a reasoning model or to a person. And the line works: both of the easy claims Jev got wrong came back at 0.37 and 0.71, below its own threshold. It did not quietly fail. It said it was unsure, and it was right to be.
What nothing caught
Three of the eighteen hard claims were missed by all four models.
One is the spliced quotation, the phrase in quotation marks that is a heading joined to a sentence from later in the paper. Every model called it supported. The words are all there, in that order, in that document. Nothing was checking whether they were ever one sentence.
The other is "sixteen points of decay", where the arithmetic is right and the thing that decayed is the accuracy of self-generated training labels rather than model capability. Every model called it supported, because the number matches and the sentence reads as though it follows.
TypeSafe's own documentation predicts that second one. Their page on where Jev is unreliable says plainly that it "is not a calculator", that it "recognizes the shape of an answer rather than tallying", and that it "reads dates as text, not as ordered quantities". A claim whose only fault is that a correct number is pointed at the wrong quantity is precisely the shape that slips through. They published that before I ran anything, and it is worth more than most benchmark results.
The third is a sentence of mine that claimed a paper's caveats are "published in the place almost nobody reads". The correct answer is that the source does not address it: a paper reports its limitations, it does not report who reads them. All four models called it unsupported, which is a reasonable-sounding verdict on a question the passage never touches.
Two of those three are errors that actually got published under my name. Nine of the eighteen were solved by every model. Three were solved by none. The interesting work is in the six in between, and Jev won most of those.
What I am doing with it
Jev goes into my gate as a pre-filter, not a replacement. Anything it flags at high confidence gets blocked. Anything below the line gets escalated. It does not pass a sentence on its own authority, because of those three.
By the rule I wrote into JevBench beforehand, it does not replace anything: it caught 78.9% of the bad sentences against the 80% bar, and GPT-5.4 caught 89.5%. It earns its place on precision and on price. When Jev says a sentence is unsupported it is right 88.2% of the time, against 84.2% for Sonnet 5, 81.0% for GPT-5.4 and 76.2% for Gemini, and it costs a fiftieth as much.
The wider thing I take from it is not about any of these products. A model's stated confidence is a separate capability from its accuracy, and the two do not arrive together. I had been treating a confidence score as a free extra that comes with an answer. Three frontier models produced one that carries no information, and a model costing a fiftieth as much produced one I can build on.
Turn the accuracy numbers round and that would still hold. A system that is slightly less accurate but tells you when to stop trusting it is the more useful of the two, because you can route the uncertain cases somewhere else.
What this is not
One run per model, no repeats, so none of these numbers has an error bar. Eighteen items carry the headline, which is enough to show a direction and not enough to put a decimal place on it. The labels come from source audits produced by another model and checked by hand against the primaries. They are not the verdict of an independent panel. One of the eighteen is marked as contested in the data, because a careful person could argue it either way.
Three design flaws I can name, now that I have read the documentation properly. I called the model by its moving alias rather than pinning a version, so a future run may not reproduce this one. My "unsupported" label merges flat contradiction with correct-but-altered, and TypeSafe's own guidance is that Jev answers the question you wrote rather than the one you meant, so a compound category is the wrong shape to hand it. And I asked one Choice question where the thing they actually sell is several questions fanned out over the same text in one pass, billed once. Any of the three could move the result, and I would not call the gap settled until they are fixed.
Gemini 3.1 Pro is a preview build. Latency for all three frontier models is end to end through OpenRouter and includes its routing hop, so it is not the providers' own. The frontier models answer into a JSON schema, which is a design choice worth knowing about: without it they spend the token budget reasoning in prose and often never reach a verdict at all.
Forty-two claims against four models took under six minutes of model time and cost 35 cents, and almost all of both was the frontier models.
Sources
The benchmark: JevBench, run 20 September 2026. Jev jev-1.13.0 via TypeSafe, and openai/gpt-5.4, anthropic/claude-sonnet-5 and google/gemini-3.1-pro-preview via OpenRouter. 42 claims, one run per model. Every number above comes from those runs and the per-claim results are recorded alongside them.
The labelled set is built from source audits of three papers: SkillsBench (arXiv:2602.12670), ReasoningBank (arXiv:2509.25140) and R-Zero (arXiv:2508.05004).
TypeSafe's own documentation, for the pricing and the model identifier: https://docs.typesafe.ai/models