Jev, a cheap AI, did two thirds of my filing: same result as Claude, 63% cheaper
Jamie Watters
Operational resilience and AI delivery practitioner.
Who this is for: you pay for AI to sort something over and over (emails, tickets, notes, documents) and you'd like the bill smaller without the results getting worse. You'll come away with what that saved on my notes, the catch, and a test to run on your own work.
Skip it if you've never paid for an AI model by the call. Most of this won't matter to you yet.
On the length: the result and the catch are in the first three paragraphs. The rest is how sure you can be of them, what didn't help, and the test.
A cheap AI model can do most of a sorting job and hand the hard cases to an expensive one. I tested that on 75 of my own notes. Jev, a model TypeSafe released on 15 September, filed each note into a folder. Where it was confident, its answer stood. Where it wasn't, the note went to Claude Sonnet 5, which costs about 76 times as much per note. Jev handled about two thirds itself.
The result was the same as giving every note to Claude: the same 65 of the 75 filed where I'd filed them, taking each model's most common answer over five runs. The model bill was 63% lower. (Every figure here is from my Test 1c results, listed under Sources.)
The catch is what the hand-off doesn't buy: a check. On the six notes where Jev was confident and wrong, Claude's usual answer was the same wrong folder on all six, so sending them up couldn't have fixed them. And Claude's own confident answers miss the 5% limit I set, so "as good as Claude" is not "safe to run unchecked".
Three tests, one question each
My notes live in a vault, filed into seven folders. Five are projects, paid work, article drafts, things I've read and reference. The other two are my book and Efformism, a philosophy framework I'm developing. (The tests also offered an eighth answer, delete.) Right now a Claude session proposes where each note goes and I say yes or no. Jev looked made for the job: you give it text and a fixed set of answers, and it returns one answer and a confidence score. Claude sessions built and ran the tests; I approved each set of rules before it ran.
Test 1, on 23 September, gave Jev and Claude each note and a short description of each folder, and Jev lost, 65.1% to 77.1%.
Test 1b, on 24 September, added ten examples to every note. They were the most similar notes already filed, found by a keyword search, each shown with its folder. Jev rose to 83.1%. But the search on its own, taking the folder most of the ten were in, got 74.7%. Jev was ahead by 14 notes to 7, a lead that 83 notes couldn't confirm (p = 0.19).
Test 1c asked whether Jev's hand-off works. It scored 75 notes that no earlier test had scored, and the rules were written down before any of them were run. One caveat belongs here rather than at the end: 54 of those 75 had already appeared as examples, with their folders, in earlier runs, including the rewording runs described below. Each call sent one message holding that note and its ten examples, and nothing from any earlier call, so that doesn't affect the Jev or Claude results. It does weaken the rewording result.
The hand-off
Jev answers every note. If its confidence is 0.80 or more, its answer stands; below that, Claude's answer is used. Jev's confidence is not the probability of its top answer. It is a number Jev derives from the shape of its whole probability spread, and here it was a little lower on 289 of 375 answers. So 22 answers with a top probability of 0.80 or more were still passed on. My first test's rules had named the top probability, so the line as run passed on slightly more than the line as written. Claude was asked to state its own confidence. Claude answered every note so the two could be compared, and each Jev run was paired with one Claude run. In use Claude would only see the notes Jev passes on, so the cost is Jev on every note plus Claude on those.
| 75 notes | Right, most common answer over 5 runs | Share of answers passed on | Model charges per note |
|---|---|---|---|
| Keyword search only | 76.0% (1 run) | n/a | $0 |
| Jev | 85.3% | n/a | $0.000119 |
| Jev, passing unsure notes to Claude | 86.7% | 32.5% | $0.00340 |
| Claude alone | 86.7% | n/a | $0.00909 |
Jev alone got 64 notes right. Passing its unsure notes to Claude fixed 3 and broke 2 that Jev had right: 65. On the answers Jev kept, it gave the same folder as Claude's paired answer 251 times in 253. The two exceptions were one note, which Claude got right in two runs. Keeping is decided per answer, not per note: 47 notes were kept in every run and 20 in none. The other 8 sat on the line and were kept in some runs, and both models filed all 8 right every time. On the answers Jev passed on, the pair's answer simply was Claude's. So the two ended on the same 65, and in no single run did the hand-off beat Claude.
That match is the finding. The rule I approved was that a standard statistical test must not show the hand-off worse than Claude, at no more than half the cost. It passed. But on 75 notes that rule could not have failed by much, because I didn't set how much worse was acceptable. With every disagreement going Claude's way, a hand-off 7 notes worse would still have passed it. With disagreements both ways, up to 15 notes worse would have passed. The evidence is the observed result, not the rule. With no disagreements in 75, the true disagreement rate could still be up to about 4% (a one-sided 95% bound, worked out afterwards). The 20 notes passed on in every run couldn't have disagreed, so that bound rests on the other 55. Averaged over the five runs rather than by most common answer, the pair got 85.9% and Claude 86.4%. Claude also cost more on the notes passed to it, $0.0101 each against $0.0091 across all 75. The hard notes are the dear ones, so a job that passes more on saves less.
Confident isn't the same as right
Of the answers Jev kept, 30 of 253 were wrong: 11.9%, against a 5% limit I set before the test. On Test 1b's notes it had been 5.5%. Jev never changed its folder for a note between runs, so the 30 are six notes, each wrong in every run. For comparison only, since the rule applied to Jev, Claude's confident answers were wrong 14 times in 259, 5.4%. TypeSafe says Jev is trained to return calibrated confidence. On the answers it kept here, its average confidence was 0.94 and it was right 88% of the time. Six notes can't settle whether that gap is the model or my labels.
Those six notes decide whether Jev can file on its own. Two are my career fact sheet and my contacts register, filed under paid work; both models said reference. Two are content ideas: a review of a source, which both put with things I've read, and an article brief, which both put under projects. The Claude session that wrote up the results thinks several look like my filing habits rather than model mistakes. That's my call, and I haven't made it. If four of the six turn out to be labels I'd change, Jev's rate falls to 4.0% and passes. Until then no labels change, because relabelling after a model disagrees is exactly the temptation the rule exists to stop.
So by the rule I set, Jev doesn't file on its own, even with me as the backstop. It would act on two thirds of its answers. Of all its answers, 8.0% would have been acted on wrongly (Test 1c verdict).
What the test couldn't separate
Narrower questions. TypeSafe recommends breaking a broad question into narrow ones and combining the answers in code. So one setup asked Jev eight yes-or-no questions in one call, one per answer. The rule for combining them, set in advance: the highest yes wins, but a note is passed on unless the top yes is at least 0.80 and no other reaches 0.50. It filed 85.3%, the same as the single question, one note better and one worse. It passed on 57.1% of its answers, almost all because no yes reached 0.80.
Rewording. Jev isn't fine-tuned on customer data; you shape it through what you send with each request. So the Claude session running the tests reworded Jev's folder descriptions from my filing rules, working only on Test 1b's 83 older notes. There Jev went from 69 right to 74. On the 75 test notes it went from 64 to 66, p = 0.5. An independent check found that some of the rewording went beyond my written rules and some was prompted by particular older notes, and the 54 overlapping notes were among the examples in the rewording runs. So I don't lean on that result.
Jev against the search. Jev got 64 notes right and the search 57. Jev was right on 13 that the search got wrong, and wrong on 6 that it got right (p = 0.167). That's a lead the test can't confirm, not a tie. The search gave no confidence, so it couldn't pass anything on; a search that hands off when its vote is close wasn't tested.
What this test can't show
Most of my notes were filed by a Claude session and accepted by me, so the labels may lean Claude's way, which makes agreement with Claude partly built in. 54 of the 75 test notes had appeared as examples in earlier runs, including the rewording runs. None of them was scored until the test itself, but they weren't new to it either. During the test, the examples were drawn from every filed note, so test notes served as examples for each other (109 of the 750 example slots, never a note for itself). That is how it would work in use, where every filed note is a candidate example, but a stricter test would keep test notes out of each other's examples. The sample is balanced across folders on purpose, so it isn't the mix my real inbox gets. The test notes include none from my book or Efformism folders, because Test 1 used all four of each. The hand-off was to Claude without a reasoning step. And the Test 1c runs took six minutes on one day, so they say nothing about whether either model behaves the same next month.
Set it up this way
BEFORE YOU LET AN AI FILE THINGS FOR YOU
1. Pick filed items to tune on and a separate set to
test on. I used 83 and 75, up to 15 per folder,
drawn with a fixed seed. Keep the test items out of
the tuning runs, including as examples. I didn't
manage that: 54 of my 75 had been examples in the
tuning runs. Count items, not runs: to show a 5%
limit is met with no errors you need about 59 kept
items.
2. Strip anything added at filing time (tags, labels).
3. For each item, find its 10 most similar filed items
from everything filed, never the item itself. I used
BM25 (k1 1.5, b 0.75) over title plus first 1,500
characters, and showed each example as title, first
500 characters, folder.
4. Give the model the item, a short description of
each folder, and the 10 examples. Get a folder and
a confidence. Jev returns its own; I asked Claude to
state one. Pin both model versions. Run it five
times; the answer is the folder given in three or
more runs, else it counts as wrong. My prompts and
folder descriptions are in the public code (Sources).
5. Before any test run, freeze the question wording
and write down three numbers: the confidence line
(I used 0.80), the most wrong answers you'll accept
above it (I used 5%), and how much worse than the
strong model the hand-off may be. I didn't set the
third, and should have.
6. At or above the line the answer stands. Below it,
pass the item to the strong model or to you.
Cost = cheap model on every item
+ strong model on the items passed on.
7. Score once, on the test set: the hand-off against
the strong model alone, and the cheap model alone
against the search's vote. I used an exact McNemar
test on the majority answers, which only counts the
items the two got differently.
8. Count wrong answers at or above the line. Over your
limit? Someone still checks those. If a label is
wrong, fix it for next time; don't fix it to pass.
For now Jev doesn't file my notes on its own. Passing its unsure third to Claude gave the same result as Claude alone for 63% less. It didn't give me a check, because Claude makes the same confident mistakes.
Sources
The notes are private, so the raw material lives in my vault at tools/jev-stack-tests/. The rules
for each test were written before its runs (test1-preregistration.md, test1b-preregistration.md,
test1c-preregistration.md). Every Test 1c figure is in results/test1c-summary.json, which
analyse1c.py regenerates from the raw runs, or in the verdict results/2026-09-24-test1c-verdict.md,
with its deviations and the two independent checks. Test 1b: results/2026-09-24-test1b-verdict.md.
Test 1: results/2026-09-23-test1-verdict.md. The code, the prompts, the folder descriptions and
the pre-registrations are public in the jev-bench filing
folder. So are the per-note results,
with every note's text, title and path removed, and reproduce.py regenerates every summary from
them. Jev jev-1.13.0 via TypeSafe; anthropic/claude-sonnet-5 via OpenRouter. Model charges:
Jev from TypeSafe's published input price, $0.042 per million tokens with output free (TypeSafe's
model docs, read 20 and 24 September 2026); Claude from OpenRouter's
reported charge. Retrieval ran locally and isn't costed.
Why examples: in the public BANKING77 experiment, a single run, Jev scored 92.4% on 77 kinds of banking request with 24 retrieved labelled examples per message. Its README, which it says GPT-6 Astra Light wrote, adds: "we did not measure a retrieval-only classifier, so we cannot quantify Jev's incremental contribution over that alternative."
The earlier pieces this builds on: Test an AI model before you rely on it, the six mistakes and the checklist these tests follow; and I said Jev beat GPT-5.4. Five reruns can't tell them apart, the fact-checking test where Jev's confident wrong answers first turned up.