# Jev's probabilities only mean something on your own data Jev, TypeSafe's System One model, answers typed questions with probabilities in about 270 ms for $0.042 per million input tokens, and tied Claude Haiku on Banking77. But its numbers only mean something once measured on your own data: as a Claude Code skill router, a written test set gave 0% false triggers and real prompts gave 22%, and the right guardrail threshold was 0.68 for one dataset and 0.09 for another. Tested against jev-1.13.0 in September 2026. Published: 2026-09-20 Canonical: https://umar.codes/testing-jev-system-one ## Revisions - 2026-09-20 — Created. - 2026-09-20 — Published the same day, outside the calendar, after a fact-check against the lab notes: narrowed the Claude Haiku tie to Banking77, added the caveats the notes mark as required, and made the headings readable alone. --- Jev, TypeSafe's "System One" model, is not a chat model. It answers typed questions with probabilities in about 270 ms, for $0.042 per million input tokens, and tied Claude Haiku on one public benchmark. But as a skill router, a written test set showed 0% false triggers and my real prompts showed 22%. Measure its numbers on your data. I tested `jev-1.13.0` on 19 and 20 September 2026. I directed the experiments; Claude Code wrote and ran the code, in one long session. All Jev calls across the twelve experiments cost about $0.86. I have no relationship with TypeSafe. The version is pinned and dated because an alias like `jev-latest` can move. ## Jev answers typed questions, not prompts A request has a `state`, which is the text or JSON the model reads, and a set of named questions. A Choice picks one option from a list and returns a probability for each. A Score rates the state on an ordered rubric. A Noul asks "is this statement true?" and returns one probability from 0 to 1. No text is generated, so there is nothing to parse. A trimmed example: ```ts const { answers } = await client.systemOne({ state: "I was charged twice for March. Please refund the second charge.", questions: { team: choice("Which team should handle this support ticket?", { billing: "Charges, refunds, invoices, tax details, or subscription payments", technical: "Bugs, errors, failed integrations, crashes, or login problems", }), wants_refund: noul("The customer asks for a refund or for money back"), }, }); answers.team.choice; // "billing" answers.wants_refund.noul; // 0.99 ``` ## Fifty questions take the same 270 ms as one On one state of about 300 words, ten sequential runs per size, the median latency was 269 ms with 1 Noul and 273 ms with 50, network round trip included. Input tokens went from 612 to 1,319. So I can ask every question that *might* be useful and let code decide which answers to use. ## Jev tied Claude Haiku on Banking77 at 1/18 to 1/64 of the cost Self-written data can flatter a model, so the main experiments used public datasets. As a re-ranker on BEIR FiQA, scoring the BM25 top 100 in one call, Jev raised nDCG@10 from 0.248 to 0.399 at $1.22 per 1,000 queries. That is a sample of 150 of the 648 test queries, and FiQA's labels are sparse. On Banking77, 770 real messages across 77 intents with no training, one Choice question scored 80.1% (95% interval 77.2 to 82.8) from label names alone and 87.7% (85.2 to 89.8) with three training examples per label. As a guardrail over 1,968 public jailbreak and injection messages it separated attacks from benign text with an AUROC of 0.97 to 0.99. I sent 154 of the Banking77 messages, two per intent, to Claude Haiku 4.5 with the same labels and examples. With examples, Jev scored 85.1% (78.6 to 89.8) and Haiku 87.0% (80.8 to 91.4). Jev's 85.1% is below its 87.7% above only because this is a smaller sample of the same data. Only Jev was right on 5 messages and only Haiku on 8: a tie at this sample size. Jev's median latency was 282 to 302 ms against 2,916 to 3,803 ms, about 10x faster, and it was 18x to 64x cheaper: $0.213 against $3.84 per 1,000 messages with examples, $0.043 against $2.75 without. Three caveats sit on the speed and cost. Haiku ran through `claude -p`, so its latency is the API time the CLI reports and includes some harness overhead; a direct API call could be faster. Haiku used thinking, a mean of 354 to 480 output tokens for a one-word answer, which is much of its time and cost; I did not test it with thinking off. And its cost is the API-equivalent price, not money I paid on a subscription. Jev's probabilities ran a little high: with examples, a stated 0.99 was right about 94% of the time. As a gate they still worked. Acting only at 0.9 or above handled 78.8% of messages at 94.6% accuracy. ## A written 60-prompt test set gave the Jev skill router zero false triggers The main experiment was a skill router for Claude Code. My setup has 127 installed skills, and the main model picks between them from their descriptions on every prompt. A `UserPromptSubmit` hook made one Jev call per prompt instead, with one Noul per skill, each carrying that skill's description. Code kept skills at 0.7 or above, three at most, and added them to the context as a suggestion. The hook sends every prompt to TypeSafe's API, so it does not belong where prompts can hold secrets. The test set had 60 labelled prompts: 20 that reuse words from a skill description, 28 that describe the need in other words, and 12 that need no skill. On the 28 hard ones Jev suggested a correct skill for 96.4% (27 of 28), Haiku for 67.9% (19 of 28) and BM25 keyword search for 35.7% (10 of 28). On the 12 that should stay silent, Jev at 0.7 and Haiku both made 0 false triggers; keyword search made 9. Haiku's prompt told it to reply with an empty list when no skill applied, and a different prompt could raise its hit rate. The weakness was in the notes at the time: the prompts, the labels and the 0.7 threshold all came from the same session that built the router. My earlier [skill trigger suite](/testing-skill-triggers) used written prompts too. A written test set tests the cases its author thought of. ## On real prompts the Jev skill router gave 22% false triggers Prompts that predate the router were already on disk: my Claude Code history, 12,572 prompts from 77 projects. A fixed sample of at most 20 per project, 160 prompts from 38 projects, was redacted, first by rules and then by Haiku, and I read the redacted text before any of it went to TypeSafe. 156 survived. Claude Sonnet labelled each one, with an instruction to be strict. The first finding came before the router ran: 84% of real prompts need no skill, 131 of 156. They are mostly short replies inside a conversation. The written set had 20%. Against that distribution the same 0.7 threshold suggested a correct skill for 88% of the prompts that needed one (22 of 25; interval 70 to 96) and fired on 22.1% of those that needed none (29 of 131; 16 to 30). | Real prompt (redacted) | Jev suggested | | --- | --- | | "env variables have been added to vercel" | `vercel:env-vars` 0.77 | | "able to sign up and login" | `vercel:auth` 0.91 | | "yes, commit and push" | `superpowers:verification-before-completion` 0.73 | | "merged [url]" | `pdf` 0.70 | Some false triggers were reasonable suggestions that a strict label rejected. Most had one cause: Jev reacts to a message's topic, not its intent. A status report about environment variables shares a topic with the env-vars skill, and no question had asked whether the message was a request. The caveats are real. An LLM made the labels, only 25 prompts need a skill, every prompt is mine, the true rate is probably below 22%, and the router saw each prompt without the conversation around it. ## A literal "is this a request?" gate blocks bug reports Fan-out makes the fix cheap: ask for the missing judgement in the same call. The first gate was one Noul, "The user message asks the assistant to do a new piece of work." It failed. "PORT 5432 is already in use" scored 0.04 and "blank page when it visit localhost:5173" scored 0.20. A bug report asks for nothing, read literally, and 12 of the 25 prompts that need a skill were of that type. The working gate adds a second Noul, "The user message reports an error, a failure, or behavior that is wrong", and code takes the maximum of the two. I split the prompts in half by hash, chose the gate threshold of 0.8 on one half and measured on the other: 76 prompts, 15 needing a skill. Without the gate, 93.3% correct (14 of 15) and 24.6% false triggers (15 of 61). With it, 86.7% (13 of 15) and 6.6% (4 of 61); the drop in hits is one prompt. The gate has a cost: on the written set it blocks 7 of the 28 hard prompts, all requests for advice. I stopped there, because each question tuned on the same prompts makes the result less honest. I also never measured whether the suggestion improves what Claude Code then does. ## The right Jev threshold was 0.68 for one dataset and 0.09 for another Two guardrail questions and a 0.5 threshold were fixed before I saw the public data. On 1,306 jailbreak messages that held: the jailbreak question alone found 89.5% of attacks (86.9 to 91.6) with 0.9% false alarms. On 662 injection messages the AUROC was 0.992, so nearly every attack ranked above nearly every benign message, yet the two questions at 0.5 found only 56.7%. Jev often gave those attacks probabilities of 0.1 to 0.4. I then chose a threshold on each train split, the highest recall with at most 2% false alarms, using three questions; the third was written after the first run. That gave 0.68 for jailbreaks and 0.09 for injections. On the test splits it found 92.8% of attacks (87.3 to 96.0) in 262 messages and 90.0% (79.9 to 95.3) in 116, with false alarms of 1.6% (0.4 to 5.7) and 1.8% (0.3 to 9.4). A Jev probability ranks well inside one task. It does not carry its meaning to the next. These are also old, public attack texts, and TypeSafe lists adversarial content as a weak area. ## Jev cannot count or add, and small tests hide it Asked whether eight amounts summed to a stated total, Jev said yes at 0.94 when the total was correct and yes at 0.92 when it was wrong by 10.00. It counted 24 of the letter "e" where there were 26, and 10 animals in a list of 30 that held 11. It passed the small versions: three amounts with an obvious digit swap, and both basic counting probes. Two probes per area is a signal, not a benchmark. Counting and arithmetic stay in code. ## Four mistakes I made testing Jev The first 14 limit probes were too easy. Jev passed 14 of 14, including areas TypeSafe lists as weak. The first re-ranker put all 30 passages in the state and pointed each question at one by index. The pilot metric looked fine. Printing the raw scores for one query showed the last 12 all at 0.7, as if the model had stopped reading. Score spread across positions 16 to 30 was 0.27 against 0.56 for the first fifteen; putting each passage inside its own question raised it to 0.49. The nDCG@10 difference, 0.333 against 0.343, could be noise at 150 queries. The defect showed in the raw scores before it showed in the metric. The router's test set was too kind, as above. And I held two explanations for the weak injection result. One was that my questions were too literal; the third question moved 56.7% to 64.3%, so partly. The other was that the labels were at fault, because many "injections" in that dataset are attacks only for the one chatbot they were sent to, whose task the classifier never sees. Then Haiku ran on a 150-message sample and found 83.7% of its attacks (41 of 49) where Jev at 0.5 found 57.1% (28 of 49), so that was only partly right too. Baselines earn their cost, as the [same-day control run](/cross-model-skill-regression) did in my skill regression test. ## Three rules for building on Jev's probabilities Get real inputs before trusting a threshold: the router gave 0% on written prompts and 22% on real ones. Measure the threshold per task and pin the model version, because 0.68 and 0.09 came from the same three questions on `jev-1.13.0`. And ask for the speech act as well as the topic, since Jev answers the question as written, literally. Within those limits it is a cheap first filter in front of an LLM, not a replacement for one.