Decision models head to head: Jev, clef-flash, and Qwen

A System One model against System Two LLMs: faster, cheaper and better overall

All eleven models on 1,514 emails. Higher is more accurate and further left is faster, on a log scale. Orange points ran in the cloud, blue points on my laptop, and diamonds are decision models.

TL;DR: I compared two decision models, TypeSafe’s Jev and Cloudflare’s clef-flash, with general-purpose LLMs on 1,514 synthetic emails. Jev answered in a median of 138 ms, against 493 ms to 1.97 s for the cloud LLMs, while Claude Sonnet 5.5 and Haiku 5.5 were a little more accurate out of the box. With clearer category definitions, Jev and Haiku tied at 99.3% on held-out emails. In this sample, Jev’s confidence score flagged all of its mistakes for review.

I ran the local models on a base-model MacBook Pro (M5, 16 GB) and the cloud models through their providers’ APIs, for about $3.47 in API costs.

System One and System Two

Daniel Kahneman’s Thinking, Fast and Slow (2011) describes two modes of thinking in our heads, using labels from the psychologists Keith Stanovich and Richard West. System 1 is fast, automatic, and effortless: it hears hostility in a voice or completes “bread and” before you decide to. System 2 is slow and effortful: it multiplies 17 by 24, fills in a tax form, and checks System 1’s quick answers when they look doubtful.

TypeSafe took these names from Kahneman’s book and calls Jev a System One model. Jev returns a choice from a fixed set of options, with a probability for each option and a confidence score, and writes no text. Cloudflare’s clef-flash works the same way. I call these decision models, the System One side.

The other models in my test, Claude, Gemini, Gemma, Qwen, and Llama, are the System Two side. They generate text one token at a time and can reason step by step before answering. Even with reasoning switched off, they answer in text, so to classify an email you ask for a category name and parse the reply. I call these general-purpose LLMs, the term Cloudflare uses in its own benchmark.

I can’t frame this as decision models against transformer models, because decision models can run on transformers too. clef-flash is a Qwen3.5 language model underneath: Cloudflare runs it once over the input, a prefill-only pass that generates no text, and then scores the valid choices in parallel. TypeSafe hasn’t published Jev’s architecture, but its launch post says Jev outputs all probabilities in parallel instead of generating token by token. The difference is in how the answer comes out. A general-purpose LLM predicts one token, appends it, and repeats until the answer is written. A decision model reads the input once and scores every allowed answer at the same time.

Malcolm Gladwell’s name for a System One judgment in people is thin-slicing, from Blink (2005), after Nalini Ambady and Robert Rosenthal’s research on “thin slices” of behavior: a judgment made from a narrow slice of information. In Blink vs Think: the first twenty seconds I wrote that thin-slicing is only as good as the patterns behind it. Sorting an email into one of nine folders is a thin-slicing task, and I wanted to know whether a System One model does it faster or more accurately than a System Two model.

My test

I chose email because one message can mix purposes: a friend’s update might mention a job opening. Each model had to pick the main purpose from nine categories: marketing promotions, transactional orders, newsletters and digests, financial billing, travel and hospitality, career and professional, community and civic, personal and social, and other or unclear.

Gemini read a sample of a real inbox to help design the categories, then wrote synthetic emails for each one; only synthetic emails reached the models. After removing near-duplicates, 1,514 remained, 353 of them deliberately ambiguous, all in English and averaging about 47 words. Each email’s ground truth is the category Gemini wrote it for. I didn’t relabel them by hand, so accuracy in this post means agreement with Gemini’s intended category.

I tested five cloud models (Claude Sonnet 5.5, Claude Haiku 5.5, Gemini 3.8 Flash, Gemini 3.5 Flash-Lite, and Jev 1.13) and six local ones: clef-flash and versions of Gemma, Qwen, and Llama, all as compressed 4-bit builds. For each I measured accuracy, median response time including network time, and API cost. I turned off hidden reasoning wherever the provider allowed it; Gemini 3.8 Flash kept it on.

Jev partway through a 100-email live run in my test app. Each bar fills as its category collects emails; red marks a wrong answer.

Jev was the fastest model

On all 1,514 emails, Sonnet scored 98.75%, Haiku 98.28%, Gemini 3.8 Flash 98.08%, Flash-Lite 97.16% and Jev 95.77%. Jev’s median response time was 138 ms, against 493 ms for Flash-Lite, the quickest cloud LLM, 808 ms for Haiku and 1,974 ms for Gemini 3.8 Flash.

All models on 1,514 emails. Sonnet’s 949 ms comes from an earlier 329-email interactive run, because its full run used the Batch API. The tuned Jev row is optimistic: I wrote those definitions after studying errors on these same emails.

I wouldn’t rank the top models by those small gaps. My rough uncertainty was plus or minus 0.9 percentage points on the full set, and timings varied too: Gemini 3.8 Flash ranged from about 1.7 to 5 seconds per email on different days.

Decision models weren’t automatically the fastest. On my laptop, clef-flash scored 95.7% at a median of 609 ms, while Gemma 4 E4B answered in 390 ms at 94.9%. clef-flash, the largest local model, also ran my 16 GB laptop out of memory and crashed it twice.

Clearer definitions raised accuracy

Most of Jev’s mistakes came from the ambiguous emails: it scored 99.31% on clear-cut messages and 84.14% on ambiguous ones, often misfiling personal notes and system reports. An ambiguous email is where thin-slicing is weakest, because the message mixes two purposes.

I rewrote the category definitions around the sender’s main purpose, without retraining any model. A personal note stays personal when it mentions a job, and a system report stays in other or unclear when it mentions a charge.

On 448 held-out test emails, Jev rose from 95.3% to 99.3% and Haiku from 98.9% to 99.3%. Median times were 167 ms for Jev and 899 ms for Haiku, and Jev stayed cheaper: $0.05 per 1,000 emails against Haiku’s $0.06 with the shorter original definitions.

By better I mean the whole package. On accuracy alone Jev tied Haiku at 99.3%, and gaps under about 1.6 points on 448 emails are noise. Jev’s edge was speed, cost, and the confidence score. I didn’t run Sonnet or Gemini 3.8 Flash with the tuned definitions.

The same 448 held-out emails, before and after the rewrite. Clearer definitions helped seven of nine models. I wrote them after studying errors across the full set, so these scores are optimistic until they hold on fresh emails.

Longer instructions hurt some models: Qwen2.5-3B fell from 73.7% to 63.2%, so I'd retest each wording change on each model.

Confidence scores showed which answers to check

Jev returns a confidence with every answer, so answers below a threshold can go to human review while the rest go through. Coverage is the share handled without human review.

Get Kundansen’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

With the original definitions and a 0.95 threshold, Jev handled 88.4% of the held-out emails automatically with no errors among them, and all 21 of its mistakes went to review. With the tuned definitions, a 0.90 threshold gave 94.4% coverage, and all three remaining mistakes went to review.

At each threshold: the share handled automatically, its accuracy, and the share sent to review with the number of Jev’s mistakes in it.

My LLM calls returned no comparable score. Other ways to estimate an LLM’s confidence exist, and I didn’t test them; Jev’s built-in score made routing a few lines of code.

This is Kahneman’s handoff: the fast system answers, and the slow system takes over when the answer is in doubt. Here the slow system would be a reviewer reading the flagged emails.

Cost barely mattered

Per 1,000 short emails at October 2026 list prices: $0.03 for Jev, $0.06 for Haiku, $0.10 for Flash-Lite, $0.66 for Gemini 3.8 Flash and $1.14 for Sonnet.

Costs come from recorded tokens and list prices, and exclude infrastructure, engineering, and review time. Some Google and TypeSafe prices came from third-party trackers.

At these prices I’d choose between Jev and Haiku on response time and confidence routing. Real mail is longer and more varied, so I’d expect higher costs, slower answers and lower accuracy, though I haven’t measured by how much.

One request and response

I called Jev through the TypeSafe SDK, sending the email as state and one multiple-choice question with the nine category definitions as the allowed answers:

from typesafe_sdk import TypeSafeClient, Choice
client = TypeSafeClient(model="jev-1.13.0")
result = client.system_one(
    state={"email": email_text},
    questions={
        "category": Choice(
            instructions="Which category best describes this email in a personal inbox?",
            criteria=CATEGORIES,
        )
    },
)

I sent it a maintenance report that mentioned a subscription renewal. With the original definitions, Jev chose financial_billing; I expected other_unclear:

{
  "answers": {
    "category": {
      "type": "choice",
      "choice": "financial_billing",
      "confidence": 0.63,
      "probabilities": {
        "marketing_promotions": 0.0,
        "transactional_orders": 0.0,
        "newsletters_digests": 0.0,
        "financial_billing": 0.67,
        "travel_hospitality": 0.0,
        "career_professional": 0.0,
        "community_civic": 0.0,
        "personal_social": 0.0,
        "other_unclear": 0.33
      }
    }
  }
}

The app reads the choice and scores directly, with no text to parse. This call returned a confidence of 0.63 and the scored run 0.58 for the same email; both would go to review at 0.95.

A different wrong answer, from a small run in the test app. Jev chose financial billing with 44% confidence; the correct travel category, in blue, received 40%.

Jev returns no written explanation, so a workflow that needs one would have to add and test it separately.

What didn’t work

Packing several emails into one request hurt accuracy for some models: clef-flash fell from 94% to 54%, though each throughput test used only 50 emails. Sending single requests in parallel raised cloud throughput without making any one request faster.

System Two reasoning didn’t help this System One task. Gemini 3.8 Flash, the one model with reasoning on, was the slowest and no more accurate. An early run of it appeared to score 32% because hidden reasoning used up the 16-token answer limit; raising the limit fixed it. Claude needed its own handling for response blocks and reasoning settings, and I rejected runs with more than 5% failed calls, which caught two broken ones.

Small LLMs scored lower: my smallest Qwen model reached 74.4% on the full set. Fine-tuning Qwen3-4B on my laptop raised it to 98.9% on held-out emails, and 97.5% on the 121 that shared little wording with the training data. I didn't test fine-tuning in the cloud.

The two clef-flash crashes came from my test process loading models one after another without fully releasing them, while the web server held a 2.3 GB model. Running each model in its own process, with a free-memory check first, fixed it.

Spend

The experiment took about 25 hours over two days and $3.32 of recorded API spend, plus about $0.15 on discarded runs, not counting my time, hardware, or development assistants.

Calling models took little effort. Most of the work went into the dataset and fair test code: generated batches repeated themselves, fake amounts could give away labels, privacy checks needed manual review, and quotas, response parsing and wrong prices all needed fixes.

What I’d test next

Next I’d test Jev against Claude Haiku 5.5 on fresh, hand-labeled emails, with long threads, ambiguous cases and more than one language. I’d set the test set aside before writing any definitions, choose wording and thresholds on separate validation data, then measure accuracy, coverage, review workload and timing in the same time windows.

Jev’s strengths were speed and a confidence score that made review easy to add, and clef-flash showed promising accuracy on a laptop. Would either advantage hold on real mail, and could a general-purpose LLM take the reviewer’s place for the emails a decision model isn’t sure about?

Further reading: Daniel Kahneman’s Thinking, Fast and Slow (2011) sets out System 1 and System 2, the split TypeSafe borrows. Nalini Ambady and Robert Rosenthal’s 1992 meta-analysis in Psychological Bulletin found that judgments from half a minute of behavior predicted outcomes about as well as judgments from four or five minutes. Malcolm Gladwell’s Blink (2005) turned that research into thin-slicing, and my post Blink vs Think looks at when a fast judgment earns trust. Diogo Almeida’s Introducing System One Models & Jev (TypeSafe, 2026) compares existing LLMs with System One models in one table. Michelle Chen’s Introducing Clef (Cloudflare, 2026) explains how Clef and clef-flash score choices on a frozen Qwen backbone.

I built the test app, scoring code and reports with Claude Code. Claude and Gemini worked directly in tandem to cross-check each other’s work, break down the tasks and share them out: Gemini designed the categories and wrote the synthetic emails, Claude checked the batches and built and ran the tests, and Gemini reviewed the code and the write-up.