Testing Clef Decision Models as a Content Filter

What Clef Is

Clef-Flash is post-trained from Qwen3.5-9B, and Clef is the 27B version. Both are Apache 2.0 on Hugging Face and on Cloudflare's Workers AI, at $0.09 and $0.24 per million input tokens. A small "joint schema head" on top of the backbone scores every option of every question in one forward pass, so there's no text to parse. A request looks like this:

{
  "state": {"question": "Will there be 500+ measles cases this year?"},
  "questions": {
    "human_toll": {
      "type": "noul",
      "instructions": "Is it about a count of cases, deaths or disaster victims?"
    }
  }
}

The answer is a probability for human_toll. Questions can also be choice (pick one of several options) or score (a point on a scale).

The Engines

Everything ran on one RTX 5080 Laptop GPU with 16 GB of VRAM. Claude listed the runners that could serve Clef-Flash, and I picked which ones to benchmark.

PyTorch won because its linear-attention kernels run Qwen3.5's gated delta-net layers faster than llama.cpp does. That needed the causal-conv1d kernel built from source for the 5080 alone. The default nine-architecture build got killed without an error, most likely out of memory. Without the kernel, PyTorch drops to 3.42 requests a second.

Batching didn't help

Batching made PyTorch slower: 3.70 requests a second at batch 1, then 3.16, 2.85 and 2.15 at batches of 4, 8 and 16. A prompt of about 1,000 tokens already keeps this GPU busy. It also changed the answers, because bitsandbytes' 8-bit mode handles outliers across the whole batch (4-bit doesn't, which confirmed the cause). So the server runs one request at a time. llama.cpp and Ollama don't batch Clef requests at all.

The 27B model

I also ran the 27B as a 3-bit GGUF on 22 test questions. It was better at decoys: all nine scored 0.73 or higher on "is the harm word only part of a name?", where Flash let several through. But it missed one positive Flash caught, ran 3.2× slower (1,202 ms per request against 378) and needed 13.9 GB of VRAM. Claude's recommendation was to use it as a second opinion on uncertain cases, not as the main filter.

Cost is just tokens

Across 17,194 timed requests, time grew in a straight line with prompt length: ms = 33 + 0.31 × input tokens, with an R² of 0.993. Sending fewer tokens is the main lever. Request count matters much less, though the 33 ms per request still adds up.

One Big Question Didn't Work

The first pilot asked one holistic question, "would a reasonable person find this insensitive?", plus a harm cross-check, a category and a severity score. It flagged 1 question out of 500, and that one was arguably wrong. Half the sample came from a keyword search for words like "death" and "war", but almost every hit was a decoy: esports kill counts, video game deaths, film titles, a team called the Earthquakes. The regex also lacked a closing word boundary, so "war" matched Warsaw.

Claude's fix was to ask for facts instead of a judgment. Narrow yes/no questions like "does it hinge on a named real person's death or illness?" or "on a count of cases, deaths or disaster victims?" do the asking, and code decides which facts count. On 22 hand-picked questions, all ten decoys scored 0.20 or lower, against 0.03–0.45 with the holistic question, and prompts were about 14% shorter. Two more lessons held for the rest of the project:

• Write the boundary cases into the question. Clef reads literally. A line saying that harm words inside a name, a title, a video game or fiction don't count did most of the work against decoys.
• Questions asked outside their premise give nonsense. "If a named person is harmed, is the harm their death?" scored 0.95 on a Minecraft player dying to a spider. Only read answers whose premise applies.

Building Something to Measure Against

Twenty-two examples couldn't tell strategies apart, so Claude suggested an eval set. I set the policy's boundaries (what a reasonable person would find in poor taste, with political questions tagged and filtered too), and Claude turned them into written criteria. The set has 565 rows. Each category (harm to a named person, human toll, violence, self-harm and sexual abuse, animals, minors, tragedy-adjacent, political) gets positives plus an equal number of controls, lookalikes first. A lookalike shares the words or topic but not the meaning: a junior football club next to a junior tennis tournament, or a storm count next to a storm death toll.

Real positives are rare in this data, so 47 came from targeted searches and 30 rows were invented for categories the data lacks, like animal fighting. The numbers in this post use real rows only: 162 positives and 212 controls. Every request was cached, so re-scoring never cost another request.

How to read the numbers

Clef returns a probability for each question. The plan was three bands: 0.7 or higher gets flagged automatically, 0.3 to 0.7 goes to a person for review, and below 0.3 is cleared.

Ways of Asking

The question bank is a tree: 10 yes/no root questions (person harmed, natural disaster, violence, self-harm or sexual abuse, animals, minors, tragedy-adjacent, a catch-all, political, and a "name only" decoy check), with 29 questions in all. The experiment was how to walk it. I proposed the first approaches (one filter question, a tree, a breadth-first walk) and later the chain and context variants. Claude formalised them, added its own, and ran all 22.

Reqs (requests) and ms are per item; "Days" projects all 884,566 question templates on my one GPU.

One question is cheap and blunt

The single-question strategies ask one gate question of about 250 tokens, alongside a political question. The narrow version asks whether the question hinges on real people or animals being harmed, killed, infected, attacked, held captive or sexually exploited, or on children competing. The broad version adds anything connected to a real death, attack, war, disaster, epidemic or abuse case, such as charges, verdicts, statements or search rankings. The broad one catches nearly everything, and 30% of disease-count lookalikes with it. The narrow one is clean but misses a lot, including some disease case counts. Neither works alone, but the broad one with a low threshold makes a good gate in front of something more careful.

Everything in one request is cheaper but vague

Asking the whole tree in one request kept false positives at 4%, but only 72% of positives scored above 0.7, so more of them would land in the middle band for a person to check.

A chain with memory is accurate, and very expensive

The most accurate setup was my idea. It asked all 29 questions in separate requests, least severe first. Each request also carried the questions already asked about that item and the model's answers to them. It put 94% of positives above 0.7 at a 7% worst false-positive rate. The cost was 29 requests and about 7 seconds per item, which works out to 72 days on my laptop or about $1,800 on Workers AI.

The front-runner

When I asked how to make that chain cheaper, Claude offered four ideas, and I had it run them alone and in combination:

• Prune: ask the 10 roots first, then only the children of roots scoring 0.3 or higher. It cut the chain 4.6×, but lookalike false positives rose (natural events went from 5% to 15%), because a child asked right after its parent leans yes.
• Gate: put the broad single question in front, so only items scoring 0.2 or higher get the chain. Its answer only routes the item and is never passed along as context. It cleared the lookalikes that pruning let in.
• Short-label context: carry earlier answers as short labels instead of the full question wording. It saved little and raised false positives, so it was dropped.
• Prefix caching: reuse the prompt prefix a chain's requests share. It helped the full chain but barely moved the pruned one.

Gate plus pruning became the front-runner: 5.5 days, or about $124 on Workers AI, for 1/13 of the cost and nearly the same quality.

What Moved the Answers

The side tests, which Claude designed, ended up more interesting than the leaderboard.

It's perfectly deterministic

Rerunning 50 items twice more gave bit-identical answers: 1,200 answers, maximum difference zero. That made the cache a faithful replay of the model.

Wording swings recall by nearly 28 points

Each gate question was asked in three wordings that meant the same thing. For the harm questions, 10–14% of answers crossed 0.5 between wordings, and recall at 0.5 moved by nearly 28 points. The ranking held up better (PR-AUC stayed between 0.90 and 0.97), so a reworded question still sorts items in roughly the same order. It just puts the line somewhere else. The political question barely moved.

The same question scores differently depending on its neighbours

The same node, asked about the same item, scored differently depending on what else was in the request. The average shift was 0.085 to 0.155, and 5–16% of answers crossed 0.5. The model scores questions jointly, so this probably shouldn't have surprised me, but it has a practical consequence: you can't tune thresholds offline. A threshold only holds for the exact wording and the exact set of questions it was tuned with.

Question order behaves the same way. Asking the chain most-severe-first against least-severe-first moved individual answers by 0.077 on average, but recall and PR-AUC came out tied.

Carried context only helps when it's the model's own precise answers

• Carrying earlier answers, which I'd asked to try on every chained strategy, fixed the depth-first tree's over-flagging (worst false positives went from 22% to 8%).
• Carrying the broad gate's "yes" into later questions made things worse, because the gate says yes to lots of disease questions and that pushed the disease-count lookalikes up.
• I had Claude plant a fake "the gate said yes" in every request to see what would happen. It pushed the false-positive rate on political lookalikes to 48%.
• Shortening the carried answers to short labels (one of Claude's cost ideas) saved only 10–26% of the time and raised false positives almost everywhere (natural-event controls went from 5% to 20%). The full wording of earlier questions is what kept later answers precise.

Claude drew one rule from this, and it held for the rest of the project. Use the gate to route, and never pass a broad question's answer on as context.

What Didn't Pay Off

Prefix caching

A chain of requests about one item shares most of its prompt, so Claude added prefix reuse to the PyTorch server: keep a snapshot of the backbone's state and only compute the new tokens. On the full 29-request chain it cut 6,995 ms to 4,368 (1.6×). On the pruned chain it saved 1–3.5%. Each forward pass costs about 85 ms however few tokens it has, and the decision head still reads every position. The answers also weren't bit-identical to the plain path (2–4% moved by more than 0.05), so a cached run would need its own thresholds. And it does nothing on Workers AI, which bills every input token.

The decoy veto

The "is the harm word only in a name or title?" root was meant to suppress harm flags on decoys. Across every strategy, it never changed a result. My guess is that the boundary-case wording inside the other questions was already doing that job.

Asking about ten items at once

I wanted to know if asking about a batch of items at once could work as a prefilter. Asking the 10 roots once about a chunk of 10 items ("does any of these…") worked, with one exception: the minors root fired on ordinary sports chunks. Without it, chunks of 10 at 0.3 cleared every clean chunk and kept 97% of the other positives. That didn't get tested at scale before I stopped.

Adding a Local Wikipedia

Some questions are only sensitive if you know what happened. A question about whether someone will be the most-searched person of the year doesn't say that person was killed. No amount of routing fixes that. I asked for a way to flag names that needed outside knowledge. Claude suggested Wikipedia as the source, and I asked for it to run locally with EmbeddingGemma 2 and LanceDB.

Claude pulled the 2026-09-01 English Wikipedia dump (about 34 GB), built a 4.9 GB LanceDB with 18.3 million titles and redirects, 7.23 million article leads and 1.24 million passages from event articles, and flagged about 221,000 pages by category (killings, crime, disasters, recent deaths and so on). Getting the flags right took a lot of exclusions. "Shooting guards" flagged 5,000 basketball players as killings, and "physiotherapists" contains "rapists".

For each item, the linker matches capitalised names against titles and redirects. Claude's design used Google's EmbeddingGemma 2 only to pick between namesakes, with a full-text phrase search over event passages as the fallback. Facts about flagged pages go into the state. It takes about 10 ms per item on the CPU.

On the eval set it did what I hoped. With the front-runner, recall at 0.7 went from 93% to 95%, and the outside-knowledge positives it caught went from 12 to 15 of 18. The worst false-positive rate went from 7% to 11%, all from a single control.

On a random sample it barely mattered. Only 1.9% of items got any facts, and only 4 of 10,000 crossed 0.5. Three of those four got worse: political questions about a sitting politician dropped from about 0.85 to 0.45, because the facts attached mentioned criminal convictions and pulled the political answer down. The fix would have been to attach facts only to harm requests, but I didn't get to it.

Ten Thousand Random Items

The last full experiment ran the front-runner on 10,000 random templates that weren't in the eval set. 19.6% passed the gate, the average was 3.14 requests and 470 ms per item, and 4.6% were flagged (0.9% for harm, 3.9% political). At that rate, the whole dataset projected to 4.8 days.

Only two strategies ran on this sample: the front-runner, and the front-runner with the Wikipedia facts from the previous section. Harm and Political are the shares scoring 0.7 or higher on each; Facts is the share that got any Wikipedia facts.

Reading the harm flags by hand turned up the next problem. About half were junior grand slam tennis matches. The policy does count competitions where most players are minors as sensitive, but many of those questions read like ordinary adult tennis until you see the event title. The minors root needed more work before the thresholds could be set.

I started the full run anyway, projected at about five days. A laptop lid closing knocked the GPU down to 45–50 W from its usual 80, and the run slowed from 2.1 items a second to 1.3. I restarted the laptop to try to get the power limit back, and the run resumed. Then the machine went down again at 152,933 of 884,566 templates, about 17%, and that's where I called it.

What I Left Undone

• A held-out test set. One question's wording was fixed after looking at the controls it was then scored on.
• Baselines outside Clef. A keyword filter, an LLM judge with the same rubric, and moderation models like Llama Guard or ShieldGemma would have put the Clef numbers in context.
• Two cheap savings. The 884,566 templates come from only 228,293 event groups, and almost every group shared one answer. Asking once per group, plus the chunk prefilter, each looked like a 3–4× cut.
• A 27B cascade for the uncertain middle band.

Wrapping Up

On the hosted side, all of those requests would have cost about $28 on Workers AI. The four days were mostly my laptop's time, not mine. If you're trying Clef or another decision model on a classification job and want to compare notes, find me on LinkedIn or email me.