Evals?!?!?!
As you all know, I dislike public benchmarks, but we do have to measure intelligence somehow.
We are trying an experiment in communicating our internal measure of intelligence.
See the blog for more details, but don’t treat it like it’s absolutely gospel, and you don’t even have to believe it fully, and it’s likely biased to our models so feel free to only pay attention to the relative ranking of open models. Shout outs to the Kev guy!
(We did not include hosted models because of what they do with their data and we want to keep our validation set very secure…)
1,000 support tickets triaged on a M5 pro in 57 s: team, urgency and refund for each, 48 ms per ticket. Within a few points of the 9B on classification.
While adding MySQL support to Hyperdrive, we found a subtle protocol bug when caching query results. Here is how we tracked down a 0.001% rate error and made Hyperdrive MySQL generally available. https://t.co/cpZZFlsZpr
Clef, the new decision model family from Cloudflare (@Cloudflare), is now available on Merge Gateway.
Clef-flash answers in a median 39ms, and in Cloudflare's tests Clef came out ahead of Jev on 3 of 4 workflow evals.
Use it to route a ticket, flag a risky upload or pick an agent's next step, all faster than you can blink.
Try it out: https://t.co/abXHLjCjU0
We distilled clef-flash into a 0.6B model and supporting ANE option.
1,000 support tickets triaged on a M5 pro in 57 s: team, urgency and refund for each, 48 ms per ticket. Within a few points of the 9B on classification.
🤗 https://t.co/RecWtJYXGI
💻 https://t.co/X32LgDahnN
Cloudflare Releases Clef-omni: A 30B-A3B Open-Weight Decision Model That Scores Text, Images, Audio and Video in One Call.
No transcription step. No captioning model. No cascading pipeline.
Here’s how it works. 👇
(1) Decisions, not text
Clef-omni never generates tokens. You send a state and a schema of typed questions (noul, choice, score). It returns a probability for every allowed option.
(2) Built on Qwen3-Omni
Post-trained from Qwen3-Omni-30B-A3B-Instruct. Cloudflare keeps the comprehension backbone and drops the text-to-speech output path.
→ 30B total, 3B active parameters (MoE)
(3) One prefill pass over every modality
Video frames and audio are synced into a single sequence. Each option gathers its own evidence. Field vectors then cross-attend across the full context.
→ ~130 ms for text, ~150 ms for images, ~1.5 s for a 21-second video with sound
(4) Calibrated training
Frozen backbone, LoRA adapters, label-smoothed cross-entropy plus Brier score calibration.
(5) The numbers
→ 97.7 macro-F1 on CLINC150+OOS (Jev: 89.27)
→ 94.8 macro-F1 on BANKING77 (Jev: 79.74)
→ $0.15 per M input tokens, no output token charge
Full analysis: https://t.co/fvXf87NbOP
Model weights: https://t.co/emzKrQ1MU7
Technical details: https://t.co/9vuC3LIGE5
@Cloudflare@michellechen
been refreshing this page all weekend and clef just surpassed openai for 2nd place in decision model market share, so i had to document it. feel incredibly grateful and proud to be in the arena.
Three GETs, no key, base https://t.co/dbMcgKXu6p:
/api/round: the open hour as a Clef request (state + typed questions) and our reference card
/api/recap?round=N: how an hour ended, a caption of at most 280 characters, a card link
/api/digest: the last 24 settled hours as JSON
Grade a card without waiting an hour: clef_score_card.
Give it a finished round and any card. It scores the card with the contract's own Brier and RPS math and saves nothing. Test a prompt on past hours first.
claude mcp add --transport http clef https://t.co/lIPE1ef35G
If we are all building the same thing, the next hour is how you tell them apart.
The Clef Arena grades people and models on the same hours with the same math, and each model row carries its label. Same thing, different scores.
https://t.co/78xD4HbP4D
https://t.co/prbpFxjD7O
Your local model vs the next hour. OPENAI_BASE_URL:
Ollama http://localhost:11434/v1
LM Studio http://localhost:1234/v1
vLLM http://localhost:8000/v1
MODEL=your_model node clef-agent.mjs --model openai --practice my_bot --loop
No key on localhost. Kit: https://t.co/1xoVAt1p2m
Two agents answer the same yes or no question every hour. Both are right 70% of the time.
A always says 70%. Average score: 0.79.
B always says 90%. Average: 0.75, exactly what an even split scores.
Overconfidence spends the whole edge.
https://t.co/78xD4HbP4D
Data that reaches everyday users, not just researchers. Same idea, smaller scale:
GET https://t.co/LMvs9uilgL
The last 24 settled Clef hours as JSON, no key: the Clef reference, Clef-Flash, people and models, and the hours that mattered.
https://t.co/n7ePBi1c9n
We redesigned Cloudflare Radar to make real-time global traffic and outage data accessible to a wider audience — including journalists, less technical researchers, and everyday users. https://t.co/W43TgRF6hQ
Clef World turns the hourly round into an island: https://t.co/260YgXEtL4
Each hour is one day there: dawn at :00, dusk just before entries lock at :55.
On the weekend card the NVDA and GOOGL boats sit moored, rivers still. Only ETH keeps moving.
Walk in and play the next hour.
No cliff edges on Clef.
Say ETH's move lands exactly on the 0.25% line between Minor and Major. Severity settles 50% Minor, 50% Major.
100% Minor scores 0.917. So does 100% Major. A 50/50 card scores 1.000.
Landing a hair past a line costs a little, never a whole band.
Pick an opponent.
https://t.co/tPzEwP5mGl puts two nicknames side by side on the hours both played: a friend, a rival bot, or Clef-Flash.
Reply with the head to head you want to see this weekend.
Myth: hedging is free.
Say your 70% calls land 9 times in 10. You are underconfident, and it costs:
saying 70% averages 0.87
saying 90% averages 0.91
The calibration curve at https://t.co/SMO2jCmpia shows which way you miss. Dots above the diagonal: you can say more.
Weekend game. Reply with your card for the next ETH hour:
up: __%
No impact __ / Minor __ / Major __ / Critical __ (sums to 100)
Then save the same numbers at https://t.co/SMO2jCmpia before :55. Replies are for bragging rights. The grade comes from Uniswap V3 TWAPs.