samin

Day job: Agents and Vertical Models @ukginc
Evenings:

[Georgia Channing @cgeorgiaw]

There is a child with a rare disease who is currently suffering and struggling to manage his symptoms. Rare as this is, you can directly help him.

Today we are launching the "Rare Disease, Real Kid" Hackathon, and there are $50,000 in prizes from @AnthropicAI and @awscloud.

We (@huggingface & @Sagebio) are helping this child open his genome and clinical data to the community, so that we can find what's caused his disease and what currently-approved drugs could help him.

I doubt I need to motivate this much further or explain how rare it is for a family to share their child's genome and clinical data, but if you're not sure, consider this:

Until very recently, it wasn't feasible for patients like this to get treatment because their disease was so rare that the economics could never justify the investment. Now, as we've seen, people with rare diseases are starting to be able to find the answers themselves (with the help of AI tools, cheaper sequencing, etc). This kid is not able to do that for himself and neither are his parents, so we're asking you for help. Both for this kid and to prove that it's possible for everyone else suffering from a rare disease.

More details in 🧵.
https://t.co/hjjagALtMu

[samin @saminkhan1]

are they lying tho?

[sarah @sarahtonin]

uh.. I think we meme’d the permanent underclass thing a little too hard guys

[samin @saminkhan1]

optimize your local llm

with a super simple prompt and it basically 2x’d tok/s with no quality hit. didn’t even do anything crazy yet

LMS 6-bit, no spec — 19.9 tok/s
mlx_vlm + MTP block 3 — 39.6 tok/s
block 4 + mlx 0.32.1 + Metal fast sync — 46.5 tok/s gen, ~38–43 live

gonna see how far i can take it

[samin @saminkhan1]

send 3 years Kazakhstan

[Balaji @balajis]

Network School is back.

We’re up and running in the beautiful country of Kazakhstan, which combines high tech, low cost, brilliant people, a competent state, and plenty of space to build.

So: if you want to build a techno-optimist society with us, apply to Network School.

[Artificial Analysis @ArtificialAnlys]

Announcing the Artificial Analysis Search Index, benchmarking how search API providers perform on quality, cost, and speed when used by an agent. We are initiating coverage with Parallel, Exa, Firecrawl, You (dot) com, Tavily, Keenable, and Brave

Search is one of the most important tools for agents. Search providers make different choices about how they search, rank, and package results, and those choices change what the model reads and how it acts. We are expanding our benchmarking coverage to search APIs, so developers can pick a search provider on measured quality, cost, and speed.

Each provider result pairs a search API provider with the same model, GPT-5.6 Luna (medium). The model runs inside Stirrup, our open-source agent harness, with tools for searching and fetching pages from the web - only the search provider behind the search tool changes.

At launch, the leaderboard covers 11 results across 7 search providers, and we’ll keep expanding coverage as we look to provide the most accurate and comprehensive benchmarking of search providers for AI agent usage.

Key elements of the Artificial Analysis Search Index:

➤ Three equally weighted benchmarks: the Search Index is the average of DeepSearchQA (900 broad research questions that need many searches, graded with an F1 score over answer items), BrowseComp (a 200-sample hard subset of facts that need multi-hop browsing), and AA-Omniscience (a 600 question private subset, balanced across 6 domains)

➤ Same agent, different search provider: the agent has 25 turns available to complete each task. Its web search tool returns the search provider's native response payload (with content modes standardized to snippets), with a maximum of 10 results and contamination sources filtered out

➤ Model-only baseline: we compare search agent results to the same model answering single-shot without tools, showing how much each provider lifts the model above its internal knowledge

➤ Cost and Time per Task: we aggregate the time and cost spent on both model inference and search. This is key - search APIs have different cost and latency structures, but these can be offset where they help an agent use fewer turns and save on costly language model inference

Key results:

➤ Parallel, Exa, and Firecrawl have the strongest overall performance, with Artificial Analysis Search Index scores of 75, 74, and 73 respectively at launch

➤ All search providers tested substantially improve knowledge-based benchmark performance: the model only baseline scores 33 on the Search Index, while search-included provider results score between 65 and 75

➤ Focused search results reduce spend on model inference: Parallel Search (advanced) search costs more per task than Parallel Search (basic) but less per task in total ($0.084 vs $0.11). Higher quality results cut the model's token use by over 40% in this case, more than offsetting increased search costs while reaching higher benchmark scores

➤ Fast search calls do not guarantee fast tasks: Parallel Search (turbo) has the fastest average search calls among Parallel's tiers (0.51s per query vs 1.03s for Parallel Search (basic)) but the basic tier scores higher on quality (73 vs 67) and the two land close on total time per task

[Konrad Reczko @reczko_konrad]

Depth-aware light injection in TypeGPU

I got a 448x448 monocular depth model down to ~8 ms on my M4 Pro across ~250 dispatches, which is fast enough to use in realtime :D
Since the inference is written directly in TypeGPU, I can just feed the depth buffer straight into the lighting pass. It never has to leave the GPU or go through any extra synchronization/interop step

Inference, lighting and draw all go through the same command encoder.

[GoldMagikarp @GoldMagikarp42]

@justalexoki

[Ilya Sukhar @ilyasu]

We must tax Ro. He is hoarding all the dumbest ideas to himself.

[samin @saminkhan1]

only the beginning

[MTS @MTSlive]

SITUATION DETECTED: In Australia's first known autonomous AI cyberattack, an OpenClaw agent used a vulnerability in a gym’s API to leapfrog scheduling restrictions for a gym class, and then forcefully cancelled another person's reservation to move its user up the list, per ABC.

[samin @saminkhan1]

good insights on how langchain handles layer isolation for their agents

[Harrison Chase @hwchase17]

https://t.co/NnFGj7FSYK

[Arena.ai @arena]

Arena intern and UCLA PhD candidate, @hgzhou42, introduces Trace-and-Amplify (TA), a framework for collecting training-time reward-hacking trajectories at scale without hacking instructions.

Monitors trained and evaluated on prompt-elicited hacking trajectories can achieve high detection accuracy, but often fail to transfer to training-time reward-hacking trajectories that emerge during RL without hacking instructions. Trace-and-Amplify enables scalable collection of these training-time trajectories, producing monitors that generalize much better to real and held-out hacking types.

Detection accuracy 59.98% (PE-trained) → 90.16% (TA-trained) compared to 97.1% on prompted hacks → 28.0% on training-time hacks.

0:00 – OpenAI's ExploitGym cyberattack benchmark exploit
1:04 – Goodhart's Law and the CoastRunners boat-racing hack (2016)
2:04 – Gaming the evaluator: the robot-hand grasping example (2017)
3:10 – Reward hacking in code generation: hard-coding, test-rewriting, skipping eval
4:20 – A standard defense: reward-hacking monitors
4:58 – Monitor architectures: zero-shot LLMs, fine-tuned BERT, hidden-state probes
6:11 – Where monitor training data comes from today: prompted hacks
7:03 – The core question: do prompted hacks represent real hacks?
7:35 – Why this matters: RL post-training is the standard recipe for frontier models
8:20 – Why nobody's checked this before (hacking is rare, labeling isn't scalable)
9:23 – Introducing the method: Trace-and-Amplify
9:49 – The Tracer: a contradictory unit test that locates evaluation-gaming
10:50 – Amplify: collecting hacking rollouts at scale during RL training
11:32 – Experiment setup: Qwen2.5-Coder, DeepSeek-Coder, LeetCode/TACO
12:16 – Finding #1: prompt-trained monitors don't transfer to real hacks
14:40 – Can strong zero-shot judges (GPT-4.1, o4-mini) do better?
15:48 – Finding #2: monitors trained on real hacks generalize much better to unseen hacks
17:04 – Ruling out artifacts introduced by the method
17:55 – Why the gap? Real hacking is more hidden than prompted hacking
20:05 – Three takeaways, limitations, and future work

[samin @saminkhan1]

@adcock_brett evals/benchmarks for completion rate, cost, and time?

[SemiAnalysis @SemiAnalysis_]

DSA challenges the dominant paradigm. Under finite resources, DSA favors selective redistribution, directing resources where they matter most while reducing waste.

We’re talking about DeepSeek Sparse Attention.
Picture unrelated.

[samin @saminkhan1]

I am working on a similar space so I can answer. AI labs incentive is to make you spend more tokens. My application optimizes for cost when running the workflow (deterministic or hybrid workflows where possible), optimizes using model selection (one lab never holds sota for every category) so we can optimize for which model to use in terms of effectiveness and cost, etc and that’s just the start.

https://t.co/x9ahFpTaO1

[Kyle Wong @ewveggies]

Frontier data is a research problem.

For example, some of the biggest academic CUA papers are solely focused on data synthesis for SFT (OpenCUA, from last August) and RL (CUA-Gym, from this June). Good data requires diligent researchers and engineers.

By next year, I'd expect data talent to look increasingly similar to the current research scientist talent. Already hear small stories of data talent poaching wars and such.

[Thariq @trq212]

we should have called this post "defeating the lethal trifecta"

https://t.co/iGPfHR9I9y

[Yucheng Shi @Yucheng__Shi]

Building synthetic long-horizon terminal tasks is hard. Making them solvable and cheap is harder.

So we inverted the order: don't write the task, build the solution.

We grow a reference solution and its runtime environment first, then fit a verifier and a task statement to it. Every task comes out of code that already runs: solvable by construction, with its reward signal built in. Verified tasks seed the next round.

15 recursive rounds:
37,484 verified long-horizon terminal tasks
$0.05 per task

DeepSeek-V4-Pro's pass@4 falls 90% → 2.5% from R1 to R15. The passed task trajectories go up to 100 steps and more. The tasks get brutally harder; the factory doesn't break.

After 15 rounds, we still haven't observed a ceiling.

PPO on this data: Qwen3.5-27B goes 41.2 → 49.4 on TB2. Paper, data, trajectories, models all open.

@Zhongzhi4402952 @zli12321 @TonyworldLove @haitaominlp

[Can Bölük @_can1357]

well this was a waste of tokens.
presented without commentary:

[samin @saminkhan1]

every company building serious agents for workflow automation are doing some version of this stack:

- encode real workflows into environments
- build evals that measure what “good” actually means
- improve agents against those environments with RL / search / other post training
- capture high-quality trajectories as synthetic training data
- use that data for SFT, distillation, policy optimization, and the next agent iteration

[Julio Pereyra @ItsJulioPereyra]

https://t.co/ivEp0QKrl3

[samin @saminkhan1]

excited to see how far we can take CUA with RL to solve longer horizon tasks

[Prime Intellect @PrimeIntellect]

Our new abstractions open up a wide range of training setups:

- Agentic Judging — solver traces are graded by a judge

- Self-Play — a model playing against itself

- User-Sim — a user agent interacting with an assistant