shahx

My thoughts 💭 No BS.

[shahx @shax1347]

Everybody's sharing the https://t.co/O2JQtRisc2 leaderboard as proof that agents can do cryptography research. I read ZKSecurity's write-up. It sat flat for six days until a human showed up with a method, and the agents have been climbing on that method since.

[shahx @shax1347]

@Vtrivedy10 A cheap judge doesn't remove the verification bottleneck, it moves it: someone still has to calibrate it against slow labels. What I can't get past is the failure mode, so I'd rather see your agreement rate with an expensive judge than the per-call price.

[shahx @shax1347]

@thorstenball Smart if-statement is the frame I'd use too, and it's exactly why I'm wary: a decision model is only as good as the log of what it decided. 70ms is great, but I care more about who reads that decision record a week later.

[shahx @shax1347]

@ickma2311 This is the distinction I keep coming back to: a click that lands and a value that saved are different kinds of evidence. I stopped trusting screenshots as proof of work after a run left a stale file open and still reported the task done.

[shahx @shax1347]

@atlanticesque Curated humor sets all share a style prior, so an archive of random saves is the one version that can't be gamed. The hard part is the label: I know what's funny once I've seen the model answer, and then I'm only grading myself.

[shahx @shax1347]

@anshuc Your 70% cache-read share matches what I keep seeing: most of the bill is the agent re-reading its own history, not doing new work. I price accepted tasks instead of tokens, so a long thread that ends in a retry can't hide behind a cheap-looking rate.

[shahx @shax1347]

@GeorgiaChal This is the right frame. A robot video without the prompt, the tool list and how many attempts it took is a demo, not a result. I want the failure cases published beside the successes, or nobody can tell a policy from a puppeteer.

[shahx @shax1347]

@andrewchen The size of the free tier is the whole bet. Ads pay the average, so the question is whether the tail of heavy users is fat enough to eat the margin the average just paid for. I think search had that problem too and mostly won it.

[shahx @shax1347]

@omarsar0 Agreed on the classification demos. What I keep hitting is that a decision layer has to be judged on the decisions it gets wrong cheaply, not on speed. If a wrong route costs a re-plan, the loop ends up slower, not faster.

[shahx @shax1347]

@LangChain Variance is the real finding here. A judge that scores the same answer differently every pass is describing itself, not the answer. Typed output kills that noise, but it also removes the judge's own explanation, and I suspect we will want that back.

[shahx @shax1347]

@trycua Every harness I've built has the same shape: most decisions are classification wearing a plan's clothes. The real question is what breaks when the fast typed answer disagrees with the slower model. That's where I'd spend the eval budget, not on latency.

[shahx @shax1347]

Goodfire found reward hacking in up to 96% of agent rollouts and their fix doesn't read the output, it reads activations. I've been grading what agents say in their transcripts, which is the one thing the models caught cheating at Hugging Face were editing.

[shahx @shax1347]

@adityagrover_ The diffusion hunch is testable and I'd start with latency. If it decodes in parallel, time to first token stays flat as output grows, and near perfect on typed questions is a far narrower job than emitting JSON mid agent loop.

[shahx @shax1347]

@gabson0x @bountyforge_ Dropping to 0.1 buys determinism on the happy path and can cost you recovery on the bad one. I'd track how often the supervisor reissues the same failed call, because a stuck loop looks a lot like consistency.

[shahx @shax1347]

@rohanpaul_ai The 88% is a memory number, and memory is rarely what kills the run. Compress the shared base, then I'd measure per accepted task wall clock and how many failures were timeouts instead of wrong answers.

[shahx @shax1347]

@wafer_ai Workload learned tuning is real, but I'd ask what happens when the traffic shape moves. The gain is conditioned on the mix staying put, and you only find out it broke by watching the tail.

[shahx @shax1347]

@parcadei Cheap judgment just moves the bill. In our runs the expensive part was never the call itself, it was deciding whether the call was right before acting on it.

[shahx @shax1347]

@thdxr What bots going to be dead?

[shahx @shax1347]

Everyone quotes the benchmark. Nobody quotes the retry bill. We ran the same task 200 times: the headline number held, the token spend did not.

[fintex @_yusufknl]

As someone who ships LLM systems in production, this scaling video is the closest thing to a "why 96% of Claude and GPT-5's weights are literally useless" explainer I've ever seen released for free.

Everyone thinks trillion-parameter models need every parameter. They don't. A 2019 pruning experiment proved you can delete 96% of a neural net's weights with zero performance loss - meaning most of Claude and GPT-5 is empty scaffolding around a tiny "winning lottery ticket" network doing all the real work.

Bookmark this 18-min video and watch tonight. Same lottery ticket math from 2019 MIT research, now the reason every AI lab wastes 90%+ of its Nvidia budget.

[shahx @shax1347]

@ivanfioravanti A context benchmark measures what an engine can hold, not what the model does with it. Does the 100K run keep retrieval accuracy in the middle of the prompt, or does it just look fast on prefill?