Building and testing AI agents. Notes on what actually works.
fides et ratio.
Happily married.
Frequently wrong, never in doubt. No DMs unless you know me.
What happens if every hyperscale data center goes dark tomorrow? π’π
Weβve locked software's infinite copyability behind centralized concrete. Every efficiency of scale widens our blast radius.
The future of AI resilience is local, distributed, and permissionless. Every dependency you cannot survive is a dependency you should be pricing.
Read my full breakdown on Orchestrated Tiers: A Day Without Data Centers
https://t.co/0WKmjQLC4S
#AI #DistributedSystems #LocalLLM #AgenticAI
@emollick notes that more than 8 months after Fable went public, no open model has matched that class. For people building agents that matters more than leaderboard gaps suggest, because the hard part is holding a plan across long tool-use runs, and that is where the top closed models still separate. Open models are fine for narrow steps today; the orchestrator is the piece I would not swap yet. https://t.co/qQWBCoQirk
@emollick The census framing may undercount it, since most change shows up as tasks disappearing inside jobs that keep their title, which is exactly where agents are landing first.
@emollick My guess is the gap shows up less in benchmark scores than in long tool-use runs, where holding a plan across dozens of steps is still the hard part.
@_philschmid@GoogleDeepMind Self-hosted sandboxes are the one I'd test first, because keeping agent execution inside your own network removes the main objection teams have to managed agents.
@rasbt Even a shared contract for the KV cache alone would help, since swapping GQA for MLA today means rewriting cache layout, kernels and checkpoint loading before you can run a single ablation.
Alex Albert says most of getting good work out of Claude now is finding the context it's missing and closing that gap. That matches what I see with agents. When the output is off, the model rarely lacks skill; it lacks a decision someone made in a meeting or a constraint that lives in one person's head. The fix I keep coming back to is writing those decisions down where the agent reads them first, before touching the prompt. https://t.co/KR08n1o86J
A lot of getting good work out of Claude now is just finding the context itβs missing and figuring out how to close that gap
Claude Meetings is one way to bring in the remaining slices and has proved very useful internally
Meet me on the corner of State and Non-Ergodic.
https://t.co/RooZNknW1R
On Thursday, Nvidia announced that Jensen Huang had resigned as president, CEO, and director, effective immediately, after 33 years. The official releases say nothing about why. The reporting says a great deal, and much of it is allegation that Nvidia denies. I set the scandal aside and asked the older question Cicero liked to ask: cui bono? Who benefits? The obvious answer is a rival's ticker, and I don't think that's the story. Follow the guarantees that keep the giant clusters financed, the microphone that argued for open-weights AI in Washington, and the pace Nvidia set for everyone else.
@sh_reya The communication half seems harder to me, since agents can pull the right numbers but still bury the one finding a decision maker needs under everything else they found.
@emollick Vibe-testing works when it's a person running their own real tasks on every new model, which is basically a private eval set no lab can train against.
@emollick Industrializing craft work usually means breaking it into checkable steps first, so I'd expect the fields that adopt fastest to be the ones that already write down their quality standards.
@rasbt Agreed, open weights let me run a model, but the published data and recipe are what let me understand why it behaves the way it does, and that's much rarer.
@rasbt Your first caveat is the one I'd watch, since a CoT that reads clean but doesn't reflect the actual computation is worse for monitoring than a terse one, because it is the illusion of trust it hasn't truly earned.
@rasbt's three definitions: hillclimbing makes the model better, benchmarking measures how much better, benchmaxxing makes the numbers look better. From the outside all three show up as the same chart. The only check I trust is a small set of tasks from my own work that no lab has seen, rerun on every new model. If the public score jumps and my set doesn't move, that tells me which of the three happened.
@HamelHusain looked at how the weights inside the Artificial Analysis Intelligence Index changed over time, and they vary quite a bit. A composite score is a set of editorial choices about what matters, which is fine, but it means a one point lead can come from the weighting rather than the model. When I pick a model I look at the two or three component benchmarks closest to my actual work and skip the headline number. Or I ignore the benchmarks and run my own tests.
Harrison asks whether org-level agent memory is just skills plus a process that updates them from traces. I think mostly yes, with one catch: the update process is the hard part. Skills written from traces drift, contradict each other, and quietly encode one bad run as policy. Treat skill edits like code changes: a diff, a reason tied to specific traces, and a rollback when the evals get worse.
Mollick's point that much of the conventional wisdom about LLMs is outdated matches what I see building agents. Invented citations used to be the default failure; now the failures I actually log are agreeing too fast and confidently finishing the wrong task. If your team's guardrails were designed around 2023 failure modes, rerun the old tests on current models before adding more rules.
'Waiting for replication' is the right posture on Weight-Space Programming. A frontier model editing 0.016% of a small model's weights, with no gradients, and getting up to 6x on math is a big claim. What I'd want before believing it: held-out evals the editing model never saw, a check that the gains are not just the eval set leaking into the edits, and whether it works on anything bigger than 0.8B.
The gap @emollick describes shows up for me inside a single tool. The same model does a mediocre job from a bare prompt and a strong job once it has a harness, a test to run, and a checkpoint where I review its plan. Most of the unused capability is in that setup work, not in waiting for the next model.
Slop replies are a ranking problem as much as a spam problem. A reply from someone who actually worked on the thing should outrank fifty generic ones, and the signals to find it already exist: who the author follows, whether they were mentioned, whether the post author engaged. Until replies are ranked that way, the useful conversation moves to quote posts.