Bhut

Data skeptic data scientist.Normcore ML.LP Enthusiast.Retweets not necessarily agreement.

[Essays @Iqiipi_Essays]

There Is Nothing New
Virtual machines (IBM, 1967), microservices, event sourcing, reactive systems: each sold as a breakthrough, each decades old. The software industry has been solving the same six problems since the 1960s, and has an extraordinary talent for forgetting
https://t.co/XS2yobe49x

[Grady Booch @Grady_Booch]

An observation, having now studied the K3 architecture.

A common pattern in all these frontier models is the use of agent swarms. As it turns out, this was one of the secret sauces of IBM's Watson Jeopardy, wherein the analysis of an answer would generate thousands of candidate questions, then through a pipeline managed by the open source framework UIMA, evidence would be gathered to supported each hypothesis and then at the end of the pipeline, ranked.

IMHO one of the failures of Watson was that, as it was commercialized, the use of UIMA was diminished, in effect breaking the spine of what architecturally made Watson successful.

All that is old is new again.

[elune @elune0x]

10 agent evals every AI engineer should know

1) golden set

a frozen set of cases you run after every prompt model or tool change

use it as your baseline to see whether the system improved or quietly broke

OpenAI Evals helps you build repeatable benchmark sets and compare model changes

→ https://t.co/dr1GZlC75R

2) llm as judge

a second model scores open ended answers against a written rubric

use it when there is no exact output to compare against

OpenEvals provides ready made evaluators for LLM applications

→ https://t.co/S2yhnByFIP

3) rubric scoring

score correctness tone safety and cost separately

one quality number hides the problem

DeepEval helps you create custom metrics and score every dimension independently

→ https://t.co/q9Z6Xmixia

4) trajectory eval

grade the path the agent took

not only the final answer

AgentEvals checks agent actions decisions and tool calls across the full trajectory

→ https://t.co/0oziAl54az

5) tool unit tests

test every tool with fixed inputs and outputs

no model in the loop

MCP Inspector helps you inspect and test MCP servers tools and responses separately

→ https://t.co/IVmt5qpWIN

6) regression suite

replay previous runs against every new prompt model or toolset

then compare the results

Promptfoo helps you run repeatable eval suites catch regressions and add checks to CI

→ https://t.co/zxi2PuWuhe

7) a b in prod

split real traffic between two versions and compare actual outcomes

GrowthBook provides feature flags controlled experiments and product analytics

→ https://t.co/DGlE3JjDD3

8) human review

sample real runs and let a person grade them honestly

use it to calibrate your automated judge

Argilla helps teams collect human feedback review outputs and build better datasets

→ https://t.co/QHWb7skWjr

9) shadow run

let the candidate run on real traffic while its output is shown to nobody

use it before a risky rollout

Langfuse helps you trace production runs compare candidates and monitor eval results

→ https://t.co/IrhDf38tRn

10) red team

attack the system before somebody else does

jailbreaks prompt injection data leaks and tool abuse

Garak scans LLM systems for vulnerabilities and unsafe behavior

→ https://t.co/w8ObyW4ZKv

offline evals tell you it works

online evals tell you it still works

you probably do not need all ten today

start with the two that would have caught your last outage

bookmark this

[Thariq @trq212]

We removed ~80% of the Claude Code system prompt for our newest models, this is what we've learned about writing system prompts, skills and Claude.MDs for them. https://t.co/6DZwSrZjE9

[Kyle Jeong @kylejeong]

Your agents suck because you fill their minds with useless tool descriptions.

The smartest agents write and execute code instead.

We've gone deep on this pattern in Browserbase Agents, read about how we think about code execution for agents.

[alphaXiv @askalphaxiv]

"Intelligence from Learnable Novelty "

This paper suggests that intelligence is not surprise itself, but the ability to turn surprise into understanding.

Since novelty search gets trapped by noise, and surprise minimization gets trapped by stasis, it splits surprise into learnable and unlearnable parts, then optimizes only the learnable part.

With a cheap differentiable estimator, it finds rule 110, grows solitons, clusters MNIST without labels, and improves RL exploration.

[harsehaj ⋆˙⟡ @harsehaj]

https://t.co/sea7QlZcLh

[alphaXiv @askalphaxiv]

Introducing OpenResearch: run parallel research agents with any model

Bring your own compute and your code + data stays with you

A virtual lab on your machine

[Braden Hancock @bradenjhancock]

The bitter lesson rejects domain-specific structure. Problem decomposition & recombination (what RLMs make first-class) is about as general-purpose of a problem-solving tool as it comes.

Reminds me of the march of processor speeds: CPUs got faster every year until ~2004, when clock speeds hit a heat wall around 3 GHz. Transistors kept getting cheaper, but the only way to keep getting gains was to decompose workloads into pieces and go parallel.

As the tasks we give LLMs get longer (context length/time horizon) and more complex, scaling up to 100T parameters will probably help...but my money is on big-O-improving innovations like this improving generalization faster.

[Sebastian Raschka @rasbt]

How can an LLM switch between low-, medium-, and high-effort reasoning? And how does an LLM learn to reason more or less?

I put together a “little” article explaining how these effort levels are implemented at inference time and during training.

[elvis @omarsar0]

Highly-recommended overview of self-improving agentic systems.

(bookmark it)

Self-improving agents are moving from research demos into deployed systems.

This survey frames a modern agent as a foundation model coupled with an operational scaffold, then formalizes self-improvement as a self-induced update operator that commits changes to either.

Paper: https://t.co/VMpAGWKMYg

Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX

[Cathryn @cathrynlavery]

every time i needed a diagram in Claude Code, it gave me the same generic garbage.

so i built a Claude Code skill to fix it.

Diagram Design (now at 2.8k stars on Github)

• 27 diagram types (architecture, flowchart, sequence, ER, swimlane, timeline, venn, org chart, and more)
• one Claude Code skill, matches your brand in 60s
• three variants per type: minimal light, minimal dark, full editorial
• pure HTML output. skip Figma entirely, no build step or JS needed

open source 👇🏻
npx skills add cathrynlavery/diagram-design
/plugin marketplace add cathrynlavery/diagram-design

[sunick @Serantych]

Anthropic just released how they build agents on Opus 4.8 with Karpathy's loop method

same task, same harness - reliability jumped from 30% to 100%

→ drop the effort a level - 4.8 defaults high, the cost curve moved down.same output, less spend

→ cut your "summarize" prompt - it narrates its own progress now

→ change the rules mid-task with a system message - your cache holds, no 10x penalty

→ send a second agent to attack the first one's answer - keep only what survives

→ edit memory in one script, not twenty tool calls - and git-track it

bookmark this ↓

[DAIR.AI @dair_ai]

// Rethinking the Evaluation of Harness Evolution for Agents //

The gains you have seen from self-evolving agent harnesses might not be coming from the harness at all.

Automatic harness evolution is what everyone is using to squeeze more out of agents.

But that approach conflates two different things. Harness evolution is itself an iterative search that keeps evaluating and revising candidates on task feedback, so its gains have to be compared against plain task-level search under the same feedback and inference budget.

And because the search and the final test share one benchmark, the reported wins can be overfitting to that task set rather than reflecting better design.

More in the paper.

Paper: https://t.co/RzSMtIyFCJ

Learn to build effective AI agents in our academy: https://t.co/LRnpZN7L4c

[Hanchen Li @lihanc02]

https://t.co/AYMHyL3JCk

[Superman @thesupermanmx]

SAM ALTMAN HAS LOST IT.

Google just shrunk 31GB of AI memory down to 4GB.

They open-sourced a vector index that fits 10 million documents in 4GB of RAM and searches them faster than FAISS.

→ 16x compression per vector
→ 10-19% faster than FAISS
→ Zero training. Zero rebuilds.

No GPU cluster. No cloud bill. No compromise.

Runs 100% locally.

[Teknium 🪽 @Teknium]

Prior to today, tool calls would work in parallel only if all were safe to parallelize. Now you gain big speedups for parallel tool calls so long as any subset are parallelizable.

Check it out

[Zhengyao Jiang @zhengyaojiang]

The first experimental evidence of recursive self-improvement (RSI).

Autoresearching the autoresearch agent for eight days.

The result beats the harness we hand-tuned for two years, on held-out benchmarks: 🧵(1/7)

[Yuzhen Mao @Mao_Yuzhen]

Strongly agree, and we recently tried to make this argument concrete at the systems level.

In DeLM, we explicitly decouple three primitives that are often bundled together: agents, a shared verified context, and a task queue.

Agents asynchronously claim subtasks, read the latest accumulated progress, perform local reasoning, and write back compact, verified updates. The agents themselves are therefore closer to interchangeable function calls over persistent shared state than owners of the workflow.

That is, in miniature, the “core IP” layer Satya is describing: not any single model or agent, but the learning loop and shared state that allow an organization’s knowledge to compound across model calls.

Check out the paper and code!
https://t.co/e0qdIfE5zw

[Eli Gottlieb @GottliebEli]

Traces are now DAGs? And prime-rl has a custom algorithms layer? Cursor style targeted self-distillation is ~basically free, out of the box

https://t.co/iyVEakiHNl
https://t.co/DPrOCy5zQw