the OpenRouter situation for anyone curious:
-someone managed to access outputs sent to MiniMax
-researcher discovered this by sending a secret links
-40 to 130 hours later someone opened the secret links
-oop, this means that someone accessed your outputs!!
-AltasCloud/Novita/StreamLake were forwarding requests to MiniMax???? what the helly?
ugh.
.@HoneyHiveAI has agreed to join @elastic. Congrats to @mohak__sharma, @ds3638 and the whole HoneyHive team!
AI has changed a lot since Dhruv and Mohak started HoneyHive in 2022, a month before ChatGPT launched. So have the needs of engineers building with it.
The team did an outstanding job of evolving the business as those needs changed underneath them, from helping startups get their first AI apps into production to supporting thousands of mission-critical use cases at Fortune 500 companies.
Their work now goes into Elastic Observability, so teams can see what an agent did right next to the rest of their stack.
@ZeroPrimeVC is proud to have backed this team. Excited to see what they build inside Elastic!
We’re excited to share HoneyHive has entered into an agreement to be acquired by Elastic.
Together, we will bring agent tracing and evaluation into Elastic Observability to help enterprises build more reliable AI.
Read more: https://t.co/SbRXBo1HA0
We've signed an agreement for @HoneyHiveAI to join @Elastic. @Mohak and I started HoneyHive four years ago, a month before ChatGPT launched. We knew these models would keep getting smarter, and that knowing when to trust them would only get harder. So we set out to build the best observability and evaluation system for AI.
Tracing the harness and evaluating tokens worked up to a point. Then providers started holding back reasoning tokens, and the harness grew from a few simple tools to the entire operating system. To really debug an agent now, you need its behavior and the rest of the stack in one place. Making sense of that data at scale is a search problem, and Elastic has been solving it for more than a decade.
To the HoneyHive team, our customers, and Mohak: thank you. Solving these problems together has been the best part of my career, and I'm glad we get to keep going. Superintelligence doesn't sleep, and neither will we till our observability is complete.
https://t.co/l0v2pHCzok
We’re releasing Multi Agent Coordination Bench, a benchmark of twelve natively multi-agent long-horizon tasks that measure the coordination capabilities of frontier models.
We find that current frontier agents coordinate poorly. A team of eight agents beats a single agent, but the gain comes from having more agents, not from coordination. Of Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, only Opus shows signs of a gain from coordinating.
areas where agents are still weak in engineering:
- distributed systems/infra
- architecture decisions (related to above)
- time awareness/prioritization
- technical writing (for humans)
if you are good at any of the above your comparative value is higher than ever
Episode dropping tomorrow with @paulfchristiano.
He thinks there’s a 1 in 3 chance of extreme superintelligence within a year (absent any slowing down).
And he'll explain what labs and governments should do right now given that possibility.
I bet against this and I bet against Dwarkesh.
He conveniently sweeps under the rug that climbing a metric involves defining it. Then, supposing you define it (in a way that avoids Goodharting, reward hacking, etc), you need data/envs, which do the lifting. On defining good metrics - this is most of the game of making AGI (converting fuzzy concepts to formalized objects you can climb without gaming, which requires costly fixes later). He elides the hard problem by saying 'AI will just automate data'.
How do you know whether a change belongs in your final recipe? You ablate at scale. When you make N changes, on even a subset of combinations, this compute cost is naively exponential. So, you need a faster signal to tell you when a task that isn't immediately verifiable improves end behavior. Finding that signal is the bottleneck, which doesn't seem to be helped by 'automating the great algorithmic thinker starting from the empty string on paper'.
OCR folks are having a heated debate about particular benchmarks in this thread. Much of the frustration stems from the unreasonable sway benchmarks have on vendor selection: nobody looks behind the top level score!
This invites moral hazard when publishing a benchmark, and understandably shines a light on motivations and incentives at play, even if there are good intentions.
As a consumer of models, if you are relying on benchmarks the least you should do is look at the traces from 3-5 failing tasks and try to make an informed decision yourself. Vik has a checklist as well below.
Don't make up your mind based on twitter arguments either! Look at the data. Also sometimes "benchmarks of benchmarks" are slop. You really need to look at the data yourself if you are making large purchasing decisions based on the numbers.
AI interfaces nowadays are still pretty much turn-based.
In my PhD thesis, I thought about what interaction would look like if we could go beyond taking turns.
One example is intra-turn interaction:
None of the labs have nailed the taxonomy for how to work with agents.
Chat = quick, 1-off task (but it can handle substantial tasks too)
Projects = knowledge work without a gut repo?
Code = code first approach likely with a repo?
Artifacts / Pages / Spaces = interactive docs?
Personal agent as a separate chat?
Muse is one core chat and then sub chats (when should I use those?)
Grok Bot style messaging inbox for everything might be best abstraction.
Repeat after me:
My job as a leader is not to make people successful.
My job is to create the conditions that give them the highest possible chance of success.
Everything else is up to them.
This is what I believe the current meta to be (disclaimer: Googler):
Fable 5.2: architecture, high ambition projects using either Fable 5.2 or Opus 5.5 subagents
Carbon, GPT-6 Astra Ultra: review Fable’s work and supply prompts
GPT-6.1 Sol Ultra: review Carbon’s work and supply prompts
Carbon: all around workhorse for medium complexity features
Carbon: performance and memory optimization
GPT-6 Astra Ultra, GPT-6.1 Ultra: sledgehammer prototyping; prove a thing is possible and hand off to Fable or Carbon
GPT-6.1 Sol low: cleanup, deslop