Top Tweets for #indirection

The moat of software AI agents is not the thin wrapper layer (Devin, SWE-Agent), but the underlying LLM. Instead of benchmarking the wrapper, I think SWE-Bench is excellent for evaluating coding LLMs instead: Hold the agent layer fixed and vary only the LLM backend. Provide all tools: LLM-friendly command line, editor, web browser. Specify a maximum token budget and internet query budget. How do GPTs, Gemini's, Claude's, Command R, Mistral, Llama compare with each other on solving real-world software engineering problems, with human-level tool access? This is a MUCH more realistic benchmark than HumanEval. It tests an LLM's ability to search StackOverflow, read documentations, self-reflect, self-correct, and carry out long-term consistent plans. In the Voyager (LLM playing Minecraft) paper, we did such an experiment where we held the Voyager algorithm fixed and swapped out the GPT-4 backend (orange curve) with 3.5 (green curve). It makes a world of difference. With a bad LLM, neither Devin nor Voyager matters - you won't get good performance any way. Conversely, with a much stronger LLM (eyeing GPT-5), we can probably get to L4-L5 software automation with just a bit of extra effort.