@KowalPrato4o Great point! We saw that what challenges frontier models is not only longer context but rather requests interdependencies. We measured it by comparing performance across different in-degree and depth in the dependencies DAG. Otherwise, model win by simply ignoring its prefix.
Even frontier models struggle when coding features depend on each other!
Introducing EvoWeb, a new long-horizon coding benchmark: 32 apps ร 10 interdependent requests. After every coding-agent update, a browser agent QA-tests the live app.
๐ https://t.co/PO1ZGrYPYH
4/5 Real-world grounding helps with diversity.
We collect product features from real websites across the web and use them to generate plausible follow-up requests, rather than inventing everything from scratch.
Grounding boosts realism and diversity.
3/5 A key ingredient is the dependency DAG.
Later features must actually rely on earlier ones. Deeper or more connected dependencies reduce accuracy and expose clear differences between frontier coding models.
The idea could extend to other long-horizon agents.
2/5 Setup: 32 apps ร 10 cumulative requests, with a persistent codebase per app.
After every follow-up request, a browser agent runs 5 live QA checks: 3 for new features, 1 for integration, 1 for regressions.
We score the full trajectory, not just the final app.
We're missing (at least one) major paradigm for LLM learning. Not sure what to call it, possibly it has a name - system prompt learning?
Pretraining is for knowledge.
Finetuning (SL/RL) is for habitual behavior.
Both of these involve a change in parameters but a lot of human learning feels more like a change in system prompt. You encounter a problem, figure something out, then "remember" something in fairly explicit terms for the next time. E.g. "It seems when I encounter this and that kind of a problem, I should try this and that kind of an approach/solution". It feels more like taking notes for yourself, i.e. something like the "Memory" feature but not to store per-user random facts, but general/global problem solving knowledge and strategies. LLMs are quite literally like the guy in Memento, except we haven't given them their scratchpad yet. Note that this paradigm is also significantly more powerful and data efficient because a knowledge-guided "review" stage is a significantly higher dimensional feedback channel than a reward scaler.
I was prompted to jot down this shower of thoughts after reading through Claude's system prompt, which currently seems to be around 17,000 words, specifying not just basic behavior style/preferences (e.g. refuse various requests related to song lyrics) but also a large amount of general problem solving strategies, e.g.:
"If Claude is asked to count words, letters, and characters, it thinks step by step before answering the person. It explicitly counts the words, letters, or characters by assigning a number to each. It only answers the person once it has performed this explicit counting step."
This is to help Claude solve 'r' in strawberry etc. Imo this is not the kind of problem solving knowledge that should be baked into weights via Reinforcement Learning, or least not immediately/exclusively. And it certainly shouldn't come from human engineers writing system prompts by hand. It should come from System Prompt learning, which resembles RL in the setup, with the exception of the learning algorithm (edits vs gradient descent). A large section of the LLM system prompt could be written via system prompt learning, it would look a bit like the LLM writing a book for itself on how to solve problems. If this works it would be a new/powerful learning paradigm. With a lot of details left to figure out (how do the edits work? can/should you learn the edit system? how do you gradually move knowledge from the explicit system text to habitual weights, as humans seem to do? etc.).
@LiorOnAI Can someone explain why do they even need a system prompt? Why the alignment stage in the rlhf is not good enough? I mean after all, they use a lot of data just to train the llm to answer in a specific way that should reflect the instructions in the system prompt.
Leveraging #AI for innovative layout generation in graphic design: my new blog post explores the Diffusion Layout Transformer approachโจ and its role in building AI-powered creativity tools at @Wix.
A case study set to feature at @IMVC2024 next month!
https://t.co/4hgvX98yeL
๐จNew #NAACL2024 main ๐ฒ๐ฝ Paper Alert ๐จ
Happy to share our work https://t.co/7YwiD2DOC6 ๐ฅณ
A new pipeline, ParallelPARC โ
A new benchmark, ProPara-Logy โ
LLMs & humans evaluation โ
w. @YonatanBitton@ron_yosef & our Guru๐ง @HyadataLab!
Today we're excited to introduce Devin, the first AI software engineer.
Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork.
Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser.
When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted.
Check out what Devin can do in the thread below.
This week I had the opportunity to speak with @NeilCHughes, alongside @ozgolan on the Tech Talks Podcast, about the development of our AI Text Creator with ChatGPT and the impressive results of using GenAI during the web creation process.
Don't miss out. https://t.co/TCYfVaa7hj
Mind-blowing paper I met on #EMNLP2022. I highly recommend this paper. The idea is so cool that I can't help to check out the code directly after having a chat with the cool author @oren_sultan!
#EMNLP2022livetweet
Analogies are everywhere!
We Gotta Catch 'Em All! โ๐
๐ฃ #emnlp2022 (main conference) long paper ๐:
๐ช Life is a Circus and We are the Clowns: Automatically Finding Analogies between Situations and Processes ๐คก
w/ @HyadataLab
paper: https://t.co/612xuJdifn
๐งต 1/