Daniel Rupawalla

RL research | prev. acq by US News / ex-YC

[Daniel Rupawalla @rupawalladaniel]

working with a few large co's to post-train a model for their use cases and it's making me more excited for the future of small models in enterprise. a few takeaways:

(1) picking which model actually matters, we played around with 5+ models in the 20-40B (and one in the ~300B parameter) range and shockingly found that bigger doesn't always equal better. note: this is where evals become increasingly important in being able to benchmark how good a model performs out of the box.

(2) data matters (lol). outside of just RL data, SFT recipes and data compositions are vastly important. picking the right experiments, data recipe, and problems to hill-climb on vs. try and swap models for is non-trivial. hyperparameter tuning was useful but only in so far as the data behind the training was good.

(3) human testing still matters. OOD examples seem to be difficult to get LLMs to do autonomously unless you prescribe exactly what it is (which ruins the point), though we are researching methods of improving model on model testing, human testing is always important to dejank performance and i don't think this has been solved just yet.

[Daniel Rupawalla @rupawalladaniel]

in an effort to push for higher data quality standards, i offer to do this for free for any lab that wants us to QA any set before training. naturally, it doesn't make sense for a lab to outsource the entire QA layer, but if you want an extra set of eyes on any set, lmk. tired of bad data.

specifically code & knowledge work

[Daniel Rupawalla @rupawalladaniel]

to people at labs that are doing QA on tasks, would love to chat (or offer to QA tasks tbh). i've looked through samples from a ton of people over the last 4 days and i'm noticing a bunch of common patterns in slopped-up tasks. notably:

(1) in order to get in-band, there are a ton of unfair or brittle checks. simple stuff like substring matches for non-deterministic outputs to more complex verifiers that are blatantly wrong or unfair (requires information that is impossible to get or subjective opinions as verifiers or only info that appears in a few rollouts by chance, causing model hallucination).

(2) rollout analysis is shockingly bad. even if the verifiers used are correct, they often penalize better trajectories since it doesn't fit the author's suspected golden path. checks between in-batch rollouts to make sure that higher scoring > lower scoring rollouts in actuality.

(3) toy-like or unrealistic worlds with tons of task leakage in the world. many people craft worlds with tasks in mind, when it should be the opposite, teaches models behaviors on unrealistic worlds.

many of these people are telling me about the large lab contracts they already have, so i'm wondering where the mismatch is.

[Daniel Rupawalla @rupawalladaniel]

i think this is where principally what @Ventali was talking about is most important. over time you build a general intuition for realistic axes to introduce vs. what not to. as you build very large eval environments, they naturally introduce complexity. just need to make sure you're always reading everything and have a clear goal in mind once you find what you're looking for (especially rollout and world analysis). if the failure you find is reproducible in different settings you can figure out if it's a factor of noise amount or a real failure mode. sometimes parsing through noise IS the failure/behavior you want to teach, but just need to make sure you know why/what you're attempting to teach through the task or capture through the eval.

[Daniel Rupawalla @rupawalladaniel]

agreed, i think the one caveat to this is emergent failure modes that can come from experimentation (under the assumption you do so in a controlled fashion without introducing unexpected or unrealistic noise). part of the way we approach eval creation is mapping failures in models through EXTENSIVE, purposeful testing to understand limitations. each model has its own quirks and weaknesses, and you may not find them unless you play around with the model enough with no explicit failures in mind. often i find my pre-existing notions of what makes models fail does not include a lot of surface area, so i try not to be too pointed in early phases.

naturally, you still need to be careful about what parameters are introduced, but i think there is value in no explicit objective being used in experimentation (though after failure modes have been mined, task creation should then be explicitly goal-oriented afterward).

[Ventali @Ventali]

agreed that one important failure mode in verifier/task generation is mistaking difficulty for signal.

a lot of “hard” tasks are just poorly constructed. they introduce noise, ambiguity, or arbitrary complexity instead of isolating the capability we actually want to measure.

i think the missing piece is that most task generation lacks an explicit objective. before generating anything, we should be able to state:

• what capability are we testing?
• what specific failure mode are we targeting?
• what evidence would convince us the model improved?

then every task should be intentionally composed to maximize signal for that objective, not just maximize difficulty.

one direction we’re excited about is treating tasks as compositions of atomic capabilities, planning, abstraction, memory, retrieval, tool use, causal reasoning, etc. but the hard part isn’t composing them, it’s composing them while preserving what you’re actually measuring.

for example, if you’re evaluating long-horizon planning, adding difficult retrieval or ambiguous instructions can easily dominate the failure. a model might fail because it forgot a fact, misunderstood the prompt, or picked the wrong tool, not because its planning was weak. the planning signal gets washed out.

ideally, composition should preserve capability attribution. every added component should either (1) be controlled so it’s unlikely to fail, (2) have independent verifiers that tell you whether that component failed, or (3) be generated from a causal graph or dependency structure where you know exactly which capabilities each subtask depends on. otherwise, once a composed task fails, you no longer know why.

benchmark generation should optimize for information gain, not just hardness. the goal isn’t to make models fail, it’s to produce failures that are interpretable enough to drive the next training iteration

[Daniel Rupawalla @rupawalladaniel]

to people at labs that are doing QA on tasks, would love to chat (or offer to QA tasks tbh). i've looked through samples from a ton of people over the last 4 days and i'm noticing a bunch of common patterns in slopped-up tasks. notably:

(1) in order to get in-band, there are a ton of unfair or brittle checks. simple stuff like substring matches for non-deterministic outputs to more complex verifiers that are blatantly wrong or unfair (requires information that is impossible to get or subjective opinions as verifiers or only info that appears in a few rollouts by chance, causing model hallucination).

(2) rollout analysis is shockingly bad. even if the verifiers used are correct, they often penalize better trajectories since it doesn't fit the author's suspected golden path. checks between in-batch rollouts to make sure that higher scoring > lower scoring rollouts in actuality.

(3) toy-like or unrealistic worlds with tons of task leakage in the world. many people craft worlds with tasks in mind, when it should be the opposite, teaches models behaviors on unrealistic worlds.

many of these people are telling me about the large lab contracts they already have, so i'm wondering where the mismatch is.

[Daniel Rupawalla @rupawalladaniel]

@pax21e8 id imagine over time it’ll become impossible for humans alone to verify the discoveries, it’ll have to be LLM on LLM, which seems like a harder verification system to construct, but if done right would be infinitely valuable

[Daniel Rupawalla @rupawalladaniel]

all of the new math discoveries signals to me that LLM research will (if it has not already) outpace that of humans. the problem i see here is that once we get to non immediately verifiable domains (non-math, non-code) the ability to advance is limited by our ability to check LLM work. id be curious to hear from scientists using LLMs in their work to understand how they are approaching verifiers. i imagine in some cases it would have to be similar to RL verifiers in non-deterministic domains, though im not quite sure.

[Daniel Rupawalla @rupawalladaniel]

@SukhaniShri too many people wanna waste their time with rl envs

[Daniel Rupawalla @rupawalladaniel]

rl data verifiers are hard to do right. i think the big reason for this is because the incentives are misaligned between data vendors and labs. specifically: i'm under the impression (feel free to jump in if you're from a lab and have a more correct/concrete take, would love to hear) most labs look for something in the realm of pass@k = 0.2-0.6, after all if a model performs stochastically on a task and it passes sometimes and fails sometimes there is signal to learn. true! but let's dig into why this causes issues in tasks.

(1) vendors remove normal checks to only leave stochastic checks. this is bad because it removes the grounding for the model, if you never check for things the model always does correctly, then the model can simply do ONLY the things you check for and essentially reward hack through (given thousands of rollouts)

(2) LLMaaJ rubrics. CAN be good, but most implementations suffer. the very easy test for if a rubric is good, is if you take the exact same rollout and grade it on the rubric 100 different times with different models, will the grade be the same. if no: rubric = flaky = bad in rollouts (since this is essentially what happens in GRPO).

the two of these compound to create tasks that may improve the model, but probably introduce some sort of noise into training as well. to clarify, i'm not saying labs are wrong about how to harness signal, and i don't think all data vendors are terrible, but there is much hill-climbing to be done on data quality alone that's likely blocking intelligence from improving even more.

[Daniel Rupawalla @rupawalladaniel]

@dsandzhiev yup, exactly. in an effort to get "lots of dense signal" with low pass rates, someone's sacrificing something, whether it's verifier quality, rubric flakiness, or some other dimension of quality

[Daniel Rupawalla @rupawalladaniel]

there are too many buzzwords that surround the RL data space that have created convoluted, perverse forms of making data. in my opinion it is extremely simple to classify good versus bad data (though infinitely harder to actually create, because models are so smart). these principles in summary are:

(1) is the behavior, and its underlying principles, beneficial to teach model? if you were to take a baby in isolation and teach it this task, what core cognitive principles are you teaching it? this is a bit of regurgitation from a previous tweet i had, but heavy search tasks teach models to over-index on searching through entire corpora instead of being efficient. similarly, tasks that can be reward hacked (esp tasks that seem otherwise good but don't have negative space criterion), can lead models to find shortcuts and end up teaching them behaviors that would deteriorate model performance in prod. it is crucial that in all cases, the behavior and the distilled principles of that behavior is correct.

(2) is the task fair? now this seems obvious, but can be non-trivial, and actually extremely detrimental if done incorrectly. in an experiment i ran, we SFT'd a model on data that had 25% of examples that included hallucinated candidates (in a name-matching experiment). as a result, the model in prod hallucinated names far more frequently, often times using the names from the training data. while this was a self-contained experiment, in practice, RL tasks that have fields or points that are unreachable without getting lucky or guessing (like the GDP-val verifier for example), can push models to hallucinate. grounding all aspects is unbelievably important.

(3) does it discriminate properly? pass/fail tasks seem to struggle with this the most, but most vendors/benchmarks assign gradient weirdly too, so lose-lose rn lol. but the point here being, if a task is pass/fail, and a model does 0% work and gets a 0, vs 90% and gets a 0, the model learns nothing. however, you can harness gradient signal by including partial-credit verifiers. this if done correctly will make tasks discriminate properly, the only issue being, people artificially drop criterion to get desired pass rates or adjust their tasks to fit certain specs for labs without holding the sanctity of the grading. the issue being you expose yourself to (1) where you're missing checks that would normally enforce regular behavior, etc. i'll go more into depth into this later.

making data is an art. it's important we treat it like that and don't desecrate what we train models on.

[Daniel Rupawalla @rupawalladaniel]

i'm conflicted about pieces like this [https://t.co/BLgAvRcgrB] because on one end it pains me that LLMs are stripping people of the outlets to do magical work, especially in math and science. however, it excites me that we are in a place where the rate of scientific innovation is going to be exponentially increasing, and the amount of new discoveries will expound faster than ever. it excites me even more that through models, the barrier to entry is far lower and the curve on being able to learn, publish, and iterate is getting faster.

in my eyes, it's a new age of becoming a leader of a "country of geniuses" beyond just being the genius only: being able to leverage agents to do things no human could ever. i do still strongly believe it's important to learn and understand concepts yourself (and feynmann's takes on understanding are even more relevant when instructing LLMs for discovery)

but then again, there is nothing i can do to stop the improvement of models, i will only contribute to their advancement, so we may as well be optimistic about it.

[Daniel Rupawalla @rupawalladaniel]

i got a text about if im worried about “giving away alpha” through the recent string of data tweets. i appreciate that, but im a little offended.

(1) i care far more about advancing models and everyone not serving bad data, than me personally doing so and gatekeeping from everybody else. collective benefit. a rising tide raises all boats. data is not a winner takes all market, and it never will be. moreover, science should be focused on advancing all of society, if bad data = slightly worse model = it takes an extra 6 months for us to solve certain loadbearing problems, that’s a net negative for everybody.

(2) if all it takes is one guy tweeting about how bad data is for you to fix your production / get into the data space and do it well, every take ive had about data being hard becomes irrelevant.

the moat in data does not come from proprietary knowledge imo, its more from rigor of quality, understanding and experience with data, and deep connections with the buyers. none of which can be distilled from anyone’s twitter.

[Daniel Rupawalla @rupawalladaniel]

"from the people that* have sent me...," wrote it on my phone

[Daniel Rupawalla @rupawalladaniel]

my takeaways from the people have sent me their datasets in the last two days to QA:

(1) there’s a fundamental misalignment with how people approach data creation, and with what’s useful, and im not entirely sure labs have been able to QA this given some of the datasets being bought. specifically, the understanding of what makes data have good signal: ive seen a LOT of “needle in the haystack” tasks where a corpus is really big and the only way the model solves it is by reading thousands of messages & docs. think about the behavior you’re training. it might be hard and stochastic, but the behavior you’re teaching a model is to ALWAYS search for that long given a big corpus, i.e. work harder not smarter. this is fine in some cases, but you’re inherently making LLMs less token efficient and how you get erroneous random api calls when trying to do corpus search in practice. this underlying principle applies to many other types of tasks that ill enumerate in other posts

(2) a common misconception is that long-horizon = good. it’s a buzzword in the industry. for non-science, non-code, non-math tasks, it’s unclear to me how to do long-horizon tasks extremely well. the large reason for this being that knowledge work doesn’t happen in a vacuum. you don’t send out an email and operate on it unless the world responds to you irl. thus, either you need to have LLMs simulate the convo (TAU-style) or a deterministic response or have the LLM operate in isolation. (A) and (B) are hard to do well, (C) is unrealistic. this is a problem ive been thinking about quite a bit, and if anyone at a lab has thoughts would love to hear it. im sure there is a way to do it well but its uncertain to me if its been done.

[Daniel Rupawalla @rupawalladaniel]

@_zach_richards_ @sid_potdar_ put more contractors on it

[Daniel Rupawalla @rupawalladaniel]

it's insane to me how many people take benchmark scores at face value. i spent 2 minutes looking at GDP-val tasks and realized there's a verifier item that checks for:

" The Word document includes the warehouse phone number 560-555-3867 (accepts common US formats such as 560-555-3867, [phone], or 560-555.3867)."

but the prompt has:

"Include the address and phone number for Gravon Shoes’ warehouse, which is 555 Waters Avenue, Austin, TX 78726, phone number 455-864-3867."

the lack of good benchmark automated verifiers is jarring*, and most code benchmarks tend to lack the same verifier quality. this isn't to say GDP-val isn't useful or decent, but stuff like this is there across the benchmark if you dig deep. the "good enough" bar for data is finally starting to catch up to model performance. we need higher quality data, we need higher quality benchmarks to track improvements. my biggest peeve with how people construct tasks is that they mistake difficulty for signal. just b/c your benchmark has a 10% pass rate, doesn't mean it's good, and often, to make a difficult benchmark, people jump through loops. it's absurd that labs aim to hill-climb all these benchmarks but many have issues that would bar a model from getting better, so either you corrupt your model to improve or buy a bunch of data that doesn't actually help and look bad on your team. we saw this with OAI's paper on swe-bench, then subsequently swe-bench-pro, and im sure we'll see even more "benchmark debunks" soon.

making good benchmarks is hard, models have saturated most of the easy stuff and indie benchmark developers or contractors are no longer the best way of approaching it, because you can't have a hodgepodge task that a model will fail on. verifiers MUST be carefully constructed, the worlds the models are playing in must be realistic and without leakage, context has to be carefully selected. we're at a point now where 99% (arbitrary number) are saturated, and it's not "experts" but real expert engineers who understand what makes good training data.

*i understand that GDP-val specifically is graded by human experts, but for anyone running the tasks at home, it is still nuts to put out verifiably incorrect rubrics especially because others will use it.

[Daniel Rupawalla @rupawalladaniel]

interpretability work is super interesting. ran an open-model (laguna XS 2.1) and read through the J-lens of the model to try and analyze the failures in coding problems. in a few cases where the model messes up, the word success shows up prematurely, leading the model to think it was correct before it actually was.

to combat, injecting the word wait and/or urging the model to check its work more frequently and in depth leads to better results (on a small sample). early success tokens hurt results, neutral tokens didn't help when injected, and the wait token improved results where early success was common. i think the implications are likely more relevant for non-code problems (as code and math can be verified deterministically). by pushing LLMs to cook for longer at a token-level, seems like it performs better (and the activation space changes to urge the model to do more work). would be curious if anyone's done anymore work looking into the J-space paper from ant (https://t.co/RciwoDXZuq).

initial experiment inspiration via: @venkat9165

[Daniel Rupawalla @rupawalladaniel]

@ArjunRajJain if the answer was able to be put into a tweet, we’d all be millionaires. hard to tell what works at scale, no one’s really figured it out well for rl envs 😕, surge solved for preference data

[Daniel Rupawalla @rupawalladaniel]

sparked from a conversation i had earlier today:

unless you have some sort of moat to the space (i.e. worked at a lab + have some deep speciality in a niche space), or have worked in data for > 1 year and understand the space, it's unclear to me that starting a data company right now is a good idea, for the following reasons:

(1) arguably the most competitive space to be in right now. although no one's really figured it out save for a few vendors in niches like code, etc, it is unbelievably hard to make good data. pretty much every benchmark released has some kind of issues, quality checks at most vendors are subpar because they don't understand what to check for. just because labs may buy what you're selling, doesn't mean the data itself is good lol. synthetic data is interesting but it's very, very difficult to do without creating bad data. i have only seen a few people do synth well, and they've been in the data space since its conception.

(2) if you're convinced data is the place to be, and you NEED to found in data, i would just join a pre-existing company. selling to labs is increasingly becoming a game of who-knows-who, and labs are rightfully becoming more skeptical of new vendors (considering most people are producing slop). breaking into this industry when there are so many dominant players is difficult without having a moat like the one mentioned above. instead, if you're convinced you have the necessary skills and want to print cash, it's better to join a company that can give you the upside, without wasting the time to build the relationships and burning months. if this is your plan feel free to reach out and i can connect you with a company that fits the profile you're looking for/help you broker.

(3) if you're young, although it may seem like a cash grab, i don't think you learn that much being a data monkey outside of data-specific problems. the exposure is minimal and if you're doing it with contractors, you are inherently an operator, not a researcher/engineer. the caveat to this is companies that are doing novel data types and are pure engineers, but even amongst those, if data is the only end result of the company, you are not learning that much beyond the walls of data. this could be your calling, but <25yo who want to be researchers, this doesn't seem like an ideal path.

naturally, i think there are always exceptions to this rule, but i've been hit up by so many YC companies or friends of friends who've founded in the space and pivoted out after 6 months. save your time, join a company, learn from the amazing engineers you're surrounded by, not everyone needs to found, especially in this space.

[Daniel Rupawalla @rupawalladaniel]

through making data, it's become abundantly clear to me that frontier intelligence is clearly not as accessible as people or labs think. i've become more convinced that LLMs are capable of almost anything reasoning-wise, it is just extremely difficult to use them at scale in ways that don't compound negatively.

(1) although i don't like people who shill prompt-engineering, there's merit in optimizing prompts & systems, and employing deterministic checks with LLMs to ensure drift doesn't happen, especially in long-horizon engagements. it's jarring to me that people bank on "intelligence" improving to cover for underspecified prompts when most workflows they'd want can be done by a much "dumber" model that has a good harness (or in unique cases, a much smaller model specifically post-trained for them, both of which are cheaper than using frontier intelligence). also, more intellect won't fix a lack of context, which seems to be most problems people have with models. given the specific context, setup, and harness you're using, you should be able to push a model to be better than itself (for me, the clear answer is if you can set up opus to make a fair, verifiable benchmark opus would fail on fully autonomously).

side note: self-distillation/synth data is interesting, and being able to make tasks that work for RL at a frontier level is non-trivial. it's signal that you're able to use LLMs well and create some sort of capability asymmetry (through only prompting and a proprietary setup, you've made the model better at verifiable tasks, i.e. more capable than the base model. this assumes the reason it's failing is not tribal knowledge, contamination, etc though).

(2) it's generally hard for people to be an "operator" for 5+ agents all at once. from engineers i've spoken to: by delegating too much to agents, you end up with shoddy work in multiple things, instead of good work in one lane. the slop compounds over usage, and eventually you get bloated repositories or work that you aren't able to comprehensively understand. i think this also breeds a sense of complacency in that you trust in models to be correct all the time, regressing your own capabilities to iterate and improve. not to say don't use AI, but more to say use it in a way that levers your own intellect and "taste" rather than recursive slop loops and thumbs up without understanding the work. in the same way models need context, as we lose context by not being in the weeds, our capabilities to monitor the agents regress as well.
i think orchestrator agents work well if done right, but very dependent on the person and how you have it surface information. this seems like something frontier labs will have to fix though, if the way they want AI to be consumed is en masse with hundreds of instances spun up.

on (2), this is a pretty widely known concept [https://t.co/MIbmKlXU64] but think it deserves restatement for LLMs, especially considering failures are so much harder to notice, and the skill loss is far more upstream.