Daniel Rupawalla

RL research | prev. acq by US News / ex-YC

[Daniel Rupawalla @danielrupawalla]

https://t.co/Vhm7Diohq5

[Daniel Rupawalla @danielrupawalla]

my lessons on data/eng from training small models frequently for different enterprise use cases:

(1) small mistakes in the data will cascade. i've noticed that it becomes abundantly clear in the model's behaviors. key examples:
- hallucination becomes common if the data includes defects, this is the most common and most detrimental one in my mind, and can be caused by any number of reasons.
- dropped recall on specific items that the model must include in requests
(2) LLMs cannot produce even simple data at scale without a well-designed harness and deterministic checks. even SOTA models are unable to follow simple synth-gen requests without either hallucination or some sort of failure in following policy. my advice here is to understand your end-goal extremely well and create scaffolding (specifically, unit-test style testing & deterministic beats for initial seeding) on each piece of data in your pipeline. then, only use LLMs to humanize the data. to be clear, this is only for simple data for small models, and an entirely different system is needed for SOTA model data.
(3) inference engineering is powerful and interesting. after training this should be your main focus on increasing latency, it is absurd how fast you can get models you fully own e2e.

[Daniel Rupawalla @danielrupawalla]

@Liam_Cashel if it was that easy / deterministic, data vendors (and subsequently labs) would prob catch it in house, otherwise, I wouldn’t even consider them a vendor lol. also doesn’t make sense for labs to outsource that kind of alpha, anything to get an edge against everyone else right

[Daniel Rupawalla @danielrupawalla]

lol i saw a tweet about separating task QA and generation into two separate companies b/c it'd be more efficient. terrible idea for the following reasons:

(1) every data vendor would just create bad tasks and offboard it to the QA company to fix
(1a) if the structure is that the QA company reviews and sends it back to the vendor, the efficiency increase is effectively null and you’re just adding more operational overhead by introducing a third middleman

(2) why would labs trust the people who review the tasks any more than the people who create them. you would add a second review layer (that may?) reduce back and forth between labs and reviewer, the same ping pong exists between reviewer and data vendor anyway

(3) QA IS the hard part of task creation, anyone create tasks, not many people can make it good. if you rip the QA out of the vendor, you rip the only valuable part out of the company. without it, you have a band of monkeys clacking keys. also, it's much easier to QA a task you created, and most IR's aren't even good. leveraging LLMs + domain knowledge is the best once you understand the system in depth.

(4) labs will need to internalize better QA systems because that's the only way to enforce standards on vendors, if they out-house their ability to QA data, they lose a massive edge in being able to decipher who's good and who's not.

amongst many other reasons.

[Daniel Rupawalla @danielrupawalla]

@dubrovnik07 i specified the contractor setup doesn’t work. agree that you can use prod data for this too instead of synth (prob better in terms of realism)

[Daniel Rupawalla @danielrupawalla]

there is a lot of buzz on how data companies are actually research companies, etc. in theory, i agree they should be. however, i wouldn't say most are as sophisticated for the following few reasons.

the contractor format of task creation doesn't track well into doing real research, as the people building the tasks would not be involved in the science. this is important because:

(1a) the contractors do not understand data, and thus cannot structure tasks in maximally optimal formats for signal

(1b) contractors do not have company buy-in and have no incentive to create tasks that are good/increasingly better

(1c) diffusing scientific principles across hundreds/thousands of people usually does not work and is hard to enforce at scale

(1d) contractor farm = mostly ops people = low engineering talent/engineers are second-rate citizens at the company and thus most engineering talent will be worse = less research = not a research partner eod

there are data companies that have done this well, by being pure engineers and researchers who understand data. however, this is much harder to pull off, and most of the startups that are <10 people aren't big enough or old enough to serve labs as a research partner yet. few companies who CAN fill this role exist now, but more will pop up and i think they will approach data through the right lens. i wholeheartedly believe data is an engineering problem and that synth data will get better as long as the foundations are built right. the issue is most people haven't spent long enough around RL data to understand what those foundations are, or spent time at the wrong companies so can't tell quicksand from concrete.

[Daniel Rupawalla @danielrupawalla]

@massivedan69 not worth it imo, get good engineers instead

[Daniel Rupawalla @danielrupawalla]

i think the lift for an engineer to understand the real work is lower than a worker to understand the engineering. also, if it’s 1:1-1:5, you cannot scale, it works great in theory. but theres a reason you need ops people to manage contractors and they don’t interface directly with eng

[Daniel Rupawalla @danielrupawalla]

ah yes agreed, i think there's value in working very closely with specific labs to develop your taste and understanding in data. i think there is also valuing in being extremely opinionated and developing your own abilities. but working as a design partner can help improve your team for sure.

in many ways it's a very enterprise-sales motion to work with labs unless you are a better researcher than they are.

[Daniel Rupawalla @danielrupawalla]

@sensho explain the second half more?

do you mean be opinionated on what tasks should be and don't build for labs?

[Daniel Rupawalla @danielrupawalla]

@zero_goliath niche pull, favorite show

[Daniel Rupawalla @danielrupawalla]

@aisafetycompute yeah, my point is just, if you're good at QA, be a data company, don't just QA, the hard part of data gen is the QA piece.

[Daniel Rupawalla @danielrupawalla]

sure, but there are n<15 total labs (n<6 frontier) so the amount of specifications needed are limited. also, no trusted "QA" layer is that easy to build, if it was, every data company would probably have this already & labs would never outsource QA b/c it doesn't make sense for them. data QA is a part of training

[Daniel Rupawalla @danielrupawalla]

@mongo_lassi dm’d!

[Daniel Rupawalla @danielrupawalla]

@DominicColligan it’s hard to do manually, pretty boring if you’re doing it manually, and absurdly difficulty to do at scale

not to mention you don’t learn a bunch either if you’re only doing data

[Daniel Rupawalla @danielrupawalla]

"why don't labs just in-house RL data creation"

they are starting to for synthetic data. i don't doubt there will be bigger pushes to in-house this over the next year but:

(1) it's illogical for engineers who are good at data to want to join labs. instead of joining a single lab why would they not just sell to all vendors at once and print. i.e. labs can't get the best talent for this market, so vendors will outpace in-house production in quality and speed.

(2) if there is any issue in the creation process for the lab, and they rely unilaterally on a single data vendor, their model will feel the results and they've put out an unstable model. inherently, by splitting your data vendors you amortize issues over large sets of data. also competition in the market makes it so there is a greater incentive to innovate on data, vs. in-house engineers have a smaller fire under them to work harder.

(3) niche datasets are hard to get, medical patient data, private financial info, etc. the datasets that haven't been trained on yet require non-linear acquisition methods, something many labs cannot do easily.

(4) there are investments into certain infrastructure that make it hard for labs to replicate what data vendors have chosen to specialize in (robotics is a good example). naturally, as a lab you can probably buy your way in, but it's more of an investment than just buying the pre-made data.

(5) the work is unsexy and many lab researchers find it painstaking and not fun. 99% of people aren't excited by the rote work of creating envs.

there are probably more reasons i can't think of immediately as well, but should provide some context to this repeatedly asked question.

[Daniel Rupawalla @danielrupawalla]

@dlobaton2 @tanayj seems like i was spot on then!

[Daniel Rupawalla @danielrupawalla]

i don’t think the mechanize acquisition is impactful to most RL data startups because they were very much in their own niche for code. most other RL data companies were not / cannot make tasks as long or high quality as theirs. though they scaled very manually, they scaled well

[Daniel Rupawalla @danielrupawalla]

i think there’s a pretty big misconception related to synthetic data.

in most cases, synthetic pre-training data seems to be low quality and mass-produced (given it’s not synthetic off of real data, but often times this is still bad). the pre-training data market seems to me to be a lot more like the wild west where real datasets are sourced from who knows where…

on the other hand, although most people can’t do this well, post-training synth data is a lot higher quality. i think it stems more from asymmetric information disparities between author and model. my mental structure surrounding post-training data is having behavior A that we as humans (or as model + human / model + harness) know is correct (verifiable through deterministic / LLMaaJ checks).

[Daniel Rupawalla @danielrupawalla]

@aisafetycompute most new gen data co’s aren’t

[Daniel Rupawalla @danielrupawalla]

@tanayj hard to say with certainty, i’ve never worked there