Parul Sinha

21 // CTO @oxfordunion // Building with interp and model science // Law Undergraduate @uniofoxford

[Parul Sinha @itsparulsinha]

Some miscellaneous (and increasingly specific) stuff about me:
- My name is Parul, I am 21

- I am currently a summer researcher at @law_ai_ . I look at the intersection between technical ML and legal/policy
- There, I am working on architectural changes that can make AI systems more inherently law following
- technical interests currently mostly lie in mech interp led RL improvements for model customisation

- Recently, outside of this work I have been thinking about AI security + parallel risks (that many institutions could be exposed to) of implementing similar multi agent stacks (Multi-Agent systems being somewhat less secure than individual agents). These systems are super useful in deployment but I am currently pondering how to make them more secure

- I have a bunch of legal interests and am currently a law student at @UniofOxford! Dm me if you would like to chat about the relationship between the rule of law and "the social contract", or have thoughts on legal realism vs formalism

- In my free time, I like to sing & write songs. I have been doing it since I was a kid and have a past life as a child actor, theatre kid and wedding singer. I play guitar (badly) and find a lot of joy in creating and sharing music. I love getting new music recommendations to listen to

- I don't think in words pictures or sounds, and I think that this really impacts my experience of living and processing the outside world. Basically I have aphantasia and anauralia. It's non-communicable brain vibes up there! I only realised this was abnormal maybe a few years ago, and thought that "internal monologues" in movies were just a lazy narrative device. Because of this I really enjoy hearing how other people's brains work and what thinking is like for others.
- If you would like to discuss what it feels like to be in your brain, DM me I would be very interested to hear
- I was very into neuroscience as a kid and had a childhood interest in neuromodulation treatments and did some interning surrounding ECT while I was in school. I eventually decided not to go into medicine, but I think this is where my interest in interp comes from

- I think a lot about what it means to be high agency and just do things that interest you and bring you joy. I only started thinking this way maybe a year or so ago so it has been a recent change for me: just ask for things// have agency// take big swings. I have started thinking a lot more about what exactly brought about this change in how I approached things after a conversation with @YesThisIsLion at the @UniofOxford before our AI Personhood debate this summer about finding joy in innovation and finding your own agency. I still try to balance this with a healthy dose of pragmatism in my day to day life but enjoy things that bring whimsy, serendipity and novelty to my day. I think this shift in mindset has been one of the most substantive things to happen to me in the last few years due to the ways I use it to shape my experience of and choices in the world

[Parul Sinha @itsparulsinha]

gonna go back to Yarm
Yarm is a great place to do kernel work

[Parul Sinha @itsparulsinha]

@xavicfu Good luck!! It may end up being equivalent to ancient/modern history. Good for understanding things and seeing patterns in the world but perhaps not *directly* applicable to your job (if those still exist)

[François Chollet @fchollet]

I hope the proposals by frontier labs to pace AI research stem from a genuine concern for safety and a recognition of the potential risks posed by future models, rather than a strategic effort to consolidate power and permanently solidify the market dominance of a few top players.

If these efforts are genuine, then oversight must take a more democratic and accountable form, with both national and international components, similar to the Nuclear Regulatory Commission in the US and the International Atomic Energy Agency internationally.

If there is only one organization responsible for safety monitoring, and it happens to be staffed by the same people as the frontier labs, and it is perfectly aligned with them both by incentives and by ideology, then it would be indistinguishable from letting frontier labs self-regulate and self-certify their own safety standards.

Two potential warning signs of regulatory capture to watch out for:

1. Calls to ban open-source AI. Open-source development currently serves as the only real counterweight to the dominance of frontier labs.

2. Attempts to hinder non-frontier research. Models that are one or two generations behind the frontier -- whose level of capability has already been deployed at scale -- are empirically known not to pose safety risks.

As long as we don't see 1 & 2, and we see real international coordination efforts, then I will optimistically believe that the proposals are benevolent.

[Bahrad Sokhansanj @bahradx]

This is totally unconfirmed. But it's important to recognize that, as I read them, the general policies at the frontier AI companies don't prohibit this (and I've advised many people of this). It's certainly likely the case that there are enterprise agreements that do go farther.

For example, I've attached a screenshot from Harvey, which I think is a decent example of a company that both fine-tune models and sells services based on API calls to model providers, and which works with customers who demand clear policies.

Of course that still means that there's data retention at Harvey or those third parties, but they'd have an obligation to strictly limit access to that data.

[Parul Sinha @itsparulsinha]

I’ve yet to see anyone ask: who owns the IP of the recent proof of Navier-Stokes?

It surfaces a much broader question for any future commercially-valuable R&D conducted through closed-model providers.

“We didn’t train on the raw user data” isn’t necessarily the end of the legal analysis if derivative/synthetic data ultimately carries commercially valuable unpublished research into later models. There’s no concrete evidence that this is what happened with Alpöge & Buckmaster’s unpublished work but I haven’t seen OAI refuting this as of yet.

For patentable work you potentially get difficult questions around derivation, inventorship, trade secrets + contractual scope if the provider later independently “discovers”/commercialises the same thing.

Increasingly strong argument for doing genuinely novel R&D on open-weight models running locally/on-prem: for commercially important innovations, you may want the entire inference/data pipeline inside your own trust boundary. Whether to go with closed or open weight is now potentially a part of your IP strategy.

Separate from all of that: it’s insane that capabilities have moved this quickly. Models are meaningfully pushing the frontier of mathematical research.

[Aidan Gomez @aidangomez]

Synthetic data derived from production user data of consumer AI tools is used for training. I’ve heard this rumour from both large labs’ employees.

In particular, if you’re doing something “interesting” like working on complex math/business/software/bio problems you’re dramatically more likely to get trained on because they filter/up-weight towards those usecases where the model has the most to learn.

Even in ZDR and “we won’t train on you” regimes, derivative data is usually carved out. The promise is only not to train on exactly the data you put in, rewritten data is fair game.

[Parul Sinha @itsparulsinha]

I’ve reliably found good mentorship in London but frankly a lack of opportunity. Network effects of being in Cali/the bay (even groups like GBX where everyone is a Brit in the states) are insane if you’re somewhat social. Seeing the sheer velocity of the people around you is both a great motivator and makes it easier to succeed. Unsure either investment or more mentorship can really fix the structural problem here.

[Parul Sinha @itsparulsinha]

I have mixed feelings on the whole topic - having a piece of knowledge work that you enjoyed (and was economically valuable) get automated can be… jarring. I have found a lot of joy in the way that AI has accelerated me and allowed me to quite quickly become more technical, but I do miss the future where I could rely on being able to conduct legal analysis to make a living. Pros and cons of living through an Industrial Revolution!

[Parul Sinha @itsparulsinha]

As long as closed is still ahead (capability wise), KYC makes a lot of sense. “I can use another worse tool to do something nefarious without an ID, why shouldn’t I be able to use a better tool to do something nefarious without an ID?”

A valid argument might be around privacy concerns

[Parul Sinha @itsparulsinha]

I completely agree. From a policy nerd perspective, I think that it was clear @deanwball was making a prediction - also it’s good that he pointed it out so that we can be aware of regulatory capture in the discourse! “Dean works at OAI!” is not a valid criticism of his argument.

I may not agree with everything that he said, but it seems that people were not analysing the merits/demerits of his statements.

[Parul Sinha @itsparulsinha]

@adumbthumb @deredleritt3r You could just pay for multiple accounts, or create burner accounts to avoid suffering the effect of a ban for dangerous usage

[Parul Sinha @itsparulsinha]

I think there could be a good chance (with fable-esque aggressive classifiers, likely). If other labs don’t release full frontier research capability publicly it seems unlikely from a business standpoint that OAI do this (potentially fuelling competition, i.e. fable not helping with frontier ML R&D). Potentially if open source capabilities increase to be more in line with frontier, OAI would have to release to stay competitive

We can likely still be hopeful but there seem to be a myriad of factors that indicate we will get a quite watered down version of the model, if anything.

I agree that it’s all quite fluid at the minute and could very well change.

[Parul Sinha @itsparulsinha]

I wonder whether we will see a sharp rise in American/western open source. If this IS the most viable business model it would make a lot of sense.

Companies like @thinkymachines and @cohere are already leading the way. It’s a shame that @Meta seems to have deprioritised open source, last year I really loved building with Llama. Lots of privacy benefits to deploying open source, it is quickly becoming MUCH cheaper, and as a model provider you don’t have to pay for inference compute.

Also, I am constantly impressed by the ingenuity of labs like deepseek in the technical improvements they put out due to how (comparatively) resource constrained they are. I agree that the ecosystem likely benefits from both open and closed source models coexisting, so this isn’t to say I’m anti-closed source.

As funders see the proof of concept as validated by labs like @moonshot I hope to see more open source led labs crop up in the west.

Whether this is desirable from a policy/risk perspective is a completely different question.

[nic carter @nic_carter]

havent seen one person from OAI or Ant address Jon's argument here.

the point is simple: the USG does not owe either of the large labs a business model. if the economics of selling tokens don't work due to distillation/cheap clones/Chinese AI magick, the American enterprise and consumer will be A-OK. they will benefit from hyperdeflation in the cost of digital cognition just like everyone else. the hyperscalers will be fine. it's just OAI and Ant that won't be – in their current forms at least. if they are willing to adapt, they can develop new business models.

so what if the token merchants don't do well? the neoclouds will be fine. the internet companies will be fine. the consumer gets cheaper queries. the enterprise will still incorporate AI.

the only world in which this isn't fine, is if you hold a quasi-religious belief that we're on the cusp of a kind of AI rapture in which one of the labs Logs On And Wins Forever, namely hits RSI and we enter some kind of sublime post economic society run by GEOTUS Dario. so to accept that Ant's business model might be suboptimal or impaired by China's commoditization is to accept the unacceptable; namely that someone other than the anointed might kick off the runaway feedback loop and that they, instead might log on and win forever.

this appears to explain the discrepancy in reaction to Deepseek Moment v254 Kimi edition. everyone has bag bias, of course. but leaving that aside, most people think it's pretty much ok if Ant and OAI suffer margin compression due to Chinese distillation / industrial sabotage via open weight models. the American economy is not reliant on those two firms. they could blink out of existence and we would pretty much be ok. the AI capex supercycle will still produce tokens, closed weight or not. American firms will consume those tokens. OAI and Ant would probably still scratch a living, due to the latent preference of some token consumers to buy domestic and face off against a known entity.

this is only unacceptable if you think AI is strongly path dependent; that is, if it really matters who the market leader is when AI reaches a breakout level of capability. this is true both in the good case (superintelligence, singularity, etc) and the bad case (this is the essence of safetyism). but if this sounds more like wishcasting than forecasting, you probably don't mind the labs being pressured economically.

now you can clearly tell which side I'm on. I think AI is a fantastic technology which is hyperdeflating the cost of cognition and will fundamentally reshape society but there are real reasons why it wont diffuse as fast as the AGI people think it well. I would prefer an American firm achieve RSI relative to a Chinese one but I think either outcome would be suboptimal; better that we don't end up with a closed oligopoly composed of Ant/OAI. China by crushing the margins of the labs is doing everyone a favor by eliminating their pricing power and empowering the buyers of AI, namely, everyone.

objections:

-but you can't celebrate America losing to China!
- in my opinion this is a minor victory for China but not necessarily an enduring one. USA still has the chip, datacenter, and neocloud advantage, not to mention, it still has the best frontier models. Chinese labs releasing open weight models have no business model of their own. so even if they hurt the US labs, they have nothing to show for it. it's profoundly unlike their successful dumping campaigns with solar panels, batteries, drones, etc where they eventually built big domestic industries. (if China kills American AI with open weight models, we can even the score the moment they try and release a proprietary model). even if open weights win, the USA can still leverage AI extremely well and potentally retain the aggregate compute advantage. yes, the US would be more assured of victory if OAI or Ant won forever, but I don't know if I want to live in that world.

- no one will ever train a model again
- this is where I think the concern is unwarranted. let's say distillation really is a golden bullet and kills big training runs. that doesn't advantage either China or the US. that's a stalemate. not to mention, the trend seems to be less focusing less on massive pretraining budgets and more on finetuning for specific genres of tasks, thinking machines style. and lastly I find it hard to believe that training runs will stop altogether. the labs can probably develop anti-distillation techniques. you could adopt a whitelist style permission for everyone using your model. different consortia could be put together to share in the cost of training a model, if it is seen as too expensive for an individual firm.

- the AI buildout is path dependent and OAI/Ant are now load bearing GDP infrastructure
- it would be a significant setback for investors if they had to cancel their IPOs and suffered big markdowns, and some neoclouds with lab based RPOs would suffer for a while, but everyone would be fine, really. does Microsoft need OAI or Ant? does Meta? does Google? ordinary Americans have ~no exposure to either OAI or Ant. would the world want any less compute if it turns out to be another order of magnitude cheaper? certainly not. as we all know at this point, consumption would go up. I don't think the economy is so dependent on the labs that it couldn't handle their margins compressing.

[Parul Sinha @itsparulsinha]

Yes! I also think that students are often not encouraged to think of these “side doors” in the right way - aimless “networking” probably won’t help you, but if you find a passion and a niche and try to meet people through that or prove your worth in other ways (more tangible than sending CVs off into the void) it can be very fruitful.

[Parul Sinha @itsparulsinha]

I guess there is a broader trend of things becoming more convenient to achieve (in a disruptive way) that means that you can both make implementation much easier but easily erode your ability to think and be intentional (the two things the original “struggle” that was replaced by tech would help you work towards)

[Parul Sinha @itsparulsinha]

This forms a big part of what I care about too - art, beauty, happiness, serendipity, doing good for others, feeling I have added to the beauty of the world. I think you need to have a good sense of “emotional sovereignty” (feeling settled and sure of how you feel and the ability to introspect on it) to be able to rely on it.

I work on AI security/safety not just because of impact, but because I find purpose and joy from that impact

[Parul Sinha @itsparulsinha]

@dineshpaii Also, I think this may be a more “age old” problem than initially meets the eye. For example, I just saw this in a talk Richard Hamming gave in 1986. Complacency within your field seems to be a persisting issue that hinders great work!

Full talk here: https://t.co/SQqL4lMBLD

[Parul Sinha @itsparulsinha]

@natolambert @joshua_saxe I wonder whether it would be tenable to have govt/govt adjacent but NGO talent (and funding to retain that talent) that can genuinely understand and legislate for the frontier.

[Parul Sinha @itsparulsinha]

@teachrobotslove @TheLynx_1 Hard agree. The media pushes limerence/infatuation as love whereas I think the real thing is intentionally created, upheld and forged.

You have to agree on the broad points and have a certain level of compatibility to begin with but I don’t really believe in singular soul mates

[Parul Sinha @itsparulsinha]

This is really true and I would do much better to just get started on things asides from the mental block. I’ve been building routines that help me get into the habit of doing my “random procrastination” tasks in the morning that my brain finds painful to look at and it’s been wonderful