Rushil Bhat

[Rushil Bhat @rushilbhat]

@sdand @thinkymachines @axiommathai Most likely. Nous’ results are pretty close.

[Nous Research @NousResearch]

Today we open source Nomos 1. At just 30B parameters, it scores 87/120 on this year’s Putnam, one of the world’s most prestigious math competitions.

This score would rank #2/3988 in 2024 and marks our first step with @hillclimbai towards creating a SOTA AI mathematician.

[Rushil Bhat @rushilbhat]

https://t.co/fw0vYID8d7

[Rushil Bhat @rushilbhat]

While writing a minimal TP setup, I noticed the loss was dropping sluggishly and the gradient norms were unstable vs the single-GPU baseline. Turns out the way I parallelised CE loss ended up erasing gradients.

New post on what went wrong + how to fix it.

[Rushil Bhat @rushilbhat]

@maharshii Useful primer on sinusoidal positional encodings:
https://t.co/2K1U5VoU5H
https://t.co/SAaegHPzEX

Nice math-focused walkthrough from @A_K_Nain:
https://t.co/mPUJMyUwjx

[Rushil Bhat @rushilbhat]

@vega_myhre The bit on parallel cross-entropy loss is really useful - can get super fiddly. I had a bug while replicating f.log_softmax + f.nll_loss like f.cross_entropy since the gradients weren't flowing back through the normalisaton factor on ranks without the predicted logprob.

[Rushil Bhat @rushilbhat]

@fabknowledge @dylan522p @asianometry S-tier audio quality

[Rushil Bhat @rushilbhat]

https://t.co/rEJZDxWNTj

[Rushil Bhat @rushilbhat]

Inspired by @jacobrintamaki's posts, I've sketched out some ideas about what could help maximise the project's impact.

@eladgil @shreyanj98

[Jacob Rintamaki @jacobrintamaki]

Made another post after having a few conversations with the great @shreyanj98 on evals for this project. It made me believe that to have great evals (if you're paranoid) you need to go all the way back to how you source your PDFs. link below

[Rushil Bhat @rushilbhat]

The recent Kimi paper gives really nice intuition on why MCTS failed for R1.

In AlphaGo the value function was necessary to steer the system towards a strategy that was difficult to exploit—optimal for adversarial play.

For language models straight-line drives of optimal actions seem actually counterproductive. ‘Mistakes’ allow for reflection and backtracking, increasing inference compute by extending the CoT and triggering more forward passes.

[Rushil Bhat @rushilbhat]

@nrehiew_ @hyperbolic_labs Any reason why TRL's implementation omits the clip function constraining the policy update?

[Rushil Bhat @rushilbhat]

https://t.co/mf0D4awNmy

[Rushil Bhat @rushilbhat]

New post: Investigating Flat-Parameter FSDP

[Rushil Bhat @rushilbhat]

https://t.co/5sY60sEl2a

[Rushil Bhat @rushilbhat]

New post: Counting backward FLOPs for GPT-2

[Rushil Bhat @rushilbhat]

Looking to migrate fully browser-side by using whisper.cpp's WASM port.

Challenges:
-Extracting audio from YT in a browser context.
-Executing the WASM module in a content script or service worker.

If anyone has any insights/solutions, please reach out. I'd appreciate the help.

[Rushil Bhat @rushilbhat]

Was watching this podcast of @nitin_gadkari with @ANI. I'm comfortable with colloquial Hindi but words like "aarthik" and "svadhinata" had me constantly pausing the video and using Google Translate.

Decided to build an extension to caption YT vids in real time using whisper.cpp.

[Rushil Bhat @rushilbhat]

Current setup - leveraging whisper.cpp server-side for transcription to avoid Whisper API costs. The extension then auto-injects the captions in real time into the YouTube player.

https://t.co/xSXiwnAgwn

[Rushil Bhat @rushilbhat]

@AaronPTweets @_buildspace Absolutely, optimising for different types of companies is crucial for this project. I'm thinking of incorporating LinkedIn profiles, as that could give insight into whether the developer has experience working with startup/larger organisations.

[Rushil Bhat @rushilbhat]

@0xSaura @_buildspace Thanks. Right now, I'm focused on building a web application that takes job descriptions as input and outputs a list of developers who best match the requirements, along with highlights of their relevant projects.