Vedant

Learning and Unlearning

[Vedant @NipaneVedant]

Ahh a poem after long long time..

Give it a try

On the same old street https://t.co/Hr3G30uoze

[Vedant @NipaneVedant]

"It might feel like focusing on the fundamentals instead of chasing novelty will hold you back, but strong science always wins out. The facts don't care about novelty. "

"It's important to remember negative results are just as valuable as positive results"

Summed it up very well!

[Cody Blakeney @code_star]

Training models is the best job in the world. It's important to remember this is supposed to be fun. It's hard, confusing 90% of the time, and almost constantly surprising. Its always crazy to me to see people talk about AI research work sometimes in such nihilistic ways. You get to melt GPUs for a living, enjoy it!

It's important to remember negative results are just as valuable as positive results, finding a strong baseline is important even if it means you don't get to run as many experiments, and always be willing to change your mind when presented with evidence.

It feels like the pressure to have big splashing success all the time is dialed up to 11 with all of the funding and attention in the past couple of years. It might feel like focusing on the fundamentals instead of chasing novelty will hold you back, but strong science always wins out. The facts don't care about novelty.

If you work on a team remember to be kind to everyone and be a source of joy. Everyone else feels the pressure of this moment as well. It means more than you think to crack a joke, or encourage people when they get a bad result.

[Vedant @NipaneVedant]

@Radhikakhanzode Ha ha.. Brobar

[Vedant @NipaneVedant]

Ups and downs
Ebbs and flows
Bursts and Silences

[Vedant @NipaneVedant]

@MayankGoel28 @jacobrintamaki Best adaptation!

[Vedant @NipaneVedant]

Hmm

[Pranav Mital @pranavmitl]

Really good. The words we use matter. https://t.co/SwN1UjPkDd

[Vedant @NipaneVedant]

Go to an extreme of a hell. To the depths. Reside there. Breathe there. Gather the gems. Next time, you’ll descend into another hell already knowing half of what it will ask of you.

[Vedant @NipaneVedant]

A spectrum of colours,
yet I see the two
That the world showed me!

[Vedant @NipaneVedant]

Few things boil down to how cleanly and quickly you rebound. Act. Fail. Take the feedback. Shrink the time it takes to recover.

[Vedant @NipaneVedant]

Theorize -> Ablate -> Experiment -> Evaluate -> Fail

Pause
Pause

Theorize the failure -> Ablate -> Experiment -> Evaluate -> ???

Repeat
Repeat
Repeat

[Vedant @NipaneVedant]

https://t.co/R0i3RJIFwq

[Vedant @NipaneVedant]

TDD -> EDD

[Harrison Chase @hwchase17]

eval driven development

[Vedant @NipaneVedant]

I mean, social image does matter. We are social beings, after all, so outright rejection of social norms is hard. Explaining ourselves is often the easier route to being understood and accepted. And therein lies another pitfall, I guess. We might end up explaining ourselves not because we want to be understood, but because we want to be accepted. So, let's just make sure we don't betray ourselves in the process.

I wish I had a one-liner for this, but alas, just rudimentary thoughts :)

[Vedant @NipaneVedant]

A bitter lesson, indeed!

[Vedant @NipaneVedant]

Tradeoffs | Balance!!

[Aalok Patwa @apatwa7]

I recently crossed 1 year working at OpenAI.

I reflected on doing good work, company culture, and the 5 biggest surprises from an action-packed year:
https://t.co/hoVOAgHzyL

[Vedant @NipaneVedant]

Witness / Observer?

[Omer Ghazi @OmerGhazi2]

Lately, I have been thinking about a very simple idea from Advaita: you can observe your thoughts, so perhaps you are not your thoughts.

Anger comes and goes. Fear comes and goes. Even our sense of who we are keeps changing. Yet there is always this strange awareness watching it all happen.

I find that idea incredibly freeing. Not every thought that enters the mind deserves to become part of your identity.

[Vedant @NipaneVedant]

"If you can meet with Triumph and Disaster / And treat those two impostors just the same"

AND

कर्मण्येवाधिकारस्ते मा फलेषु कदाचन। मा कर्मफलहेतुर्भूर्मा ते सङ्गोऽस्त्वकर्मणि॥

Looks like Krishna and Kipling conveyed something similar :)

[Vedant @NipaneVedant]

[6/6] Open benchmarks are only as trustworthy as the clarity of what they measure.

If you’re evaluating telecom LLMs or running leaderboard audits, it's worth knowing how the teletables task actually works on this leaderboard vs. what the source paper specifies.

@GSMA @Dorialexander

[Vedant @NipaneVedant]

[1/8] Speaking of flawed evals…

We have been evaluating our telecom domain-specific LLM on the GSMA open telco leaderboard benchmark and reached top-5. But our entry got removed.

Here’s what we found when we dug into why. 🧵

[Yafah Edelman @YafahEdelman]

We audited 15 benchmarks and labeled 9 flawed:
- In Terminal Bench 4.0 we found 45.5% of tasks to be broken after reviewing github issues.
- In HLE, 46% of the 48 questions we randomly sampled were broken.
- In DeepSWE 1.1 we found a bug that can break grading for every task.

[Vedant @NipaneVedant]

[5/6] The evaluation protocol difference between the source paper and the leaderboard isn’t documented anywhere in the repository.

We only discovered it when our submission was removed. While we are working with the leaderboard maintainers to republish our model results, we feel it is important to highlight such flaws.

Practitioners running evals against this benchmark have no way to know they’re measuring something different from what the paper describes.

(There are also broader questions about submission consistency across models on this leaderboard - but that’s a separate thread for another day.)