Vetha

Applied Scientist @Amazon • CausalNLP/Fairness/Privacy/Safety • Previously @Columbia @Adobe @IITGuwahati 💚✨

[Jack Morris @jxmnop]

people are underestimating what a big deal this is

this is the ONLY open-weight model that's trained without distilling from OpenAI or Anthropic

• Kimi distills
• GLM distills
• Qwen distills
• Nemotron distills (Kimi & DeepSeek, which counts)

basically a fully different tech stack. the first pure open frontier coding model. very exciting

[Jack Morris @jxmnop]

BREAKING: ArXiv changes toolbar color from #AE2A24 to #1C1A17

RIP "ArXiv Crimson Red" (1991 – 2026) 🟥 🔴 ❤️

[Vetha @Vethssvikas]

@natolambert very interested! Thank you!!

[Vetha @Vethssvikas]

@Sylvia_Sparkle @togethercompute Congrats Siyan!!! 🥳🥳

[Siyan Sylvia Li ✨ @Sylvia_Sparkle]

This work got the best poster runner up award at the AIX summit yesterday!! Thanks to all the collaborators!! ❤️

[Vetha @Vethssvikas]

@Sylvia_Sparkle Congratulations Siyan!!!!🥳🥳

[Joël Niklaus @joelniklaus]

Nice blog post by Jack Morris on embedding models!

Main learnings:
- Embedding models have not been scaled up as much as generative models
- Scaling data does not work well naively because inevitably we will get (wrong) gradients on false negatives
- Human labels are super expensive, but we can approximate them with LLMs.

Full procedure:
1. Gather all the text pairs you can find.
2. For each query Q, filter the top-k using BM25 and embeddings. Gain some set of K documents {D_1, ..., D_K}, relevant to Q.
3. Run each pair (D_*, Q) through an LLM to figure out if D is truly a negative example or not.
4. Output: a perfectly-labeled retrieval dataset that's arbitrarily scalable.
5. Train the model using exact softmax and coordinate ascent.

https://t.co/nYZ1Oqizl9

[Shane Gu @shaneguML]

True "AI Scientist" work. Big congrats to the team and scientists of the world.

[NASA's Johnson Space Center @NASAJohnson]

Mission accomplished, Suni. 🚀

After 27 years at @NASA, astronaut Suni Williams retires following a career that helped redefine human spaceflight. With 608 days in space, three missions to the International @Space_Station, and a record-setting 62+ hours of spacewalks, her legacy will inspire generations to come. Thank you, Suni. 🌍✨ https://t.co/vT77K6OjCO

[Vetha @Vethssvikas]

@niloofar_mire Thanks for sharing! This was super helpful

[Vetha @Vethssvikas]

ending the year on a sinkhorn-knopp note
thank you @deepseek_ai

[nor @norxornor]

Quick read through of Deepseek's new Manifold-Constrained Hyper-Connections paper:

- You want to increase residual size from 1×C to n×C (n streams instead of 1). Earlier residual update: x' = x + layer(x). Make the x be n×C, and use x' = Ax + B layer(Cx) instead. A, B, C are all dependent on x and are small matrices (n×n, n×1, n×1). A seems the most impactful. This is Hyper-Connections (HC).

- HC has the same issue as other residual modification schemes - eventually the product of the learned A matrices (along the identity path) blows up/vanishes.

- To fix this, they project the A matrices onto the Birkhoff polytope (simpler words: transform it, after exp to make elements positive, to a matrix whose row sums and column sums become 1 - called a doubly stochastic matrix). This has nice properties - products of these types of matrices still have row and column sum 1 (due to closure), so things don't explode (spectral bound), and the invariant is that the sum of weights across streams is 1. For n = 1, this becomes the standard residual stream, which is nice. Their transformation method is simple - alternatively divide rows and columns by row and column sums respectively for 20 iterations (converges to our desired matrix as iterations go to infinity). They find 20 is good enough for both forward and backward pass (across 60 layers, maximum backward gain is 1.6 as opposed to 3000 from usual HC, and 1.6 is not very off from 1).

- Composing these matrices (convex hull of all permutation matrices) leads to information mixing as layer index increases, which is a nice piece of intuition and is also shown very clearly in their composite matrix for 60 layers. I believe overall we get a weighted sum of residual paths (thinking of gradients), where logically group-able paths have weights summing to 1. Quite principled approach IMO, also makes gains (forwards and backwards) very stable.

- Interesting thing to note - lot of "pooling"-like mixing in the first half compared to the second half of the layers. Second half of layers treat different channels more precisely/sharply than the first half, quite intuitive.

- They also change parameterization of B and C (sigmoid instead of tanh, to avoid changing signs probably, and a factor of 2 in front of B, I believe to conserve mean residual multiplier, C doesn't need this because input is pre-normed anyway).

- Cool systems optimizations to make this op fast - they do kernel fusion, recomputation in the mHC backward pass, and even modify DualPipe (their pipeline parallelism implementation).

- Only 6.7% overhead in training when n = 4, loss goes down by 0.02 and improvements across benchmarks.

[Siyan Sylvia Li ✨ @Sylvia_Sparkle]

🤔Virtual Teaching Assistants (VTAs) have incredible potential, but how to evaluate them?

Our EMNLP Findings paper addresses the lack of standardized pedagogical eval frameworks for VTAs.

‼️When using LLMs as automated evaluators, they struggle with these nuanced assessments!🧵

[Jack Morris @jxmnop]

OpenAI hasn’t open-sourced a base model since GPT-2 in 2019. they recently released GPT-OSS, which is reasoning-only...

or is it?

turns out that underneath the surface, there is still a strong base model. so we extracted it.

introducing gpt-oss-20b-base 🧵

[Vetha @Vethssvikas]

@jxmnop Qwen 3!

[Vetha @Vethssvikas]

@demishassabis this is so amazing! blown away by the benchmarks! thanks for all the hard work 🙂‍↕️🙏

[Vetha @Vethssvikas]

hopefully with feedback and multi-agents, it’s starting to be less about perfect steps and more about recovery

[Benjamin Todd @ben_j_todd]

Why can AIs code for 1h but not 10h?

A simple explanation: if there's a 10% chance of error per 10min step (say), the success rate is:

1h: 53%
4h: 8%
10h: 0.002%

@tobyordoxford has tested this 'constant error rate' theory and shown it's a good fit for the data

chance of success declines exponentially

[DSPy @DSPyOSS]

If you're at NAACL, don't miss Siyan's talk.

Her work introduces PAPILLON, an AI system that protects privacy by teaching a tiny local LM to use powerful remote LLMs as tools.

Optimizing PAPILLON with DSPy's MIPRO boosts quality by up to 41% and can reduce leakage by up to 68%.

[Siyan Sylvia Li ✨ @Sylvia_Sparkle]

Excited to see everyone at @naacl !! PAPILLON will be presented in Session J on Friday at 9 am (first talk in the session!!)

Here's my map of the Breaking Bad Universe locations near the venue: https://t.co/BSdUgsT0xN

[Vetha @Vethssvikas]

@superbabyanushk 🥰🥰✨

[Vetha @Vethssvikas]

@superbabyanushk aesthetic as always