Sidharth Pulipaka

I really like Deep Learning

[Sidharth Pulipaka @psidharth567]

This was so fun. Would be the most memorable social event at #icml

We spontaneously decided to walk 5km to the bank of Hangang river and it was ~50 AI mech interp researchers walking on the streets of Seoul at midnight.

[Celeste @celestepoasts]

For all those excited about

Activation Oracles, LoRAcles, Natural Language Autoencoders, Introspection Adapters...

Come meet the authors at the scalable interp and meta models mixer!

[Sidharth Pulipaka @psidharth567]

@thkostolansky I believe Anthropic being more "safety-aligned lab" does not put much optimisation pressure to reduce the length of CoT.
OpenAI claims they don't train against CoT traces for monitorability, but they also previously mentioned that training to compress the CoT length is acceptable

[Sidharth Pulipaka @psidharth567]

@gaurav_ghosal You would still have shared experts retained during inference. This would also perform "joint learning", which seems to be your primary selling point.
"Shared information is reinforced in the always-on backbone."

[Sidharth Pulipaka @psidharth567]

@gaurav_ghosal Why can't you use a simple MoE (with shared experts), route through a particular set of experts at each layer during training on the target documents and remove those experts at inference time.

[Sidharth Pulipaka @psidharth567]

@yacineMTB Check this benchmark out which evaluates LLMs on this behavior:
https://t.co/LzcxSbDoLA

[Sidharth Pulipaka @psidharth567]

@sahar_abdelnabi While all other models go a bit beyond, be helpful and do the distractor tasks.

It would be nice to see Opus 4.6 and 4.8 results too here.
They were not hated for such behavior, and my guess is that they would be more compliant here than 4.7

[Sidharth Pulipaka @psidharth567]

@sahar_abdelnabi Opus-4.7 was widely hated for not understanding the user's intent, 'being autistic', hard to work with.
While GPT-5.5 had very positive reactions from the community, it's pleasant to work with.

And Opus-4.7 is the least compliant model in the results significantly.

[Sidharth Pulipaka @psidharth567]

@ayhozade There are variants of Soft thinking which claim successful training with improved performance.

I'd like to see performance and CoT length of MUX in comparison to SofT-GRPO.

[Sidharth Pulipaka @psidharth567]

"PersistBench: When Should LLMs Forget Long-Term Memory?" got accepted at ICML!

[Sidharth Pulipaka @psidharth567]

@yacineMTB This is so stupid. The whole premise of Anthropic is the same. "Criminals are usually not very skilled" and Mythos makes then skilled enough to find Zero days.

[Sidharth Pulipaka @psidharth567]

@thekitze > Elo score

Even Llama-4 was #1 on LMsys/LMArena when it first released.

[Sidharth Pulipaka @psidharth567]

Read Chollet's tweet above it. You were agreeing with Chollet. Your first word of the tweet was 'yep', agreeing to Chollet's claim of requiring a special harness and now you deny providing special harness and just providing a standard harness that we see in the webui.
Also you claimed they could *only* solve most of the easy/trivial tasks but we know they can also solve the hard tasks.

Incredible misinformation.

[Sidharth Pulipaka @psidharth567]

> If these were AGI systems with superhuman reasoning as frontier labs claim
And you proceed to evaluate o4-mini and Kimi-K2. Make it make sense.

No labs claim GPT-5.4 and Opus-4.6 are super human reasoning systems. And you thought o4-mini is an AGI system with superhuman reasoning?

[Sidharth Pulipaka @psidharth567]

@fchollet That's what they say the models did: https://t.co/otJ9Pauo9O

[Lossfunk @lossfunk]

7/ After the paper was finalized, we ran agentic systems that mimic how humans would learn to solve problems in esoteric languages.

We supplied our agents with a custom harness + tools on the same benchmark.

They absolutely crushed the benchmark.

Stay tuned 👀

[Sidharth Pulipaka @psidharth567]

@aidaniil You are literally one Claude update away from being killed

[Sidharth Pulipaka @psidharth567]

@jxmnop Gemini Embedding did use the Gemini LLM to judge the correctness of hard-negatives and filter them out.

[Sidharth Pulipaka @psidharth567]

Yann LeCun: Autoregressive modelling is doomed.
If you are interested in human-level AI, don't work on LLMs.

Also Yann LeCun:

[alphaXiv @askalphaxiv]

Yann LeCun 🤝 Saining Xie

insane crossover of the 2 biggest visual representation researchers in the AI field

“Beyond Language Modeling: An Exploration of Multimodal Pretraining”

Right now, most multimodal models are basically a language model with a vision adapter bolted on, so they can describe images, but they don’t really think in images or video.

This paper shows what happens when you do it the hard way: train one model from scratch on text, images, and video with a unified setup.

They key idea is if you give the model a good visual internal format and it can use vision for both understanding and generating.

Additionally, multimodal data can improve language instead of distracting it, and mixture-of-experts lets you scale vision’s huge data intake without bloating everything else.

This paves the way towards changing the vision paradigm from “captioning add-on” model to native multimodal foundation model.