gururaj kosuru

Husband of a lovely lady, Father of a mini terminator. Also sometimes a Robotics and software engineer, jumping roles and over tadpoles.

[Sakana AI @SakanaAILabs]

We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible.

https://t.co/ApVzVsBuv1

By training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly internalize new information or adapt to new tasks.

Biological systems naturally rely on two key cognitive abilities: durable long-term memory to store facts, and rapid adaptation to handle new tasks given limited sensory cues. While modern LLMs are highly capable, they still lack this flexibility. Traditionally, adding long-term memory or adapting an LLM to a specific downstream task requires an expensive and time-consuming model update, such as fine-tuning or context distillation, or relies on memory-intensive long prompts.

To bypass these limitations, our work focuses on the concept of cost amortization. We pay the meta-training cost once to train a hypernetwork capable of producing tasks or document specific LoRAs on demand. This turns what used to be a heavy engineering pipeline into a single, inexpensive forward pass. Instead of performing per-task optimization, the hypernetwork meta-learns update rules to instantly modify an LLM given a new task description or a long document.

In our experiments, Text-to-LoRA successfully specializes models to unseen tasks using just a natural language description. Building on this, Doc-to-LoRA is able to internalize factual documents. On a needle-in-a-haystack task, Doc-to-LoRA achieves near-perfect accuracy on instances five times longer than the base model's context window. It can even generalize to transfer visual information from a vision-language model into a text-only LLM, allowing it to classify images purely through internalized weights.

Importantly, both methods run with sub-second latency, enabling rapid experimentation while avoiding the overhead of traditional model updates. This approach is a step towards lowering the technical barriers of model customization, allowing end-users to specialize foundation models via simple text inputs. We have released our code and papers for the community to explore.

Doc-to-LoRA
Paper: https://t.co/87xEEpf0GN
Code: https://t.co/zBfQi2L9LW

Text-to-LoRA
Paper: https://t.co/emLRZ4Vdvo
Code: https://t.co/b9mrdoWWRB

[Rohan Paul @rohanpaul_ai]

Uber CEO Dara Khosrowshahi believes we are approaching a "Flip Point" where the ROI (Return on Investment) of a human salary will be lower than the ROI of an AI agent.

IMO, this will apply to any and all intellectual jobs.

[Andrej Karpathy @karpathy]

It is hard to communicate how much programming has changed due to AI in the last 2 months: not gradually and over time in the "progress as usual" way, but specifically this last December. There are a number of asterisks but imo coding agents basically didn’t work before December and basically work since - the models have significantly higher quality, long-term coherence and tenacity and they can power through large and long tasks, well past enough that it is extremely disruptive to the default programming workflow.

Just to give an example, over the weekend I was building a local video analysis dashboard for the cameras of my home so I wrote: “Here is the local IP and username/password of my DGX Spark. Log in, set up ssh keys, set up vLLM, download and bench Qwen3-VL, set up a server endpoint to inference videos, a basic web ui dashboard, test everything, set it up with systemd, record memory notes for yourself and write up a markdown report for me”. The agent went off for ~30 minutes, ran into multiple issues, researched solutions online, resolved them one by one, wrote the code, tested it, debugged it, set up the services, and came back with the report and it was just done. I didn’t touch anything. All of this could easily have been a weekend project just 3 months ago but today it’s something you kick off and forget about for 30 minutes.

As a result, programming is becoming unrecognizable. You’re not typing computer code into an editor like the way things were since computers were invented, that era is over. You're spinning up AI agents, giving them tasks *in English* and managing and reviewing their work in parallel. The biggest prize is in figuring out how you can keep ascending the layers of abstraction to set up long-running orchestrator Claws with all of the right tools, memory and instructions that productively manage multiple parallel Code instances for you. The leverage achievable via top tier "agentic engineering" feels very high right now.

It’s not perfect, it needs high-level direction, judgement, taste, oversight, iteration and hints and ideas. It works a lot better in some scenarios than others (e.g. especially for tasks that are well-specified and where you can verify/test functionality). The key is to build intuition to decompose the task just right to hand off the parts that work and help out around the edges. But imo, this is nowhere near "business as usual" time in software.

[Rohan Paul @rohanpaul_ai]

🚨 BREAKING: DeepSeek dropped a core Transformer architecture improvement.

A traditional transformer is basically a long stack of blocks, and each block has a “main work path” plus a “shortcut path” called the residual connection that carries the input around the block and adds it back at the end.

Each block in this original transformer architecture does some work (self attention or a small feed forward network), then it adds the block’s input back onto the block’s output, which is why people describe it as a “main path” plus a “shortcut path.”

Hyper-Connections is a drop-in change to that shortcut path, because instead of carrying 1 stream of activations through the stack, the model carries a small bundle of parallel streams, then it learns how to mix them before a block and after a block.

Standard Transformers pass information through 1 residual stream. Hyper-Connections turn that into n parallel streams, like n lanes on a highway. Small learned matrices decide how much of each lane should mix into the others at every layer.

In a normal residual connection, each layer takes the current hidden state, runs a transformation, then adds the original back, so information can flow forward without getting stuck.

In this new Hyper-Connections, the layer does not see just 1 hidden state, it sees a small bundle of them, and before the layer it learns how to mix that bundle into the input it will process.

So in a traditional transformer block, wherever you normally do “output equals input plus block(input),” Hyper-Connections turns that into “output bundle equals a learned mix of the input bundle plus the block applied to a learned mix,” so the shortcut becomes more flexible than a plain add.

After this learned layer, the "Hyper-Connections" mechanism again learns how to mix the transformed result back into the bundle, so different lanes can carry different kinds of information, and the model can route signal through the shortcut in a more flexible way.

The catch is that if those learned mixing weights are unconstrained, stacking many blocks can make signals gradually blow up or fade out, and training becomes unstable in big models.

This paper proposes mHC, which keeps Hyper-Connections but forces every mixing step to behave like a safe averaging operation, so the shortcut stays stable while the transformer still gets the extra flexibility from multiple lanes.

---

The paper shows this stays stable at 27B scale and beats both a baseline and unconstrained Hyper-Connections on common benchmarks.

HC can hit about 3000x residual amplification, mHC keeps it around 1.6x.

[MIT CSAIL @MIT_CSAIL]

Andrej Karpathy, Eureka Labs founder & former Director of AI at Tesla, breaks down how LLMs like ChatGPT "download the Internet."

Watch the full beginner-friendly breakdown here: https://t.co/doph3alGnE

[Akshay 🚀 @akshay_pachaar]

DeepSeek cracked the O(L²) attention bottleneck.

Their new V3.2 model introduces DeepSeek Sparse Attention (DSA), and it's the only architectural change they made. That tells you how important this is.

What does it solve:

Standard attention scales quadratically. Double your context length, quadruple the compute. This is why long-context inference gets expensive fast.

DSA brings complexity down from O(L²) to O(Lk), where k is fixed.

How it works:

A lightweight Lightning Indexer scores which tokens actually matter for each query. Small number of heads, runs in FP8, computationally cheap. Then a selection mechanism retrieves only the top-k key-value entries.

The key insight: only 2,048 tokens get selected per query, regardless of context length. The expensive attention computation happens on this small subset, not the full 128K sequence.

Less attention, better results. DeepSeek V3.2 just proved it.

The results:

At 128K context, prefilling costs drop from ~$0.65 to ~$0.35 per million tokens. Decoding drops from ~$2.4 to ~$0.8.

And performance? Stays the same. On some long-context benchmarks, V3.2 actually scores higher.

Sparse attention isn't new. Making it work without losing quality is hard.

DeepSeek cracked it with a two-stage training process, first aligning the indexer using KL-divergence, then training the full model to adapt to sparse patterns.

This is how you scale context without scaling costs.

Why this matters beyond DeepSeek:

Ilya Sutskever recently mentioned in a podcast that scaling data and compute has hit a wall. You can throw more GPUs at the problem, but natural data is finite.

The next leap comes from architectural breakthroughs and research, not just scaling.

DSA looks like a step in that direction.

What are your thoughts?

I have linked the paper in the next tweet.

[Vector Institute @VectorInst]

⚡ Computational Efficiency: Smarter, not just bigger

LookWhere (Evan Shelhamer): 17x FLOP reduction while maintaining accuracy

EAGLE-3 (Hongyang Zhang): 6.5x LLM inference speedup through smarter token prediction

Proving that intelligent design beats brute force scaling.

[Open Review @openreviewnet]

Statement Regarding API Security Incident - November 27, 2025

[Sebastian Raschka @rasbt]

With the release of the Kimi Linear LLM last week, we can definitely see that efficient, linear attention variants have seen a resurgence in recent months. Here's a brief summary of what happened.

First, linear attention variants have been around for a long time, and I remember seeing tons of papers in the 2020s.

I don't want to dwell too long on these older attempts. But the bottom line was that they reduced both time and memory complexity from O(n^2) to O(n) to making attention much more efficient for long sequences.

However, they never really gained traction as they degraded the model accuracy, and I have never really seen one of these variants applied in an open-weight state-of-the-art LLM.

In the second half of this year, there was a bit of a revival of linear attention variants. The first notable model was MiniMax-M1 with lightning attention, a 456B parameter mixture-of-experts (MoE) model with 46B active parameters, which came out back in June.

Then, in August, the Qwen3 team followed up with Qwen3-Next, which I discussed in more detail above. Then, in September, the DeepSeek Team announced DeepSeek V3.2 with sparse attention.

All three models (MiniMax-M1, Qwen3-Next, DeepSeek V3.2) replace the traditional quadratic attention variants in most or all of their layers with efficient linear variants. (DeepSeek's sparse attention it's not strictly linear but still subquadratic).

Interestingly, there was a recent plot twist, where the MiniMax team released their new 230B parameter M2 model (discussed in section 13) without linear attention, going back to regular attention. The team stated that linear attention is tricky in production LLMs. It seemed to work fine with regular prompts, but it had pure accuracy in reasoning and multi-turn tasks, which are not only important for regular chat sessions but also agentic applications.

This could have been a turning point where linear attention may not be worth pursuing after all. However, it gets more interesting. Last week, the Kimi team released their new Kimi Linear model with linear attention. The tag line is that compared to regular, full attention, it has a 75% KV cache reduction and up to 6x decoding throughput.

Kimi Linear shares several structural similarities with Qwen3-Next. Both models rely on a hybrid attention strategy. Concretely, they combine lightweight linear attention with heavier full attention layers. Specifically, both use a 3:1 ratio, meaning for every three transformer blocks employing the linear Gated DeltaNet variant, there's one block that uses full attention as shown in the figure below.

However, Kimi Linear modifies the linear attention mechanism of Qwen3-Next by the Kimi Delta Attention (KDA) mechanism, which is essentially a refinement of Gated DeltaNet. Interestingly, it also replaces the standard full attention module by multi-head latent attention.

There's no direct comparison to Qwen3-Next in the Kimi Linear paper, but compared to the Gated DeltaNet-H1 model from the Gated DeltaNet paper (which is essentially Gated DeltaNet with sliding-window attention), Kimi Linear achieves higher modeling accuracy while maintaining the same token-generation speed.

Of course, I couldn't resist and added it to my The Big LLM Architecture Comparison article, which has grown to >10,000 words now (basically becoming book!?).

[Matt Shumer @mattshumer_]

Introducing: AI Researcher 🧪

A Gemini 3-powered multi-agent AI system that autonomously runs ML experiments

Just give it a research question, and it will:
- Design experiments
- Spin up specialist agents with their own GPUs to run them
- Write a paper

And it's open-source!

[Brie Wensleydale🧀🐭 @SlipperyGem]

Dude, this guy is making some crazy comfyUI nodes.

Made SAM3, SAM 3D Body and SAM 3D Object all in like a day, then Depth Anything 3, also a cool 3D Geometry pack, but then also this thing.

This is a lot of pressure man. I want to test them all...

https://t.co/5pyknWTepQ

[gururaj kosuru @glusus]

@Lufthansa259005 I have messaged my phone number

[gururaj kosuru @glusus]

@Lufthdnsa I have messaged my phone number

[Oriol Vinyals @OriolVinyalsML]

Exciting times, welcome Gemini (and MMLU>90)! State-of-the-art on 30 out of 32 benchmarks across text, coding, audio, images, and video, with a single model 🤯

Co-leading Gemini has been my most exciting endeavor, fueled by a very ambitious goal. And that is just the beginning! A long 🐍 post about our Gemini journey & state of the field.

The biggest challenges in LLMs are far from trivial or obvious. Evaluation and data stand out to me. We've moved beyond the simpler "Have we won in Go/Chess/StarCraft?" to “Is this answer accurate and fair? Is this conversation good? Does this complex piece of text prove the theorem?” Exciting potential coupled with monumental challenges.

The field is less ripe further down the model pipeline. Pretraining is relatively well understood. Instruction tuning and RLHF, less so. In AlphaGo and AlphaStar we spent 5% of compute in pre-training and the rest in the very important RL phase, where the model learns from its successes or failures. In LLMs, we spend most of our time on pretraining. I believe there’s huge potential to be untapped. Cakes with lots of cherries, please 🎂

@Google has demonstrated its ability to move fast. It has been an absolute blast to see the energy from my colleagues and the support received. A “random” highlight is coauthoring our tech report with a co-founder. Another is coleading with @JeffDean. But beyond individuals, Gemini is about teamwork: it is important to recognize the collective effort behind such achievements. Picture a room full of brilliant people, and avoid attributing success solely to one person.

On a personal note, recently I celebrated my 10 year anniversary at Google, and it’s been 8 years since @quocleix and I co-authored “A Neural Conversational Model”, which gave us a glimpse of what was, has, and is yet to come. Back then, that line of work received a lot of skepticism. Lessons learned: whatever your passion is, push for it!

Zooming back out, there’s lots of change in our field, and the stakes couldn’t be higher. Excited for what’s to come from Gemini, but humbled by the responsibility to “get it right”. 2024 will be drastic. Welcome Gemini!

https://t.co/X4ZLpcUiiL

[gururaj kosuru @glusus]

@letterlitter New Bob Gooder than past Bob

[Amazon Web Services @awscloud]

Build your business on a solid foundation with #AWS & #GenerativeAI. ☁️💡⚡️

Now GA, Amazon Bedrock is the easiest way to build & scale generative #AI applications with foundation models.

🚀 https://t.co/4bB7t2T4M8

[gururaj kosuru @glusus]

@letterlitter Ya ya yay ohwhy ayyy ayy

[Harbhajan Turbanator @harbhajan_singh]

Apke baap ki Umar ke hai kam se kam Umar ka hi dhyaan rakho.. 😡

[gururaj kosuru @glusus]

@letterlitter You can faludify your understanding with the-belly

[Raphael Pollock @DrRPollock]

Lifestyle factors like alcohol intake play a big role in #cancer development and prevention. This study is a reminder to providers that we need to be vigilant in sharing this important information with our patients. https://t.co/VDCthzkWgm