Kushal Patil

Inference @ Autoloops, prefill minning and tps maxxing | IIT Bombay CSE

[Kushal Patil @LatentKush]

The fastest API for Gemma 4 26B is live on https://t.co/QVN4HHBxcu.

The idea is simple, generic platforms like vllm were based on a world where we couldn't write custom code for each new model.

Now we can optimize Model x Workload x Hardware. Fuse kernels, custom KV caching, scheduling decode/prefill on the GPU all of it.

[Anirudh @anieasyy]

The oddly named inference engine that powers the best provider API for Gemma-4-26B https://t.co/u1aJ8aQgOy

[Kushal Patil @LatentKush]

@LukePiette Where do I get in?

[Kushal Patil @LatentKush]

@subygan Fixed my link. Thanks!!

They really do think alike

[Kushal Patil @LatentKush]

Any open-weight model can be Jev now.

Jevify gives it a Jev-compatible API: typed decisions and calibrated probabilities, without losing what made you choose that model: 1M context, images, multimodality.

Here’s Not Hotdog running on jevified Gemma 4 which takes images as input👇 (Had to add manual delay because jev is so fast <150ms)

https://t.co/fT3RRZ3XYn

[Kushal Patil @LatentKush]

@melvynx Check this out -&gt; Converting Gemma to a jev like api.
Multimodal as well as a long context window
https://t.co/pM4kP5QMtX

[Kushal Patil @LatentKush]

Any open-weight model can be Jev now.

Jevify gives it a Jev-compatible API: typed decisions and calibrated probabilities, without losing what made you choose that model: 1M context, images, multimodality.

Here’s Not Hotdog running on jevified Gemma 4 which takes images as input👇 (Had to add manual delay because jev is so fast <150ms)

https://t.co/fT3RRZ3XYn

[Kushal Patil @LatentKush]

@gregpr07 Which model do you use? Would love to give it a shot

[Kushal Patil @LatentKush]

@ivanburazin Wouldn’t it be hypocritical for Daytona to host GPUs, then?

[kyutai @kyutai_labs]

We're releasing Voice of Reason, a speech-native model that does math out loud. Give it a spoken problem without transcription nor text LLM in the loop, and it reasons and answers in speech. GSM8K goes from 27.3% for GLM-4-Voice to 77.1%. Link in 🧵

[Kushal Patil @LatentKush]

Two models are trained today. Anyone can use the recipe in the repo to train a different model.

There is scope for a lot of improvement in the training recipe. Its the most basic thing that could be done.

Gemma 4 E4B — 8B, 128k context, 16 GB; runs on a 24 GB Mac
https://t.co/eRXlue5rCN
Gemma 4 26B-A4B — 26B MoE, 4B active, 256k context, 52 GB
https://t.co/8yJRvucf7l

Both accept images. Adapters: 140–180 MB.

[Kushal Patil @LatentKush]

why train LoRAs
Raw chat models choose accurate answers, but their confidence is broken: they say ~0.99 even when wrong.

So I trained small LoRAs on 47k probability targets—no teacher model.

ECE: .218→.028 (E4B), .235→.032 (26B)
Accuracy: .745→.823, .757→.821

[Anirudh @anieasyy]

Our inference platform has crossed 1 billion tokens in traffic handled today.

🚀🚀 w/ @LatentKush

[Kushal Patil @LatentKush]

@trikcode What's the problem in a MoE

[Kushal Patil @LatentKush]

@Ramanean People forget who introduced transformers. OpenAI takes all the credit. That's the way it works

[Kushal Patil @LatentKush]

@gregpr07 Add a state for reasoning. Jev will tell you when it wants to reason, and you call Fable once

[Kushal Patil @LatentKush]

@InvestwithFoxx @finkd they are on track to win - consumer

[Kushal Patil @LatentKush]

@kai_brokering Yes

[Kushal Patil @LatentKush]

@thdxr you are the only one

[Kushal Patil @LatentKush]

My entire weekly limit went away in a single prompt!!

Its becoming unusable at this point

[Kushal Patil @LatentKush]

@ivanburazin @daytonaio when are b200s coming?

[Anirudh @anieasyy]

Proud of the work we have done with @cekuraAi!

If you are building voice agents, you can get the API they use at https://t.co/VWMh0lqLmO.

LLMs are the most brittle part of any voice agent, and we have leapfrogged the market on low-latency, high concurrency workload management.