Garvin Bhati

AI Solutions Engineer

[Garvin Bhati @BhatiGarvin]

@evilmathkid @arcprize @fchollet @mikeknoop Great work!
How long scoring takes on 240 tasks of test?

[Garvin Bhati @BhatiGarvin]

This is scoring 54.08% on IndicParam which is the highest I have seen after Gemini.

[OpenCode @opencode]

Ox Alpha (stealth model) is free for the next week

- 1M Context
- Multi-modal
- Zero Data Retention

Generous rate limits, near unlimited usage

We have capacity for 100T tokens per day, lets see what you can do

[Garvin Bhati @BhatiGarvin]

Scores for Qwen3.8-27B!
• MILU: 58.22%
• IndicParam: 36.06%
• Strict IndicIFEval: 54.65%
• Equal-task mean: 49.64%

The gap is especially visible for Tamil, Telugu, Gujarati, and Kannada.
very strong general purpose reasoning/coding model, but a clear Indic language capability gap.

[Garvin Bhati @BhatiGarvin]

Just finished evaluating Inkling-Small on the same frozen 5,242-prompt Indic evaluation.

Inkling-Small (276B total, 12B active)
• MILU: 69.90%
• IndicParam: 42.42%
• Strict IndicIFEval: 70.10%
• Equal-task mean: 60.81%

Gemma 4 Indic RYS (ours, 31B base)
• MILU: 76.4%
• IndicParam: 44.4%
• Strict IndicIFEval: 68.4%
• Equal-task mean: 63.1%

Gemma 4 Indic RYS is better overall

[Garvin Bhati @BhatiGarvin]

@grok are these correct?

[Garvin Bhati @BhatiGarvin]

This is crazy!!
Qwen 3.8 27B is better than Kimi K2.6, a 1-trillion parameter Mixture-of-Experts (MoE) model from 4 months ago across various benchmarks.

[Garvin Bhati @BhatiGarvin]

Need to check these numbers once ->

[Garvin Bhati @BhatiGarvin]

Just finished evaluating Inkling-Small on the same frozen 5,242-prompt Indic evaluation.

Inkling-Small (276B total, 12B active)
• MILU: 69.90%
• IndicParam: 42.42%
• Strict IndicIFEval: 70.10%
• Equal-task mean: 60.81%

Gemma 4 Indic RYS (ours, 31B base)
• MILU: 76.4%
• IndicParam: 44.4%
• Strict IndicIFEval: 68.4%
• Equal-task mean: 63.1%

Gemma 4 Indic RYS is better overall

[Garvin Bhati @BhatiGarvin]

@LottoLabs just 4 hours to go

[Garvin Bhati @BhatiGarvin]

@parth_mahajan20 @agarwalpacker @rameshagrwl

[Garvin Bhati @BhatiGarvin]

@MohapatraHemant Okay need to comeback and see what are you testing with this retweet.

[Garvin Bhati @BhatiGarvin]

After the update today at #sarvamepoch this holds for now.

[Garvin Bhati @BhatiGarvin]

*Introducing gemma-4-31b-indic-rys*: a 61-layer Gemma 4 model for Indic language tasks.

@SarvamAI probably releases new SOTA on Indic Benchamrks at Sarvam Epoch (AI Conference) in few hours, until then this is best option on Indic in <100B size.

[Garvin Bhati @BhatiGarvin]

It’s crazy how good some updates are, especially voice.

[Shreyas Dixit @Shreyas__Dixit]

Man I love what @SarvamAI is doing it’s the dream to work on things like this !!!

[Garvin Bhati @BhatiGarvin]

This is a research/development result on an inspected split, not an untouched benchmark claim. Independent evaluation on more benchmarks is welcome.

[Garvin Bhati @BhatiGarvin]

*Introducing gemma-4-31b-indic-rys*: a 61-layer Gemma 4 model for Indic language tasks.

@SarvamAI probably releases new SOTA on Indic Benchamrks at Sarvam Epoch (AI Conference) in few hours, until then this is best option on Indic in <100B size.

[Garvin Bhati @BhatiGarvin]

https://t.co/cSfvP5Krtx
This is not a standard 60-layer Gemma checkpoint. The model has 61 decoder layers and a gated repeated block, so use the custom Transformers implementation or the supported vLLM runtime included with the release.

[Garvin Bhati @BhatiGarvin]

Inkling is the strongest model among the seven shown.
Nemotron’s within-family gap is substantial.
Kimi K2.6 reaches 69.3% on strict IFEval—but MCQ answer validity hurts

[Garvin Bhati @BhatiGarvin]

Four more models are now on our frozen Indic evaluation—and the winner changes by task.

[Garvin Bhati @BhatiGarvin]

We added:
- `thinkingmachines/Inkling`
- `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16`
- `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16`
- `moonshotai/Kimi-K2.6`

[Garvin Bhati @BhatiGarvin]

If you have a setup, benchmark or failure mode to test next, please share it. Specific suggestions are especially welcome.

[Garvin Bhati @BhatiGarvin]

The most important result from this LLM evaluation was not the leaderboard. It was a **68.6× output-token gap** on the exact same 5,242-prompt Indic evaluation.

Inference behavior changed the benchmark.
#IndicAI #LLMEvaluation #OpenSourceAI #AIResearch #MultilingualAI

[Garvin Bhati @BhatiGarvin]

The scores are useful—but they are not an apples-to-apples architecture or efficiency comparison. Sarvam used FP8 with native thinking; Gemma and Qwen used BF16 with direct answers. This is a comparison of the models in the deployment modes we could run reliably on one 96GB GPU.