Divyansh

LLM Inference

prev research @adobe | cs grad @IITKanpur

[Divyansh @divyansh_token]

@__tinygrad__ since these are consumer cards ( and not server)
how many years will they last if ran continuously??

[Divyansh @divyansh_token]

@thsottiaux is this a promotional move to incentivize luna/terra usage or these are completely optimization gains??

[Divyansh @divyansh_token]

@OpenAI this is phenomenal, practically oss models aren't the cheaper options anymore!!
OAI is really going hard on cost efficiency

[Divyansh @divyansh_token]

@__tinygrad__ wait is that ~780 gigs of vram for $160k??
am I seeing it wrong?

[Divyansh @divyansh_token]

@ClementDelangue was this ui generated by grok 4.5??

[Divyansh @divyansh_token]

@thsottiaux in my office as well

[Divyansh @divyansh_token]

@teortaxesTex any reference numbers from other models to get an idea of scale up?

[Divyansh @divyansh_token]

@mehulmpt also ttft is 10 seconds for a prompt of size of ~128k tokens

[Divyansh @divyansh_token]

@mehulmpt will you pay for this K3 service:
$2/1M in and $7/1M out (cache hits are free)
only tradeoff is that speed will be ~10-12 tok/sec

[Divyansh @divyansh_token]

@suchenzang for some reason this makes me think that @thinkymachines will really make it in long term, they are doing( and releasing) good research work despite all these setbacks and that goes long way!

[Divyansh @divyansh_token]

@__tinygrad__ @AMD @sgl_project @Kimi_Moonshot why bfloat16?

[Lisan al Gaib @scaling01]

I guess it's on DeepSeek alone to make intelligence too cheap to meter

[Divyansh @divyansh_token]

yeah, now 800B activated for mythos doesn't feel surprising

[Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) @teortaxesTex]

Kimi K3 has 104B active parameters.

[Divyansh @divyansh_token]

@teortaxesTex restraint

[Divyansh @divyansh_token]

@osanseviero 1. pls obv maitain the token efficiency (unlike qwen) its the best thing
2. a ~100B class MoE model with excellent knowledge work and agentic/coding capabilities (eg: laguna ?)
3. in house trained dspark drafters?
4. long running tasks, eg: 100-200 coherent tool calls?

[Divyansh @divyansh_token]

@mitsuhiko but isn't it significantly token efficient than opus 4.8? should be net cheaper then?

[Divyansh @divyansh_token]

@davis7 > Opus 5 feels like a strange hybrid of GPT-5.6-sol and Fable
precise!!

[Divyansh @divyansh_token]

@teortaxesTex now I'm unsure

[Divyansh @divyansh_token]

@teortaxesTex agree! looks like to achieve gpt-5.6ish cold efficiency you need to sacrifice the chalant opus soul

[Divyansh @divyansh_token]

@andrew_n_carr medium thinking has been best in my experience as well