Djordje Madic

LLM inference

[Djordje Madic @dmadic]

@GrayBeamTech @tenstorrent Mr @moritzthuening is having lots of fun predicting protein structures :)

[Djordje Madic @dmadic]

we need more TUIs in this world

[Djordje Madic @dmadic]

@artem_aero @tenstorrent We need a reward for the first OSS contribution coming from this :)

[Djordje Madic @dmadic]

@OfficialLoganK @HCSolakoglu Gemini app has 900 million MAU 🤯

[Djordje Madic @dmadic]

so SGLang supports platform plugins now

I wanted to get a feel for the per-token overhead in the streaming phase
For mocked model streaming 500 tok/s the I measured 0.3 ms overhead per token

https://t.co/ZZBRWGgjmF

What's not yet perfect is having to build sglang from sources to use platform plugins

[Djordje Madic @dmadic]

I was surprised that Dynamo does not sit on top vLLM/SGLang HTTP servers but uses them as Python API engines

[Djordje Madic @dmadic]

Made a PoC today to understand how Mooncake works

I was able to run Qwen 0.6B on CPU with Mooncake

Nano vLLM inspired
2xPrefill 1xDecode nodes
TCP transport mechanism

https://t.co/x1A44VWzaP
Entrypoint pd_two_prefill_chat.py

[Djordje Madic @dmadic]

👀

[Cirrascale Cloud Services @Cirrascale]

What if you could cut inference costs in half without giving up production-scale performance?
That's what @Tenstorrent Galaxy Blackhole, now live on the Cirrascale AI Innovation Cloud, was built to deliver:
✅ Approximately half the cost of leading GPU alternatives
✅ Bare-metal access (no virtualization overhead)
✅ 90% of HuggingFace models run as-is, no rewrites required
✅ Latency-optimized for large-context LLM inference and video generation

👉 Request access: https://t.co/C5PscwsnqX

[Djordje Madic @dmadic]

@kevinmi920 @tenstorrent @prodia How can we try this out

[Kevin Mi @kevinmi920]

Introducing Infinite Studio ♾.

Last week, @tenstorrent x @prodia announced the fastest Wan 2.2 video generation in the world.

We built a demo to show what that speed unlocks: directing an infinite movie in real time.

Demo 👇

[Djordje Madic @dmadic]

WHAT

A

DAY

@tenstorrent

[Djordje Madic @dmadic]

@wesbos Better home page 🙏

[Djordje Madic @dmadic]

it's super fun working on this

can't wait to ship some more amazing things to TT customers

[Sally Ward-Foxton @sallywf]

.@tenstorrent is launching Galaxy Blackhole servers and clusters for fast LLM inference (DeepSeek-671B at up to 350 t/s/u), among other applications - no disaggregation needed, according to @jimkxa:

https://t.co/G2yu85tXs2

[Djordje Madic @dmadic]

perfect weekend

[Djordje Madic @dmadic]

how do you test your monitoring?

[Djordje Madic @dmadic]

@daniel_mac8 Reminds me of this talk counter strike players complained about high ping and they just decreased it by fixed number :D

[Djordje Madic @dmadic]

Fixed

[Djordje Madic @dmadic]

Fyi @openclaw heartbeats can cost you even if heartbeat.md is empty

Created a bug https://t.co/cD0Vre676Y

[Djordje Madic @dmadic]

How quickly we forgot all the amazing things Claude guys made

They just also have bugs

It would be crazy not to have shipping at this speed

[Djordje Madic @dmadic]

@steipete maybe stupid question but I'm curious

[Djordje Madic @dmadic]

Why don't we just fine-tune a model to be a good claw?