Topi Santakivi

Architecting AI solutions @LuotoCompany

[Topi Santakivi @sandst1]

@ivanfioravanti Yup, rtk is great! Using it in pretty much all of my LLM workflows 👌 Configuring it for unknown/custom commands is a bit of work but it handles the common patterns nicely and the coverage is growing too.

[Topi Santakivi @sandst1]

@ChicouTiMix @ivanfioravanti Mem bandwith 600 Gb/s, DGX Spark has 273 Gb/s, so on a single decode stream that's 2x the speed.

[Topi Santakivi @sandst1]

@_Aria_2210 32. n x n x 2

[Topi Santakivi @sandst1]

@hicasamadim 72. n x n x 2

[Topi Santakivi @sandst1]

@antirez Ha, missed the details of this thread earlier. Nice approach!

[Topi Santakivi @sandst1]

@waltonoemi @Youssofal_ 4-bit

[Topi Santakivi @sandst1]

@agitbackprop @antirez My experiences with it are mixed. Sometimes works, but also breaks down, often when the model has to make multiple edits to the same file, or larger edits. The usual error is using old hashes. Larger/cloud LLMs are better at it. Also interested to hear @antirez thoughts.

[Topi Santakivi @sandst1]

@PoetzlPtzl @aijoey only cloud versions published so far.

[Topi Santakivi @sandst1]

@mr_r0b0t @NVIDIAAI @Alibaba_Qwen Working nicely! Tried this model yesterday and got pretty much the same numbers 🥳 The unsloth NVFP4 version has 0% acceptance (i guess MTP heads in wrong format or sth), sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP is working nicely.

[Topi Santakivi @sandst1]

@karpathy Wow, that’s awesome! Congrats!

[Topi Santakivi @sandst1]

@MichaelZima @KuittinenPetri @antirez On Mac I use OMLX, a bit MTPLX too. It runs nicely, lot of progress in the past months, also the multi-token prediction stuff has been boosting inference speeds on Mac too. If you already have a Studio, plenty of good stuff to run.

[Topi Santakivi @sandst1]

@KuittinenPetri @MichaelZima Qwen 3.6 27B, different quants, atm via llama-cpp as it just landed MTP support. Also the DwarfStar by @antirez is running nicely on the GB10.

Those are my main models atm, then also a bit of Qwen 3.6-35B-A3B and Qwen3-Coder-Next.

[Topi Santakivi @sandst1]

@KuittinenPetri @mr_r0b0t wait: did Spark / sm121 NVFP4 support already land on vllm main branch? or did you build some custom setup?

[Topi Santakivi @sandst1]

@MichaelZima @KuittinenPetri That's exactly what i'm doing atm

[Topi Santakivi @sandst1]

@techedgedaily @rohanpaul_ai Yeah. Now anybody can vibe-code their software into existence, and if it's good software, they can have other people pay them to use it. Oh wait..

[Topi Santakivi @sandst1]

@MemoryReboot_ The only thing missing is the M3 Ultra in stock.

[Topi Santakivi @sandst1]

@indes_yo @nash_su MTP is speculative decoding, specifically for token generation.

[Topi Santakivi @sandst1]

@0xSero With MoE, the experts are not activated just based on prompts, the expert selection is done for _each token_ separately. This is why the full MoE usually needs to be loaded in memory even if a fraction of it is used per token.

[Topi Santakivi @sandst1]

@usr_bin_roygbiv @LottoLabs Parallel processing. The mem bandwith is low for a single token stream but there's a lot of compute in that box to handle X tasks at a time. Parallel subagents etc. If you only need single-user single-stream, go M5 Max. https://t.co/6MjBCDXZ2T