LordOfTheYaml

Belgian geek • Opinions are my own •
« Les convictions sont des ennemis de la vérité plus dangereux que les mensonges. » — Friedrich Nietzsche

[Elon Musk @elonmusk]

@yunta_tsai A petition is underway to reinstate Jensen and I’m signing it too

[Benjamin De Kraker @BenjaminDEKR]

"oh, THOSE kind of Chinese models."

[LordOfTheYaml @Tycale]

@nb4ld Je focus deepseek de mon côté. Le moteur d'inférence github lxe/dsx est facile à hacker. Et tu peux faire tourner ça en cluster facilement vu l'architecture de DS

[LordOfTheYaml @Tycale]

@nb4ld @djafidjaf 1bpw?

[LordOfTheYaml @Tycale]

@LurioNineFive The memory bandwidth is great, but the instruction set isn't optimized for LLMs. You can essentially divide it by four, since you have to deal with larger memory chunks.

So in the end, the effective memory bandwidth is on par with a DGX Spark.. For the moment.

[LordOfTheYaml @Tycale]

@loktar00 Yup, same conclusion here. Even with a vision model.
Ripple 65B Q4_K_M + MiniCPM-V 4.7 for the vision; claude code for the harness.
281 628 tokens

[LordOfTheYaml @Tycale]

@lxe @pocketmodelai @RicursiveAI @reflection_ai Tried Ripple as well, pretty let down.
65B dense felt like the ideal sweet spot, runs great on 64GB with 2x RTX 5090s hitting ~150 tok/s dense over P2P.

[LordOfTheYaml @Tycale]

@MiaAI_lab Hmm 🤔 still big 🥺

[LordOfTheYaml @Tycale]

@0xSero Noctua <3

[LordOfTheYaml @Tycale]

@LLMJunky @samsaied55 That's optimistic! I would say less than 5 tok/s 😬

[LordOfTheYaml @Tycale]

@MiaAI_lab @NVIDIAAI Crazy 😳

[LordOfTheYaml @Tycale]

@lxe Funny experiment with DSX: tried bumping the bpw, ran out of memory, so I offloaded part of the compute to a host one hop away over Ethernet :')

Not ideal on paper, but performance didn't take that big of a hit!

dropped from 65 to 55 tok/s with ~15 GB remote.

[LordOfTheYaml @Tycale]

@lxe Thanks for the tip!

[LordOfTheYaml @Tycale]

@lxe I meant on my end. I may try again later by taking your files, thanks !

[LordOfTheYaml @Tycale]

@lxe Not really good results, ahah 😅

[LordOfTheYaml @Tycale]

@lxe Thanks, really appreciate it! I’m testing a code-focused calibration and giving a few expert layers a little more precision to see if it helps on the Java and SQL tasks I run locally 😊 (tot. 192GB vram, getting ~60tok/s)

[LordOfTheYaml @Tycale]

@jmartinprin

[LordOfTheYaml @Tycale]

@nb4ld Quel modèle ?

[LordOfTheYaml @Tycale]

@nb4ld Si tu connais en Europe qqun qui fait des opérations VRAM.. Je suis intéressé 🤗