Fabiano Anemone

I reverse engineer things, find vulnerabilities and write FOSS under pseudonyms.
Local LLM entusiast, I do experiments with the RTX 6000 Pro, CMP HX170, M5 Max

[Fabiano Anemone @anoane]

@AndriusStepona1 @sfxnz Yeah, speculative decode should be tested on real stuff, like a real coding repo or so...

BTW, a M5 Max with 3x Thunderbolt 5 M.2 with fast Gen4 x4 NVMEs can go from ~14gbps to ~30/35gbps if you tune streaming properly with 4x replicas 👀

[Fabiano Anemone @anoane]

@Hikari_07_jp Crying in octachannel

[Fabiano Anemone @anoane]

@Sentdex Mind bending

[Fabiano Anemone @anoane]

@MiaAI_lab I prefer Deepseek V4.1 Flash over GLM 5.3 Flash (at least for my use cases). Running your EXL3 3.0bpw quant btw

[Fabiano Anemone @anoane]

@itsjustmarky Very nice!

I'm going with EPYC + TURIND8X-2T/500W
1x RTX 6000 Pro WS
4x CMP 170HX 64GB

I'll squeeze them inside a WS case with limited Watt (260W for the CPU, 325W for the 6000, 160W for the 170HX)

Buying more 6000 Pro at this price it's not worth it IMHO, I'd rather CMP-MAX

[Fabiano Anemone @anoane]

@itsjustmarky @schnelle Any straight or 90 degree bracket will do it

[Fabiano Anemone @anoane]

@MiaAI_lab Totally agree!

[Fabiano Anemone @anoane]

@0xSero Any "cheap" 6000 Pro WS? 🤣

[Fabiano Anemone @anoane]

@alphatozeta8148 @TheZachMueller I'm buying hardware cause I get censored even by Opus 4.6 nowadays. We are not the same, I guess...

[Fabiano Anemone @anoane]

@ashxhart -for disk offload -> ability to use different disks (with replicas of the model) to stream faster (like a fake raid) [required proper cpu/thread/ram management as well since the type of op could be sync, async, batched, etc, needs proper scheduling]
-some auto-sweep ->best config

[Fabiano Anemone @anoane]

@ashxhart -knob to choose how much ram to use, how much for experts/layers to be streamed, how much for engrams
-fast ssd offload (FreeToken style) with several backend depending on the action (e.g. mixing io_uring, odirect, pread, mmap, depending if it's model load, expert stream, engram)

[Fabiano Anemone @anoane]

@antirez You can apply for Daybreak Blue and you will not be annoyed further (unlike Opus/Fable)

[Fabiano Anemone @anoane]

@adamdotenv @0xIlyy Sam *man...

Same same.

[Fabiano Anemone @anoane]

@regsky_ Spot on
https://t.co/EznHCilh5o

[Fabiano Anemone @anoane]

@melvindvivas 500$? I'd rather cancel, Astra is smart but doesn't get any work done...

[Fabiano Anemone @anoane]

@juliuss1907 Accurate

[Fabiano Anemone @anoane]

@superalesha @trymirai @PrismML IMHO the real 2 bit war is GSQ-RCO vs EXL3 😄

[Fabiano Anemone @anoane]

@ashxhart Local Deepseek V4.1 Flash is amazing for cyber-sec work, even better than GLM-5.3 Flash.
Meanwhile Anthropics does this (I have Apple CVEs and cyber-sec is all I do, so...)