Thought Provoking Vile VR Man

[Thought Provoking Vile VR Man @tompettyflacko]

@jun_song show me any model running on an 128gb m5 and I can show you it running it faster on 128gb strix halo that's half the cost

[Thought Provoking Vile VR Man @tompettyflacko]

@MiaAI_lab 🙋‍♂️

[Thought Provoking Vile VR Man @tompettyflacko]

@AnushElangovan could you also please fully port oga to linux? would love to run onnx hybrids on my linux os strix!!

[Thought Provoking Vile VR Man @tompettyflacko]

@Italianclownz @barackomaba @sudoingX @LarryAGuy1 with the work from you too and i have 35b at peaks of 137.9 t/s (~83 mean) and 27b at peaks of 51.4 t/s (~32 mean)

Thats matching/beating 2x 3090 setups, but mine are co-loaded, and have access to a small army of helper, speech, img, and video models in parallel on the npu 🤘

[Thought Provoking Vile VR Man @tompettyflacko]

@LottoLabs hey lotto! my agent accidentally made some mistake (or was being overly literal) and lots of my 5060ti egpu runs on localmaxxing are also showing on the strix halo boards. despite patching my submissions they still show there. I don't want to slop ur site up, plz help

[Thought Provoking Vile VR Man @tompettyflacko]

@AMD please fully port oga to Linux, not being able to run onnx hybrid is a travesty

[Thought Provoking Vile VR Man @tompettyflacko]

@Italianclownz beast

[Thought Provoking Vile VR Man @tompettyflacko]

@bwaltens based alert

[Thought Provoking Vile VR Man @tompettyflacko]

@barackomaba @rocketman110us @sudoingX @Italianclownz yes and only p2, afaik mtp bugs at >2

[Thought Provoking Vile VR Man @tompettyflacko]

@populartourist @barackomaba @Italianclownz @rocketman110us @sudoingX @DJLougen WOW. now those are impressive numbers. and a killer idea. my latest frustration has been the bottleneck hermes seems to put on the model in the 10-15 t/s range, now going to try and claw some of that back. ty!

[Thought Provoking Vile VR Man @tompettyflacko]

@Italianclownz @barackomaba @rocketman110us @sudoingX @DJLougen @populartourist I dont see why not. maybe a multi task bench testing small runs on code, creative, extraction, facts, multi-turn convo, short gen, long gen, etc? score each section. select for configs w best robust performance bias towards stable gen across tasks > bursty peaks on select tasks

[Thought Provoking Vile VR Man @tompettyflacko]

@barackomaba @Italianclownz @rocketman110us @sudoingX @DJLougen :( spec draft type stacking i picked up from @populartourist . but relative success does depend on task type.
I have a coherence baseline quick bench that gets run to keep things sane. I just give all the flags I want to stack/test and toss into an autoreseach hill climb loop.

[Thought Provoking Vile VR Man @tompettyflacko]

@Italianclownz @rocketman110us @sudoingX @barackomaba @DJLougen same config with 137.91 tg/s , just uploaded lots of my runs to localmaxxing . across the board with different models im beating or matching 2x 3090 setups

[Thought Provoking Vile VR Man @tompettyflacko]

@Italianclownz @rocketman110us @sudoingX @barackomaba @DJLougen same idea! I had the saber 35b shards already on my box when you dropped your repo. Just ran them through your quantizer at the strix lean profile 🤘

[Thought Provoking Vile VR Man @tompettyflacko]

@rocketman110us @sudoingX @Italianclownz i went back to double check and 35b top was actually 🔥128.88 tg/s🔥

[Thought Provoking Vile VR Man @tompettyflacko]

@rocketman110us @sudoingX @Italianclownz the receipe:

[Thought Provoking Vile VR Man @tompettyflacko]

@barackomaba @Italianclownz @DJLougen from same tests had strix lean orstein 27b at mean 34.25 and max of 46 !!!

[Thought Provoking Vile VR Man @tompettyflacko]

@barackomaba @Italianclownz @DJLougen forgot I got long bench running rn but for stats on a previous run my strix lean 35 had a mean tps of 82.38 and max of 122.80. pp 772 @ 50k, 1101 @ 10k

f16 kv
b 32768
ub 2048
t 32
tb 16
draft-mtp , ngram-mod
draft n max 6
draft p min .25
ngram-mod match 32
ngram-mod max 64

[Thought Provoking Vile VR Man @tompettyflacko]

@barackomaba @Italianclownz @DJLougen and yeah the nsc ace saber series is insane work. now if we could only get the 9b with mtp to round it out... @DJLougen 👀😁