Ian Hailey

Learning

[Ian Hailey @IanHailey]

@vinitpaul6 @0xSero “for the Spark”

[Ian Hailey @IanHailey]

@Hikari_07_jp Nice work.

[Ian Hailey @IanHailey]

@Hikari_07_jp @sakurayukiai Good idea to prioritise the key precision. Could be interesting to run a needle in a haystack (NIAH) test on the variants. https://t.co/QmHGU4v0PB

[Ian Hailey @IanHailey]

@0xSero Seems @__tinygrad__ take these apart when fitting them into their TinyBox.

[Ian Hailey @IanHailey]

@Hikari_07_jp Actually it seems what you did with BF16 is the preferred solution as this is what Stepfun themselves have now added!

[Ian Hailey @IanHailey]

@0xSero Well deserved IMO.

[Ian Hailey @IanHailey]

@Hikari_07_jp I will be sad if it no longer fits it 192GB

[Ian Hailey @IanHailey]

@0xSero Not looking good for 192GB people :(

[Ian Hailey @IanHailey]

@Hikari_07_jp Would using the FP8 MTP weights (which I guess are 50% smaller) make much difference to the over decode speed (vs these BF16)?

[Ian Hailey @IanHailey]

@Hikari_07_jp Did you reconstruct the MTP layers from the base model?

[Ian Hailey @IanHailey]

@Hikari_07_jp Nice, thanks.

[Ian Hailey @IanHailey]

@minviable_org @gospaceport Which is not a UPS! Apart from having multi 100ms switch overtime’s it does not offer any line conditioning or protection.

[Ian Hailey @IanHailey]

@0xSero Nice, this must have taken ages to put together!

[Ian Hailey @IanHailey]

@0xSero Neat that they provided a docker image.

[Ian Hailey @IanHailey]

@Hikari_07_jp Any ideas why NVFP4 quant is missing MTP layers unlike the base and FP8?

[Ian Hailey @IanHailey]

@StepFun_ai Thanks for release this.

[Ian Hailey @IanHailey]

@0xSero @StepFun_ai @scaling01 I hope it’s not a nightmare of vllm patches to run like DSv4 on the 6000!

[Ian Hailey @IanHailey]

@realcheeker This is with max-model-len 65535 as per your run_dsv4.sh? Have you tried it with 256k max len?

[Ian Hailey @IanHailey]

@realcheeker @unwitty Excellent, been trying to get a docker image build for this for a while, which HF model did you try, any with MTP (e.g. Canada-Quant or LordNeel)?

[Ian Hailey @IanHailey]

@0xSero Do you have a DSv4 compatible docker runtime image you can share for the 6000 also?