Ashrf Ali

Building & breaking things in Python & AI. Local AI, SOTA AI & workflows.

[Ashrf Ali @ashrftvm]

everyone benchmarks on a 4090. I have an 18GB macbook pro, same as most of you. some numbers from a year of models that didn't fit.

[Ashrf Ali @ashrftvm]

@DogukanUrker a 1 point SWE-bench gap means less than the quant you actually run. 3060 is 12gb, what quant are you fitting once the pr merges? seen Q4_K_M vary up to 2gb between publishers for the same label, worth checking the actual file size not just the name.

[Ashrf Ali @ashrftvm]

@Beamsters1 kv cache quant is slept on. what's the raw kv size before 8bit, and what card you running this on for 80-100 tok/s at 1m context?

[Ashrf Ali @ashrftvm]

@MatthewParrott the Q4 quant of Qwen 3.8 27B is 17.77GB, doesn't fit in 16GB on its own. the only 16GB-fitting build I've seen measured is the 2-bit UD-IQ2_XXS at 9GB, and that ran 7 tok/s on an M2 Pro. what quant/backend gets you to 80

[Ashrf Ali @ashrftvm]

@ivanfioravanti does the m5 ultra tuning show up mostly at longer context or does short-prompt speed jump too

[Ashrf Ali @ashrftvm]

@Oluwaphilemon1 which q4 build, bartowski's is 17.77gb and ggml-org's is close to 19 for the same label, matters a lot for m5 ultra bandwidth math. did the 1800 prefill hold with a real document-length context or just a synthetic 8k test

[Ashrf Ali @ashrftvm]

@ivanfioravanti 42 decode is great but prompt processing is where unified memory hurts, one M3 Ultra test took ~3 min to eat a 65k prompt, 8x slower than a 5090. does DSpark move ingest at all or just decode?

[Ashrf Ali @ashrftvm]

@wesbos that's actually a cool use case, I was thinking of getting an OSS repo and classify its existing issues to look for "good-first-issue". Let me know if you do find who was the runaway demo in yours.

[Ashrf Ali @ashrftvm]

I do love OMP alot. But one thing I have noticed is that I was using it out of the box no config changes or skill toggle. Then claude and openai token were depleting much faster than antigravity ones. Have you made any optimisations for that? Which model are you primarily running with omp?

[Ashrf Ali @ashrftvm]

@btsouth Just out of curiosity, when you say 5 agents what kind of work are they doing (coding, audit, review)? Do you have your harness setup from before or did u start one randomly to use deepseek?

[Ashrf Ali @ashrftvm]

@TeksEdge 30b fitting in memory and 30b running at a usable tok/s are different claims. snapdragon x elite's npu does 14 tok/s on a tuned 4b, ryzen's npu barely beats 10 on a 20b while the plain cpu beats it. whats the actual tok/s on this dimensity chip, not just the param count it holds?

[Ashrf Ali @ashrftvm]

@ItsmeAjayKV @Zai_org @Kimi_Moonshot @deepseek_ai @Alibaba_Qwen @Eschalabs not yet. exl3 is cuda only so that's out on my mac, everything I run is gguf or mlx on metal. do the GSQ or eschalabs quants have a metal path, or are those nvidia too? if there's a mac-friendly one I'll test it on the 18GB and post numbers

[Ashrf Ali @ashrftvm]

@bigaiguy phi-4-mini is the one that surprised me, 94.6% on math-500 in about 5gb of memory. which of these are you actually running day to day vs just benchmarking once

[Ashrf Ali @ashrftvm]

@Oluwaphilemon1 same chip family only hit 32 tok/s on deepseek v4-flash, but that was the older 128gb/395 board, nobody's published real numbers on the new 192gb/495 yet. what output tok/s are you seeing on qwen3.8-flash there

[Ashrf Ali @ashrftvm]

@jovan_kis neither, q4 doesn't fit my 18GB mac. ran unsloth's 2-bit UD-IQ2_XXS at 9GB, got 7.11 tok/s. works, slow. what are you running it on that both fit? and did the 58% cut hold accuracy on your side or did you see it slip anywhere

[Ashrf Ali @ashrftvm]

@mr_r0b0t @NVIDIAAI 3.0 is probably the sweet spot yeah. Qwen3.6 27B at 3-bit did about 10 tok/s on 18GB, 4-bit needed 14 to 16GB and started swapping. haven't stacked a draft model on top yet. which draft are you pairing, and is that exl3 or gguf? exl3 is dead on metal

[Ashrf Ali @ashrftvm]

@mr_r0b0t @NVIDIAAI looking forward to it!

[Ashrf Ali @ashrftvm]

@mr_r0b0t @NVIDIAAI gsm8k 0-shot is table stakes now yeah. multiturn at low is the one I'd watch, does it still hold constraints from 4 turns back or start dropping them? which quant and hardware are you on, I want to know if this fits an 18GB mac

[Ashrf Ali @ashrftvm]

@WescheNex1q glm-5.3-flash is 320b total, 18b active. smallest build unsloth ships is still 120gb, needs a 128gb box. what's your spark setup running, quant and size on disk?