Amin Heidari

Computer Science

[Amin Heidari @aminlight51]

@ivanfioravanti Would you also upload your quantized model to hf? That would be awesome

[Amin Heidari @aminlight51]

@mehdizxx ما که جنسامونو خریدیدم خداروشکر

[Amin Heidari @aminlight51]

@Alibaba_Qwen My GPU is getting wrecked trying to quantize it. Lets goooo

[Amin Heidari @aminlight51]

@populartourist Why is the 3.5 in every section better than the 3.6??

[Amin Heidari @aminlight51]

@Prince_Canuma @jtdavies I will do it.

[Amin Heidari @aminlight51]

@Prince_Canuma @jtdavies Yes. I created a bug report on the gh

[Amin Heidari @aminlight51]

@Prince_Canuma @jtdavies I will do that. The generation looks like this: ir generates some words, then it stops, and them again… the intervalls are like every second. The power usage is 40-60w for the whole system(very very low) and gpu utilization max 80%

[Amin Heidari @aminlight51]

@ivanfioravanti I also got that feeling. Still, I saw some posts that the ds4 was running on M5 Max with like over 35 t/s, which is way higher than on the antirez ds4. If you could tell me where I can find that fork of the mlx_lm, I would very much appreciate it

[Amin Heidari @aminlight51]

@jtdavies @Prince_Canuma What model exactly??

[Amin Heidari @aminlight51]

@ivanfioravanti using deepseek v4 flash with ca 50K tokens I was able to patch the mlx_vlm so that I can run osmapi/Step-3.7-Flash-OptiQ-3.7bpw-mlx . this is just awesome

[Amin Heidari @aminlight51]

@MiaAI_lab @NVIDIAAI That's awesome

[Amin Heidari @aminlight51]

@MiaAI_lab @NVIDIAAI How does it hold up on the longer context sizes?? like over 60k tokens?

[Amin Heidari @aminlight51]

How did I sleep on AnythingLLM @AnythingLLM for so long? This think is just amazing

[Amin Heidari @aminlight51]

@Michaelzsguo Just use Tailscale.

[Amin Heidari @aminlight51]

@ivanfioravanti And it's just the 60 core version.

[Amin Heidari @aminlight51]

@ivanfioravanti lets gooooo, its like 600 t/s

[Amin Heidari @aminlight51]

@vmarquesa @ivanfioravanti Yes we are talking about the 4bit mlx.