IHC

[IHC @ihc_sim]

@KuittinenPetri @jun_song You can find one on backmarket for half this price from time to time. But you need to be quick!

[IHC @ihc_sim]

@jun_song Agree! M1 Ultra even better for a slight premium, 800GB/s bandwidth, 2x GPU. 64GB is the sweet spot for quantized models + KV cache. 128GB great for ~100B MoE but GPU can’t handle dense >40B at usable speed. Best price to perf hardware right now

[IHC @ihc_sim]

@ivanfioravanti Probably using ngram-mod parameters :
https://t.co/CtzAR3WZpt