@_weilix I hit the exact same problem but it is on apple silicon, so i have no ram to dump GDN state (every ram is used by gpu) and have to left on ssd. How can you manage that part? My bottleneck is that I cant reduce ssd writing without sacrifing the re-prefill gap.
@_ryu15_ It would have been a big work from my side lol. I am running on apple's native metal kernel not the llama cpp. Let me figure out how I possibly can get there. Keep in touch! Thanks buddy!
[1/5] I am happy to share JustFit v2: the local mac inference runtime.
Same coding agent, same task, same 24 GB Mac. Only the inference engine changed.
JustFit v2 finished in 66 min with 163K tokens of context and zero compactions, while the other two engines were still going.
[4/5]
How: compressed (TQ4) KV that attention reads directly, components loaded only for the phase that needs them, committed state that survives restarts, and CPU/GPU/Neural Engine coordination per chip.