timelapse #35 (17 hrs):
- trained a theoretical reasoning chatML expert for later MoE merge
- broke apart multi-head latent attention
- sorted through HQQ, HQQ+, int8 optimizer quantization
- touched up on kv-cache optimization w/ paged attention
- planned for new years party in SF
- we cursed a qwen tokenizer then fixed it
- kicked off a merge for instructs, math, coders, and generalist 1.5B models into MoE
- learned how to only change chat templates without touching weights