Stephen Panaro

making coffee and other things. @BrewTimerApp

[Stephen Panaro @flat]

“We won’t run it in digital because we’re purists and maniacs.”

[Stephen Panaro @flat]

Excellent rundown of the current state of LLM quantization.

[Artur Chakhvadze @norpadon]

We are doing really cool hard tech at @trymirai, but until recently our social media feeds were full of linkedinish cringe. We decided to fix it and share more technial content

I am currently working on our quantization pipeline, so here is a thread about LLM quantization

[Stephen Panaro @flat]

@mweinbach Thought it was a cleaner summary. Possible I’ve just never looked beyond that though 🤔

[Stephen Panaro @flat]

@mweinbach Huh weird. I also got it from Opus today too but had never seen it before. Assumed it was an Anthropic issue

[Stephen Panaro @flat]

@mattcassinelli @tylerangert Oh for sure some cool non-MLX stuff this year. When Google released their on-device base model + LoRA, I crossed my fingers Apple would do the same. And they did!

Am actually curious now if anyone has shipped an AFM LoRA.

[Stephen Panaro @flat]

@mattcassinelli @tylerangert Having been away from MLX for a bit, what did I miss? (Or is this about non-MLX like CoreML/Foundation Models?)

[Stephen Panaro @flat]

https://t.co/gpuCdSfrzu

[Stephen Panaro @flat]

This ButterflyQuant paper looks neat, but also a little sus:
- no code
- no comparison against its closest relative (SpinQuant)

A good test project for coding agents?

[Stephen Panaro @flat]

@anemll How’s the speed compare to float16 inputs?

[Stephen Panaro @flat]

@anemll

[Stephen Panaro @flat]

Incoming new coremltools looks like it has some nice bits:
- 8 bit input/output tensors (previously all 8bit compute was kept internal)
- >1 input can be enumerated shapes (👀ANE)

[Stephen Panaro @flat]

https://t.co/i3sCue1Prt

[Stephen Panaro @flat]

Looks like it’s happening!

[Stephen Panaro @flat]

Wonder if we’re gonna get a new version of coremltools. Last year it dropped on Monday.

[Stephen Panaro @flat]

@AaronWeiHuang 👋 Any chance y’all plan to release code for DBellQuant?

[Stephen Panaro @flat]

@cloneofsimo Can use this for quantization too:

https://t.co/6YjulZBg0J

[Stephen Panaro @flat]

🐙: https://t.co/M6F5EjzFrS

📄: https://t.co/aaPSq8iCmq

(R₅ is a rotation matrix, so its transpose is its inverse and it naturally cancels out in [email protected])

[Stephen Panaro @flat]

Turns out you don’t need R₅⁻¹ at all. 🫠 Fusing into Q and K is enough!

Cool paper from Qualcomm explains this and a few similar transforms.

No code in the paper, so gist proof👇

[Stephen Panaro @flat]

Liking the line of research where you multiply LLM weights by rotation matrices and the model still works.

Most do it in between layers, but you can also sneak one between Q/K and RoPE.

Extra parameters? None.
Useful? …Maybe.
Cool? I think so.

(See R₅ below.)

[Stephen Panaro @flat]

The python library is interesting too. “Download files”:

https://t.co/swlL1C8AFf

[Stephen Panaro @flat]

Curious about the Apple Foundation Model architecture? I updated my netron fork to visualize the draft model*.

*they say it might differ from the real model but looks convincing to me

[Stephen Panaro @flat]

See for yourself:

1. Get the adapter training toolkit: https://t.co/FBoPMOyOFd
2. Clone: https://t.co/KBnvGSQh5I
3. Edit https://t.co/NwMlsQrn2a:
- delete all functions except the first
- rename it to: func main<ios18>(
4. Follow readme to start netron, and open the .mil