Praful🔮

ML @VISA | Working on Small Models, LLD & Open Source | Ex-@ProdigalTech @OmdenaAI, BimaPe (now @VerakInsurance)

[Praful🔮 @PrafulMohanan]

Had trouble finding generation parameters for a small language model, built a small tool to solve for that

Release blog - https://t.co/27IPvT3MQo

[Praful🔮 @PrafulMohanan]

@benno_krojer Yes, the degradation trend is minor here except for BnB NF4 quantization. It could be stronger for logitlens in the early layers due to the hidden states not being yet aligned towards next token prediction

[Praful🔮 @PrafulMohanan]

Wrote up a follow up experiment to LatentLens (@benno_krojer et al.) examining how quantizing a VLM's LLM backbone affects interpretability of visual tokens.
Tested Qwen2-VL-7B across multiple quantization setups.
Few things I found:
- GPTQ INT4 preserves visual-token representations better than BnB INT8
- Degradation is not uniform across layers, it's concentrated at a specific mid-layer

Read more about the experiment on my blog 👇🏼

[Praful🔮 @PrafulMohanan]

https://t.co/TrX9Z81wuN

[Praful🔮 @PrafulMohanan]

Use Partial Dependency Plots
- to understand global feature importance
- to check if any non-linear relationship exists between feature & prediction

Use SHAP
- to understand local feature importance -- takes interactions between features into account
- to answer the question --- why did the model predict what it did?

[Praful🔮 @PrafulMohanan]

TIL 💡
Using Option + Left/Right Arrow key can cycle you through start/end of a word in an IDE.
Extremely useful while writing code, have been using Command + Left/Right to navigate code.

[Stanford AI Lab @StanfordAILab]

arXiv -> alphaXiv

Students at Stanford have built alphaXiv, an open discussion forum for arXiv papers. @askalphaxiv

You can post questions and comments directly on top of any arXiv paper by changing arXiv to alphaXiv in any URL!

[Praful🔮 @PrafulMohanan]

@bhutanisanyam1 @Meta This is huge, Congrats Sanyam!

[Praful🔮 @PrafulMohanan]

@StasBekman Would be interesting to know how this fares on a single user single request case, how fast are these compared to something like exllama_v2 (excluding throughput)

[Praful🔮 @PrafulMohanan]

architecture is largely the same with minor modifications
• tokenizer built has better compression rate --> can represent more data in less tokens (optimizing on compute)

[Praful🔮 @PrafulMohanan]

llama 3.1 technical report is full of insights
as usual more & better data(& money) is key

- trained on 10x more tokens than llama 2
- use a lot of rules(heuristics + model) to filter data and keep high quality data
- use annealing (sample higher quality data while decreasing lr to 0 while training)

[Praful🔮 @PrafulMohanan]

• use scaling law both to identify what is the lowest loss that they can achieve given a compute budget -- and they do almost accurately predict it!
• they use annealing both while initial pre-training(although they don't mention it specifically) & at the last stage
• pre-train on long context separately in the 2nd stage

[Praful🔮 @PrafulMohanan]

@swapnilsinha07 Love the setup!
Congrats!
Worked with Nayhrit & Adithya earlier, great folks to be around!

[Praful🔮 @PrafulMohanan]

Ways to serve a exllama_v2 quantized model in realtime using batching at blazing fast speed

1. Use EricLLM with exllama_v2 quantization - https://t.co/HrvESxtUah
2. Use Aphrodite - https://t.co/N6PkV9Wfff

Yet to test both of these out and how it fairs with vLLM.
#LLM

[Praful🔮 @PrafulMohanan]

massively underrated stuff if you're in tech
1. having a util library of the most commonly used code - loading & saving data, parallelising function calls (asyncio, threading, multiprocessing), batching, etc
2. having a really good local dev setup - ssh server + linters + copilot

[Sayak Paul @RisingSayak]

Wow!

This was raw and fulfilling.

https://t.co/C4XbIxxOrs

[Praful🔮 @PrafulMohanan]

Use exllama-v2 if you are looking for a very fast inference engine to use in a real-time setting.

On average for a 7B model you get ~85 t/sec on a RTX A4000, which is good enough. For comparison raw HF inference in bfloat16 gives you ~25 t/sec.

#LLMs

[Praful🔮 @PrafulMohanan]

Optimising for latency & cost while using OpenAI APIs in JSON mode:

Make the model
1. Output a minified version of JSON
2. Avoid outputting keys which have null values
3. Focus on making the output as short as you can

cutting x% of output tokens saves ~x% of latency!

[Praful🔮 @PrafulMohanan]

@_kothariji why is that

[Praful🔮 @PrafulMohanan]

met a good doggo 🐶 on the way, accompanied us almost till the end

[Praful🔮 @PrafulMohanan]

should do treks more often ⛰️