Prasanna Kanagasabai (PK)

[Prasanna Kanagasabai (PK) @prasannain]

@vllm_project built a KVConnectorBase_V1 connector that restores evicted blocks faster than GPU cache hits.

pip install tierkv
https://t.co/5vMYPxIJtm

#vLLM #LocalLLM #LLMInference #opensource

[Prasanna Kanagasabai (PK) @prasannain]

Your LLM just recomputed 30,000 tokens it already did 5 minutes ago.

Every cache eviction = full prefill from scratch. 10–30 seconds. Again.

tierKV fixes this. Here's how 🧵

[Prasanna Kanagasabai (PK) @prasannain]

6/ Getting started:

pip install tierkv

One command. Hooks into vLLM or EXO automatically. No source changes.

📖 https://t.co/6aI0jq7v0k

⭐ https://t.co/VteKzPlgzX

[Prasanna Kanagasabai (PK) @prasannain]

@wuhanbat @alexabelonix Works well in larger prompts and repetitive use like iterations over a report. Shared more on metrics here : https://t.co/Ke2nd54ppP

[Prasanna Kanagasabai (PK) @prasannain]

Results: 95.2% detection (2k payloads, 110 cats), 14× FP drop, 8.6 ms p95.

Paper: https://t.co/fWJ1l5C3nr

Endorsement: https://t.co/zqDJ2xJAH5

Mech-interp thoughts welcome! #MechInterp #LLMSecurity

cc @NeelNanda5 @edoardo_debe @goodside
2/2

[Prasanna Kanagasabai (PK) @prasannain]

What if we stopped treating Sparse Autoencoders as fragile single-feature detectors for prompt injection?

Instead, I mined **conjunctive co-activation patterns** — groups of features that only fire together in real attacks.

On Gemma Scope (layers 6/12/18). 1/2

[Prasanna Kanagasabai (PK) @prasannain]

🔗 Full technical breakdown: https://t.co/nnPqhsnPZT
🤗 Ready-to-use model (HF): PKSGIN/qwen3-30b-selective-quant-MixedMPW-mlx

[Prasanna Kanagasabai (PK) @prasannain]

🧵 MLX said "pick one precision for all experts." We needed 9 at FP16, 119 at 4-bit. So we split what wasn't meant to be split. Here's how we got Qwen3-MoE-32B running in 64GB on Apple Silicon 👇 1/🧵

[Prasanna Kanagasabai (PK) @prasannain]

Why this works on MLX:
✅ gather_mm and gather_qmm are independent kernels ✅ Each block only sees local indices [0-N]
✅ mx.where on Metal GPUs is basically free

[Prasanna Kanagasabai (PK) @prasannain]

Total cost: ~$2 GPU rental, one weekend, 4 Python scripts.
Model: PKSGIN/qwen3-30b-selective-quant on HuggingFace
Full technical writeup (profiling, quantization, MLX conversion, benchmarks): https://t.co/PFL0JE80py
Security teams deserve local models that actually work.

[Prasanna Kanagasabai (PK) @prasannain]

Running a 30B LLM for security analysis means choosing between:

Cloud APIs (ship sensitive data off-prem)
Local quantized models (too degraded to be useful)

I spent a weekend building a third option.
47 tok/s on a MacBook. Zero data leakage.
🧵

[Prasanna Kanagasabai (PK) @prasannain]

Unified memory changes the game.
Traditional GPU: model must fit in VRAM (RTX 4090 = 24GB) Apple Silicon: CPU/GPU/Neural Engine share 32GB pool
18 GB model loads once. GPU runs inference in-place. Headroom for OS + KV cache.
Air-gapped ready.