Adarsh Singh

💻 MERN Stack Developer | Backend & AI 🤖
🚀 Building APIs, automation & scalable apps
⚡ JavaScript • Node.js • React
📍 Mumbai, India

[Adarsh Singh @AdarshSingh1606]

Building an LLM inference engine from scratch in C++ to run Llama 3.2 3B locally.

Next up: forward pass, KV cache and quantization. I'll share tokens/sec as it gets faster.

#buildinpublic #cpp #LLM

[Adarsh Singh @AdarshSingh1606]

@ParthBhovad @OjasSharma276 +1

[Adarsh Singh @AdarshSingh1606]

@ParthBhovad Yeah , you are right !

[Adarsh Singh @AdarshSingh1606]

I have a idea ,when a person adds products to their cart multiple e-commerce platforms and social media platforms .Later when they want to buy the product have to visit multiple platforms. to find the product and then buy it To solve this problem, I'm introducing. Universal Cart

[Adarsh Singh @AdarshSingh1606]

Llama 3.2 3B engine in plain C++: a hand-written AVX2 kernel for the 4-bit matmul took it from 1.7 to 4.1 tok/s on a Ryzen 5 5500U.

int8 activations + maddubs = 32 multiply-adds per instruction.

Next: a logit check against Hugging Face.

https://t.co/YqsvE2gwGf

[Adarsh Singh @AdarshSingh1606]

@_Sec0ps @ETHGlobal @world_id @abhishekislinux @0xborax @zimalkumud97 Congratulations bro 🎉

[Adarsh Singh @AdarshSingh1606]

Llama 3.2 3B now chats from my own C++ engine.
• 4-bit weights, GQA, KV cache, SwiGLU
• Python chat wrapper over the C++ binary
• 1.7 tok/s on a Ryzen 5 5500U, no SIMD yet

Next: AVX2 kernel for speed, then a logit check vs HF.

https://t.co/YqsvE2gwGf
#buildinpublic #LLM

[Adarsh Singh @AdarshSingh1606]

@aksenHQ Not measured on this build yet. The 4-bit matvec has a bug I'm fixing first, so a comparison now wouldn't mean much. I'll post the max logit diff vs HF once it's in.

[Adarsh Singh @AdarshSingh1606]

Llama 3.2 3B in plain C++, today:
• forward pass: GQA attention + KV cache, SwiGLU
• sampling: temperature, top-k, top-p

Next: fix the 4-bit matvec and match logits with HF.

Open to AI inference and backend roles.

https://t.co/YqsvE2gwGf
#buildinpublic #LLM

[Adarsh Singh @AdarshSingh1606]

Llama 3.2 3B engine in plain C++: RMSNorm and RoPE are in.

RoPE uses the half-split layout (dim i pairs with i+64), which the Hugging Face weights expect. Pair adjacent dims and attention breaks silently.

Next: attention + KV cache.
https://t.co/YqsvE2fYQH

[Adarsh Singh @AdarshSingh1606]

@imThejas18 @Cyient Interested in the Full-Stack track, Node.js, ~11 months exp. Could you follow back so I can DM my resume?

[Adarsh Singh @AdarshSingh1606]

@gol_tiro Fair question. That comparison isn't in the public repo yet. I'll push the full forward pass with the test and report the logit error both before and after 4-bit quantization, so it's clear what the quant costs.

[Adarsh Singh @AdarshSingh1606]

Update on my Llama 3.2 3B engine in plain C++:
• 4-bit group-wise quant: 6.4 GB down to 2.2 GB
• KV cache + GQA (about 470 MB at 2K context)
• OpenMP + AVX2 on a Ryzen 5 5500U
• Logits match Hugging Face within 6e-5
Next: hand-written SIMD.
https://t.co/YqsvE2fYQH

[Adarsh Singh @AdarshSingh1606]

@aksenHQ No target number yet. I'd rather measure than guess. Hand-written AVX2 SIMD is next, then I'll post real tokens/sec from my Ryzen 5 5500U. At batch 1 it's memory-bound, so 4-bit weights (2.2 GB vs 6.4 GB) should help a lot.

[Adarsh Singh @AdarshSingh1606]

Building an LLM inference engine from scratch in C++ to run Llama 3.2 3B locally.

Next up: forward pass, KV cache and quantization. I'll share tokens/sec as it gets faster.

#buildinpublic #cpp #LLM

[Adarsh Singh @AdarshSingh1606]

@velgberggroup Weights only for now. The KV cache is still full precision, about 470 MB at 2K context even with GQA (8 KV heads vs 24 query). Quantizing it is tempting, but hand-written SIMD and tokens/sec numbers come first.

[Adarsh Singh @AdarshSingh1606]

@abhinayguptha On it. KV cache and 4-bit quant are both in now, so tokens/sec numbers are next.

[Adarsh Singh @AdarshSingh1606]

@deeponailabs Exactly. That's why I went 4-bit: 6.4 GB down to 2.2 GB, so roughly 3x less to read from RAM per token. Thanks!

[Adarsh Singh @AdarshSingh1606]

@helloshiva0801 x86 first. I'm on a Ryzen 5 5500U, and GCC vectorizes the 4-bit loop with AVX2. Apple Silicon comes later.

[Adarsh Singh @AdarshSingh1606]

Why C++ and not just an API? I want to understand what actually happens per token: matmuls, attention, memory layout.

Current stack: plain C++, no frameworks. Target: Llama 3.2 3B on a laptop CPU.

If you've built something similar, I'd love your tips.