Vaibhav

high school drop out doing ai, self taught. alum @fastdotai.

[Vaibhav @vbnakrani]

https://t.co/7WtAQ0tRWb

[Vaibhav @vbnakrani]

5, post training gemma 4 for a kaggle comp

the comp only allows the 4-bit (int4) gemma 4 checkpoints. was setting up sft with a lora.

what's stored

take one weight matrix: the query projection (q_proj) of layer 0 in gemma 4 e2b. it's 2048 × 1536, so 3.1M weights.

the file doesn't hold 3.1M numbers. it holds two tensors:

weight_packed: int32, shape 2048 × 192
weight_scale: bf16, shape 2048 × 48

why int32 if the weights are int4? because there is no 4-bit data type, on GPUs or in files. the smallest normal integer is 8 bits. so 8 four-bit numbers get packed side by side into one 32-bit integer. that's why 1536 columns become 192 (1536 / 8).

unpacking one int32 by hand

the first int32 in row 0 is 0x8D38A7F5. read it 4 bits at a time, starting from the right (lowest bits first):

5, 15, 7, 10, 8, 3, 13, 8

4 bits hold 0 to 15. subtract 8 to get signed values, -8 to 7:

-3, 7, -1, 2, 0, -5, 5, 0

those are 8 weights, still as plain integers. to get real values, multiply by a scale. each run of 32 weights in a row shares one bf16 scale (1536 / 32 = 48 scales per row). here the scale is 0.0019836:

-0.00595, 0.01392, -0.00198, 0.00397, 0, -0.00989, 0.00989, 0

so each weight costs 4 bits, plus one 16-bit scale shared by 32 weights. that's 4.5 bits per weight instead of 16, or 3.6× smaller.

when serving

vllm's int4 kernel unpacks on the fly inside the matrix multiply and computes W · x. the LoRA is two small bf16 matrices, A and B, and adds its own term:

output = W · x + B · A · x

training where all the "caveats" are lol.

vllm's int4 kernels only compute W · x, forward. there's nothing in torch, transformers or peft that runs a backward pass on packed int4 for this format:

compressed-tensors (the format's own library) removed its layer that computed on packed weights. it now raises "CompressedLinear is no longer supported"

peft has LoRA layers for bnb, gptq, awq, hqq and torchao checkpoints, but none for compressed-tensors

transformers can unpack the checkpoint at load time (run_compressed=False)

so for training, every int4 weight gets unpacked once at load: weight = (int − 8) × scale, stored as bf16. now they're ordinary matrices, and autograd handles the transpose.

e2b becomes ~10 GB on the GPU, which fits on my 5090.

does that change the model? only by bf16 rounding of the product: 7 × 0.0019836 = 0.013885 is stored as 0.01392. the serving kernel also turns each weight into a 16-bit value before it multiplies, so training and serving start from the same integers and scales. the LoRA trains against the model that gets served.

i served the int4 model with the trained adapter and replayed 6 real agent requests at temperature 0, once with the adapter and once without. vllm loaded it with no unmatched-module warnings, and all 6 outputs differed. same on e4b. so the adapter is being applied.

e4b is bigger: 16.7 GB unpacked. 5.6 GB of that is one per-layer embedding table, a plain lookup with no gradient. i offloaded it to the cpu, along with the unused audio and vision parts. that leaves 9.5 GB on the GPU, and the longest 32k-token sequence trains at 29 GB.

training quant models is something.

[Vaibhav @vbnakrani]

@Wbackdown Not by much. When it comes to throughput.

Other cards have more vram but then get slower with more parallel agenst running.

I guess most optimal way would be to run on multiple 5090’s instead of one gpu with huge vram. But 5090’s are never available.

[Vaibhav @vbnakrani]

4, post training gemma 4 for a kaggle comp

side quest: renting GPUs on RunPod to test faster.

my 5090 runs 3 gemma 31b agents at once. a 4th thrashes the kv cache (343 preemptions). that's ~38 tasks/hour. so i rented bigger cards.

setup: claude code drives runpod through its MCP server. it checks prices, creates the pod, sets it up over ssh and deletes it after. the model weights come straight from hugging face . vllm runs on the pod, the agent sandboxes stay on my machine, connected by an ssh tunnel.

A40 48 GB, $0.59/h, ~1 hour, $0.57 total:

- 20 GB of kv cache vs 7 locally.

- 8 agents at once: 30 tasks/hour, no preemptions.

- 12 at once: 34 tasks/hour, but 10 of 12 hit the 20 min task limit.

RTX PRO 6000 96 GB, $2.49/h, ~50 min, ~$2.14:

- 65 GB of kv cache. 24 agents at once.

- 83 tasks/hour, ~2x my 5090.

- but each task ran ~2.7x slower (24 s per tool call vs 9 locally), so 14 of 57 tasks hit the time limit.

i you want to run smaller model with more parallel execution one big vram is not ideal. it has to share the same bus.

i am leaning towards using the a40 to its max potential. good enough vram, reasonable prices, and these machine are actually available when needed.

also benched h100, a 6000 etc.

bang_for_buck maxxing as they say.

[Vaibhav @vbnakrani]

Yea was doing trace analysis. Either they solve it in all 8 rollouts or they don’t. Only a few where the model does a hit and miss.

Bery hard for rl to get a signal. I guess more temperature. + someone said the other day be patient and let it run, one good completion all of a sudden and they start learning.

[Vaibhav @vbnakrani]

@Recursive_SI @RichardSocher I have a competitor for flint research loop that I use to do kaggle comps. 😄

[Vaibhav @vbnakrani]

@airuptwastaken yea 32k context.

[Vaibhav @vbnakrani]

@airuptwastaken 5090

[Vaibhav @vbnakrani]

@airuptwastaken yea vllm 4 bit 31b gemma 4. model takes 17-18 gigs

[Vaibhav @vbnakrani]

@ClementDelangue @netflix @OpenAI @huggingface they mighty be maxxing their views here. scandals and dooming sells way better ?

[Vaibhav @vbnakrani]

Goat

[DeepSeek @deepseek_ai]

🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.

🔗 https://t.co/JGAik1kgk0

1/4

[Vaibhav @vbnakrani]

replaced mac fans control app with this cli claude made me.

i type "fan 90" fans go brrr at 90%, type in "fan auto" and it hands back control to mac.

[Vaibhav @vbnakrani]

@mistyotter_ai I am still trying to figure out how to do the graph embeddings. Outside data works lesser until we get this sorted.

Tried like 10 different models with no result. Every flavour of gnn lol.

[Vaibhav @vbnakrani]

3, post training gemma 4 for a kaggle comp.

if i want to train on outside data (SWE-Gym, SWE-smith), every task needs what the Kaggle tasks ship with: a code graph and code embeddings.

the graph: every function, method and class in the repo is a node, and an edge means "A calls B". the embeddings: one 256-number vector per node. the agent gets 3 tools on top of these: search similar code, get a function's neighbours, and pull a small subgraph around it.

are they worth it? i ran an A/B on e2b, with and without them, they do solve more when given access to these tools so might as well keep it.

so they stay, and outside data has to look the same.

the call graph was the easy part. my builder parses the code and matches all 127 official graphs exactly: ~368k nodes, ~226k edges. its a simple call graph and can be made by parsing and transforming the ast.

the embeddings are the hard part. the hosts haven't said how they were made, so i reverse engineered what i could from 368k official vectors:

they come from a graph neural net over raw code text: a node's own text plus its neighbours' text. same text with no neighbours always gives the same vector; same text with different neighbours gives a different one.

i tested ~30 public encoders (EmbeddingGemma, CodeGemma, Qwen3-Embedding, UniXcoder, CodeBERT, CodeT5+, BGE, E5, nomic...). none is the source. they all land at R² 0.28-0.42, the same as plain char TF-IDF (0.40).

so i'm training a small GNN to imitate the official vectors. untuned it agrees 0.84 on ranking within a repo, but only 0.08-0.24 on a repo it hasn't seen, and outside repos are the whole point.

now running a larger 60-config sweep (loss, input features, depth/width) to get as close else the final option is to drop the search similar code tool.

[Vaibhav @vbnakrani]

day 2

benched all four models gemma 31b, 12b, e4b and e2b + qwen 3.8 27b as well.

i see clear scaling of model solve rate out of 120 training task as the size increases.

also the max parallel agents in can run on my 5090 also doubles with decreasing size.

qwen can run much more parallel compared to gemma 4 because it has many linear attn layers. easily handle 8 at a time. smaller kv cache.

since e2b is not hopeless thinking of setting up the rl loop on this model for a start.

[Vaibhav @vbnakrani]

Haven’t vibe coded parts of games.

I play a moba on mobile. ( please domt call me noob 😭 mobile is super accessible )

So yesterday i made a character map + item/emblem map with claude and pulled all synergies/counters between characters.

Now before a match i tell Claude what characters are already selected before my pick and inform me the best during match making.

Then after that i ask it to give me a custom build path to counter the opposite team effectively.😁

[Vaibhav @vbnakrani]

@mistyotter_ai God speed.

[Vaibhav @vbnakrani]

day 1: post-training Gemma 4 in public for a Kaggle comp.

the task: build a SWE-bench-style coding agent, scored on hidden tasks from several repos. Four Gemma 4 sizes are allowed: 31b, 12b, e4b, e2b. ( all 4 bit int 4 )

constraints are real 32k context, and 12 h on 4×L4 for ~120 tasks, so about 6 minutes per task. The 31b writes ~32 tok/s there. I ran tasks on Kaggle to measure setup and tool-call time, subtracted it from total, and that leaves ~6k output tokens per task.

so far, 31b is benchmarked and the other three sizes are running. 31b i can get 3 agents parallel on my 5090. Every run is traced and logged with its exact settings, locally and on RunPod. Surprisingly, prompt changes have barely moved the score.

Next: curating really good data and setting up an environment for RL.

[Vaibhav @vbnakrani]

day 2

benched all four models gemma 31b, 12b, e4b and e2b + qwen 3.8 27b as well.

i see clear scaling of model solve rate out of 120 training task as the size increases.

also the max parallel agents in can run on my 5090 also doubles with decreasing size.

qwen can run much more parallel compared to gemma 4 because it has many linear attn layers. easily handle 8 at a time. smaller kv cache.

since e2b is not hopeless thinking of setting up the rl loop on this model for a start.

[Vaibhav @vbnakrani]

day 1: post-training Gemma 4 in public for a Kaggle comp.

the task: build a SWE-bench-style coding agent, scored on hidden tasks from several repos. Four Gemma 4 sizes are allowed: 31b, 12b, e4b, e2b. ( all 4 bit int 4 )

constraints are real 32k context, and 12 h on 4×L4 for ~120 tasks, so about 6 minutes per task. The 31b writes ~32 tok/s there. I ran tasks on Kaggle to measure setup and tool-call time, subtracted it from total, and that leaves ~6k output tokens per task.

so far, 31b is benchmarked and the other three sizes are running. 31b i can get 3 agents parallel on my 5090. Every run is traced and logged with its exact settings, locally and on RunPod. Surprisingly, prompt changes have barely moved the score.

Next: curating really good data and setting up an environment for RL.

[Vaibhav @vbnakrani]

@thatGTMdude thats why the fancy restaurant line. if they can go to a fancy restaurant then they can afford i guess.