Vaibhav

high school drop out doing ai, self taught. alum @fastdotai.

[Vaibhav @vbnakrani]

https://t.co/7WtAQ0tRWb

[Vaibhav @vbnakrani]

@Wbackdown Not by much. When it comes to throughput.

Other cards have more vram but then get slower with more parallel agenst running.

I guess most optimal way would be to run on multiple 5090’s instead of one gpu with huge vram. But 5090’s are never available.

[Vaibhav @vbnakrani]

4, post training gemma 4 for a kaggle comp

side quest: renting GPUs on RunPod to test faster.

my 5090 runs 3 gemma 31b agents at once. a 4th thrashes the kv cache (343 preemptions). that's ~38 tasks/hour. so i rented bigger cards.

setup: claude code drives runpod through its MCP server. it checks prices, creates the pod, sets it up over ssh and deletes it after. the model weights come straight from hugging face . vllm runs on the pod, the agent sandboxes stay on my machine, connected by an ssh tunnel.

A40 48 GB, $0.59/h, ~1 hour, $0.57 total:

- 20 GB of kv cache vs 7 locally.

- 8 agents at once: 30 tasks/hour, no preemptions.

- 12 at once: 34 tasks/hour, but 10 of 12 hit the 20 min task limit.

RTX PRO 6000 96 GB, $2.49/h, ~50 min, ~$2.14:

- 65 GB of kv cache. 24 agents at once.

- 83 tasks/hour, ~2x my 5090.

- but each task ran ~2.7x slower (24 s per tool call vs 9 locally), so 14 of 57 tasks hit the time limit.

i you want to run smaller model with more parallel execution one big vram is not ideal. it has to share the same bus.

i am leaning towards using the a40 to its max potential. good enough vram, reasonable prices, and these machine are actually available when needed.

also benched h100, a 6000 etc.

bang_for_buck maxxing as they say.

[Vaibhav @vbnakrani]

Yea was doing trace analysis. Either they solve it in all 8 rollouts or they don’t. Only a few where the model does a hit and miss.

Bery hard for rl to get a signal. I guess more temperature. + someone said the other day be patient and let it run, one good completion all of a sudden and they start learning.

[Vaibhav @vbnakrani]

@Recursive_SI @RichardSocher I have a competitor for flint research loop that I use to do kaggle comps. 😄

[Vaibhav @vbnakrani]

@airuptwastaken yea 32k context.

[Vaibhav @vbnakrani]

@airuptwastaken 5090

[Vaibhav @vbnakrani]

@airuptwastaken yea vllm 4 bit 31b gemma 4. model takes 17-18 gigs

[Vaibhav @vbnakrani]

@ClementDelangue @netflix @OpenAI @huggingface they mighty be maxxing their views here. scandals and dooming sells way better ?

[Vaibhav @vbnakrani]

Goat

[DeepSeek @deepseek_ai]

🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.

🔗 https://t.co/JGAik1kgk0

1/4

[Vaibhav @vbnakrani]

replaced mac fans control app with this cli claude made me.

i type "fan 90" fans go brrr at 90%, type in "fan auto" and it hands back control to mac.

[Vaibhav @vbnakrani]

@mistyotter_ai I am still trying to figure out how to do the graph embeddings. Outside data works lesser until we get this sorted.

Tried like 10 different models with no result. Every flavour of gnn lol.

[Vaibhav @vbnakrani]

3, post training gemma 4 for a kaggle comp.

if i want to train on outside data (SWE-Gym, SWE-smith), every task needs what the Kaggle tasks ship with: a code graph and code embeddings.

the graph: every function, method and class in the repo is a node, and an edge means "A calls B". the embeddings: one 256-number vector per node. the agent gets 3 tools on top of these: search similar code, get a function's neighbours, and pull a small subgraph around it.

are they worth it? i ran an A/B on e2b, with and without them, they do solve more when given access to these tools so might as well keep it.

so they stay, and outside data has to look the same.

the call graph was the easy part. my builder parses the code and matches all 127 official graphs exactly: ~368k nodes, ~226k edges. its a simple call graph and can be made by parsing and transforming the ast.

the embeddings are the hard part. the hosts haven't said how they were made, so i reverse engineered what i could from 368k official vectors:

they come from a graph neural net over raw code text: a node's own text plus its neighbours' text. same text with no neighbours always gives the same vector; same text with different neighbours gives a different one.

i tested ~30 public encoders (EmbeddingGemma, CodeGemma, Qwen3-Embedding, UniXcoder, CodeBERT, CodeT5+, BGE, E5, nomic...). none is the source. they all land at R² 0.28-0.42, the same as plain char TF-IDF (0.40).

so i'm training a small GNN to imitate the official vectors. untuned it agrees 0.84 on ranking within a repo, but only 0.08-0.24 on a repo it hasn't seen, and outside repos are the whole point.

now running a larger 60-config sweep (loss, input features, depth/width) to get as close else the final option is to drop the search similar code tool.

[Vaibhav @vbnakrani]

day 2

benched all four models gemma 31b, 12b, e4b and e2b + qwen 3.8 27b as well.

i see clear scaling of model solve rate out of 120 training task as the size increases.

also the max parallel agents in can run on my 5090 also doubles with decreasing size.

qwen can run much more parallel compared to gemma 4 because it has many linear attn layers. easily handle 8 at a time. smaller kv cache.

since e2b is not hopeless thinking of setting up the rl loop on this model for a start.

[Vaibhav @vbnakrani]

Haven’t vibe coded parts of games.

I play a moba on mobile. ( please domt call me noob 😭 mobile is super accessible )

So yesterday i made a character map + item/emblem map with claude and pulled all synergies/counters between characters.

Now before a match i tell Claude what characters are already selected before my pick and inform me the best during match making.

Then after that i ask it to give me a custom build path to counter the opposite team effectively.😁

[Vaibhav @vbnakrani]

@mistyotter_ai God speed.

[Vaibhav @vbnakrani]

day 1: post-training Gemma 4 in public for a Kaggle comp.

the task: build a SWE-bench-style coding agent, scored on hidden tasks from several repos. Four Gemma 4 sizes are allowed: 31b, 12b, e4b, e2b. ( all 4 bit int 4 )

constraints are real 32k context, and 12 h on 4×L4 for ~120 tasks, so about 6 minutes per task. The 31b writes ~32 tok/s there. I ran tasks on Kaggle to measure setup and tool-call time, subtracted it from total, and that leaves ~6k output tokens per task.

so far, 31b is benchmarked and the other three sizes are running. 31b i can get 3 agents parallel on my 5090. Every run is traced and logged with its exact settings, locally and on RunPod. Surprisingly, prompt changes have barely moved the score.

Next: curating really good data and setting up an environment for RL.

[Vaibhav @vbnakrani]

day 2

benched all four models gemma 31b, 12b, e4b and e2b + qwen 3.8 27b as well.

i see clear scaling of model solve rate out of 120 training task as the size increases.

also the max parallel agents in can run on my 5090 also doubles with decreasing size.

qwen can run much more parallel compared to gemma 4 because it has many linear attn layers. easily handle 8 at a time. smaller kv cache.

since e2b is not hopeless thinking of setting up the rl loop on this model for a start.

[Vaibhav @vbnakrani]

day 1: post-training Gemma 4 in public for a Kaggle comp.

the task: build a SWE-bench-style coding agent, scored on hidden tasks from several repos. Four Gemma 4 sizes are allowed: 31b, 12b, e4b, e2b. ( all 4 bit int 4 )

constraints are real 32k context, and 12 h on 4×L4 for ~120 tasks, so about 6 minutes per task. The 31b writes ~32 tok/s there. I ran tasks on Kaggle to measure setup and tool-call time, subtracted it from total, and that leaves ~6k output tokens per task.

so far, 31b is benchmarked and the other three sizes are running. 31b i can get 3 agents parallel on my 5090. Every run is traced and logged with its exact settings, locally and on RunPod. Surprisingly, prompt changes have barely moved the score.

Next: curating really good data and setting up an environment for RL.

[Vaibhav @vbnakrani]

@thatGTMdude thats why the fancy restaurant line. if they can go to a fancy restaurant then they can afford i guess.