THIS IS A PURE F*CKING GOLDMINE
karpathy’s open-source loop runs ~100 ai experiments while you sleep.
this guide shows the part nobody tells you: a lot of those “wins” can be pure luck.
the obvious keep rule overstated its own result in 1,999 of 2,000 simulated runs. dashboard +2.37. reality +1.16.
what’s inside:
→ the 6 parts every ai loop needs
→ why the evaluator is the real product
→ how agents cheat their own tests
→ the keep rule that kills lucky numbers
→ the folder + a 15-line runner you can copy
prompts were the old skill. loops are the new one.
watch the balloon pop, then read the article ↓
save it before your next overnight run
your ai loop has been gaslighting you.
+2.37 on the dashboard. +1.16 in reality.
i caught it lying in 1,999 out of 2,000 runs.
same change, tested 5 times: +2.00, +1.58, +1.25, +0.90, +0.48. a loop that runs once keeps whichever number it drew. that’s the whole bug.
the dumbest version is even worse. half the changes it kept made things worse, and 46% of those loops ended below where they started. the dashboard still said green.
one rule fixes most of it. it costs 6× the evals and it’s worth every one.
full breakdown in the article below. watch the spider first.
THIS IS A PURE F*CKING GOLDMINE
karpathy’s open-source loop runs ~100 ai experiments while you sleep.
this guide shows the part nobody tells you: a lot of those “wins” can be pure luck.
the obvious keep rule overstated its own result in 1,999 of 2,000 simulated runs. dashboard +2.37. reality +1.16.
what’s inside:
→ the 6 parts every ai loop needs
→ why the evaluator is the real product
→ how agents cheat their own tests
→ the keep rule that kills lucky numbers
→ the folder + a 15-line runner you can copy
prompts were the old skill. loops are the new one.
watch the balloon pop, then read the article ↓
save it before your next overnight run
prompts are one line of the system now.
this is the rest: the evaluator, memory, stop rules, and the noise that made one loop report 2× its real gain.
19 diagrams, a folder you can copy, and the runner. https://t.co/GwEWizyMd4
prompts are one line of the system now.
this is the rest: the evaluator, memory, stop rules, and the noise that made one loop report 2× its real gain.
19 diagrams, a folder you can copy, and the runner. https://t.co/GwEWizyMd4