Germán Milano

Data scientist, tech enthusiast, music lover from Uruguay 🇺🇾🎧🤖📈🎶 Passionate about innovation and progress through data and technology

[Martin Bel @__mbel__]

We reached #1 at the SWE-bench-Multimodal benchmark with the @Globant Code Fixer Agent.

Kudos to team that made it happen!
@german_milano, @__mbel__, @mperezjodal, @RodolfoAni72304
@GMilano @jlamasrios

[José Lamas @jlamasrios]

Congrats to the Data Science team behind Code Fixer at Globant!! After ranking #1 on SWE-Bench Lite in Nov ’24, their major upgrades in multimodal preprocessing, prompt design, and large codebase navigation (especially for complex frontend/backend stacks) made it #1 on SWE-Bench Multimodal.

@german_milano, @__mbel__, @mperezjodal, @RodolfoAni72304

https://t.co/zF3g327az2

[Andrej Karpathy @karpathy]

I don't have too too much to add on top of this earlier post on V3 and I think it applies to R1 too (which is the more recent, thinking equivalent).

I will say that Deep Learning has a legendary ravenous appetite for compute, like no other algorithm that has ever been developed in AI. You may not always be utilizing it fully but I would never bet against compute as the upper bound for achievable intelligence in the long run. Not just for an individual final training run, but also for the entire innovation / experimentation engine that silently underlies all the algorithmic innovations.

Data has historically been seen as a separate category from compute, but even data is downstream of compute to a large extent - you can spend compute to create data. Tons of it. You've heard this called synthetic data generation, but less obviously, there is a very deep connection (equivalence even) between "synthetic data generation" and "reinforcement learning". In the trial-and-error learning process in RL, the "trial" is model generating (synthetic) data, which it then learns from based on the "error" (/reward). Conversely, when you generate synthetic data and then rank or filter it in any way, your filter is straight up equivalent to a 0-1 advantage function - congrats you're doing crappy RL.

Last thought. Not sure if this is obvious. There are two major types of learning, in both children and in deep learning. There is 1) imitation learning (watch and repeat, i.e. pretraining, supervised finetuning), and 2) trial-and-error learning (reinforcement learning). My favorite simple example is AlphaGo - 1) is learning by imitating expert players, 2) is reinforcement learning to win the game. Almost every single shocking result of deep learning, and the source of all *magic* is always 2. 2 is significantly significantly more powerful. 2 is what surprises you. 2 is when the paddle learns to hit the ball behind the blocks in Breakout. 2 is when AlphaGo beats even Lee Sedol. And 2 is the "aha moment" when the DeepSeek (or o1 etc.) discovers that it works well to re-evaluate your assumptions, backtrack, try something else, etc. It's the solving strategies you see this model use in its chain of thought. It's how it goes back and forth thinking to itself. These thoughts are *emergent* (!!!) and this is actually seriously incredible, impressive and new (as in publicly available and documented etc.). The model could never learn this with 1 (by imitation), because the cognition of the model and the cognition of the human labeler is different. The human would never know to correctly annotate these kinds of solving strategies and what they should even look like. They have to be discovered during reinforcement learning as empirically and statistically useful towards a final outcome.

(Last last thought/reference this time for real is that RL is powerful but RLHF is not. RLHF is not RL. I have a separate rant on that in an earlier tweet
https://t.co/RMIpFPVpuM)

[François Chollet @fchollet]

If your learning algorithm is based on correlation rather than causation, it will struggle with overfitting. To understand something is to identify its minimal sufficient causal mechanisms. Parsimony isn't just elegance, it's generalization robustness.

[Germán Milano @german_milano]

8/8
Get full access to the report here: https://t.co/yRiV3G2GXD

We’d love to hear if anyone has done similar studies!

[Germán Milano @german_milano]

1/8
Everyone's doing great work building agents for bug fixing. But I'm curious—does anyone have insights on which types of issues your agents are good at, and which ones pose the biggest challenges? 🧵

Cc: @__mbel__ @jlamasrios @RodolfoAni72304 @GMilano @mperezjodal

[Germán Milano @german_milano]

7/8
We’re committed to evolve the @Globant Code Fixer Agent and expand its capabilities beyond bug fixing, while actively collaborating with the community.

[elvis @omarsar0]

The hype around o3 is out of control.

It’s not AGI, it’s not the singularity, and you definitely don’t have to change your worldview.

In fact, the public doesn’t even have access to the models so how can anyone claim any of the above.

I appreciate how the OpenAI researchers presented o3.

I encourage folks to checkout the original presentation on YouTube.

Don’t fall for all the hype threads you see here on X.

OpenAI made it clear that there lots of things to improve on.

It’s exciting yes but the headlines are misleading and benchmark results don’t really say much these days.

Hoping these words balance your timeline a bit. Share if you think it helps.

[Germán Milano @german_milano]

6/6

Also (very) important: have some strategic, midterm objectives which promise to get you one step closer to your “north star”—deliver them piecewise to create value ASAP.

[Germán Milano @german_milano]

During our talk today, someone asked us for advice on building an AI Agentic System such as Code Fixer.

Instead of saying 'this is what you should do,' we want to share what worked for us and the lessons we’ve learned (so far!). 🧵

[Microsoft AI Frontiers @ms_aifrontiers]

Happening now!

cc: @Globant @GMilano @mperezjodal @german_milano @RodolfoAni72304 @__mbel__ @jlamasrios

[Germán Milano @german_milano]

5/6

Always be mindful of the scaling laws + The Bitter Lesson!

[Germán Milano @german_milano]

4/6

Whatever looks promising, evolve it one step further. At each step, ask yourself: Can we quickly validate improvements / perform error analysis? If we’ve improved, it’s still promising, so evolve it again. Otherwise, discard and go back to reading.

[Germán Milano @german_milano]

3/6

Our ethos: Applied research FTW! Read. Test. Fail. Repeat (a lot, and fast)

[Germán Milano @german_milano]

2/6

X is a great knowledge source, but it’s important to vibe-check everything to clear the hype out of it.

[Germán Milano @german_milano]

1/6

Navigating the solution space, beyond the hype: there are just too many possibilities (architectures, foundation models, etc.). How do we know what’s best when working in SOTA topics?