Vincent Hellendoorn

AI for Code Researcher at Google DeepMind. Formerly Assistant Professor @ CMU.

[Luís F. Gomes @luisfgomes24]

Happy to announce our paper has been accepted at @ICSEconf 🎉
💭 What if your quick whiteboard sketch could generate working code?

Check out all the details here: https://t.co/Mo3tTTU7HE

@VHellendoorn
@JAldrichPL
@rmaranhao

#AI4SE #ICSE2025 #SketchToCode #SE #Software

[Vincent Hellendoorn @VHellendoorn]

FYI, I’ll be migrating research updates over to Bluesky (same handle).

[Vincent Hellendoorn @VHellendoorn]

Just 2 months till the ASE'24 Industry track deadline!
Have exciting & practically grounded Automated SE research to share? Submit it here:
https://t.co/byi19V0o23
Interested in reviewing 3-4 papers? Apply here:
https://t.co/21Mgd4xOTF
Senior PhDs w/internship expertise welcome!

[Vincent Hellendoorn @VHellendoorn]

@solvay_1927 Hi! The HE numbers they report are on a relatively easy subset of the benchmark’s problems (see page 4), presumably as a kind of obfuscation, so the pass rates are about 2-3x higher than on the full set.

[Vincent Hellendoorn @VHellendoorn]

For fun, a back-of-the-envelope stab at GPT-4's size (assuming it's a dense Transformer) gets me around 0.5 Trillion parameters trained with 2T tokens.

The details: Fig. 2 in the paper (https://t.co/GN4rS7q0wL) shows a series of much smaller models to establish a scaling law 1/?

[Nikitha Rao @nikitharao006]

Copilot showed that language models can help programmers. Can we do the same for low-code programmers? In our new work we show that we can, via smarter search! Collaboration with @IBMResearch @jsntsay, Martin Hirzel, Kiran Kate, and @vhellendoorn. https://t.co/MO021V8YCM (1/8)

[Vincent Hellendoorn @VHellendoorn]

@modest_maker Voice might help with the “slow” part, but I’m thinking more contextual. There are tons of intuitive interfaces that don’t rely on language already, like light switches, remote controls, cameras. Less language-bound predictive AI could power all sorts of next-gen applications

[Vincent Hellendoorn @VHellendoorn]

I’m looking forward to LLM-caliber AI that doesn’t use chat as its interface.

Language is a very general medium (& clearly makes for convincing demos!) but it’s also quite clunky and slow. For many tasks, more context aware models could skip chatting altogether.

[Vincent Hellendoorn @VHellendoorn]

@MarkHolum @BlancheMinerva So the natural next step is to focus on new ways to improve, like multimodal learning, perhaps embodied agents, and perhaps leaving behind SGD+back-prop. Some of those will take quite a while to work at the scale that LLMs do now, hence the potential inflection point.

[Vincent Hellendoorn @VHellendoorn]

As we enter the $B stage of AI scaling, it feels like we’re hitting an inflection point. LLMs need exponential growth in investments to improve, which won’t be realistic much longer, while RLHF style tuning is capped by human effort (& leads to boring, poorly calibrated models).

[Ethan Caballero @ethanCaballero]

Anthropic plans to build a model tentatively called Claude-Next 10X more capable than today’s most powerful AI that'll require spending $1Billion over the next 18 months.

“Companies that train the best models will be too far ahead for anyone to catch up”
https://t.co/YLCfNlXDnX

[Vincent Hellendoorn @VHellendoorn]

@MarkHolum @BlancheMinerva The big picture here is that each generation of GPT-2 -> 3 -> 4 has cost about 100x as much to train and used at least as much more compute. But (a) we can’t easily go another 100x, and (b) it wouldn’t help as much if we did, per the GPT-4 scaling findings shown below.

[Vincent Hellendoorn @VHellendoorn]

@ElanRosenfeld @zacharylipton Haha yeah, no science here. But let’s say 70/30. The question we debated back then was whether math requires a kind of uniquely human creativity or whether it mainly takes *a lot* of exploration. ML could always do the latter, while LLMs now imitate the former surprisingly well.

[Daye Nam @daye_nam]

Can GPT-4 help developers learn a new library? We built a plug-in to bring GPT-4 into the IDE. Want to participate in a @SCSatCMU @S3DatCMU user study to help us understand the future of software development? Please apply by filling out this form: https://t.co/fepLBl98zP

[Vincent Hellendoorn @VHellendoorn]

@AiEleuther Love this framework, glad I could contribute. But full credit to @tri_dao on this one, both for creating FlashAttention and making it so easy to port that this barely took me an hour!

[Vincent Hellendoorn @VHellendoorn]

Regardless, rumor has it that the next models are already training, now using 25K A100 GPUs. With improved efficiency, that likely means another 3x the compute (300K PFLOP/s days in 2mo of training). The key bottleneck now looks to be the amount of clean data. 13/end

[Vincent Hellendoorn @VHellendoorn]

Another: their data might have much less Python in the mix so each of the HE results actually required more training & larger models than I guesstimated. At the low end, they might have used "only" twice the compute of PALM-540B. 12/?