Pete Walsh

Research Engineer @allen_ai | Python | Rust | Neovim

[Pete Walsh @epwalsh]

@saurabh_shah2 I think you’re doing it wrong

[Pete Walsh @epwalsh]

@soldni @finbarrtimbers This is it. But if you want to use dataclasses AND be able to ser/deserialize to/from json/yaml, I wrote a little library for that: https://t.co/8zPux4GAVG

Eventually we're going to replace omegaconf with that in olmo-core.

[Pete Walsh @epwalsh]

@saurabh_shah2 @finbarrtimbers We use FSDP2 now (what torchtitan uses) because it’s just DTensor under the hood, which plays nicely with other axes of parallelism. I don’t think there’s much difference in performance between the old FSDP and the new.

[Ai2 @allen_ai]

Introducing olmOCR, our open-source tool to extract clean plain text from PDFs!

Built for scale, olmOCR handles many document types with high throughput. Run it on your own GPU for free—at over 3000 token/s, equivalent to $190 per million pages, or 1/32 the cost of GPT-4o!

[Pete Walsh @epwalsh]

@soldni using Rust I hope

[Kyle Lo @kylelostat]

kicking off 2025 with our OLMo 2 tech report while payin homage to the sequelest of sequels 🫡

🚗 2 OLMo 2 Furious 🔥 is everythin we learned since OLMo 1, with deep dives into:

🚖 stable pretrain
🚔 lr anneal 🤝 data curricula 🤝 soups
🚘 tulu post-train
🚜 compute infra

👇🧵

[Pete Walsh @epwalsh]

@Tanishq97836660 Hey Tanishq, really interesting work. I'm curious if your low-precision training setup involves keeping the main copy of the weights in full precision (like with how torchao's Float8Linear works)?

[Niklas Muennighoff @Muennighoff]

Releasing OLMoE - the first good Mixture-of-Experts LLM that's 100% open-source
- 1B active, 7B total params for 5T tokens
- Best small LLM & matches more costly ones like Gemma, Llama
- Open Model/Data/Code/Logs + lots of analysis & experiments

📜https://t.co/Vpac2q90CS
🧵1/9

[Jesse Dodge @JesseDodge]

Congrats to our team for winning two paper awards at #ACL2024!

OLMo won the Best Theme Paper award, and Dolma won a Best Resource Paper award!

All the credit goes to the whole team for the massive group effort 🎉🎉

[Pete Walsh @epwalsh]

@lambdaviking @AnanyaHarsh @davidjwadden @allen_ai I’m on the twitter now 😉

[Pete Walsh @epwalsh]

@saurabh_shah2 Being driven around by a ghost? 👻 yikes I would bail

[Pete Walsh @epwalsh]

@saurabh_shah2 I try my best!

[Pete Walsh @epwalsh]

I attempted the Three Sisters Ski Traverse in one day with a buddy earlier this week. Despite the seemingly endless number of transitions between booting, skinning, and skiing, there were some great moments like standing on top of Middle Sister and skiing perfect corn snow ⛷️

[Pete Walsh @epwalsh]

Thankfully we got to the road just before sunset and got watch this very cool moonrise above Mt Bachelor to close out the day 🌔

[Pete Walsh @epwalsh]

But there were also many "wtf are we doing moments" like walking for hours to get to the snow line or realizing our planned route up the final peak (South Sister) and down to our pickup location wasn't going to be straightforward.

[Pete Walsh @epwalsh]

It took us 14 hours to cover 18.5 miles with 7.5k of vertical gain, though we had to bail short of the summit of South Sister as we were running out of daylight.

[Pete Walsh @epwalsh]

Hey babe want to watch W&B curves tonight?

[Pete Walsh @epwalsh]

@saurabh_shah2 Oh cool I was just looking for some light reading material

[Pete Walsh @epwalsh]

@saurabh_shah2 Blasphemy