Roger Waleffe

Computer Sciences PhD student at the University of Wisconsin-Madison

[Bryan Catanzaro @ctnzr]

Today, @NVIDIA is launching the open Nemotron 3 model family, starting with Nano (30B-3A), which pushes the frontier of accuracy and inference efficiency with a novel hybrid SSM Mixture of Experts architecture. Super and Ultra are coming in the next few months.

[Bryan Catanzaro @ctnzr]

Today we're releasing NVIDIA Nemotron Nano v2 - a 9B hybrid SSM that is 6X faster than similarly sized models, while also being more accurate.

Along with this model, we are also releasing most of the data we used to create it, including the pretraining corpus.

Links to the models, datasets, and tech report are here:

https://t.co/NqSYULoiW3

[Bryan Catanzaro @ctnzr]

Nemotron-H: A family of Hybrid Mamba-Transformer LLMs.
* Hybrid architecture means up to 3X faster at the same accuracy
* Trained in FP8
* Great for VLMs
* Weights and instruct versions to come soon.

https://t.co/h3dLuDuiUz

[Bryan Catanzaro @ctnzr]

A 8B-3.5T hybrid SSM model gets better accuracy than an 8B-3.5T transformer trained on the same dataset:
* 7% attention, the rest is Mamba2
* MMLU jumps from 50 to 53.6%
* Training efficiency is the same
* Inference cost is much less
https://t.co/x62otbC5uN

[Roger Waleffe @RWaleffe]

@WesleyYue @ctnzr The authors of the RULER benchmark observed something similar for some Transformers they tested.

[Roger Waleffe @RWaleffe]

@WesleyYue @ctnzr See our discussion here:

[Theo Rekatsinas @thodrek]

Data pruning to reduce pertaining costs is hot, but fancy pruning can take just as long to select data as to train on all of it! Patrik, @Rwaleffe, and @vmageirakos's work at #ICLR2024 tomorrow shows how a simple, low-cost tweak to random sampling outperforms trendy methods!

[Roger Waleffe @RWaleffe]

@DisseminatePod Thanks for having me on the podcast Jack!

[Disseminate: The Computer Science Research Podcast @DisseminatePod]

🚨 "MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks" with @RWaleffe is available now!

🎧 Listen on Spotify ➡️ https://t.co/PgTrDblzJx

☕️ Support the podcast ➡️ https://t.co/tVHEIk5EgN

[Roger Waleffe @RWaleffe]

@rishiyer Thanks for sharing this!! Continuous random exploration had also been part of our motivation.

[Roger Waleffe @RWaleffe]

Not convinced about using random sampling for data pruning? Consider twice! In our recent work, we introduce Repeated Sampling of Random Subsets: https://t.co/jk2dWHpocl, where we sample a subset of data at each epoch of training instead of only once at the beginning!

[Roger Waleffe @RWaleffe]

@BlackHC Regardless of which ‘viewpoint’ one chooses to look at our method with, this algorithm had yet to be studied extensively (empirically and theoretically).

[Roger Waleffe @RWaleffe]

@BlackHC If the sampling of S’ across rounds is done without replacement (instead of with replacement), then our method can also be seen as training on the full dataset but with early stopping after a few epochs (discussed in the paper). This version is particularly useful for analysis.

[Roger Waleffe @RWaleffe]

Joint work with Patrik Okanovic @vmageirakos Kostis Nikolakakis @aminkarbasi @DKalogerias @nmervegurel @thodrek

[Roger Waleffe @RWaleffe]

See the preprint here: https://t.co/kT897KdZgz for extensive evaluations together with the convergence analysis and discussion on its generalization.

[PyKEEN @keenuniverse]

Marius, another amazing KGE (and more) library is now auto-formatting its code with black as of https://t.co/bCxU19K73t 🚀

@JasonMohoney @RWaleffe nice job :)