Adam Ibrahim

[Adam Ibrahim @ai_phd]

Our tech report for Zamba-7B-v1 is out. We manage to come close to Llama 3 8B, Mistral 7B and others' level of performance, with only 1T tokens, with faster inference and less memory usage at a fixed context length. Read up to learn about our not-so-secret sauce!

[Quentin Anthony @QuentinAnthon15]

Zyphra is dropping the tech report for Zamba-7B, along with:
- Model weights (phase 1 and final annealed) at https://t.co/971YoPRdVI
- Inference/generation code (both pure PyTorch and HuggingFace) at https://t.co/ygeY3nNtOY and https://t.co/vuYlLM6fyo

Tech report: https://t.co/VWDXu27HWG

Intermediate checkpoints and improved phase 1 dataset (+dataset paper!) coming soon.
Intermediate optimizer states available on request.

Some highlights:

[Rylan Schaeffer @RylanSchaeffer]

Another #ICML2025 paper!

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?

TLDR: Predicting language model performance with scale on multiple choice question-answer (MCQA) benchmarks is made difficult b/c ...

1/3

[Rylan Schaeffer @RylanSchaeffer]

Excited to announce our paper ⬇️ was selected as an **Outstanding** paper at @TiFA_ICML2024 🔥🔥🔥

What did the paper show?

Let's try to summarize the paper in a single tweet!!

1/3

[Rylan Schaeffer @RylanSchaeffer]

❤️‍🔥❤️‍🔥Excited to share our new paper ❤️‍🔥❤️‍🔥

**Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?**

w/ @haileysch__ @BrandoHablando @gabemukobi @varunrmadan @herbiebradley @ai_phd @BlancheMinerva @sanmikoyejo

https://t.co/q9VoUMp9qu

1/N

[Adam Ibrahim @ai_phd]

@xwang_lk Hey Xin long time no see (well since UCSB) ! Anyway maybe our model shows the reply might be "yes" ? 😄https://t.co/QTkHRgs9l3

[Beren Millidge @BerenMillidge]

Extremely excited to announce Zamba! A 7B SSM with a novel architecture competitive with Gemma-7B and Mistral-7B and significantly beating Llama2-7B trained on only 1T open training tokens.

[Adam Ibrahim @ai_phd]

Worth noting that we're working with @huggingface to release the model over the next week. Stay tuned !

[Adam Ibrahim @ai_phd]

something cool we did

[Beren Millidge @BerenMillidge]

Extremely excited to announce Zamba! A 7B SSM with a novel architecture competitive with Gemma-7B and Mistral-7B and significantly beating Llama2-7B trained on only 1T open training tokens.

[Adam Ibrahim @ai_phd]

@QuentinAnthon15 @BerenMillidge @yury_tokpanov And kudos to you too, especially for suffering through @BerenMillidge and my memes for a month😁

[Adam Ibrahim @ai_phd]

@Ar_Douillard @MassCaccia DIstributed CApacity PRIOritisation ?

[Adam Ibrahim @ai_phd]

@BrandoHablando @benjamintherien @AiEleuther @laion_ai @ThomasScialom Note that from the EN->DE experiments and some feedback I had from people working on adapting to time-series, stronger shifts seem to require more replay than EN->EN (>5%). But again, relative performance of replay%appears fast enough that searching this hyperparameter is viable.

[Adam Ibrahim @ai_phd]

@BrandoHablando @benjamintherien @AiEleuther @laion_ai I don't know a way to do it automatically, but so far @ThomasScialom's work (Continual T0) and ours show 1-5% is often good for EN->EN. Our Fig 4. shows that fortunately, comparing relative performance of different choices of replay % reqs only ~10B tokens https://t.co/Qc3j9qexzv

[Benjamin Thérien @benjamintherien]

To mitigate forgetting, we study the effect of replaying previous data with a fixed compute budget across two distribution shifts and many replay percentages. We find that an appropriate amount of replay significantly mitigates forgetting, even for hundreds of billions of new tokens. 4/N

[Adam Ibrahim @ai_phd]

@BrandoHablando @benjamintherien @AiEleuther @laion_ai Which reweightings ?

[Adam Ibrahim @ai_phd]

@BrandoHablando @benjamintherien @AiEleuther @laion_ai Note that Llama i+1 is a separate setting however since there are architectural changes involved (context length, GQA, ...), besides dataset changes. If it were only dataset changes, I believe our Pile -> SP experiments suggest it could have worked to continue pretraining.

[Adam Ibrahim @ai_phd]

@BrandoHablando @benjamintherien @AiEleuther @laion_ai We pretrained both 405M and 10B sized models from scratch on the 300-600B tokens of each of these modalities. Our results are consistent at both scales with those 2 actual EN pretraining datasets, but also in the entirely separate case of the stronger EN-> DE (200B tokens!) shift

[Adam Ibrahim @ai_phd]

@BrandoHablando @benjamintherien @AiEleuther @laion_ai If your dataset is much smaller than the ones we consider though (e.g. few billions), I'd treat it more like a fine-tuning dataset and not a pretraining dataset (the ones considered in our work/experiments are hundreds of billions of tokens). I'd bet replay still works though.

[Adam Ibrahim @ai_phd]

@BrandoHablando @benjamintherien @AiEleuther @laion_ai The LR scheduler results are even more likely to be agnostic of the setting, since those are more tied to fundamentals of optimisation with transformers at scale, Adam, and cos decays.