Alex Stern

Pretraining intern @Cohere | ML MSc @UCL_DARK 🇵🇹🇮🇳 🇨🇭 🇬🇧

[Alex Stern @alexkrstern]

@KumbongHermann @IddahMlauzi @HazyResearch @Azaliamirh @simran_s_arora Did you try doing something similar and training ARivley with these concept tokens as discrete semantic ones? Though diffusion intuitively makes more sense for this!

[Alex Stern @alexkrstern]

@KumbongHermann @IddahMlauzi @HazyResearch @Azaliamirh @simran_s_arora I imagine the VQ Vae part was hard to get right. Congrats! I worked on a project with similar idea but not with diffusion. Took an embedding model to discretize concepts in tiny stories, then simply pretrained with these extra concept tokens prepended to context of fine tokens.

[Alex Stern @alexkrstern]

@YueSong48287250 Cool stuff am wondering: usually activations growing is correlated with lm loss at least for short horizons. In your experience rotations are enough? Also when you rotate, do you also enforce the magnitude to stay fixed?

[Alex Stern @alexkrstern]

@dominik_schnaus Do this with dolphin Gemma next

[Alex Stern @alexkrstern]

@wangsw5653 I see. Did you run at this setting 18 L with H same as the D at 12? Wondering if with more H, D could be deeper and not lose perf.

[Alex Stern @alexkrstern]

@wangsw5653 Specifically referring to this table. Thanks!

[Alex Stern @alexkrstern]

@wangsw5653 Hey, very through interesting paper. I was wondering though: why do you think the ublooped depth 18 experiment you did hit higher loss than the depth 12 unlooped baseline? Were they trained on different token counts? otherwise more flops feels like it should have performed better

[Alex Stern @alexkrstern]

@paradigmainc Cool work, am very curious how useful the MUDD modules were? Generally curious on insight into these architecture changes you made

[Alex Stern @alexkrstern]

@industriaalist Cool work. So from my understanding the boundary operator innovation is what allows you to increase depth even further, both in the looping and unlooped case?

[Alex Stern @alexkrstern]

@Dorialexander Yeah ofc very dependent on the synthetic pipeline. But assuming some fixed relatively optimal pipeline (optimal prompts even etc), there must be a compute optimal ratio. Were you able to tune it?

[Alex Stern @alexkrstern]

@Dorialexander Wow ~40% FLOPs budget spent on synthetic data gen. Did you tune this ratio? Have been thinking about whether a compute optimal ratio exists.

[Alex Stern @alexkrstern]

@michaelyli_ complexity is observer dependent alas

[Alex Stern @alexkrstern]

@_rockt @PMinervini Big day. Wonder how @PrimeIntellect prime-agent harness plus a worse base model would perform on net hack.

[Alex Stern @alexkrstern]

@alvarobartt cool stuff! in your experience how many questions is too many questions to probe laya at once (assuming compute allows ofc)?

[Alex Stern @alexkrstern]

@jaredpalmer @modal Cool, thanks for this work. How many parallel questions can these models answer at the same time?

[Alex Stern @alexkrstern]

@_potatodonkey_ Am wondering how prime agent from prime intellect benchmarks on this, with each of these models.

[Alex Stern @alexkrstern]

@kickingkeys Wonder if this can be applied to music. RL an LLM to use a music production environment like LMMS https://t.co/JVN6agKA0r

[Alex Stern @alexkrstern]

@pradheepraop Cool sounds relevant to slow run folks @industriaalist

[Alex Stern @alexkrstern]

@PrimeIntellect This the beauty of pi. Anyone can make a harness.