Samuel Arcadinho

ML Research Scientist

[Samuel Arcadinho @SSamDav]

@m_newhaus Do you plan to release a model with input size bigger then 512?

[Samuel Arcadinho @SSamDav]

@singularity_sah Amazing I was checking GLiNER today, do you plan to release a version with bigger ctx size?

[Samuel Arcadinho @SSamDav]

@NielsRogge I was thinking if we couldn't use one of the BidirLM with masking objective 🤔 we even have omni version in huggingface

[Samuel Arcadinho @SSamDav]

@Ekaeoq Trying to enter the split keyboard work and I just came across the cornix lp split keyboard.

[Samuel Arcadinho @SSamDav]

Awesome research! Highly recommend for anyone interested in looping models to read it.

It suggests that the main bottleneck holding back loop models against a fully unlooped model is the memorization.

[Benhao Huang @huskydogewoof]

🔥
𝐍𝐞𝐰 𝐛𝐥𝐨𝐠: 𝐓𝐨𝐰𝐚𝐫𝐝𝐬 𝐋𝐨𝐨𝐩𝐞𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 𝐃𝐨𝐧𝐞 𝐑𝐢𝐠𝐡𝐭 — 𝐏𝐚𝐫𝐭 𝐈

Looped models reuse the same weights across depth, promising a better compute–parameter trade-off, especially for reasoning.

𝐁𝐮𝐭 𝟏) 𝐝𝐨 𝐭𝐡𝐞 𝐠𝐚𝐢𝐧𝐬 𝐬𝐮𝐫𝐯𝐢𝐯𝐞 𝐰𝐡𝐞𝐧 𝐛𝐨𝐭𝐡 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐅𝐋𝐎𝐏𝐬 𝐚𝐫𝐞 𝐦𝐚𝐭𝐜𝐡𝐞𝐝? 𝟐) 𝐀𝐧𝐝 𝐰𝐡𝐢𝐜𝐡 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐚𝐥 𝐜𝐡𝐨𝐢𝐜𝐞𝐬 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐦𝐚𝐭𝐭𝐞𝐫?

We run 𝐚𝐩𝐩𝐥𝐞𝐬-𝐭𝐨-𝐚𝐩𝐩𝐥𝐞𝐬 ablations spanning Ouro to Huginn. Huginn performs better overall, with the largest gains coming from the loop-in-the-middle (sandwich) design and input injection, though they provide different benefits.

Trained on 𝟓𝟎𝟎𝐁 tokens, an 𝟖𝐁-𝐀𝟎.𝟖𝐁 Huginn MoE approaches or surpasses a 𝟑𝟐𝐁-𝐀𝟑.𝟐𝐁 feedforward MoE on several reasoning benchmarks, including GSM8K (83.6% vs. 80.8%), while using 𝟕𝟓% 𝐟𝐞𝐰𝐞𝐫 resident parameters under 𝐦𝐚𝐭𝐜𝐡𝐞𝐝 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 FLOPs.

More details and the blog link in the thread ↓

[Samuel Arcadinho @SSamDav]

@minchoi When will this feature be released? Can wait for it 😅

[Samuel Arcadinho @SSamDav]

@TrelisResearch Yes I it would be nice!

[Samuel Arcadinho @SSamDav]

@LoubnaBenAllal1 Nice! Was the talk recorded?

[Samuel Arcadinho @SSamDav]

@RidgerZhu Hey nice work! I was reading the paper and I found this part interesting. Does the "Ouro-Thinking" having better score than the original model means we could still add more loops? (Assuming you trained with reasoning traces)

[Samuel Arcadinho @SSamDav]

@tensorqt Im also exploring looped transformers and I don't the improvements I was expecting from them (probably skill issue 😅), Energy transformers seems a good alternative but their efficiency is still a issue.

[Samuel Arcadinho @SSamDav]

@tensorqt Cool post!! Would love to see the impact of position embeddings like ROPE, my intuition is that they reduce the contribution of distance tokens (so initial tokens contribute less) but probably they aren't enough to reduce this bias. 🤔

[Instituto de Telecomunicações @itnewspt]

💡LxMLS 2025 is almost here!
From July 19 to 25, @istecnico will once again host the Lisbon Machine Learning Summer School (LxMLS), a flagship event in the ML community. IT is among the event supporters.

🔗 https://t.co/D0rWLmCloj

#SummerSchool

[Samuel Arcadinho @SSamDav]

@tydsh Nice work! Do you plan to release the code? I would love to experiment using the same setting.

[Samuel Arcadinho @SSamDav]

@BlinkDL_AI @SakanaAILabs Isn’t this model specialized to solve sudoku tho?

[Samuel Arcadinho @SSamDav]

@tensorqt https://t.co/ptgRC0Tab4

[Samuel Arcadinho @SSamDav]

@tensorqt Then the question I ask is if we really need RL 🤔

[Samuel Arcadinho @SSamDav]

@tensorqt I am very bullish on ttc on latent space, would love to fine tune these model https://t.co/P2icyjBRCX on math dataset to know how they compare with reasoning models.

[Ana Moreno @_heyana]

˗ˏˋ bop and it's live! ˎˊ-

In a world full of cookie-cutter design, we're bringing character to products.

And, we just launched our first website!

Link below ↓

[Samuel Arcadinho @SSamDav]

To learn more about our work please follow:
https://t.co/Tss0Dmca9l

[Samuel Arcadinho @SSamDav]

What an incredible week at #EMNLP2024! It was such a pleasure connecting with familiar faces.