Defa Zhu

AI Researcher |

[Defa Zhu @zhudefa]

@teortaxesTex Interesting take! Curious how you arrived at that conclusion?

[Defa Zhu @zhudefa]

@aHpaBean Got it, thanks for the clarification! That's a very interesting finding. Congrats again on the great work! :)

[Defa Zhu @zhudefa]

Which Chinese AI model has the best shot at joining the “Big Three” this year (GPT, Claude, Gemini)?

[Defa Zhu @zhudefa]

@natolambert Why are they concerned about ByteDance?

[Defa Zhu @zhudefa]

@aHpaBean Thanks! I’ll look forward to your update once the paper is online.
And thank you for your kind words about our Hyper-Connections work — I really appreciate your interest. Looking forward to seeing more of your work in this direction as well!

[Defa Zhu @zhudefa]

I hope this is not a whale fall, but just a temporary stranding.

[Defa Zhu @zhudefa]

After reading the v4 tech report, a model of this scale really shouldn't be this unstable. They need to look into the issue thoroughly. I think there are indeed some internal problems at DeepSeek, but I hope they can bounce back soon.

[Defa Zhu @zhudefa]

God is silicon-based.

[Defa Zhu @zhudefa]

@xiaolonw This is a solid piece of work. I found reading your paper very inspiring. Congratulations on such a great result—I must admit, I'm a bit jealous of this work!

[Defa Zhu @zhudefa]

@ytz2024 👍👍

[Defa Zhu @zhudefa]

@_arohan_ @YouJiacheng https://t.co/su6VBwGfG6

[Defa Zhu @zhudefa]

@_arohan_ @YouJiacheng While we independently developed over-encoding, we acknowledge that Ngrammer introduced a similar technique earlier. However, the primary contribution of our work, the Over-Tokenized Transformer, lies in the discovery that the input ngram vocab follows a log-linear scaling law.

[Defa Zhu @zhudefa]

@leijun I am looking forward to buy a MI car.

[Defa Zhu @zhudefa]

@MadHermitHimbo The computation of concatenating is non - negligible. The computation of the unembedding layer is quite substantial. When concatenating, this part of the computation will be magnified by n times.

[Defa Zhu @zhudefa]

🎉 Thrilled to announce that our paper "Hyper-Connections" has been accepted at ICLR 2025! 🚀

💡 Discover how Hyper-Connections improve performance in LLMs & vision models. Faster convergence, better results! 💪

📄 Paper: https://t.co/bJwYFTpCMC
#LLM #LLMs #AI

[Defa Zhu @zhudefa]

@mbalunovic How about Part II

[Defa Zhu @zhudefa]

🚀 Check out UltraMem, accepted at ICLR 2025 🌟

🔑 Highlights:

2-6x faster inference vs. MoE 🚀
State-of-the-art performance with minimal memory access 🧠
Scales better than MoE models 📈https://t.co/NUccimBKzk

#LLMs #MachineLearning #NLP #Transformers

[Defa Zhu @zhudefa]

🚀 Excited to share "Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling"! 🎉

Discover how scaling input vocabularies with multi-gram tokens improves LLM performance. 💡

Read more: https://t.co/pMOkbFY6y4
#AI #LLM #NLP