Qijun Luo

CS PhD student in CUHKSZ | Ex-Research intern @MSFTResearch | Efficient AI & optimization

[Qijun Luo @Clear_Fing]

@cHHillee @YouJiacheng Hi Horace, thanks for sharing it! We will check the code :)

[Qijun Luo @Clear_Fing]

@YouJiacheng Thank you for recognizing StreamBP! We will provide more discussion on BPT and include the detailed memory analysis in the next version of the manuscript.

[Qijun Luo @Clear_Fing]

@YouJiacheng @cHHillee Thanks for sharing the cool repo. It works for CE (i.e. SFT) but is not applicable to GRPO. The idea of decomposing the objective in sequence dimension is similar though.

Yes - DPO and other non-separable objectives requires gradient analysis as in StreamBP.

[Qijun Luo @Clear_Fing]

@YouJiacheng Yes, it's the checkpointed layer inputs X, which is released from memory when the layer's grad computation is done. That's actually the major activation memory cost in StreamBP.

[Qijun Luo @Clear_Fing]

@YouJiacheng BPT's remat+scan implementation indeed avoid saving full activation. Thanks for pointing out!

StreamBP can be much faster though:
- It uses flash-attn for fast attention computation & parallelization on SRAM.
- No loop on kv; the HBM throughput (of KV's weight) is much lower.

[Qijun Luo @Clear_Fing]

@YouJiacheng Please feel free to experiment with our code: https://t.co/CRj2FGn6hq

[Qijun Luo @Clear_Fing]

From implementation perspective, StreamBP avoids unnecessary computation of attention score in masked position, achieving acceleration over the gradient checkpointing:

[Qijun Luo @Clear_Fing]

Introducing 𝗦𝘁𝗿𝗲𝗮𝗺𝗕𝗣, a memory-efficient exact backpropagation (BP) method. Compared to gradient checkpointing, it

- Increases max sequence length by approx. 5 times.

- Costs comparable or even less BP time.

Paper: https://t.co/UqetxZmXQI
Code: https://t.co/CRj2FGnE6Y

[Qijun Luo @Clear_Fing]

A linear transformation example.

To compute dW1 and dW2, StreamBP sequentially computes Yi and dYi, accumulate Xi^T dYi and Yi^TdZi, then drop Yi and dYi. Consequently, StreamBP does not store full Y and dY in the memory.

[Jing-Jing Li @drjingjing2026]

1/3 Today, an anecdote shared by an invited speaker at #NeurIPS2024 left many Chinese scholars, myself included, feeling uncomfortable. As a community, I believe we should take a moment to reflect on why such remarks in public discourse can be offensive and harmful.

[Ziniu Li @ZiniuLi]

🚀 Excited to share our paper "Why Transformers Need Adam: A Hessian Perspective", which is accepted at #NeurIPS2024!

We delve into why SGD lags behind Adam, uncovering the "block heterogeneity" in Transformers' Hessian spectra that Adam navigates more adeptly. 🧠💡

Our findings show that Adam's coordinate-wise rates are crucial for handling this heterogeneity, a game-changer for optimizing Transformers!

Paper Link: https://t.co/jFWs8eIW72

[Qijun Luo @Clear_Fing]

@YiMaTweets I suppose the it comes out only when we can answer the principle of our brain intelligence.

[Llama Factory @llamafactory_ai]

🚀We've integrated the Adam-mini optimizer into LLaMA-Factory, slashing the memory footprint of full-finetuning Qwen2-1.5B from 33GB to 24GB! 🔥

Image source: https://t.co/y8ip5W7D7N

[Richard Sutton @RichardSSutton]

The one-step trap (in AI research)

The one-step trap is the common mistake of thinking that all or most of an AI agent’s learned predictions can be one-step ones, with all longer-term predictions generated as needed by iterating the one-step predictions. The most important place where the trap arises is when the one-step predictions constitute a model of the world and of how it evolves over time. It is appealing to think that one can learn just a one-step model and then “roll it out” to predict all the longer-term consequences of a way of behaving. The one-step model is thought of as being analogous to physics, or to a realistic simulator.

The appeal of this mistake is that it contains a grain of truth: if all one-step predictions can be made with perfect accuracy, then they can be used to make all longer-term prediction with perfect accuracy. However, if the one-step predictions are not perfectly accurate, then all bets are off. In practice, iterating one-step predictions usually produces poor results. The one-step errors compound and accumulate into large errors in the long-term predictions. In addition, computing long-term predictions from one-step ones is prohibitively computationally complex. In a stochastic world, or for a stochastic policy, the future is not a single trajectory, but a tree of possibilities, each of which must be imagined and weighted by its probability. As a result, the computational complexity of computing a long-term prediction from one-step predictions is exponential in the length of the prediction, and thus generally infeasible.

The bottom line is that one-step models of the world are hopeless, yet extremely appealing, and are widely used in POMDPs, Bayesian analyses, control theory, and in compression theories of AI.

The solution, in my opinion, is to form temporally abstract models of the world using options and GVFs, as in the following references.

Sutton, R.S., Precup, D., Singh, S. (1999). Between MDPs and semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning. Artificial Intelligence 112:181-211.

Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., Precup, D. (2011). Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In Proceedings of the Tenth International Conference on Autonomous Agents and Multiagent Systems, Taipei, Taiwan.

Sutton, R. S., Machado, M. C., Holland, G. Z., Timbers, D. S. F., Tanner, B., & White, A. (2023). Reward-respecting subtasks for model-based reinforcement learning. Artificial Intelligence 324.

[Qijun Luo @Clear_Fing]

BAdam's implementation supports ZeRO-3 now! One can do full-parameter finetune on 70B models with only 8 RTX-3090 GPUs / 3 A100 GPUs! Using BAdam only requires adding 2 lines on existing code: https://t.co/a9BqXc30Ff. One can also directly run BAdam on https://t.co/lYCxlo2ECX.