Zexi Li

AI PhD student at Zhejiang University

[Prateek Yadav @prateeky2806]

🔍 A thread on the latest @iclr_conf 2024 papers on
- Mixture of Experts
- Modular Models
- Compositional Generalization
- and related topics:

Dive into the latest papers from #ICLR2024 through the list below!

Let me know if I missed some relevant papers.

[🧵Thread ⬇️]

[Sebastian Raschka @rasbt]

I ran hundreds if not thousands of LoRA & QLoRA experiments to finetune open-source LLMs, and here’s what I learned:

1. Despite the inherent randomness of LLM training (or when training models on GPUs in general), the outcomes remain remarkably consistent across multiple runs.

2. QLoRA presents a trade-off that might be worthwhile if you're constrained by GPU memory. It offers 33% memory savings at the cost of a 33% increase in runtime.

3. When finetuning LLMs, the choice of optimizer shouldn't be a major concern. While SGD on its own is suboptimal, there's minimal variation in outcomes whether you employ AdamW, SGD with a scheduler, or AdamW with a scheduler.

4. While Adam is often labeled a memory-intensive optimizer due to its introduction of two new parameters for every model parameter, this doesn't significantly affect the peak memory demands of the LLM. This is because the majority of the memory is allocated for large matrix multiplications rather than retaining extra parameters.

5. For static datasets, iterating multiple times as done in multi-epoch training might not be beneficial. It often deteriorates the results, probably due to overfitting.

6. If you're incorporating LoRA, ensure it's applied across all layers, not just to the Key and Value matrices, to maximize model performance.

7. Adjusting the LoRA rank is essential, and so is selecting an apt alpha value. A good heuristic is setting alpha at twice the rank's value.

8. 7B models can be finetuned efficiently within a few hours on a single GPU possessing 14 Gb of RAM.
With a static dataset, optimizing an LLM to excel across all benchmark tasks is unattainable. Addressing this requires diverse data sources, or perhaps LoRA might not be the ideal tool.

[Zexi Li @TomLeeeeee]

@prateeky2806 @iclr_conf Many thanks!

[Ziming Liu @ZimingLiu11]

Spent a whole day skimming through titles of 9k+ papers submitted to ICLR this year🥲, I picked around 200 papers that piqued my interest, including "science of deep learning", "deep learning for science", "voodoos", "interpretability" and/or "LLM".
link: https://t.co/MUb1oLt7pi

[Prateek Yadav @prateeky2806]

🔍 Searching for @iclr_conf 2024 paper on Model Merging/Fusion & related topic:

Dive into the latest advancements in model merging, fusion, and weight interpolations from #ICLR2024 through the list below!

Let me know if I missed some relevant papers.

[Thread ⬇️]

[Zexi Li @TomLeeeeee]

Very nice experience in #ICML2023 !
First time to give an in-person poster presentation, and it is wonderful.

Our paper "Revisiting Weighted Aggregation in Federated Learning with Neural Networks" is published online. Stay tuned for the future code release.

[Zexi Li @TomLeeeeee]

@AnLiGentile Dear chair, the submission website of WWW Industry track is early closed, but there is still one hour left before the official deadline （announced as 11.14 end of the day, AoE). Will it possible to reopen?

[Zexi Li @TomLeeeeee]

@AnLiGentile Dear chair, the submission website of www2023 industry track is early closed, but there is still one hour left before the official deadline （noticed as 11.14 AoE, end of the day). Will it possible to reopen?

[Zexi Li @TomLeeeeee]

@TheWebConf @AnLiGentile The submission website is early closed, but there is still one hour left before the official deadline. Will it possible to reopen?

[Zexi Li @TomLeeeeee]

@TheWebConf Industry submission entrance is already early closed, and there is still one hour left. Will it possible to reopen the website？

[Zexi Li @TomLeeeeee]

笑迎百廿＃Looking forward to the 120th birthday of Zhejiang University on May 21st！