Di Luo

AI+Science, Quantum Computation, Quantum Many-body Physics

[Zhuo Chen @Zhuoc3Chen]

🚀 Excited to share our latest paper: L²M: Mutual Information Scaling Law for Long-Context Language Modeling

📌We identified a new scaling law that emerges as a general pattern in long texts: this leads to important insights in designing more powerful and efficient large language models

🔍 Key Highlights:
· 📈 Discovered a fundamental pattern in how information is shared across text at different scales, showing power-law growth as texts get longer
· 🧠 Proved that LLMs need growing memory capacity to effectively handle longer documents—similar to how humans need to remember more when reading a book versus a sentence
· 🔮 Our findings show that promises of "infinite context length" face natural limits—how well models handle longer texts depends on how their memory capacity scales
· ⚙️ Tested these findings across different model architectures, confirming the link between memory scaling and long-text performance
· 💡 Offers practical insights for building better AI models that can work with everything from short messages to entire books

📝 Read our paper: https://t.co/9ZaGZxnOQ8
🤗 See our paper on Huggingface: https://t.co/sImTrrVRx1
💻 Explore our code: https://t.co/V3dFlkZgBv

With Zhuo Chen, Oriol Mayné i Comas, Zhuotao Jin, Di Luo, Marin Soljačić
#MachineLearning #AI #LanguageModels #LLMs #LongContextModeling #InformationTheory

[Zhuo Chen @Zhuoc3Chen]

🚀 Thrilled to share that QuanTA, our innovative approach to highly parameter-efficient high-rank fine-tuning of LLMs, has been accepted to NeurIPS 2024! Looking forward to connecting on December 13!

📄 https://t.co/q9kud4Cwns

💻https://t.co/cKJEsI3Ti3

[Di Luo @DiLuo28]

Explore our poster: https://t.co/oJTsauh7fv
Read our paper: https://t.co/fdTaDDMZ4O

See you there! 🚀 #MachineLearning #AI for Science #PDEs #ICML

[Di Luo @DiLuo28]

We are excited to present our work on "TENG: Time-Evolving Natural Gradient for Solving PDEs with Deep Neural Nets toward Machine Precision" at #ICML2024! Join us at Hall C 4-9, Poster #210, on July 24th, 11:30 am-1 pm CEST and virtually at Zoom: https://t.co/6Do62edO7S.

[Di Luo @DiLuo28]

Welcome attempts and feedback on our SciCode!

[Minyang Tian @MinyangTian1]

SciCode is our new benchmark that challenges LMs to code solutions for scientific problems from advanced papers. The challenges were crafted by PhDs;

~10% of our benchmark is based on Nobel-winning research.

GPT-4 and Sonnet 3.5 get <5% ACC.

https://t.co/OtNadtSICO 🧵 1/6

[Di Luo @DiLuo28]

@BitValentine Thanks for the interest! We are quite encouraged by the advancement from quantum algorithms and AI, which can benefit a wide range of real-world applications such as high-dimensional modeling and optimization.

[Di Luo @DiLuo28]

🚀 Excited to share our latest paper: "QuanTA: Efficient High-Rank Fine-Tuning of LLMs with Quantum-Informed Tensor Adaptation"!

📄 Read the paper: https://t.co/wmBlFxbgC3

💻 Explore the code: https://t.co/8W4wMkbnmK

#NLP #LLM #Finetuing #MachineLearning #Quantum #GenAI

[Di Luo @DiLuo28]

https://t.co/VJPKOaJsA1
@Zhuoc3Chen @charlotteloh_ @dangovski @OwenDugan @ Marin Soljačić

[Di Luo @DiLuo28]

By leveraging a structure inspired by quantum circuits, our QuanTA approach surpasses the limitations of Low-Rank Adaptation (LoRA), enabling a novel, easy-to-implement, efficient high-rank fine-tuning method for large language models.

[Rumen Dangovski @dangovski]

Paper: https://t.co/cBRkGwI6Mf. Discussion: https://t.co/mpTKNFjfV9. Di Luo (@DiLuo28) (Co-first authors), Jiayu Shen (Co-first authors), Rumen Dangovski (@dangovski), and Marin Soljačić (https://t.co/4Uadbtcwuu).