Long Minh

CS PhD student at @WSUPullman. Former AI Research Resident at @fpt_software.

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

If you've followed my recent posts on model merging, I just published a long-form survey on this topic. It covers 50+ papers from the 1990s until now, including everything from basic concepts to the recent application of model merging to LLM alignment. See image for details!

[Yilun Du @du_yilun]

Excited to share Equilibrium Matching (EqM)!

EqM simplifies and outperforms flow matching, enabling strong generative performance of FID 1.96 on ImageNet 256x256.

EqM learns a single static EBM landscape for generation, enabling a simple gradient-based generation procedure.

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

SimpleFold: Folding Proteins is Simpler than You Think

"we introduce SimpleFold, the first flow-matching based protein folding model that solely uses general purpose transformer blocks. Protein folding models typically employ computationally expensive modules involving triangular updates, explicit pair representations or multiple training objectives curated for this specific domain. Instead, SimpleFold employs standard transformer blocks with adaptive layers and is trained via a generative flow-matching objective with an additional structural term."

[Stanford AI Lab @StanfordAILab]

arXiv -> alphaXiv

Students at Stanford have built alphaXiv, an open discussion forum for arXiv papers. @askalphaxiv

You can post questions and comments directly on top of any arXiv paper by changing arXiv to alphaXiv in any URL!

[Xiaolong Wang @xiaolonw]

Cannot believe this finally happened! Over the last 1.5 years, we have been developing a new LLM architecture, with linear complexity and expressive hidden states, for long-context modeling. The following plots show our model trained from Books scale better (from 125M to 1.3B) than Mamba and Transformer, and our 1.3B model works better and better with longer context.

The idea, Test-Time Training (TTT), is something we have been working on for over 5 years. I still remember when I started as a postdoc, discussing with Alyosha what to work on, he asked me to talk to Yu Sun on TTT. And that is when everything begins.

Although this TTT is quite a different thing now. It is a network layer, replacing the hidden state of an RNN with a machine learning model. Instead of using a feature vector to represent a memory, our TTT layer maintains a small neural network to compress the input tokens. As each token comes in sequentially, the compression is done via gradient descent on this token to update the small neural network.

This is currently applied to language modeling, but imagine applying it to videos. In the future, when modeling long videos, instead sampling 1 FPS, we can sample the frames densely, and these dense frames would be a burden for Transformer but a blessing for TTT layer. As they are essentially just augmentations in time to train a better network inside TTT.

Please check: https://t.co/XNK1mlxpUQ

[Johannes Brandstetter @jo_brandstetter]

Introducing Vision-LSTM - making xLSTM read images 🧠It works ... pretty, pretty well 🚀🚀 But convince yourself :) We are happy to share code already!

📜: https://t.co/XaC3EdaHrw
🖥️: https://t.co/dVucXk8s5j

All credits to my stellar PhD @benediktalkin

[Johannes Brandstetter @jo_brandstetter]

Poseidon: Efficient Foundation Models for PDEs

Exciting new multiscale operator transformer from ETH PDE folks:

arXiv: https://t.co/gI8abmKg8P

[Tan Minh Nguyen @TanNguyen689]

Our paper, "Revisiting Kernel Attention with Correlated Gaussian Process Representation", has been accepted at UAI 2024. This is great joint work with the students @LongMinh210, @Tho_Tran99 , Đinh Công Duy, and Professor @nghiaht87.

[Stanley H. Chan @stanley_h_chan]

I wrote a tutorial on diffusion models for undergrad and grad students. I tried my best to give intuitive explanations for complicated equations.

Your feedback is much appreciated

Thanks to those who suggested various reading materials to me

https://t.co/fWv111nG5M

[Yam Peleg @Yampeleg]

BREAKING!

AI21 just dropped an open source Mamba!

- MoE with 52B parameters.
- Active parameters: 12B.
- 256K Context length.
- Competitive performance to Mixtral!
- Open weights: Apache 2.0.

[I am visiting them in an hour so i'll get u some info!]

https://t.co/Vq5W4Ctxcy

[Arthur Zucker @art_zucker]

`transformers 4.39` is out, with fixed gemma/llama ram usage that were reported!
new: Llava1.6, Cohere, Mamba, Pvtv2, MusicGen, Udop, StarCoder2, SegGPT, Galore optimizer 🤗
Cheers to the community for the help! To people to tag, grateful for everyone!
https://t.co/Ml5cYvvKiJ

[Alex Yanko 🇺🇦 @LeopolisDream]

State Space Models: A Modern Approach

This is an interactive textbook on state space models (SSM) using the JAX Python library.

https://t.co/GZSaypO2o6

[Arthur Zucker @art_zucker]

`mamba` is now available in transformers. PEFT finetuning example: https://t.co/2vPQBgA9iu
Thanks @_albertgu and @tri_dao for this brilliant model! 🚀 and the amazing `mamba-ssm` kernels powering this!

[elvis @omarsar0]

🎓Stanford CS224N: NLP with Deep Learning | 2023

Very exciting to see new lectures for one of my favorite NLP courses of all time.

Topics range from attention and Transformers to multimodal deep learning.

It's worth checking out the entire set of lectures.

https://t.co/fAWQTP8oAg

[parm🔬 @parmita]

This is an essential point people seem to misrepresent.

[AK @_akhaliq]

GaLore

Memory-Efficient LLM Training by Gradient Low-Rank Projection

Training Large Language Models (LLMs) presents significant memory challenges, predominantly due to the growing size of weights and optimizer states. Common memory-reduction approaches, such as low-rank

[Aleksandar Botev @botev_mg]

We present Griffin: A hybrid model mixing a gated linear recurrence with local attention. This combination is extremely effective: it preserves all the efficient benefits of linear RNNs and the expressiveness of transformers. Scaled up to 14B!
https://t.co/uptKapilDM

[Sasha Rush @srush_nlp]

Very nice Mamba explainer blog post from @jackcookjack (and a good title). https://t.co/ti2Ez3MefD

[Sebastian Raschka @rasbt]

Just wrote a new article on "Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch": https://t.co/OnGPyHi8tD
I am super excited about DoRA, and based on my early experiments (and the original research paper itself), it's definitely worth trying. As my article above shows, it's also straightforward to implement. So, why not give it a shot the next time you think about finetuning an LLM!

[Jascha Sohl-Dickstein @jaschasd]

Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.