Alexander Popkov

[Rohan Paul @rohanpaul_ai]

Beautiful Paper.

A comprehensive survey of post-training methods including fine-tuning, reinforcement learning, and test-time scaling to refine LLMs reasoning.

Methods Explored in this Paper 🔧:

→ Systematically explores fine-tuning techniques that adapt LLMs for specific tasks, but acknowledges risks of overfitting and forgetting.

→ Reinforcement Learning from Human Feedback, are examined for aligning LLMs with human preferences and improving response quality.

→ Test-time scaling strategies, such as chain of thought prompting and tree of thought, are discussed to enhance reasoning during inference without retraining, by dynamically adjusting computation based on query complexity.

→ The paper also investigates reward modeling, policy optimization algorithms like Proximal Policy Optimization and Direct Preference Optimization, and efficient fine-tuning approaches to optimize LLMs post-training.

[𝚐𝔪𝟾𝚡𝚡𝟾 @gm8xx8]

LLM Post-Training: A Deep Dive into Reasoning Large Language Models

[Rohan Paul @rohanpaul_ai]

Wild idea in this paper 🤯

Current LLMs are stateless between tokens which lead to many many problems requiring reasoning across tokens (like unreliable CoT). Even a tiny bit of memory should help.

How might we store knowledge affordably yet comprehensively? Memory³ proposes an intriguing method - compressing factual data separately. Introduces a third form of memory in addition to the implicit knowledge stored in model parameters and the short-term working memory used during inference (context key-values).

👨‍🔧 LLMs struggle with inefficient knowledge storage and retrieval, leading to high training and inference costs. The paper aims to address this by introducing a more efficient memory format.

📌 Memory3 introduces explicit memory as a third memory format for LLMs, alongside model parameters (implicit memory) and context key-values (working memory). This explicit memory is implemented as sparse attention key-values, allowing for more efficient knowledge storage and retrieval.

📌 Defines a memory hierarchy for LLMs: plain text (RAG) → explicit memory → model parameters. As you move up this hierarchy, write cost increases while read cost decreases. The goal is to optimize knowledge placement across this hierarchy based on usage frequency.

📌 Memory3's architecture involves converting reference texts into explicit memories before inference. During inference, these memories are retrieved and integrated into self-attention layers. This design allows for smaller model size while maintaining performance.

📌 The explicit memory format uses intense compression to save space. It selects only the first half of attention layers as memory layers, uses grouped query attention to reduce key-value heads, and selects only 8 out of 128 tokens for each key-value head based on attention weights.

📌 The training process involves a two-stage approach: a warmup stage without explicit memory, followed by a continual train stage with explicit memory. This approach was necessary as starting with explicit memory from the beginning rendered the memories useless.

📌 Introduces a "memory circuitry theory" to formalize the concept of knowledge in LLMs. It defines knowledge as circuits (equivalence classes of subgraphs) in the computation graph, categorizing them as specific or abstract knowledge.

📌 The Memory3 model achieved better performance than larger models and RAG models on various benchmarks, while maintaining higher decoding speed. It showed particular improvements in factuality and reduced hallucination.

[James @heyoliverjane]

NVIDIA just released free online courses on AI.

No payment or fees are required!

Here are 6 courses you don't want to miss in 2024:

[Rohan Paul @rohanpaul_ai]

How much GPU VRAM do I need to be able to serve Llama 70B?

Clean formulae

[Rohan Paul @rohanpaul_ai]

A classic paper "Physics of Language Models"

📌 The paper explores scaling laws for LLMs, highlighting a unique perspective by quantifying the knowledge capacity in bits rather than using conventional benchmarks or loss metrics. It pinpoints a remarkable finding: LLMs can store a maximum of 2 bits of knowledge per parameter.

Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined.

"Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined."

When we say the model stores knowledge, it isn’t word-by-word memorization. Instead, the knowledge is flexibly extractable.

How training time affects model capacity, as per the Paper.

Achieving a 2bit/param capacity requires each knowledge piece to be visited 1000 times during training, termed 1000-exposure to differentiate from traditional “1000-pass” terminology, as a single data pass can expose a knowledge piece 1000 times.

---

📌 **Knowledge Representation:** The research categorizes factual knowledge into tuples (name, attribute, value), providing a structured and quantifiable approach to understand and measure knowledge within LLMs.

The researchers generate synthetic knowledge-only datasets by uniformly at random generating (name, attribute, value) tuples from a knowledge base and converting them into English descriptions. They pretrain language models (e.g., GPT-2, LLaMA, Mistral) on these texts using a standard auto-regressive objective from random initialization, and “estimate” the learned knowledge. By varying the number of knowledge pieces and model sizes, we outline a knowledge capacity scaling law.

💡 The paper's insights can streamline the design of LLMs, suggesting that beyond a certain threshold, adding parameters does not linearly increase a model's knowledge base.

It underscores the significance of thoughtful data preparation and architectural choices, emphasizing that models can achieve high knowledge efficiency with proper training regimes and data annotation strategies.

💡 This finding is great. They found that even when parameters are quantized to int8, which inherently reduces the granularity of the information that these parameters can represent, the model retains its knowledge storage efficiency. This resilience implies that the essential knowledge representation within the model is preserved effectively, even in a reduced precision format.

[Valeriy M., PhD, MBA, CQF @predict_addict]

An interesting paper on the applications of Kolmogorov Arnold Network to evolutionary algorithms for optimisation shows strong potential of KANs and significant superior data fitting capabilities compared to MLPs.

Complete with GitHub code 🧑‍💻

#KAN

[Sebastian Raschka @rasbt]

In my monthly research write-up, I am covering 3 new papers related to instruction finetuning and parameter-efficient finetuning with LoRA in large language models (LLMs): https://t.co/DVaZszqqj8
I work with these methods practically every day, so it's always exciting to see new research that provides practical insights.

1st, we will explore whether and how prompt masking strategies affect the performance of finetuning LLMs. Counterintuitively, not masking the instruction prompt can improve the performance of LLMs on downstream tasks by a substantial amount.

2nd, we will review some lessons and learnings from experiments with low-rank adaptation for parameter-efficient finetuning. In short, LoRA learns less and forgets less. This means that while LoRA is a great tool for instilling instruction-following capabilities into an LLM, it is not the right tool when we aim to instill new knowledge.

3rd, we will examine a new alternative to LoRA, which addresses one of the shortcomings mentioned above: instilling new knowledge. This LoRA alternative can improve the uptake of new knowledge (for example, in continued pretraining) by several percentage points.

[Kacper Kapuśniak @KKapusniak1]

If data lives on a manifold, how do we design meaningful interpolations between marginals? We present Metric Flow Matching (MFM)…

@PPotaptchik @TeoReu @leoeleoleo1 @AlexanderTong7 @mmbronstein @bose_joey @Francesco_dgv

🔗Dive in here: https://t.co/XwaU2nyaMT
🧵 (1/12)

[Oana-Maria Camburu @oanacamb]

💫 Check out our #NAACL2024 paper on linear relational concepts in LLMs 👇

[Ziming Liu @ZimingLiu11]

MLPs are so foundational, but are there alternatives? MLPs place activation functions on neurons, but can we instead place (learnable) activation functions on weights? Yes, we KAN! We propose Kolmogorov-Arnold Networks (KAN), which are more accurate and interpretable than MLPs.🧵

[Sharan Narang @sharan0909]

I'm personally super excited to share the progress on the 400B+ training. So proud of the entire team that worked tirelessly to make this model a reality. There's lots more to come including a full research paper soon!🚀🦙🦙

[Andrew Carr 🤸 @andrew_n_carr]

Language models are bad a basic math.

GPT-4 has right around 0% accuracy rate on 5 digit multiplication.

Most open models can't even add. Why is that?

There are a few reasons why numbers are hard. The main one is Tokenization. When training a tokenizer from scratch, you take a large corpus of text and find the minimal byte-pair encoding for a chosen vocabulary size.

This means, however, that numbers will almost certainly not have unique token representations. "21" could be a single token, or ["2", "1"]. 143 could be ["143"] or ["14", "3"] or any other combination.

A potential fix here would be to force single digit tokenization. The state of the art for the last few years is to inject a space between every digit when creating the tokenizer and when running the model. This means 143 would always be tokenized as ["1", "4", "3"].

This helps boost performance, but wastes tokens while not fully fixing the problem.

A cool fix might be xVal! This work by The Polymathic AI Collaboration suggests a generic [NUM] token which is then scaled by the actual value of the number!

If you look at the red lines in the image above, you can get an intuition for how that might work.

It doesn't capture a huge range or high fidelity (e.g., 7.4449 vs 7.4448) but they showcase some pretty convincing results on sequence prediction problems that are primarily numeric.

For example, they want to train a sequence model on GPS conditioned temperature forecasting

They found a ~70x improvement over standard vanilla baselines and a 2x improvement over really strong baselines.

One cool side effect is that deep neural networks might be really good at regression problems using this encoding scheme!

[Andrej Karpathy @karpathy]

Promising. Everyone should hope that we can throw away tokenization in LLMs. Doing so naively creates (byte-level) sequences that are too long, so the devil is in the details.

Tokenization means that LLMs are not actually fully end-to-end. There is a whole separate stage with its own training and inference, and additional libraries. It complicates the ingest of additional modalities. Tokenization also has many subtle sharp edges. Few examples:

That "trailing whitespace" error you've potentially seen in Playground? If you end your (text completion API) prompt with space you are surprisingly creating a big domain gap, a likely source of many bugs:
https://t.co/f2PBaw2iA8

Tokenization is why GPTs are bad at a number of very simple spelling / character manipulation tasks, e.g.:
https://t.co/XR3d5g4uwp

Tokenization creates attack surfaces, e.g. SolidGoldMagikarp, where some tokens are much more common during the training of tokenizer than they are during the training of the GPT, feeding unoptimized activations into processing at test time:
https://t.co/y72eaIeRrP

The list goes on, TLDR everyone should hope that tokenization could be thrown away. Maybe even more importantly, we may find general-purpose strategies for multi-scale training in the process.

[Turing Post @TheTuringPost]

A free book on Computer Vision!

"Computer Vision: Algorithms and Applications" by Richard Szeliski.

Read it on the author's website: https://t.co/9ZXytUsIv7

[James Zou @james_y_zou]

SHAP is a good feature attribution algo BUT it identifies wrong features even in linear regression.

Our #NeurIPS2022 paper finds where SHAP goes wrong and proposes a simple fix *WeightedSHAP* that greatly improves attribution https://t.co/3htjJp5mCt
Code: https://t.co/fvmoMVNJKH

[Mo Sarwat @MoSarwat]

Ever struggled using @PyTorch to train deep learning on Geospatial data? Say no more. We developed a system (GeoTorch) that makes spatiotemporal deep learning in PyTorch a cakewalk👇🏽

https://t.co/qfdKTwXqKV

#geospatial #DeepLearning #ArtificialIntelligence #DataScience

[MIT Brain and Cognitive Sciences @mitbrainandcog]

Researchers use neural networks, a computing system loosely modeled on the brain that forms the basis of many AI systems, to learn how the brain performs certain tasks. MIT scientists are now urging more caution when comparing neural networks to the brain. https://t.co/lRfNknqzSp

[Stanford AI Lab @StanfordAILab]

Causal abstraction provides a powerful set of tools for accurate, human-interpretable explanations of AI models. Check out our latest from Atticus Geiger, @ZhengxuanZenWu, @KarelDoostrlnck, @ElisaKreiss, Noah Goodman, Thomas Icard, and @ChrisGPotts: https://t.co/MNpsIva9zf

[Karol Hausman @hausman_k]

Our 2021 CS330 (https://t.co/v74IYFBlJN) lectures are online: https://t.co/bYtLzcz0YB

It was a pleasure to co-teach this class with @chelseabfinn. Topics incl. meta-learning, MTL, few-shot learning, deep RL (incl. multi-task, meta, goal-conditioned, hierarchical and offline RL)