Marianne Stecklina

Deep Learning Engineer @omniusHQ | @[email protected]

[Alex Volkov @altryne]

We had a "Cursor tips & tricks" meeting today with my colleagues at @wandb and I figured I'd share what we 'discovered' & shared between us in a 🧵

If you haven't chatted with your team about how YOU all use Cursor, you def should, but meanwhile here are our insights 👇

[Siyan Zhao @siyan_zhao]

Have you wondered how the decision boundary of in-context learning in LLMs compares to traditional models like Decision Trees and KNN? 🤔

Our research uncovers unexpected irregularities and non-smoothness in LLMs' in-context decision boundaries. 🔍

📄: https://t.co/aGtzL9hjON

[Aaron Defazio @aaron_defazio]

Schedule-Free paper is up!
https://t.co/vduzoGL5EP

Joint work with collaborators @alicey_ang @HarshMeh1a @konstmish @akhaledv2 @AshokCutkosky

We have some strong small-scale experiments on Transformers, comparing to chinchilla-style cosine 10x reduction schedules.

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

🧱DBRX🧱 is so good that it forced 3-4 companies to release "competing" LLMs in the last two days (and we've barely heard about them). Some of my thoughts are summarized below...

Prior research from Mosaic. DBRX is the next model in the series of Open LLMs released by Mosaic. Two versions of the model are released—a base model (DBRX base) and a finetuned model (DBRX Instruct)—under an open license (i.e., the Databricks open model license). Mosaic entered the space of open LLMs with their proposal of MPT-7B in May of 2023—less than a year ago! Since then, Mosaic has:

1. Been acquired for $1.3B
2. Trained thousands of specialized LLMs for customers
3. Found time to release several open LLMs (DBRX being the latest)
4. Improved the end-to-end efficiency of LLM pretraining by over 4X

The MoE-based DBRX model achieves new state-of-the-art performance for open language models in terms of both quality and efficiency across a massive suite of evaluations.

Model architecture. DBRX is a Mixture-of-Experts (MoE) model that is pretrained on 12T tokens of curated text. Compared to popular models like ST-MoE that use a smaller number of large experts, DBRX takes an opposite strategy, using a fine-grained MoE architecture with a larger number of small experts. The model has 132B parameters in total with 36B parameters active at any given time (i.e., each MoE layer has 16 experts in total and four experts are chosen in each forward pass). Relative to models like Mixtral and Grok-1 that have only eight experts (and two active experts), DBRX has 65X more potential combinations of experts that can be used, which is found to improve model quality.

“We estimate that our new pretraining data is at least 2x better token-for-token than the data used to train MPT-7B. In other words, we estimate that half as many tokens are necessary to reach the same model quality.” - from DBRX writeup

More DBRX details. Although the pretraining dataset for DBRX is much larger than prior models, authors also claim that the data used to train DBRX is higher-quality. As a result of this quality increase, the statistical training efficiency of DBRX is higher than normal—training is faster because we achieve higher accuracy with a smaller number of tokens. Additionally, a curriculum learning strategy (i.e., meaning that the data mixture is dynamically changed throughout pretraining) is adopted for training DBRX and found to improve model quality. DBRX has a longer context length of 32K and uses the GPT-4 tokenizer—available via the tiktoken package—instead of the GPT-NeoX tokenizer used by prior MPT models, which is found to improve performance.

Better efficiency. The proposal of DBRX comes with large improvements in terms of both training and inference efficiency. As mentioned above, DBRX is trained over an optimized dataset that improves training efficiency by roughly 2X. However, a variety of other factors yield further boosts in training efficiency:

- The MoE architecture, which is found in smaller-scale experiments (i.e., on DBRX MoE-B, a 23.5B parameter model with 6.6B active parameters) to require 1.7× fewer FLOPS during training compared to a performance-matched LLaMA-2-13B model.
- Other modifications to the decoder-only architecture (i.e., RoPE, GLU activation, and GQA).
- “Better optimization strategies” (hopefully more details coming soon).

When considering all data, architecture, and optimization changes, the end-to-end training process of DBRX requires 4X less compute than prior models released by Mosaic. To determine this number, authors compare a smaller variants of DBRX, DBRX MoE-A (i.e., a 7.7B parameter LLM with 2.2B active parameters) to MPT-7B, finding that DBRX MoE-A achieves similar performance on the Databricks Gauntlet while using 3.7× fewer FLOPS during training.

DBRX is efficient for both training and inference—up to 2X faster than LLaMA-2-70B at 150 tokens per second per user in load tests. Mosaic has an optimized serving infrastructure that uses TensorRT-LLM and 16 bit precision, which is very fast. The MoE architecture of DBRX also aids inference efficiency due to the relatively low number of active parameters; e.g., DBRX is 40% of the size of Grok-1 in both total and active parameters.

“Using an MoE architecture makes it possible to attain better tradeoffs between model quality and inference efficiency than dense models typically achieve.” - from DBRX writeup

Empirical evaluation. In comparison to other open LLMs, we see that DBRX Instruct achieves the best performance on composite benchmarks by a large margin compared to Mixtral; see above. Notably, DBRX has impressive programming skills, outperforming models like Grok-1 that are more than twice its size and specialized coding models like CodeLLaMA-70B (despite being a general-purpose LLM!). DBRX also has impressive reasoning/math skills.

Compared to closed models, DBRX surpasses the performance of GPT-3.5 and is competitive with Gemini Pro. Gemini Pro only outperforms DBRX on GSM8K, while Mixtral-Medium performs better on a select few tasks that are considered; see above. At a high level, DBRX seems to be good at general knowledge and commonsense reasoning tasks, as well as programming and math. DBRX also is shown to perform competitively with models like Mixtral and GPT-3.5/4-Turbo on tasks involving RAG and long context.

The research process. DBRX was created in less than three months using a cluster of 3K H100 GPUs with a budget of around $10M. Mosaic is an applied research team, meaning that a lot of their work is customer-focused—they have trained thousands of specialized LLMs for various Databricks customers. However, this focus upon applications is exactly what makes the team so interesting—it forces researchers to sift through the noise of AI research and find techniques that work. Plus, the team still finds enough free time to conduct legitimate research and has had a massive impact on the trajectory of open-source LLMs through the MPT/DBRX models.

[muhtasham @xmuhtasham]

A short thread about changes in the transformer architecture since 2017.

Reading articles about LLMs, you can see phrases like “we use a standard transformer architecture.”

But what does "standard" mean, and have there been changes since the original article?

(1/6)

[Omar Sanseviero @osanseviero]

Introducing...🥁🥁Mamba models are now compatible with 🤗transformers! Mamba models are super fast (scale well!) and have solid quality⚡️

- Generation utilities
- PEFT fine-tuning
- TRL support

Check the repos here!
https://t.co/EbcjkwltgS

[Saurabh Srivastava @_saurabh]

More than 50% of the reported reasoning abilities of LLMs might not be true reasoning.

How do we evaluate models trained on the entire internet? I.e., what novel questions can we ask of something that has seen all written knowledge? Below: new eval, results, code, and paper.

Functional benchmarks are a new way to do reasoning evals. Take a popular benchmark, e.g., MATH, and manually rewrite its reasoning into code, MATH(). Run the code to get a snapshot that asks for the same reasoning but not the same question. A reasoning gap exists if a model’s performance is different on snapshots. Big question: Are current SOTA models closer to gap 0 (proper reasoning) or gap 100 (lots of memorization)?

What we find: Gaps in the range of 58% to 80% in a bunch of SOTA models. Motivates us to build Gap 0 models.

We’re releasing the paper, code, and 3 snapshots of functional MATH() today.

arxiv draft: https://t.co/KtvWPc0R72
github repo: https://t.co/gzDVaxZ9yg

1/🧵

[Tianle Cai @tianle_cai]

How much value does your fine-tuning add? Believe it or not, just 1 bit 🤏 Thrilled to unveil BitDelta, a super simple yet effective method for compressing fine-tuning deltas into a single bit while barely touching performance. This approach slashes storage and GPU memory demands by over 10x for numerous fine-tunes on the same base model—a frequent practice, and enables efficient multi-tenant serving.

[Bob West @cervisiarius]

In our new preprint, we ask: Do multilingual LLMs trained mostly on English use English as an “internal language”? - A key question for understanding how LLMs function.

“Do Llamas Work in English? On the Latent Language of Multilingual Transformers”
https://t.co/C9a907ByAL

[Florian Brand @xeophon]

@vboykis https://t.co/C1gfErer0l is the biggest push towards safetensors with some commentary and a 57 page security audit

[Ines Montani @_inesmontani]

People often ask me how I make my talk slides or for recommendations for getting started, so here are some of my not-so-secret secrets and beginner-friendly tips!

⚡ A beginner’s guide to making beautiful slides for your talks
https://t.co/Zeh5By58GL

[Sam Rose @samwhoo]

Finally got round to updating the links page on my site. It lists the best things I've seen on the Internet. It's not exhaustive, but most of what I love is on there.

If any of it resonates with you, I'd love to hear about it.

https://t.co/hf1tsImEyH

[Lior Alexander @LiorOnAI]

Big. An RNN-based LLM just outperformed transformers.

Eagle-7B is a new attention-free LLM with 1 Trillion Tokens Across 100+ Languages.

It's RWKV-v5 architecture uses RNNs instead of the transformer architecture allowing 10-100x lower inference cost, speed, and longer context windows.

- Outperforms all 7B class models in multi-lingual benchmarks

- Approaches Falcon (1.5T), LLaMA2 (2T), Mistral (>2T?) level of performance in English evals

[Nathan Benaich @nathanbenaich]

🪩The @stateofai 2023 is now here.

Our 6th installment is one of the most exciting years I can remember. The #stateofai report covers everything you *need* to know, covering research, industry, safety and politics.

There’s lots in there, so here’s my director’s cut 🧵

[Yung-Sung Chuang @YungSungChuang]

(1/5)🚨Can LLMs be more factual without retrieval or finetuning?🤔 -yes✅

🦙We find factual knowledge often lies in higher layers of LLaMA
💪Contrast high/low layers can amplify factuality & boost TruthfulQA by 12-17%

📝https://t.co/HyyIV782Tl
🧑‍💻https://t.co/KI56Gk3zRT

#NLProc

[TimDarcet @TimDarcet]

Vision transformers need registers!
Or at least, it seems they 𝘸𝘢𝘯𝘵 some…
ViTs have artifacts in attention maps. It’s due to the model using these patches as “registers”.
Just add new tokens (“[reg]”):
- no artifacts
- interpretable attention maps 🦖
- improved performances!

[Bindu Reddy @bindureddy]

The Ongoing Case For Open Source LLMs
Custom LLMs, long context, and efficient inference

Some folks believe that training open-source LLMs is a losing battle and a complete waste of time.

They argue that the gap between closed models like GPT-4 and open models like Llama will widen and these open-source models may never catch up.

Yes, closed models like Google's Gemini or Open AI's Gobi promise to be way more powerful than GPT-4, so what hope does open source have?

To start with inference on GPT-4 is very expensive. These very large models may be performant but aren't cost-effective. At Abacus, we routinely use fine-tuned versions of LLama-2 and smaller models when we need to run 1M+ API calls a day for standard enterprise applications Q/A, summarization, and NLP at scale. GPT-4 would cost > $100K a day in these cases.

Instruct-tuned LLMs can match the performance of GPT-4 for a specific task. Instruction tuning is a technique that aims to improve the capabilities and controllability of LLMs. It involves further training these models on a dataset consisting of (instruction, output) pairs in a supervised manner. This bridges the gap between the next-word prediction objective of LLMs and the users' objective of having LLMs adhere to human instructions.

For example, we have instruct-tuned open-source models for tasks like Q/A, NER, and classification. These instruct-tuned models are better at generalizing the task to new data and can do so in a resource-efficient manner.

Another shortcoming of currently available closed models like GPT-4 is that they have relatively short context lengths. 8K tokens are default and this means that you can't pass it large documents and ask it to extract the results from there.

Luckily the open-source community has been busy solving practical problems like this. Earlier this week, the paper LongLoRA introduced an ultra-efficient fine-tuning approach to significantly extend the context windows of pre-trained LLMs.

LongLoRA adopts LLaMA2 7B from 4k context to 100k, or LLaMA2 70B to 32k on a single 8x A100 machine, and basic implementation only takes 2 lines of code.

Open-source LLMs have been the focus of the GPU-poor and constraining resources typically have magical effects - efficient, elegant, and simple innovations that solve the problem!

Open-source LLMs have emerged as cheap and efficient alternatives for enterprise AI use cases and will continue to play an important role in the space.

Some have argued that open-sourcing LLMs is dangerous and they may be misused by bad actors.

There is no historical precedent for this argument.

Traditionally, open-source technology has spurred innovation, transparency, and the creation of safe and robust systems. Linux, triumphed over Unix in the OS world, largely because it is open-source and has a huge developer community.

Open source promotes collaboration, community oversight, rapid iteration, and benchmarking all essential for responsible AI development. Open-source developer communities tend to be great at detecting and plugging vulnerabilities.

Disappointingly, big players like OpenAI (despite their name) and Google, haven't open-sourced a lot of their technology. Luckily for the open-source community, Meta has created accessible open-source LLMs. In spite of Meta open-sourcing the powerful 70B LLama-2., the doomsday scenarios outlined by the anti-open-source crowd haven't come true.

Finally, multimodal LLMs (MLLM) are around the corner and if the GPU-rich won't outsource a MLLM, we can always enhance an existing open-source LLM and convert it into a multi-modal model.

In summary, open-source LLMs play a role in the real-world application of AI and are crucial for the democratization of this technology, transparency, and AI alignment

[Owain Evans @OwainEvans_UK]

Does a language model trained on “A is B” generalize to “B is A”?
E.g. When trained only on “George Washington was the first US president”, can models automatically answer “Who was the first US president?”
Our new paper shows they cannot!

[Andrej Karpathy @karpathy]

Speculative execution for LLMs is an excellent inference-time optimization.

It hinges on the following unintuitive observation: forwarding an LLM on a single input token takes about as much time as forwarding an LLM on K input tokens in a batch (for larger K than you might think). This unintuitive fact is because sampling is heavily memory bound: most of the "work" is not doing compute, it is reading in the weights of the transformer from VRAM into on-chip cache for processing. So if you're going to do all that work of reading in all those weights, you might as well apply them to a whole batch of input vectors. I went into more detail in an earlier thread:
https://t.co/Lbtpq4VDeY

The reason we can't naively use this fact to sample in chunks of K tokens at a time is that every N-th token depends on what token we sample at time at step N-1. There is a serial dependency, so the baseline implementation just goes one by one left to right.

Now the clever idea is to use a small and cheap draft model to first generate a candidate sequence of K tokens - a "draft". Then we feed all of these together through the big model in a batch. This is almost as fast as feeding in just one token, per the above. Then we go from left to right over the logits predicted by the model and sample tokens. Any sample that agrees with the draft allows us to immediately skip forward to the next token. If there is a disagreement then we throw the draft away and eat the cost of doing some throwaway work (sampling the draft and the forward passing for all the later tokens).

The reason this works in practice is that most of the time the draft tokens get accepted, because they are easy, so even a much smaller draft model gets them. As these easy tokens get accepted, we skip through those parts in leaps. The hard tokens where the big model disagrees "fall back" to original speed, but actually a bit slower because of all the extra work.

So TLDR: this one weird trick works because LLMs are memory bound at inference time, in the "batch size 1" setting of sampling a single sequence of interest, that a large fraction of "local LLM" use cases fall into. And because most tokens are "easy".

References
https://t.co/sIBCSmsyKN
https://t.co/uSpmTzfWhR
https://t.co/7t7orHBybo

[Marco Mascorro @Mascobot]

Big improvement in Python:

The CPython's memory management will now be thread-safe, allowing truly multi-threaded programs to take full advantage of multiprocessor systems.

- Deployed as early as in 3.13 or 3.14 as experimental
- Long-term (5+ years) no-GIL build should be the only build with backward compatibility 🤯