David Lim

christian, science enthusiast, educator of masses in science and mathematics

[Sentient @sentient_agency]

Download an entire country once, switch off your signal, and keep navigating.

Organic Maps gives you offline driving, hiking, and cycling maps using OpenStreetMap data.

No ads. No tracking. No Google quietly saving every place you visit.

I love tools like this because they solve a real problem without turning the user into the product.

Download the maps before your trip, then use them without roaming charges or hunting for a signal in the middle of nowhere.

It is completely free and open source.

If I were travelling somewhere with unreliable internet, this would be on my phone before the flight.

https://t.co/AH1rQ1EYb3

[David Lim @davidzlim]

@AndrewYNg @SaveToNotion #Thread #Agent

[David Lim @davidzlim]

@karinanguyen @aiDotEngineer @SaveToNotion #Thread

[Deedy @deedydas]

Tiktok's Monolith paper is a must read.

It shows that you don't need a social graph to make an addictive product if you nail real-time recommendations that update AS each user scrolls.

Very few great applied CS papers exist and even fewer that made a 1B+ user product!

[Muratcan Koylan @muratcan]

Anthropic has just revealed some exciting news about Claude Sonnet. They've successfully identified how millions of concepts are represented inside this massive model!

LLMs are usually treated as black boxes.

We feed them data, they give us responses, but understanding the 'why' behind their actions is challenging.

Their research used dictionary learning, a technique to find patterns of activated neurons or features.

These features provide a window into the model's understanding of various concepts.

Anthropic mapped out the internal state of Sonnet, finding features for specific entities like cities, people, and atomic elements.

These features are multimodal and multilingual, responding to text and images in multiple languages.

They even found abstract features related to concepts like inner conflict, relationship breakups, and logical inconsistencies.

By manipulating these features, researchers could alter Claude's behavior. For example, activating the "Golden Gate Bridge" feature made Claude obsessed with the bridge, mentioning it in almost every response.

Read more here -> https://t.co/rjXDwtPQDj

[Andrew Ng @AndrewYNg]

This week, Google announced a doubling of Gemini Pro 1.5's input context window from 1 million to 2 million tokens, and OpenAI released GPT-4o, which generates tokens 2x faster and 50% cheaper than GPT-4 Turbo and natively accepts and generates multimodal tokens. I view these developments as the latest in an 18-month trend. Given the improvements we've seen, best practices for developers have changed as well.

Since the launch of ChatGPT in Nov 2022, with key milestones that include the releases of GPT-4, Gemini 1.5 Pro, Claude 3 Opus, and Llama 3-70B, many model providers have improved their capabilities in two important ways: (i) reasoning, which allows LLMs to think through complex concepts and and follow complex instructions; and (ii) longer input context windows.

The reasoning capability of GPT-4 and other advanced models makes them quite good at interpreting complex prompts with detailed instructions. Many people are used to dashing off a quick, 1- to 2-sentence query to an LLM. In contrast, when building applications, I see sophisticated teams frequently writing prompts that might be 1 to 2 pages long (my teams call them “mega-prompts”) that provide complex instructions to specify in detail how we’d like an LLM to perform a task. I still see teams not going far enough in terms of writing detailed instructions. For an example of a moderately lengthy prompt, take a look at Claude 3’s system prompt. It’s detailed and gives clear guidance on how Claude should behave.

This is a very different style of prompting than we typically use with LLMs’ web user interfaces, where we might dash off a quick query and, if the response is unsatisfactory, clarify what we want through repeated conversational turns with the chatbot.

Further, the increasing length of input context windows has added another technique to the developer’s toolkit. GPT-3 kicked off a lot of research on few-shot in-context learning. For example, if you’re using an LLM for text classification, you might give a handful — say 1 to 5 examples — of text snippets and their class labels, so that it can use those examples to generalize to additional texts. However, with longer input context windows — GPT-4o accepts 128,000 input tokens, Claude 3 Opus 200,000 tokens, and Gemini 1.5 Pro 1 million tokens (2 million just announced in a limited preview) — LLMs aren’t limited to a handful of examples. With many-shot learning, developers can give dozens, even hundreds of examples in the prompt, and this works better than few-shot learning.

When building complex workflows, I see developers getting good results with this process:
- Write quick, simple prompts and see how it does.
- Based on where the output falls short, flesh out the prompt iteratively. This often leads to a longer, more detailed, prompt, perhaps even a mega-prompt.
- If that’s still insufficient, consider few-shot or many-shot learning (if applicable) or, less frequently, fine-tuning.
- If that still doesn’t yield the results you need, break down the task into subtasks and apply an agentic workflow.

I hope a process like this will help you build applications more easily. If you’re interested in taking a deeper dive into prompting strategies, I recommend Microsoft's Medprompt paper (Nori et al., 2023), which lays out a complex set of prompting strategies that can lead to very good results.

[Original text (with links): https://t.co/UOtLDza1Vh ]

[David Lim @davidzlim]

@aparnadhinak @arizeai @ArizePhoenix @huggingface @OpenAI @AnthropicAI @ClementDelangue @julien_c @Thom_Wolf @AiEleuther @GoogleAI @GoogleDeepMind @SaveToNotion #thread #LLMs

[Jim Fan @DrJimFan]

This may be Apple's biggest move on open-source AI so far: MLX, a PyTorch-style NN framework optimized for Apple Silicon, e.g. laptops with M-series chips.

The release did an excellent job on designing an API familiar to the deep learning audience, and showing minimalistic examples on OSS models that most people care about: Llama, LoRA, Stable Diffusion, and Whisper.

I expect no less from my former colleague @awnihannun, spearheading this effort at Apple. Thanks for the early Christmas gift! 🎄🎁

MLX source: https://t.co/GlNJDOu9NE
Well-documented, self-contained examples: https://t.co/UazqGZ0nhZ

[Jim Fan @DrJimFan]

This article offers a rare view into the inner workings of Google, and sheds light on why OpenAI spearheaded the LLM revolution even though most of the foundation tech originated at Google.

Google is right in the thick of the Innovator's Dilemma. I am sure they know very well the power of foundation models. After all, they invented Transformer, AlphaGo (RL), and Flamingo (multimodal) - the key ingredients of GPTs.

Yet it is so difficult to justify diverting resources away from the ongoing profitable products, or potentially even cannibalize Search to promote LLMs. Citing the article, "Google is an ever-shifting web of goals and efforts". You need all the relevant high-ups to agree on the agenda, actively fight for enormous resources, and push back against all the other parties that want the minimal disruption. Too many stars need to align. That is a very tall order.

That being said, I don't think it's fair to blame anyone. The institutional bureaucracy is a natural "emergent property" of companies at this scale. 10 years from now, OpenAI and Anthropic may suffer from the same if they grow to that order of magnitude.

I'll end with an encouraging quote from the blog. This advice holds universally true for anyone who wants to push change in a big org:

Amazing things are possible at Google, if you play the right game. What this means is to first do the work that is given to you. But once that's under control, to reach out into the vast Google network, to learn what's being planned and invented, to coalesce a clear image of the future, to give it shape through docs and demos, to find the leaders whose goals align with this image, and to sell the idea as persistently as you can.

Blog: https://t.co/Yjx6Oi4Ewr @shreyans___

[David Lim @davidzlim]

@mpshanahan @Nature @SaveToNotion #Tweet #LLMs

[Murray Shanahan @mpshanahan]

My new @Nature paper "Role Play with Large Language Models" was briefly behind a paywall. It's now free-to-read: https://t.co/3XHJ8pLaIc

[David Lim @davidzlim]

@jerryjliu0 @SaveToNotion #tweet #llms

[David Lim @davidzlim]

@polak_jasper @SaveToNotion #Tweet #consulting

[David Lim @davidzlim]

@_akhaliq @SaveToNotion #thread #llm #maths

[David Lim @davidzlim]

@kareem_carr @SaveToNotion #Thread #regression

[David Lim @davidzlim]

@Yampeleg @SaveToNotion #tweet #LLMs

[Yam Peleg @Yampeleg]

The open source models arena

I try not to post too much about open models until we reach a point where there will no longer be any debate about if they are at the level of closed models.

So let's make it brief.

----
LLaMA 2

The open models arena heated up last week with the release of LLaMA 2.

The model is a direct continuation of the base model for the LLaMA from Meta and marks a significant leap from the first model that can be attributed mainly to the double amount of data on which the model was trained on.

(And probably also the quality of this data, which remained confidential)

The model was also released in a chat version but I will not expand on it at this stage, that chat version suffers from several problems and it is considered not very useful at the moment.

(But the paper do list many tricks used when training the chat version and those tricks are especially interesting and useful. recommended)

----
Update Versions

Since the release, nearly all groups working on open source models have updated their models to use the new base model.

We received updated versions from some of the most powerful open models today:

- New WizardLM model: https://t.co/dd4SqtDtwF
(from @WizardLM_AI)

- New Airoboros model: https://t.co/qZxDhkNj9U (from @jon_durbin)

- New Hermes model: https://t.co/BvXu1h4ADo (from @NousResearch)

----
The most powerful model today: Stable Beluga 2
(from @StabilityAI & @carperai)

Last week we also got one of the most powerful open models we've seen so far from StabilityAI.

The model is comparable to ChatGPT in almost [1] all measurable metrics and is currently holding the first place on Huggingface's leaderboard.

- Stable Beluga 2: https://t.co/eObXUcvV5O

-----
Long Models

The research on extending the context window length also continues in full force we received longer versions of the base model itself, which you can find here:

- LLaMA 7B 16K: https://t.co/E8KuF1W081 (from @EnricoShippole)

- LLaMA 13B 16K: https://t.co/51GeYxMawl (from @EnricoShippole)

- LLaMA 7B 32K: https://t.co/iq6Qxt0fJz (from @togethercompute)

-----
Small Powerful Models

Another interesting model we got is a 3B-parameter model that is as powerful as a 7B model.

This goes to show what most of us have feel for a long time: We have not yet reached the limit of these models. There is more to push.

You are probably thinking that the trick is simply more data: as always.

Surprise.

The tricks are:

- A large (600B) fully de-duped dataset: SlimPajama. This is the first model to train on slim pajama end 2 end [1]
- SwiGLU [2]
- ALiBI [3]
- Variable Sequence Length (2 stage training: short then long) [4]
- Maximal update parameterization (muP): Allows you to "guess" the best hyper-parameters before starting to train. [5]

- BTLM-8K-Base: https://t.co/wMkbIU8tS3 (from @cerebras)

---
[1] https://t.co/ipWKikv9Dg
[2] https://t.co/cZwmBK1sAl
[3] https://t.co/Zfz2hEw9YN
[4] https://t.co/vgvKVrxUr6
[5] https://t.co/EVF2XzG9xn (remember that on GPT-4's paper, the loss was "predicted" before the training started?)
---

-----
Open model defeats ChatGPT in MMLU

Although a single number on a single test does not reflect reality, Last week for the first time we got a model that defeats ChatGPT at MMLU.

(And is not particularly trained to do so. That's why it's impressive)

The model: https://t.co/KzYHPeSTxr

-----
Multi-Turn Chats

One of the main differences setting apart open source models at the moment is multi-turn conversations, (in my humble opinion) we already got to the point where in a single turn our models compete with the closed models but when it comes to multi-turn conversations: Open models tend to go off topic.

This is about to change.

A particularly interesting model was released serveral days ago: Another model trained according to Orca's methods (adding detailed explanations to each answer) BUT for multi-turn long chats.

The model was created with a window length of 8K and according to initial impressions it is one of the best models released.

The model: https://t.co/Epgb04d7EW (from @Shahules786)

-----
Chinese models are putting up a fight

Even before the release of Lemma 2, it was clear that the Chinese models are already particularly strong.
And you are advised to try them, their English is excellent and some of them are very useful.

The most powerful programming model for its size: CodeGeex2

The Chinese coding model CodeGeeX2 comes to us with a tiny size of 6B parameters (trained from the flagship Chinese model ChatGLM) and overtakes all models of this size scoring 35.9 on HumanEval (Pass@1).

The model: https://t.co/HnRREpqCNj (from @thukeg)

-----
More details on the Chinese models

Lately I find myself reading more and more posts translated from Chinese via Google Translate.

Infrastructure for LLMs training and dealing with data coming from China is often times particularly high-quality and also incorporate techniques that do not exist in more popular code bases.

(Such as: training with masking, bidirectional training, architecture improvements and delicate tokenizer work to support the Chinese language)

I recommend everyone in the field to also read about the advances of the Chinese models.

-----
Are the open models already at the level of ChatGPT?

Short answer: Not yet, but they are on their way.

Usually when someone say that open source models are on the level of ChatGPT, immediately someone comes up with the hardest test they can think of to show that the open source models are not on the same level as ChatGPT.
According to rumors, ChatGPT was trained on somewhere around 6 times the data of LLaMA 2.

If you want to find holes, you will find them.

Nonetheless, in the real world: I use open models everyday they are just as good as ChatGPT.

-----
What is still left to do?

After the release of the Stable Beluga 2 model I wrote a post summarizing all the metrics from all the datasets where the model's results still don't crush ChatGPT.

You can find it here: https://t.co/aDNFtCHhyl

-----
How do we measure the quality of models?

There are several main "holes" that separate the open models from the closed models, you can read about these holes and the differences between the various models here: https://t.co/5vLhrCf4Eb

-----
From the news: Open source model from OpenAI?

On the background of this, we recently received an interesting news article about the efforts within OpenAI to release an open model.

Link: https://t.co/5NQVxpfBIa

It is not known if the information in this article is correct, but according to the article, open source models are putting pressure on OpenAI they are currently working on an open model as a response to the LLaMA 2.

The model does not compete with the quality of ChatGPT (and certainly not with GPT-4).

[David Lim @davidzlim]

@thealexbanks @threadreaderapp unroll

[David Lim @davidzlim]

@thealexbanks @SaveToNotion #thread#plugins

[David Lim @davidzlim]

@Jeande_d @SaveToNotion #tweet #LLMs