cornelistools

Building advanced language technologies and robust AI systems for the Dutch language 🇳🇱 Research & development in #nlp #nlu #ai #rl #conversationalAI

[cornelistools @cornelistools]

Effectiever trainen van Nederlandstalige taalmodellen door onderzoek in niet-opknipbare zelfstandig naamwoorden en het verminderen van ambiguïteit in tokenizers. https://t.co/m7gVxWAoXg
#LLMs #AI #tokenizers #linguistics #NLP

[cornelistools @cornelistools]

Again an amazing @MLStreetTalk with the great @LauraRuis about reasoning and agency in AI Models and her paper "Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models". How Do AI Models Actually Think? https://t.co/u5tCd2OEnZ

[Stephen Wolfram @stephen_wolfram]

What's really going on in machine learning? Just finished a deep dive using (new) minimal models. Seems like ML is basically about fitting together lumps of computational irreducibility ... with important potential implications for science of ML, and future tech...
https://t.co/OfevpJezi7

[Andrej Karpathy @karpathy]

# RLHF is just barely RL

Reinforcement Learning from Human Feedback (RLHF) is the third (and last) major stage of training an LLM, after pretraining and supervised finetuning (SFT). My rant on RLHF is that it is just barely RL, in a way that I think is not too widely appreciated. RL is powerful. RLHF is not. Let's take a look at the example of AlphaGo. AlphaGo was trained with actual RL. The computer played games of Go and trained on rollouts that maximized the reward function (winning the game), eventually surpassing the best human players at Go. AlphaGo was not trained with RLHF. If it were, it would not have worked nearly as well.

What would it look like to train AlphaGo with RLHF? Well first, you'd give human labelers two board states from Go, and ask them which one they like better:

Then you'd collect say 100,000 comparisons like this, and you'd train a "Reward Model" (RM) neural network to imitate this human "vibe check" of the board state. You'd train it to agree with the human judgement on average. Once we have a Reward Model vibe check, you run RL with respect to it, learning to play the moves that lead to good vibes. Clearly, this would not have led anywhere too interesting in Go. There are two fundamental, separate reasons for this:

1. The vibes could be misleading - this is not the actual reward (winning the game). This is a crappy proxy objective. But much worse,
2. You'd find that your RL optimization goes off rails as it quickly discovers board states that are adversarial examples to the Reward Model. Remember the RM is a massive neural net with billions of parameters imitating the vibe. There are board states are "out of distribution" to its training data, which are not actually good states, yet by chance they get a very high reward from the RM.

For the exact same reasons, sometimes I'm a bit surprised RLHF works for LLMs at all. The RM we train for LLMs is just a vibe check in the exact same way. It gives high scores to the kinds of assistant responses that human raters statistically seem to like. It's not the "actual" objective of correctly solving problems, it's a proxy objective of what looks good to humans. Second, you can't even run RLHF for too long because your model quickly learns to respond in ways that game the reward model. These predictions can look really weird, e.g. you'll see that your LLM Assistant starts to respond with something non-sensical like "The the the the the the" to many prompts. Which looks ridiculous to you but then you look at the RM vibe check and see that for some reason the RM thinks these look excellent. Your LLM found an adversarial example. It's out of domain w.r.t. the RM's training data, in an undefined territory. Yes you can mitigate this by repeatedly adding these specific examples into the training set, but you'll find other adversarial examples next time around. For this reason, you can't even run RLHF for too many steps of optimization. You do a few hundred/thousand steps and then you have to call it because your optimization will start to game the RM. This is not RL like AlphaGo was.

And yet, RLHF is a net helpful step of building an LLM Assistant. I think there's a few subtle reasons but my favorite one to point to is that through it, the LLM Assistant benefits from the generator-discriminator gap. That is, for many problem types, it is a significantly easier task for a human labeler to select the best of few candidate answers, instead of writing the ideal answer from scratch. A good example is a prompt like "Generate a poem about paperclips" or something like that. An average human labeler will struggle to write a good poem from scratch as an SFT example, but they could select a good looking poem given a few candidates. So RLHF is a kind of way to benefit from this gap of "easiness" of human supervision. There's a few other reasons, e.g. RLHF is also helpful in mitigating hallucinations because if the RM is a strong enough model to catch the LLM making stuff up during training, it can learn to penalize this with a low reward, teaching the model an aversion to risking factual knowledge when it's not sure. But a satisfying treatment of hallucinations and their mitigations is a whole different post so I digress. All to say that RLHF *is* net useful, but it's not RL.

No production-grade *actual* RL on an LLM has so far been convincingly achieved and demonstrated in an open domain, at scale. And intuitively, this is because getting actual rewards (i.e. the equivalent of win the game) is really difficult in the open-ended problem solving tasks. It's all fun and games in a closed, game-like environment like Go where the dynamics are constrained and the reward function is cheap to evaluate and impossible to game. But how do you give an objective reward for summarizing an article? Or answering a slightly ambiguous question about some pip install issue? Or telling a joke? Or re-writing some Java code to Python? Going towards this is not in principle impossible but it's also not trivial and it requires some creative thinking. But whoever convincingly cracks this problem will be able to run actual RL. The kind of RL that led to AlphaGo beating humans in Go. Except this LLM would have a real shot of beating humans in open-domain problem solving.

[cornelistools @cornelistools]

@aidangomez One-Shot or Few-Shot?

[Yann LeCun @ylecun]

🥁 Llama3 is out 🥁
8B and 70B models available today.
8k context length.
Trained with 15 trillion tokens on a custom-built 24k GPU cluster.
Great performance on various benchmarks, with Llam3-8B doing better than Llama2-70B in some cases.
More versions are coming over the next few months.

https://t.co/EkU9aIHdZE

[François Chollet @fchollet]

At least once a year I come across the argument that "scale is all you need: the more neurons a species has, the more intelligent it is; humans have over 2x more neurons than gorillas and that makes all the difference; future AIs will have even more neurons than us! If we give them 1,000x more neurons they will be 1,000x more intelligent!"

As a reminder, the species with the most neurons are whales and African elephants (a whopping 3x more than us). And for any particular neuron count, you will find species with wildly different levels of cognitive ability.

[cornelistools @cornelistools]

@fchollet I agree. The strength of the Stack Overflow is still a community with diversity, vision and experiences of many programmers around the world. And one answer is often not an answer. In coding the why is at least important as the how. LLMs can't provide that vision yet.

[Google DeepMind @GoogleDeepMind]

We’re releasing Gemma 2B and 7B, which achieve best-in-class performance for their sizes compared to other models, and can run on a developer laptop or computer.

They also surpass much larger models on key benchmarks while meeting our standards for safe and responsible outputs.

[cornelistools @cornelistools]

@pkuhar @GaryMarcus Indeed, with RAG you limit the allowed "knowlegde" of the model to "talk" about only the retrieved results provided in a prompt. As the retrieval system is basically search (or "semantic" or whatever search), it has many limitations on reasoning, nor sense of the total dataset.

[cornelistools @cornelistools]

@GaryMarcus I mean, if the expectation is a talking search engine, it works "ok", but mainly defined by the quality of the external retrieval system. If the expectation is reasoning about the question and results, the RAG method is useless. LLMs do nothing very smart here & even can be SLMs.

[cornelistools @cornelistools]

@karpathy I tried managing my schedule with an LLM and had the same result 😉

[cornelistools @cornelistools]

@GaryMarcus As you need to insert with RAG your company's knowlegde externally into the LLM, most of the preliminary work is done in the R-mechanism (retrieval) outside the LLM. Using a LLM to generate pre-presented data is actually a waste of the knowlegde it has. SLMs work fine here too.

[cornelistools @cornelistools]

@christineliebr @KvW Ik gebruik deze ook en denk dat deze aan de goede kant zit. Deze kristalliseert (voornamelijk eigenschap van kwalitatieve honing volgens https://t.co/7feqr0KtHu), komt uit bloemen en echte honing is idd niet geschikt voor heel jonge kinderen.

[Cohere Labs @Cohere_Labs]

Today, we’re launching Aya, a new open-source, massively multilingual LLM & dataset to help support under-represented languages. Aya outperforms existing open-source models and covers 101 different languages – more than double covered by previous models.

https://t.co/0WsC2i9C8a

[cornelistools @cornelistools]

@ohmypy And for many scripting languages you don't really need a configuration language at all. Having a config .py or .php file is not very wrong. However, to separate code and configuration language completely or having a compiled language, JSON would be fine and a well-known standard.

[cornelistools @cornelistools]

@Justin_Halford_ @fchollet If the task you want to solve is more beneficial than the energy consumption, it may not. However, knowing spending a 500ml bottle of water when I have a conversation with ChatGPT, I would like to see more earth friendly solutions. There is a lot that needs to be improved there.

[cornelistools @cornelistools]

@fchollet Also:
Human: "Let's eat 2 slices of bread with peanut butter for the morning"
Tesla bot: "The production version of the Optimus bot will be equipped with a 2.3-kilowatt battery pack"

[cornelistools @cornelistools]

@fchollet 12 watts is about 2x Raspberry Pi 4B's at full load 💪

I love to see people already experimenting with some 7B GGUF models on PIs, and I hope to see way more focus on energy efficient models.

[cornelistools @cornelistools]

Mijn gevoel dat Chain-of-Thought prompting vooral werkt is omdat het deels "Chain-of-Completion" is. LLMs zijn getraind voor completion (next token prediction) en door een redenatie vraagstuk stap voor stap uit te schrijven, word redenatie een behapbaarder completion vraagstuk.