Ludovic Samper

#datascience #bike #computerscience #research

[Gael Varoquaux 🦋 @GaelVaroquaux]

🧑‍💻🧑‍🏫 I'm recruiting a post-doc to work on Tabular Foundation Models, one of the hotest topics in AI, where we are at the leading edge.

This is an opportunity to develop the next-level tabular AI, blending deep learning and tables.

[Valerie Faudon @ValerieFaudon]

La ministre allemande de l’énergie K. Reiche: sortir du nucléaire était une grosse erreur. Cette énergie nous manque car les coûts étaient très abordables. La seule manière d’assurer notre sécurité d’approvisionnement électrique est désormais le gaz. https://t.co/PuPf0Ldp6x

[Ludovic Samper @lsamper]

@PierreCol Voilà qui te donnerait presque envie de re-travailler! ;-)

[Ludovic Samper @lsamper]

@ylecun @paulg What about the emergence of social networks?

[Ludovic Samper @lsamper]

@Hugo_Baup En utilisant l'onglet "Following" trié par date "sort by recent" plutôt que l'onglet "For you", on limite la portée de l'algorithme twitter qui choisit ce qu'on voit.

[Antoine FLAHAULT @FLAHAULT]

La chercheuse @DgCostagliola que je connais depuis très longtemps incarne l’intégrité et l’excellence scientifique comme personne.
Elle a le courage de ses idées et la force de ses convictions.
Rien ne peut ébranler sa détermination à défendre la science, la liberté et l’équité.

[kyutai @kyutai_labs]

Want to contribute to the future of #AI? Our lab is offering #research internships in 2025, providing a unique opportunity to work on groundbreaking projects. Don't miss out 👀! #internship
https://t.co/t47yE4i4Iz

[Gael Varoquaux 🦋 @GaelVaroquaux]

Ici Vinci Autoroute, lobby de la voiture, finance une campagne pour rendre vélo responsable de l'insécurité routière.

Vinci créé ainsi l'animosité pour s'assurer que le vélo ne menace pas ses profits autoroutiers

Rappelons que la voiture tue les piétons, pas le vélo
@LCyclable

[𝙿𝚊𝚝𝚛𝚒𝚌 𝙹𝙴𝙰𝙽 🇺🇦 @patricjean]

Pour tout comprendre en une minute trente.

[William Acker @Rafumab]

Remplacez "gens du voyage" par n'importe quelle autre "communauté".

Antitsiganisme et antisémitisme s'articulent depuis quelques siècles maintenant, mais c'est toujours aussi surprenant que tout le monde s'en fout.

[François Gemenne @Gemenne]

APPEL À TÉMOIGNAGES

Vous travaillez au sein du groupe TF1 et vous avez des informations concernant les intérêts fossiles derrière la campagne de désinformation contre les voitures électriques à laquelle votre chaîne se livre depuis plusieurs semaines : contactez-nous ici.

[Christophe Monnier @CH_Monnier]

Les habitants se mettent au #vélo dès lors qu’ils bénéficient de voies sécurisées +60% de fréquentation à #Lyon en 5 ans 👇 @AMPMetropole à vous de jouer ! @BFMMarseille

[Ludovic Samper @lsamper]

@ClementDelangue strava configuré en miles ? 😮

[Thomas Wolf @Thom_Wolf]

There was a super impressive AI competition that happened last week that many people missed in the noise of AI world. I happen to know several participants so let me tell you a bit of this story as a Sunday morning coffee time.

You probably know the Millennium Prize Problems where the Clay Institute pledged a US$1 million prize for the first correct solution to each of 7 deep math problems. To this date only one of these, the Poincaré conjecture, has been solved by Grigori Perelman who famously declined the award (go check Grigori out if you haven't the guy has a totally based life).

So this new competition, the Artificial Intelligence Math Olympiad (AIMO) also came with a US$1M prize but was only open to AI model (so the human get the price for the work of the AI...). It tackle also very challenging but still simpler problems, namely problems at the International Math Olympiad gold level. Not yet the frontier of math knowledge but definitely above what most people, me included, can solve today.

The organizing committee of the AIMO is kind-of-a who-is-who of highly respected mathematicians in the world, for instance Terence Tao widely famous math prodigy widely regarded as one of the greatest living mathematicians.

Enter our team, Jia Li, Yann Fleuret, and Hélène Evain. After a successful exit in a previous startup (that I happen to have know well when I was an IP lawyer in a previous life but that's for another story) they decided to co-found Numina as a non-profit to do open AI4Math.

Numina wanted to act as a counterpoint to AI math efforts like DeepMind's but in a much more open way with the goal to advance the use of AI in mathematics and make progress on hard, open problems. Along the way, they managed to recruit the help of some very impressive names in the AI+math world like Guillaume Lample, co-founder of Mistral or Stanislas Polu, formerly pushing math models at OpenAI.

As Jia was participating in the code-model BigCode collaboration with some Hugging Face folks, came the idea to collaborate and explore how well code models could be used for formal mathematics.

For context, olympiad math problems are extremely hard and the core of the issue is in the battle plan you draft to tackle each problem. A first focus of Numina was thus on creating high quality instruction Chain-of-Thought (CoT) data for competition-level mathematics. This CoT data has already been used to train models like DeepSeek Math, but is very rarely released so this dataset became an unvaluated ressource to tackle the challenges.

BigCode's lead Leandro put Jia in touch with the team that trained the Zephyr models at Hugging Face, namely, Lewis, Ed, Costa and Kashif with additional help from Roman and Ben and the goal became to have a go at training some strong models on the math and code data to tackle the first progress prize of AIMO.

And the trainings started:

Jia being an olympiad coach, was intimately familiar with the difficulty level of these competitions and able to curate an very strong internal validation set to enable model selection (Kaggle submissions are blind). While iterating on dataset construction, Lewis and Ed from Hugging Face focused on training the models and building the inference pipeline for the Kaggle submissions.

As often in competition it was an intense journey with Eureka and Aha moments pushing everyone further.
Lewis told me about a couple of them which totally blow my mind. A tech report is coming so this is just some "along the way" nuggets that will be soon gathered in a much more comprehensive recipe and report.

Learning to code: The submission of the team relied on self-consistency decoding (aka majority voting) to generate N candidates per problem and pick the most common solution. But initial models trained on the Numina data only scored around 13/50... they needed a better approach. They then saw the MuMath-Code paper (https://t.co/9KGmjGJvT7) which showed you can combine CoT data with code data to get strong models. Jia was able to generate great code execution data from GPT-4 to enable the training of the initial models and get to impressive boost in performance.

Taming the variance: Another Ahah moment came at some point when a Kaggle member shared a notebook showing how DeepSeek models worked super well with code execution (the model breaks down the problem into steps and each step is run in Python to reason about the next one).

However, when the team tried this notebook they found this method had huge variance (the scores on Kaggle varied from 16/50 to 23/50).

When meeting in Paris for a hackathon to improve this issue (like the HF team often does) Ed had the idea to frame the majority voting as a "tree of thoughts" where you'd progressively grow and prune a tree of candidate solutions (https://t.co/dkKtBMrIPm).

This had an impressive impact on the variance and enabled them to be much more confident in their submissions (which showed in how the model ended up performing extremely well on the test set versus the validation set)

Overcoming compute constraints: the Kaggle submissions had to run on 2xT4s in under 9h which is really hard because FA2 doesn't work and you can't use bfloat16 either. The team explored quantization methods like AWQ and GPTQ, finding that 8-bit quantization of a 7B model with GPTQ was best

Looking at the data: a large part of the focus was also on checking the GPT-4 datasets for quality (and fixing them) as they quickly discovered that GPT-4 was prone to hallucinations and failing to correctly interpret the code output. Fixing data issues in the final week led to a significant boost in performance.

Final push: The result were really amazing and the model climbed to the 1 place. And even more, while tying up for first place on the public, validation leaderboard (28 solved challenges versus 27 for the second place), it really shined when tested on the private, test leaderboard where it took a wide margin solving 29 challenges versus 22 for the second team.

As Terence Tao himself set it up, this is "higher than expected"

Maybe what's even more impressive about this competition, beside the level of math these models are already capable of is how ressource contraint the participants were actually, having to run inference in a short amont of time on T4 which only let us imagine how powerful these models will become in the coming months.

Time seem to be ripe for GenAI to have some impact in science and it's probably one of the most exciting thing AI will bring us in the coming 1-2 year. Accelerating human development and tackling all the real world problems science is able to tackle.

[Ludovic Samper @lsamper]

@VeloReparade @Adava_Aix @aixmaville Arf on est repassés en 5ème position avec l'@Adava_Aix

[CIQ Venelles @CIQ_Venelles]

Rdv à #Venelles dimanche matin pour aller à #vélo à la fête du vélo à Aix
https://t.co/7RvQCLBmXc

[Andrew Ng @AndrewYNg]

This week, Google announced a doubling of Gemini Pro 1.5's input context window from 1 million to 2 million tokens, and OpenAI released GPT-4o, which generates tokens 2x faster and 50% cheaper than GPT-4 Turbo and natively accepts and generates multimodal tokens. I view these developments as the latest in an 18-month trend. Given the improvements we've seen, best practices for developers have changed as well.

Since the launch of ChatGPT in Nov 2022, with key milestones that include the releases of GPT-4, Gemini 1.5 Pro, Claude 3 Opus, and Llama 3-70B, many model providers have improved their capabilities in two important ways: (i) reasoning, which allows LLMs to think through complex concepts and and follow complex instructions; and (ii) longer input context windows.

The reasoning capability of GPT-4 and other advanced models makes them quite good at interpreting complex prompts with detailed instructions. Many people are used to dashing off a quick, 1- to 2-sentence query to an LLM. In contrast, when building applications, I see sophisticated teams frequently writing prompts that might be 1 to 2 pages long (my teams call them “mega-prompts”) that provide complex instructions to specify in detail how we’d like an LLM to perform a task. I still see teams not going far enough in terms of writing detailed instructions. For an example of a moderately lengthy prompt, take a look at Claude 3’s system prompt. It’s detailed and gives clear guidance on how Claude should behave.

This is a very different style of prompting than we typically use with LLMs’ web user interfaces, where we might dash off a quick query and, if the response is unsatisfactory, clarify what we want through repeated conversational turns with the chatbot.

Further, the increasing length of input context windows has added another technique to the developer’s toolkit. GPT-3 kicked off a lot of research on few-shot in-context learning. For example, if you’re using an LLM for text classification, you might give a handful — say 1 to 5 examples — of text snippets and their class labels, so that it can use those examples to generalize to additional texts. However, with longer input context windows — GPT-4o accepts 128,000 input tokens, Claude 3 Opus 200,000 tokens, and Gemini 1.5 Pro 1 million tokens (2 million just announced in a limited preview) — LLMs aren’t limited to a handful of examples. With many-shot learning, developers can give dozens, even hundreds of examples in the prompt, and this works better than few-shot learning.

When building complex workflows, I see developers getting good results with this process:
- Write quick, simple prompts and see how it does.
- Based on where the output falls short, flesh out the prompt iteratively. This often leads to a longer, more detailed, prompt, perhaps even a mega-prompt.
- If that’s still insufficient, consider few-shot or many-shot learning (if applicable) or, less frequently, fine-tuning.
- If that still doesn’t yield the results you need, break down the task into subtasks and apply an agentic workflow.

I hope a process like this will help you build applications more easily. If you’re interested in taking a deeper dive into prompting strategies, I recommend Microsoft's Medprompt paper (Nori et al., 2023), which lays out a complex set of prompting strategies that can lead to very good results.

[Original text (with links): https://t.co/UOtLDza1Vh ]

[Ludovic Samper @lsamper]

En plus c'est un événement inclusif: même s'il vous manque une phalange, vous pouvez faire du vélo ;-) (image d'IA generative)

[ADAVA Pays d'Aix @Adava_Aix]

[Roule ta ville] Ce vendredi à Venelles à partir de 16h30, un parcours de motricité en draisienne, un stand de réparation de vélos, un vélo-smoothie et enfin les départs de 2 balades sécurisées à 17h45 https://t.co/qNKWJVpNg7 @CIQ_Venelles @Arnaud_Mercier_
#VenellesEnTransition