Gustavo Führ

PhD, Senior Machine Learning Engineer, Computer Vision Specialist.

[Gustavo Führ @gustfuhr]

I decided to centralize my work in a personal website, which include my posts and projects related to Computer Vision. Check it out at https://t.co/4diDWwvd13

[Pensar a História @historia_pensar]

Há 5 anos, durante a à pandemia de COVID-19, o sistema de saúde de Manaus entrava em colapso. A falta de oxigênio matou dezenas de pessoas — e a tragédia teria sido ainda maior não fosse a ajuda prestada pela Venezuela. Leia mais no @operamundi

1/20

https://t.co/p4hGQXz8FD

[Gustavo Führ @gustfuhr]

@skalskip92 @onuralpszr Currently working on that, will also check performance in ort and tensorrt

[Double Down News @DoubleDownNews]

Everything That's Wrong with Liberal Centrism in Under 3 mins

As told by the late great David Graeber

[Gustavo Führ @gustfuhr]

@ai_for_success Are you using LLMs as a news source? This site is getting worse by the minute

[Gustavo Führ @gustfuhr]

@ogremiodetodos Não é possível divulgar o relatório aqui?

[Gustavo Führ @gustfuhr]

I'm doing some experiments using Large Language Models (LLMs) I wanted to compare the different models available. So I came up with a very simple wrapper for text generation:

https://t.co/BmdhlYbbYQ

It supports GPT4, Claude Sonnet, Llama (running locally), and Google Gemini.

[Gustavo Führ @gustfuhr]

@ocornut @guilhermefickel

[Gustavo Führ @gustfuhr]

@edujedii Kkkk não tinha visto

[Andrej Karpathy @karpathy]

# RLHF is just barely RL

Reinforcement Learning from Human Feedback (RLHF) is the third (and last) major stage of training an LLM, after pretraining and supervised finetuning (SFT). My rant on RLHF is that it is just barely RL, in a way that I think is not too widely appreciated. RL is powerful. RLHF is not. Let's take a look at the example of AlphaGo. AlphaGo was trained with actual RL. The computer played games of Go and trained on rollouts that maximized the reward function (winning the game), eventually surpassing the best human players at Go. AlphaGo was not trained with RLHF. If it were, it would not have worked nearly as well.

What would it look like to train AlphaGo with RLHF? Well first, you'd give human labelers two board states from Go, and ask them which one they like better:

Then you'd collect say 100,000 comparisons like this, and you'd train a "Reward Model" (RM) neural network to imitate this human "vibe check" of the board state. You'd train it to agree with the human judgement on average. Once we have a Reward Model vibe check, you run RL with respect to it, learning to play the moves that lead to good vibes. Clearly, this would not have led anywhere too interesting in Go. There are two fundamental, separate reasons for this:

1. The vibes could be misleading - this is not the actual reward (winning the game). This is a crappy proxy objective. But much worse,
2. You'd find that your RL optimization goes off rails as it quickly discovers board states that are adversarial examples to the Reward Model. Remember the RM is a massive neural net with billions of parameters imitating the vibe. There are board states are "out of distribution" to its training data, which are not actually good states, yet by chance they get a very high reward from the RM.

For the exact same reasons, sometimes I'm a bit surprised RLHF works for LLMs at all. The RM we train for LLMs is just a vibe check in the exact same way. It gives high scores to the kinds of assistant responses that human raters statistically seem to like. It's not the "actual" objective of correctly solving problems, it's a proxy objective of what looks good to humans. Second, you can't even run RLHF for too long because your model quickly learns to respond in ways that game the reward model. These predictions can look really weird, e.g. you'll see that your LLM Assistant starts to respond with something non-sensical like "The the the the the the" to many prompts. Which looks ridiculous to you but then you look at the RM vibe check and see that for some reason the RM thinks these look excellent. Your LLM found an adversarial example. It's out of domain w.r.t. the RM's training data, in an undefined territory. Yes you can mitigate this by repeatedly adding these specific examples into the training set, but you'll find other adversarial examples next time around. For this reason, you can't even run RLHF for too many steps of optimization. You do a few hundred/thousand steps and then you have to call it because your optimization will start to game the RM. This is not RL like AlphaGo was.

And yet, RLHF is a net helpful step of building an LLM Assistant. I think there's a few subtle reasons but my favorite one to point to is that through it, the LLM Assistant benefits from the generator-discriminator gap. That is, for many problem types, it is a significantly easier task for a human labeler to select the best of few candidate answers, instead of writing the ideal answer from scratch. A good example is a prompt like "Generate a poem about paperclips" or something like that. An average human labeler will struggle to write a good poem from scratch as an SFT example, but they could select a good looking poem given a few candidates. So RLHF is a kind of way to benefit from this gap of "easiness" of human supervision. There's a few other reasons, e.g. RLHF is also helpful in mitigating hallucinations because if the RM is a strong enough model to catch the LLM making stuff up during training, it can learn to penalize this with a low reward, teaching the model an aversion to risking factual knowledge when it's not sure. But a satisfying treatment of hallucinations and their mitigations is a whole different post so I digress. All to say that RLHF *is* net useful, but it's not RL.

No production-grade *actual* RL on an LLM has so far been convincingly achieved and demonstrated in an open domain, at scale. And intuitively, this is because getting actual rewards (i.e. the equivalent of win the game) is really difficult in the open-ended problem solving tasks. It's all fun and games in a closed, game-like environment like Go where the dynamics are constrained and the reward function is cheap to evaluate and impossible to game. But how do you give an objective reward for summarizing an article? Or answering a slightly ambiguous question about some pip install issue? Or telling a joke? Or re-writing some Java code to Python? Going towards this is not in principle impossible but it's also not trivial and it requires some creative thinking. But whoever convincingly cracks this problem will be able to run actual RL. The kind of RL that led to AlphaGo beating humans in Go. Except this LLM would have a real shot of beating humans in open-domain problem solving.

[Gustavo Führ @gustfuhr]

I just made three random scripts:

1. One to annotate single objects using SAM.

2. One that takes all arXiv PDFs from a folder and rename them to the papers titles.

3. One that visits the Real Madrid website to check when tickets are first available.

https://t.co/qNDOkE4Igm

[Gustavo Führ @gustfuhr]

@skalskip92 Hello, I follow your work here, it's quite impressive. I'm trying to do a comprehensive review about CV in sports. Do you have any particular papers that you would recommend? Thanks!

[Análise Política (início) @Analise2023]

🚨 GRAVE: imprensa mente novamente, associa alta do dólar a Lula e ignora possível manipulação da moeda. Entenda:

Nos últimos 30 dias, o dólar aumentou 8%, saindo de 5,25 para 5,66. Segundo a imprensa, as críticas de Lula ao BC, feitas desde janeiro de 2023, são o motivo. (+)

[Atlético @Atletico]

Ingressos vendidos: 36.871
Valor arrecadado: R$ 666.090,00

O Galo e a @MRVoficial agradecem a Massa por essa fantástica demonstração de solidariedade!

#FutebolPeloSul | #SolidariedadeEmMassa 🏴🏳️

[François Chollet @fchollet]

This is something that people often misunderstand about early-stage autocracy. The most immediate consequence of an autocrat reaching power isn't a police state and gulags as per popular imagination -- that's more of a long-term thing. The immediate consequence is corruption.

The autocrat, his family and his allies will pillage the coffers, first thing. All appointments will be via loyalty or family ties (nepotism) and all appointees will have self-enrichment as their primary goal. Political power and even foreign policy will be wielded to serve personal interests exclusively. The rule of law will break down -- first and foremost to shield the autocrat's camp from any consequences.

[Andrew Ng @AndrewYNg]

I'd like to respectfully point out that the logic in this argument is based on a flawed model for how scientists think. Scientists don't just take a weighted average of others' opinions to form their own.

A good scientist takes as input lots of data, including others' opinions, and then ultimately has to reason, build their own internal model of the world, and draw their conclusions from that model.

I give your opinion a lot of weight. And, after having heard many opinions including yours, my internal model tells me that there is essentially no risk of AI human extinction.

So I don't follow the logic that because @ylecun or anyone else disagrees with you (and other AI extinctionists, which IMO are in the minority) that they gave your opinion very little weight, unless we think scientists arrive at conclusions by taking a weighted average of what everyone else thinks.

Since you've gone against the grain many times in your own career -- often brilliantly so -- I assume you're also familiar with what it feels like to give someone's opinion weight, but then ultimately to draw a different conclusion!

[Bindu Reddy @bindureddy]

Prediction: We are soon going to prove that LLMs will reach a ceiling when they match human level intelligence and search engine level retrieval.

There will be no where else to go, after that....

THAT'S IT!

That's all AI can ever be capable of and there is no chance of AI becoming "super intelligent"or whatever other doomsday scenario some folks are imagining...

No chance it will destroy humanity.

AI is going to do the heavy lifting for us... and we humans will supervise. This is the most logical conclusion given our understanding of LLMs.

The burden of proof should be on the AI doomers.

There is no proof that some "super intelligence" exists or can exist in the known universe.

[Leandro Demori @demori]

O @HananyaNaftali, "digital influencer" do governo Netanyahu, postou que Israel tinha atacado um hospital em Gaza pra matar terroristas. Depois apagou o post e começou a dizer que foi o Hamas.

A ONU confirmou 1h atrás que Israel pediu a evacuação do hospital, o que era impossível de se fazer, segundo a ONU.

Neste caso, como se vê, nenhuma fonte é do Hamas. Uma é a ONU, a outra é o influencer Naftali, que trabalha diretamente pela propaganda digital de Netanyahu.