Agentic Balloon

AI Agentic Engineer
Football Enjoyer

[Agentic Balloon @agenticballoon]

@sace0281 Just realized today is Wednesday 😂

[Agentic Balloon @agenticballoon]

@colombiarmcf Imagínate ser marrón y ser del Madrid

[Tom Yeh @ProfTomYeh]

RLHF by hand ✍️ ~ 15 steps walkthrough below

Train a model on human text and it inherits human bias. It will assume a doctor is a "him", because the data says so.

RLHF is the correction. A human marks one preference, doc is them over doc is him, and the weights move.

But one correction is not the point. The hope is that the model learns the value behind it, gender neutrality, and applies it to professions nobody ever mentioned.

How does it work?

Goal: train a reward model from a single human comparison about doctors, then turn it on CEOs, filling in every cell yourself.

= 1. Given =

A reward model, an LLM, and two (prompt, next) pairs.

= 2. Preferences =

A human reads both pairs and picks a winner: (doc is, them) beats (doc is, him). The loser is not bad grammar, it is gender bias, and that is the whole signal.

= 3. Word embeddings =

Let us look up each word of the loser pair. These vectors are the reward model's input.

= 4. Linear layer =

We multiply by the reward model's weights and add its biases. Out come feature vectors, one per position.

= 5. Mean pool =

Let us multiply by [1/3, 1/3, 1/3], which averages the three positions into one sentence embedding.

= 6. Output layer =

We map that sentence down to a single number. Reward = 3.

= 7. The winner, the same way =

Let us repeat steps 3 to 6 on the winning pair. Reward = 5.

= 8. Winner minus loser =

We take the gap: 5 - 3 = 2. The reward model wants this positive and as large as it can make it.

= 9. Loss gradient =

Let us squash the gap into a probability, σ(2) ≈ 0.9, and subtract the target of 1. The gradient is -0.1, and it goes back through the purple weights. The reward model is now trained.

= 10. A prompt it has never seen =

We start the second half with "[S] CEO is". The feedback in step 2 was about doctors. Nothing connects a CEO to a doctor except what the reward model generalised.

= 11. Transformer =

Let us push it through attention and a feed forward layer, one vector per position.

= 12. Output probabilities =

We map each vector to a score over the vocabulary.

= 13. Sample =

Let us take the highest score. The model completes "CEO is" with "him", which is the same bias the human penalised in step 2.

= 14. Score it with the reward model =

We feed the new pair (CEO is, him) through steps 3 to 6. Reward = 3, exactly the score it gave "doc is him" in step 6. Nobody taught it about CEOs. The value transferred.

= 15. Loss gradient =

Let us set the loss to the negative of the reward, so minimising the loss maximises the reward. The gradient is a constant -1, and it goes back through the red weights.

The outputs:
Loser reward = 3, winner reward = 5
Reward gap = 2, predicted σ ≈ 0.9, reward model gradient = -0.1
LLM samples "him", reward = 3, LLM gradient = -1

Congrats! You just calculated RLHF by hand.

And you watched a value generalise: one comparison about doctors, and the model marks down "CEO is him" unprompted.

💾 Save this post!

[Agentic Balloon @agenticballoon]

@Lofcfm It's IP-locked, only people from the event can access

[Agentic Balloon @agenticballoon]

@ct_efootball They might have locked the access to only allow people from a specific IP in Thailand to enter the page

[Agentic Balloon @agenticballoon]

@Mercado_Ingles Kante

[Agentic Balloon @agenticballoon]

@ESPNDeportes De vuelta a aguantar a estos mexicanos madridistas de mierda

[Karun Singh @karun1710]

We're hiring a Research Engineer at @Arsenal ⚽🔴⚪ to work directly with our Men's First Team!

We're building state-of-the-art AI models for the football domain. This role will focus on building the application layer for our research to advance coaching and analysis workflows.

[Agentic Balloon @agenticballoon]

@potrero_app Mantengan

[Agentic Balloon @agenticballoon]

@alexthecabo Y sí, debió haber sido expulsado se dice y no pasa nada, paredes también mereció haber sido expulsado antes del 120, aquí nadie ha robado a nadie, lo que pasa es que los árbitros son todos unos retrasados mentales, con unos se aplica la norma y con otros no

[Agentic Balloon @agenticballoon]

@A_SuarezDoreski @thenumber0neboy No agus no, no les dejes ganar, recoge cable porfavor, eres un gran profesional, borra el twitt hermano.

[Agentic Balloon @agenticballoon]

@yosoyemeve La debilidad de España es la misma que la del Barcelona, la línea adelantada, Julián ya mató al barça aprovechándose de esa debilidad

[max fu @maxxfuu]

Day 13/90 of Inference Engineering

I built a MNIST classifier in C.

Not proud of it, man enough to say it. It took me 5 days to fully understand and get the intuition behind a matmul with 1D vectors.

It’s just 3 nested for-loops, and somehow, it just didn’t click. It was not intuitive at all. Here are some things I wish I knew before starting.

Practically speaking, I think sometimes drawing things out, creating different visualizations, and talking to language models to build that understanding can help out a lot, if done correctly.

I spent around 6 hours just tracing the code, drawing the data flow, predicting the shapes, and understanding the math. None of which required any coding. Once I fully understood the flow of MNIST, writing the code became blazingly fast. Implementing MNIST in C took about 1 hour, understanding everything took 6 hours.

Mindset-wise, I think learning about Deep Learning truly is a different game. Coming from a fullstack engineering perspective, there's almost a playbook on building robust backends and building tasteful frontends. I’ve found that diving straight into things and figuring it out along the way teaches you more than reading the documentation.

However, learning Inference, CUDA, and building a simple project like MNIST just isn’t the same as fullstack engineering. There's this big hurdle of understanding why something works and building an intuition of things. Moving forward, I’m definitely spending my tokens on understanding the fundamentals instead of coding.

On the technical side, here's the intuition that made the triple for-loops implementation of the matmul_forward() click! Since the matrices are in the form of a 1D vector, selecting the right cell is just figuring out which of the three indexing variables corresponds to the step, stride, and pin.

This isn't formal terminology, but it just makes sense.

The step is the axis you're moving on. It could span row-wise or column-wise. Which way you span is determined by the pin. The stride is how big of a jump you make on the 1D array with each step.

Take a look at B[l*k + j] that's in the animation. The pin is j which pins you to one column, so the only free variable is the step, l. The iteration flows top to bottom down that column. Since B lives in a 1D array, "down one row" isn't one cell over, it's a full stride or row width of k.

While which variable plays the stride, pin, and step depends entirely on how the for-loops are structured, I've been able to generalize variations of matmuls using this framework.

Starting tomorrow, I'll be able to rewrite this C code in CUDA!

[Agentic Balloon @agenticballoon]

@yosoyemeve MV debe ser el youtuber más buena honda, que tipazo

[Harshit Khosla @Harry_The_Nerd]

Publishing this soon (gonna be an epic series ✨️)

[Ding @dingyi]

tldraw 竟然发布了一个离线桌面版。

https://t.co/H5wReePFXT

[0xSero @0xSero]

One of my favorite resources for understanding llm concepts
https://t.co/tMSEP7SVMV

[Agentic Balloon @agenticballoon]

@TouchmineX Who is him anyways? 😂

[Tom Yeh @ProfTomYeh]

Transformer by hand ✍️ ~ 6 steps walkthrough below

Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking.

Which of those actually make the car run?

Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model.

So I drew and calculated those two parts entirely by hand.

Goal: push five features through one transformer block, filling in every cell yourself.

1. Given

Five positions of input features, arriving from the previous block.

2. Attention matrix

Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own.

3. Attention weighting

We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on.

4. First layer

Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4.

Note that every position goes through the same weight matrix. That is what "position-wise" means.

5. ReLU

We cross out the negatives. They become zeros.

6. Second layer

Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again.

You have just calculated a transformer block by hand. ✍️

The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run.

💾 Save this post!

#AIbyHand #Transformers #DeepLearning

[tldraw @tldraw]

We're training neural nets in tldraw offline