gily

Fighting my attention span. One day at a time.

[Andrej Karpathy @karpathy]

# on shortification of "learning"

There are a lot of videos on YouTube/TikTok etc. that give the appearance of education, but if you look closely they are really just entertainment. This is very convenient for everyone involved : the people watching enjoy thinking they are learning (but actually they are just having fun). The people creating this content also enjoy it because fun has a much larger audience, fame and revenue. But as far as learning goes, this is a trap. This content is an epsilon away from watching the Bachelorette. It's like snacking on those "Garden Veggie Straws", which feel like you're eating healthy vegetables until you look at the ingredients.

Learning is not supposed to be fun. It doesn't have to be actively not fun either, but the primary feeling should be that of effort. It should look a lot less like that "10 minute full body" workout from your local digital media creator and a lot more like a serious session at the gym. You want the mental equivalent of sweating. It's not that the quickie doesn't do anything, it's just that it is wildly suboptimal if you actually care to learn.

I find it helpful to explicitly declare your intent up front as a sharp, binary variable in your mind. If you are consuming content: are you trying to be entertained or are you trying to learn? And if you are creating content: are you trying to entertain or are you trying to teach? You'll go down a different path in each case. Attempts to seek the stuff in between actually clamp to zero.

So for those who actually want to learn. Unless you are trying to learn something narrow and specific, close those tabs with quick blog posts. Close those tabs of "Learn XYZ in 10 minutes". Consider the opportunity cost of snacking and seek the meal - the textbooks, docs, papers, manuals, longform. Allocate a 4 hour window. Don't just read, take notes, re-read, re-phrase, process, manipulate, learn.

And for those actually trying to educate, please consider writing/recording longform, designed for someone to get "sweaty", especially in today's era of quantity over quality. Give someone a real workout. This is what I aspire to in my own educational work too. My audience will decrease. The ones that remain might not even like it. But at least we'll learn something.

[gily @meanymaxx]

Thinking of starting the Hugging Face Deep RL course in parallel with working through Sutton & Barto. I think it'll provide a good balance - building the mathematical intuition while getting hands-on experience from the very start.

[gily @meanymaxx]

I've wanted to study RL from scratch for a long time. Glad I finally took the first step and got past the impostor syndrome. Excited for all the math ahead!
Looking forward to connecting with more people in the community.

[gily @meanymaxx]

Finished Chapter 1 of Sutton & Barto's RL book.

Brushed up my understanding of the core components of a reinforcement learning system.

In short: the agent observes a state, chooses an action, and the environment responds with a reward and a new state. Then the cycle repeats.

The setup has five key pieces:

• Policy — the agent's behaviour: what action it takes in each state. This is what gets learned.

• Reward — a single number from the environment indicating how good the last action was.

• Value — how good a state is in the long run: the expected cumulative reward from that point onward. The agent has to estimate this; it's never given directly.

• Model (optional) — predicts the next state and reward before acting, enabling planning.

• State — what the agent perceives about the environment at each step.

The interaction loop is straightforward.

The agent observes a state, the policy selects an action, and the value function helps evaluate how promising different choices are. If the agent has a model, it can predict future states and rewards to plan ahead. If it's model-free, it simply acts and learns from experience.

The reward then feeds back into the policy, nudging it toward actions that lead to better long-term outcomes, while the environment returns a new state and the cycle continues.

[gily @meanymaxx]

The next chapters are where the math starts. Gonna be fun.

[gily @meanymaxx]

Finished Chapter 1 of Sutton & Barto's RL book.

Brushed up my understanding of the core components of a reinforcement learning system.

In short: the agent observes a state, chooses an action, and the environment responds with a reward and a new state. Then the cycle repeats.

The setup has five key pieces:

• Policy — the agent's behaviour: what action it takes in each state. This is what gets learned.

• Reward — a single number from the environment indicating how good the last action was.

• Value — how good a state is in the long run: the expected cumulative reward from that point onward. The agent has to estimate this; it's never given directly.

• Model (optional) — predicts the next state and reward before acting, enabling planning.

• State — what the agent perceives about the environment at each step.

The interaction loop is straightforward.

The agent observes a state, the policy selects an action, and the value function helps evaluate how promising different choices are. If the agent has a model, it can predict future states and rewards to plan ahead. If it's model-free, it simply acts and learns from experience.

The reward then feeds back into the policy, nudging it toward actions that lead to better long-term outcomes, while the environment returns a new state and the cycle continues.

[gily @meanymaxx]

One structural point that's easy to miss: value-based vs. evolutionary is a different question from model-free vs. model-based. They're two independent axes, so all four combinations exist. The tic-tac-toe player, for instance, is value-based and model-free—but we could also be value-based and model-based (planning with a learned model), or evolutionary either way.

The idea that stuck with me the most: a value is always the value of a state under some particular policy, against some particular environment (in tic-tac-toe, that environment is your opponent). It's never an intrinsic property of the state itself. What a position is "worth" depends entirely on the context that determines what happens next. Change our policy or our opponent, and the exact same state can be worth something completely different.

[gily @meanymaxx]

Finished Chapter 1 of Sutton & Barto's RL book.

Brushed up my understanding of the core components of a reinforcement learning system.

In short: the agent observes a state, chooses an action, and the environment responds with a reward and a new state. Then the cycle repeats.

The setup has five key pieces:

• Policy — the agent's behaviour: what action it takes in each state. This is what gets learned.

• Reward — a single number from the environment indicating how good the last action was.

• Value — how good a state is in the long run: the expected cumulative reward from that point onward. The agent has to estimate this; it's never given directly.

• Model (optional) — predicts the next state and reward before acting, enabling planning.

• State — what the agent perceives about the environment at each step.

The interaction loop is straightforward.

The agent observes a state, the policy selects an action, and the value function helps evaluate how promising different choices are. If the agent has a model, it can predict future states and rewards to plan ahead. If it's model-free, it simply acts and learns from experience.

The reward then feeds back into the policy, nudging it toward actions that lead to better long-term outcomes, while the environment returns a new state and the cycle continues.

[gily @meanymaxx]

Estimating value isn't the only way to do RL.

Evolutionary methods skip value entirely. We maintain a whole population of fixed policies, run each one through full episodes, keep the ones with the best total score, then breed and mutate copies to form the next generation. Improvement happens between generations, not within a single run—a policy never learns mid-game; it just gets scored and either survives or doesn't.

The tradeoff is real and goes both ways. Value learning extracts information from every single step, so it's more sample-efficient, but it leans on having a good Markov state—which is why aliasing breaks it. Evolution only needs one thing: a total score per policy, which is always well-defined no matter how partial the agent's perception is. So it's robust exactly where value methods struggle—but because it throws away all the step-by-step structure and only looks at final scores, it needs much more data to get there.

[gily @meanymaxx]

The sole purpose of creating this account is to stay locked in and share what I've learned every day.

The goal is to become more informed, think more clearly, and preserve the dwindling attention span.

My primary focus is on AI (want to do deep dive into RLHF), but I also want to deepen my understanding of finance and expand my general knowledge along the way.