Olivia
Online
shaan☄️
@MehtaDontStop
consistency + iteration
New York, NY
Joined November 2014
327 Following
235 Followers
861 Posts
Pinned Tweet
Day 117 diving into ML. Read the OG DQN paper today and exploring RL applications in various industries. Cool ones:
Nuclear fusion: https://t.co/2L6u42xdL9
Ride matching at Lyft: https://t.co/Yfn8ZPQFtB
Cooling data centers: https://t.co/5XJV5uDhYt
Day 116 diving into ML. Read the NetHack Learning Environment (NLE) paper. Key takeaway:
to get generalizing agents, create envs which are procedural + stochastic worlds and build memory + structured encoders into the policy from the start. also, use unseen seeds during eval (seeds reserved for eval)
to expand on structured encoders from above, don't flatten the entire obs array, encode each modality into the right format like map data into a CNN, text based data into a language encoder, etc. feeding these into your policy allows it to learn richer representations like a human would, rather then sending a single giant vector
I built an RL environment for Flappy Bird and trained an agent that beats the game with 100% accuracy
@AnkitaDuri Thank you Ankita!
Day 118 diving into ML. Read the ReAct paper for agent building and had a long LLM session trying to understand how RL plays into the LLM landscape.
I'm at the point where applying what I've learned through a live product / problem is going to be what gives me the next learning leap. A project isn't going to suffice - needs to be something with economic stakes. The forced constraints will lead me to run into issues which I haven't premeditated, and I'll see the bigger picture.
At this point, the I understand the fundamentals very well and it almost feels suboptimal / small picture / wasting my time to learn more algorithm optimizations. I feel like I have the world's sharpest sword and I'm in search of a worthy beast to slay with it.
My brain thinks in systems and paradigms and I want to work on engineering problems at that scale.
Day 117 diving into ML. Read the OG DQN paper today and exploring RL applications in various industries. Cool ones:
Nuclear fusion: https://t.co/2L6u42xdL9
Ride matching at Lyft: https://t.co/Yfn8ZPQFtB
Cooling data centers: https://t.co/5XJV5uDhYt
Day 116 diving into ML. Read the NetHack Learning Environment (NLE) paper. Key takeaway:
to get generalizing agents, create envs which are procedural + stochastic worlds and build memory + structured encoders into the policy from the start. also, use unseen seeds during eval (seeds reserved for eval)
to expand on structured encoders from above, don't flatten the entire obs array, encode each modality into the right format like map data into a CNN, text based data into a language encoder, etc. feeding these into your policy allows it to learn richer representations like a human would, rather then sending a single giant vector
Day 115 diving into ML. Read DeepMind’s 2018 Quake III CTF paper. Notes:
- Population based training: they trained a league of agents at once instead of one at time
- Self-play with match making: generate an Elo score and match make between similar caliber agents so curriculum auto scales
- Reward shaping for sparse wins: instead of only win/lose at the end, they learn a dense reward from in-game events (flags, pickups, progress) to make credit assignment tractable
- Generalization via variation: procedurally generated maps + changing teammates/opponents forces transferable skills (this theme holds from XLand)
Starting to see a similar recipe between these multi agent papers. Varied worlds + auto curricula baked into training via self play or skill based match making + learnable reward shaping
Day 115 diving into ML. Read DeepMind’s 2018 Quake III CTF paper. Notes:
- Population based training: they trained a league of agents at once instead of one at time
- Self-play with match making: generate an Elo score and match make between similar caliber agents so curriculum auto scales
- Reward shaping for sparse wins: instead of only win/lose at the end, they learn a dense reward from in-game events (flags, pickups, progress) to make credit assignment tractable
- Generalization via variation: procedurally generated maps + changing teammates/opponents forces transferable skills (this theme holds from XLand)
Starting to see a similar recipe between these multi agent papers. Varied worlds + auto curricula baked into training via self play or skill based match making + learnable reward shaping
Day 114 diving into ML. Read OpenAI’s Hide-and-Seek paper (“Emergent Tool Use from Multi-Agent Autocurricula”). Notes:
- Key takeaway: to create emergent complexity, hand authoring tasks is not the move, instead set up incentives + rich physics playground, and let agents generate curriculum for you
- Pure competition created an autocurriculum: each side keeps inventing harder problems for the other, so difficulty self-scales
- They didn't specify a reward for tool use , but tool use still emerged bc it was the shortest path to victory
- Team rewards produced coordination + division of labor (agents specialize: builder, blocker, distractor)
- Intrinsically motivated baselines (ie reward for exploration are not as effective as auto curriculum via competition (agent is forced to learn / explore new strategies to beat opponent)
Lots of good, practical ideas in this paper regarding environment design as well. Worth the read.
Day 114 diving into ML. Read OpenAI’s Hide-and-Seek paper (“Emergent Tool Use from Multi-Agent Autocurricula”). Notes:
- Key takeaway: to create emergent complexity, hand authoring tasks is not the move, instead set up incentives + rich physics playground, and let agents generate curriculum for you
- Pure competition created an autocurriculum: each side keeps inventing harder problems for the other, so difficulty self-scales
- They didn't specify a reward for tool use , but tool use still emerged bc it was the shortest path to victory
- Team rewards produced coordination + division of labor (agents specialize: builder, blocker, distractor)
- Intrinsically motivated baselines (ie reward for exploration are not as effective as auto curriculum via competition (agent is forced to learn / explore new strategies to beat opponent)
Lots of good, practical ideas in this paper regarding environment design as well. Worth the read.
Day 113 diving into ML. Read DeepMind’s XLand paper (“Open-Ended Learning Leads to Generally Capable Agents”). Key notes:
- “General capability” came from an open-ended training loop that keeps making new challenges
- The curriculum was auto-adjusting via constant feedback so that difficulty was always just right / training didn't stall
- Multi-agent dynamics (coop + comp) act like a robustness regularizer
- XLand is a procedurally-generated 3D task universe (not a fixed task list), so skills have to transfer across endless variations
- Though, the policy adapts across variations within Xland, it doesn't necessarily transfer to all games (obvious) but worth noting
- Cheap fine tuning for new games on this generalizing policy
- World generator was the key here vs having a static environment, they could create endless scenarios
Day 113 diving into ML. Read DeepMind’s XLand paper (“Open-Ended Learning Leads to Generally Capable Agents”). Key notes:
- “General capability” came from an open-ended training loop that keeps making new challenges
- The curriculum was auto-adjusting via constant feedback so that difficulty was always just right / training didn't stall
- Multi-agent dynamics (coop + comp) act like a robustness regularizer
- XLand is a procedurally-generated 3D task universe (not a fixed task list), so skills have to transfer across endless variations
- Though, the policy adapts across variations within Xland, it doesn't necessarily transfer to all games (obvious) but worth noting
- Cheap fine tuning for new games on this generalizing policy
- World generator was the key here vs having a static environment, they could create endless scenarios
Day 112 diving into ML. Read the 2018 OpenAI "Learning Dexterous In-Hand Manipulation" paper. Key notes:
- They trained a dexterous hand control policy entire in sim using PPO, then deployed it to the real robot (no irl demos)
- Domain randomization across their sims was the real unlock
- Randomization helped bridge gap between real world and sim, as there are many more variances IRL than could be modeled. Randomization helped capture some of that noise / variability
- LSTM policy matters because randomizations persist across an episode
- Main takeaway: for sim-to-real, creating a perfect simulator might be the wrong goal, instead train on distribution of worlds and memory enabled policies so robustness emerges by design
https://t.co/bHLKtOSoXE
Day 112 diving into ML. Read the 2018 OpenAI "Learning Dexterous In-Hand Manipulation" paper. Key notes:
- They trained a dexterous hand control policy entire in sim using PPO, then deployed it to the real robot (no irl demos)
- Domain randomization across their sims was the real unlock
- Randomization helped bridge gap between real world and sim, as there are many more variances IRL than could be modeled. Randomization helped capture some of that noise / variability
- LSTM policy matters because randomizations persist across an episode
- Main takeaway: for sim-to-real, creating a perfect simulator might be the wrong goal, instead train on distribution of worlds and memory enabled policies so robustness emerges by design
Day 111 diving into ML. Read the OG AlphaGo paper. Key notes:
- They used a combo of SL + RL + MCTS
- Policy net was good for approximating best moves, but value net helped with long-horizon position evaluation (less reliance on rollouts)
- MCTS caches edge stats (N, Q) and uses policy priors to focus search
- Essentially NNs give learned approximations, but search allows deeper strategic evaluation and decision making over longer time horizons (ie this move is less obvious right now but its tactically advantageous later)
- Main takeaway: learn heuristics, then wrap them in search so we get consequence-aware decisions (not just “best-looking” moves)
Day 111 diving into ML. Read the OG AlphaGo paper. Key notes:
- They used a combo of SL + RL + MCTS
- Policy net was good for approximating best moves, but value net helped with long-horizon position evaluation (less reliance on rollouts)
- MCTS caches edge stats (N, Q) and uses policy priors to focus search
- Essentially NNs give learned approximations, but search allows deeper strategic evaluation and decision making over longer time horizons (ie this move is less obvious right now but its tactically advantageous later)
- Main takeaway: learn heuristics, then wrap them in search so we get consequence-aware decisions (not just “best-looking” moves)
Day 110 diving into ML. Exploring CUDA. Hunting for the next summit to climb.
Day 110 diving into ML. Exploring CUDA. Hunting for the next summit to climb.
Day 109 diving into machine learning. Deciding where to go from here. Some options:
1. Design another RL env that's more complex than Flappy
2. Maybe do a paper / algo implementation. PPO?
3. Take crash course on CUDA / GPUs ?
4. Something else
Spent time exploring potential envs to build and read these two today:
Puffing Up PPO: https://t.co/nIompViFTv
NeuralMMO: https://t.co/B5zUsJbYzl
@iAnonymous3000 @near_ai @ilblackdragon Thank you, I have been looking for secure alternatives. Couldn't trust OpenClaw with internet access while there was a 30% chance it'll leak my API keys on the first web search
Day 109 diving into machine learning. Deciding where to go from here. Some options:
1. Design another RL env that's more complex than Flappy
2. Maybe do a paper / algo implementation. PPO?
3. Take crash course on CUDA / GPUs ?
4. Something else
Spent time exploring potential envs to build and read these two today:
Puffing Up PPO: https://t.co/nIompViFTv
NeuralMMO: https://t.co/B5zUsJbYzl
Day 108 diving into machine learning. Realized a massive oversight in Flappy RL and fixed it (didn't have an LSTM 😂). Completed my Flappy blog post. Read four papers / blogs and had long LLM session about PPO / GAE. Here's what I read:
PPO paper - https://t.co/dtaZp19gWA
PPO implementation blog post - https://t.co/4etoWy0HAH
GAE (skimmed) - https://t.co/wPzvXKUGDj
PufferLib Sweep Algo Blog Post - https://t.co/sx3HbVFS0m
Still wrapping my head around some of the finer math details, but getting a solid intuition on how the algorithms work. PPO is deceptively simple.
Day 108 diving into machine learning. Realized a massive oversight in Flappy RL and fixed it (didn't have an LSTM 😂). Completed my Flappy blog post. Read four papers / blogs and had long LLM session about PPO / GAE. Here's what I read:
PPO paper - https://t.co/dtaZp19gWA
PPO implementation blog post - https://t.co/4etoWy0HAH
GAE (skimmed) - https://t.co/wPzvXKUGDj
PufferLib Sweep Algo Blog Post - https://t.co/sx3HbVFS0m
Still wrapping my head around some of the finer math details, but getting a solid intuition on how the algorithms work. PPO is deceptively simple.
Day 107 diving into machine learning. Consolidation day. Writing about Flappy RL. Read a couple of blog posts / papers.
Day 107 diving into machine learning. Consolidation day. Writing about Flappy RL. Read a couple of blog posts / papers.
Day 106 diving into machine learning. I'm marking the Flappy Bird RL env and agent solved. Notable takeaways:
1. Solidifying curriculum into a single, repeatable training loop.
2. Switching from discrete phases in curriculum to a continuous ramp up of difficulty.
3. Holding the env steady and easy at beginning of training so the agent doesn't start with a moving target vs continuous ramp up from beginning
4. Simplifying reward design from 5 components to just +1 for passing pipe and -1 for dying.
5. Simplifying observations to just the base, objective data points (bird pos, dist from pipe, gap midpoint, gap height).
6. LR decay so we don't unlearn things as training goes on.
These changes almost doubled performance of the agent. Simplicity is king in RL. Some things almost feel like philosophy (how would I teach a baby to do something).
I'm putting together a blog post that covers my build process and takeaways. Stay tuned.
Last Seen Users on Sotwe
Most Popular Users
Elon Musk
@elonmusk
241M followers
Barack Obama
@barackobama
119.2M followers
Cristiano Ronaldo
@cristiano
112M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98M followers
NASA
@nasa
92.2M followers
Justin Bieber
@justinbieber
91.2M followers
KATY PERRY
@katyperry
88.4M followers
Taylor Swift
@taylorswift13
82.3M followers
Lady Gaga
@ladygaga
73.8M followers
Virat Kohli
@imvkohli
71.1M followers
Kim Kardashian
@kimkardashian
70.2M followers
YouTube
@youtube
68.7M followers
Bill Gates
@billgates
64.3M followers
Neymar Jr
@neymarjr
64M followers
The Ellen Show
@theellenshow
62.4M followers
CNN
@cnn
61.8M followers
Selena Gomez
@selenagomez
61.5M followers
X
@x
60.8M followers