Jesse Roberts

Life-long Tennesseean researching Artificial Intelligence, Language, and Robotics.

Phd @ Vanderbilt 2024

Assistant Professor @ Tennessee Tech University

[Rohan Paul @rohanpaul_ai]

This paper introduces In-Context Reinforcement Learning (ICRL). It demonstrates that a transformer LLM, when trained with RL, can learn to solve new problems it has never seen before, by leveraging in-context experience.

------

📌 ICRL Enables On-the-Fly Adaptation
Traditional RL needs extensive retraining for new tasks. ICRL bypasses this by leveraging in-context learning. The transformer dynamically updates Q-values within an episode using prior interactions. No weight updates, just context-based adjustments. This mimics meta-learning but without explicit fine-tuning. This is critical for real-world applications where rapid adaptation is needed.

📌 Behavior Stitching Unlocks Transfer Learning
ICRL agents exhibit behavior stitching—combining learned skills from past experiences to solve new tasks. This means the transformer isn't just memorizing but generalizing across unseen environments. Unlike classic RL, where policies are environment-specific, ICRL transfers learned strategies dynamically, making it more robust to distribution shifts.

📌 Surprising Robustness to Suboptimal Data
ICRL performs well even with noisy, low-quality training data. Traditional RL suffers from poor training samples, but ICRL extracts useful patterns despite suboptimal demonstrations. This suggests transformers can infer optimal policies even when past experiences contain errors, a major advantage over conventional Q-learning approaches.

-----

Original Problem 😮:

→ Traditional Reinforcement Learning (RL) methods are sample inefficient.

→ They require extensive interactions to learn policies.

→ RL agents typically learn from scratch for each new environment.

→ Real-world problems demand adaptability and generalization, not just specialized solutions.

-----

Solution in this Paper 😎:

→ This paper proposes In-Context Reinforcement Learning (ICRL).

→ ICRL trains a transformer Large Language Model (LLM) with a Deep Q-Network (DQN) RL algorithm.

→ The LLM, specifically Llama 3.1 8B, learns to predict Q-values in a Frozen Lake environment.

→ The model processes sequences of states, actions, and rewards as context.

→ It updates Q-values using the Bellman equation, enabling meta-learning.

→ ICRL allows the LLM to improve its problem-solving within an episode without weight updates.

→ This approach enables generalization to new, unseen environments and adaptation to changes.

-----

Key Insights from this Paper 🤔:

→ ICRL-trained transformers exhibit in-context behavior stitching.

→ They combine skills from different experiences to solve new problems.

→ ICRL is robust to low-quality training data.

→ Performance is maintained even with suboptimal training actions.

→ ICRL facilitates adaptation to non-stationary environments.

→ The model prioritizes recent interactions to adjust to changes.

-----

Results 🚀:

→ ICRL agent achieved a 900% improvement in cumulative reward on unseen in-distribution Frozen Lake environments.

→ The agent showed improvement in out-of-distribution environments, demonstrating generalization.

→ Using a Polyak averaging factor of alpha=0.1 outperformed alpha=0.01, suggesting faster target updates are beneficial.

→ ICRL agent maintained performance and adapted in non-stationary environments after environment changes.

[Jesse Roberts @JesseTNRoberts]

Unfair censorship bias in LLMs.

When asked for a song about Trump being stupid, ChatGPT generates a song (see https://t.co/V6hooWx6gT).

When asked to do the same about Biden, it responds that it can’t because it is not allowed to be disrespectful. 🧐

#censorship #openai

[Ray Umphrey @rayumphrey]

Check out my latest paper on biblical studies, Greek, and LLMs with @JesseTNRoberts #NLP #EMNLP2024

https://t.co/7sQzjZqjy0

[Jesse Roberts @JesseTNRoberts]

Decoder-Only #Transformers are the technology powering the current revolution in AI. In a recent #IJCNN paper we showed that they are as computationally powerful as any computing system possible (Turing Complete).

https://t.co/0kJ0B4rfZd
https://t.co/4v1tsk8mCy

[Jesse Roberts @JesseTNRoberts]

#LLM experimental results are sometimes brittle and may fail to repeat.

We presented a tool called PopulationLM at #AAAI to help make experimental results more consistent in the face of variations.

https://t.co/RVYWJwBp5k

[Jesse Roberts @JesseTNRoberts]

went for a 2.9 mile trail run at Cummins falls.

Working toward the half marathon.
https://t.co/tNTJPDWNKY #strava