sam ritter

Currently into episodic memory for deep RL agents, planning, and meta-RL. On the neuroscience team at DeepMind.

[Vithu Thangarasa @vithursant19]

@JeffDean discussed the importance of sparse computation, adaptive computation and dynamically-changing neural networks @sparsenn. He thinks "dense models are going to give way to these efficient sparse models"...I agree 💯

[sam ritter @ritterstorm]

@TalkRLPodcast @DeepMind Had a blast chatting with @robinc on @TalkRLPodcast!

[TalkRL Podcast @TalkRLPodcast]

Episode 24
@ritterstorm of @DeepMind on Neuroscience and RL, Episodic Memory, Meta-RL, Synthetic Returns, the MERLIN agent, decoding brain activation, and more!
https://t.co/GRb7vWZul0

[Google DeepMind @GoogleDeepMind]

Presenting today @iclr_conf, a new frontier for meta-RL research, where agents must explore & build models on-the-fly for planning. The paper introduces new benchmarks & agents that learn to navigate held-out StreetLearn maps within a single episode: https://t.co/uOjbcU7q3Q (1/)

[Luiz Felipe Vecchietti @lfvecchietti]

The paper, a collaboration with the brilliant Minah Seo, is available in the following link.

https://t.co/4YxqiUx2zn

[Google DeepMind @GoogleDeepMind]

Most RL agents assume that rewards are caused by recent actions, and learn slowly when this isn't true.

This new method speeds up learning in tasks with delayed reward by learning to link related events - regardless of how much time separates them.
https://t.co/aSSy43gCiN

[sam ritter @ritterstorm]

@jsuarez @DeepMind @dnraposo @santoroAI @theophaneweber @hado Very cool. I think the simplest tweak would be to subsample memory storage (i.e. save only every k steps), and let the TD-algorithm assign credit within those k steps.

We never tried this (out of lack of necessity), but in principle logic behind the algorithm still holds.

[sam ritter @ritterstorm]

@robinc Yes, SR at its core does linear regression, so it's fundamentally correlational. For RL I think this is ok - if giving mints is an action, your exploration algorithm should yield data points for H1 without the mints. If not, I'd think of it as an exploration issue.

[sam ritter @ritterstorm]

@robinc What sort of confounding events did you have in mind? We're definitely considering various ways of "confounding" this algorithm (and how to improve it!) in future work.

[sam ritter @ritterstorm]

@michal_sustr @DeepMind We haven't applied our work to Atari-57, but we're planning to! Once that's done, then we'll have a more direct comparison with RUDDER.

[sam ritter @ritterstorm]

@michal_sustr @DeepMind Here we focused on the reward variance problem (key-door), interpretable synthetic returns, and getting SOTA on a major benchmark. TVT does similarly with key-door, but doesn't go for a benchmark. RUDDER beats its baseline on Atari-57, but doesn't show any learning in Skiing.

[sam ritter @ritterstorm]

@jsuarez @DeepMind @dnraposo @santoroAI @theophaneweber @hado Most benchmarks don't have episode lengths long enough to require this. But! This is changing, see e.g., https://t.co/LEROoCAeo8. I'm excited to see what memory systems we'll build as new benchmarks push us towards longer episodes.

[Tim Rocktäschel @_rockt]

@kaixhin @BlundellCharles @ritterstorm I'm quite sure NetHack would need long-term memory of specific events. Episodes in NetHack are often 100'000 steps long and the player has to revisit early-stage dungeons when ascending.

[sam ritter @ritterstorm]

@jsuarez @DeepMind @dnraposo @santoroAI @theophaneweber @hado Thanks! I think that memory _compression_ will be key as episode lengths increase. The idea is to embed big chunks of time in small representations to which we can assign credit, then we can assign credit hierarchically to the component timesteps.

[sam ritter @ritterstorm]

Fascinating talk from the president of Estonia at the Northstar conference #northstarai

[sam ritter @ritterstorm]

@marwinsegler i found your quote here (https://t.co/FsyONx9c7U) to be really insightful "there is no reaction mnist". Any idea why not? The reaction databases are proprietary/expensive?

[sam ritter @ritterstorm]

so awesome that the monzo app tells me exactly why my online transaction didn't go through -- saved me a bunch of time today! @monzo @smarius

[sam ritter @ritterstorm]

today i learned about faculty research awards at google -- a super cool program https://t.co/EypqjQx2pH