Trong-Thang Pham

PhD @UARK.
Former: Bachelor @ HCMUS.
Substack:
These are my personal opinions unless otherwise noted.

[Tengyu Ma @tengyuma]

😊 I also just uploaded a new version of the lecture notes. https://t.co/bpIvdgG8KO (it's still, and perhaps will always be work-in-progress, given the field is moving fast. Any mistakes are mine :) and any feedback is more than welcome :).

[Trong-Thang Pham @trongthangpham]

@roydanroy 😂 here I was wondering why it is "11" and suddenly remembering the old joke.

[Raila @raila_san]

Some notes #RLC2026

> Streaming RL now works reliably, a meaningful shift.
> Curiosity-driven learning gaining traction - No explicit task or reward, agent resolves its own targets.

Doina Precup:

> In-context adaptation is a very simplified, non-parametric form of continual learning.
> Drawbacks of this temporary memory system (in-context learning) is addressed by harness.

"And the harness is going to do things like, you know, remember stuff. Oh, I solved this math problem. It was an interesting lemma, let me put it in a file system right now. I can retrieve it from that file system.

Or, you know, I don't know how to do a A* search. I wrote the little program by myself that does that. Let me put it in a file system and then practically treat it. Okay, so that's a form of continual learning. It's using an external memory"

> On why this isn't automated: The learning loop currently includes of talented research engineers doing clever data mixture engineering.

"I think there is a false idea in many people in the industry's head that we are doing something safer by not allowing the agent to adaptively update? Because what if we somehow get some wrong data and now it becomes poison. Right.

What if they do something very bad? And now there's a big scandal and so on. Now, if you've been following the press for the last couple of weeks, you're going to notice that this is happening anyway because these agents are doing bad things, not because they're bad agents. Because in my opinion, they cannot fix themselves. And they don't have the right understanding of the world because all they get to do is interact with some fixed data sets and some very limited kinds of tasks that we train them on."

[Zachary Yu @thezacharyyu]

continual learning is a bullshit word. i’ve heard at least these 4 different things be called continual learning. we should have distinct names for these:

- in context learning -> technically continual learning assuming that in the future you will eventually have infinite context windows and can manage these infinite context windows

- harness and memory level improvements -> GEPA - DSPy stuff / skill and tool generation based on production data at some arbitrary interval

- ultra frequent fine tuning -> model / adapter updates via SFT, RL, or distillation from production data at some arbitrary interval

- inference time weight updates -> usually some new architecture proposed by some neo lab or smth. for example maybe for each output token you can do a zero order computation to update the model weights continuously

[Trong-Thang Pham @trongthangpham]

Probably thanks to TPU. But yeah this is crazy fast.

[Philipp Schmid @_philschmid]

It is not only a good and cost effective! It is blazing fast! 🔥

[Trong-Thang Pham @trongthangpham]

@xwang_lk It is always good to stop sometimes, back a few steps and look at the big picture

[Trong-Thang Pham @trongthangpham]

@iScienceLuvr @TimDarcet @MedARC_AI Oh nice. Will this be a series?

[Massimiliano Viola @massiviola01]

JEPA is one of the hottest buzzwords in machine learning right now.

Are you still unsure what that means for video models?

Then I've got you covered!😌

In this week's post, we examine the shift in video self-supervised learning from pixel-space to latent-space modeling.

[Trong-Thang Pham @trongthangpham]

Same trend. People need to learn verification skills. What do they expect when they vibecode anything.

[Greg Brockman @gdb]

the bottleneck is increasingly knowing what you want

[Trong-Thang Pham @trongthangpham]

I wish more providers will do this. This should at least force academic people to read when they do review and write paper at least.

[M1 @M1Astra]

Anthropic says new Claude models will embed invisible watermarks in all generated text, everywhere Claude is offered.

The watermark is part of the text, it isn't metadata: "it will travel with the text when it's copied and pasted elsewhere, and may persist through some editing."

This starts with models launched on or after August 2, 2026, under an EU AI Act code Anthropic signed. Anthropic is still working on adding it to current models. The rollout is worldwide.

[Trong-Thang Pham @trongthangpham]

Damn "ultrathink" is such a good word to use here. I'm gonna steal this.

[Lucas Beyer (bl16) @giffmana]

@danliu Ok then, as a xoogler, think about _who_ concretely would be able/allowed to tweet that? Really, ultrathink it.

[Trong-Thang Pham @trongthangpham]

One of the main reason I stay with codex for this full summer even though I use to love claude so much. The web search and Haiku are just not good features anymore in claude code. :(

[Simon Willison @simonw]

Claude Haiku is my current least favorite model - it hallucinates wildly, and is out-performed now by other similarly priced models like GPT-5.6-Luna

Even worse: it seems to still be used by the Claude Code WebFetch tool, which means hallucination risk any time you fetch a URL!

[Josh Han @JoshHanHi]

How does a model understand and explore a world it has never seen?

We introduce 🔭VISTA🔭, a visual harness that gives a VLM long-horizon vision for reasoning in an interactive world. With Claude Opus 5.0, it reaches 100% RHAE on @arcprize's ARC-AGI-3, perfectly solving all 25 public games.

Blog post: https://t.co/bYUWwi0KoQ 🧵

[Kosta Derpanis @CSProfKGD]

This is completely normal. Happens to all of us, including me after all these years. We all tend to overthink these situations. The first step is accepting that nerves are a normal part of public speaking. You’re nervous because you care.

For a talk, it’s especially important to have your first few slides memorized thoroughly. Once you get into a rhythm, you usually settle in and the nerves begin to fade.

Of course, the foundation is practicing your talk. The better prepared you are, the more confident you’ll feel.

Good luck!

[Trong-Thang Pham @trongthangpham]

Limit breaker

[Tibo @thsottiaux]

What is my title at OpenAI

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

This is a *really* good blog post by @reza_byt about how SIGReg works (the main component of @ylecun's LeJEPA)

link: https://t.co/vqzq8wzvzj

[Trong-Thang Pham @trongthangpham]

/export the full session would be great.

[Tibo @thsottiaux]

What should we improve on Codex to improve the everyday experience? Nothing too small

[Sebastian Raschka @rasbt]

The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts.

1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now)

2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.

3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details).

4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost.

5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know.

6. Kimi K3 now also has native multimodal support, which is great!

There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.

[ali @waterloo_intern]

https://t.co/aivc7B4A7Z

[Trong-Thang Pham @trongthangpham]

@thaoshibe Not sure if it’s relevant, but this reminds me of a game where you lay out images from your collection, and people try to guess your personality based on them. I think there is even a subreddit about this but for books, r/BookshelvesDetective