Pyuyi

mmm(Pyuyi) | Intern @Huawei(RA)｜(She/her)

|math|AI Interpretability|Causal Inference|

looking for PHD/Intern/RA

[Pyuyi @Pyuyi2333]

Hi, there!

I’m Pyuyi, mmm as well (she/her). This account is for academic sharing.

What I mainly post here:
1. Paper notes and thoughts on AI interpretability, causal inference, math and learning theory.

2. Summaries and discussions of recent work from venues — including causal model reduction, RL interpretability, topology & replicability, latent reasoning as dynamical systems, etc.

3.Occasional updates on my own related work and open-source experiments.

If you’re interested in math, AI interpretability, or causal inference, follow@Pyuyi2333--would love to chat and exchange iddeas!

[Pyuyi @Pyuyi2333]

Most representation learning methods rely on prediction objectives, external rewards or fixed datasets.

This paper from ICML26 explores a different principle: an agent should represent the world according to its capacity to influence it. Building on information-theoretic empowerment, the authors maximize mutual information between an agent's skills and their future outcomes.

Their analysis reveals two complementary representations: a forward representation describing which outcomes can be influenced and a backward representation describing how states can be reached. Under specified assumptions, both become invariant to control-irrelevant information without requiring task rewards or expert demonstrations.

In experiments involving Point-Maze, Ant, and visual manipulation, learned representations show at least 1000× greater sensitivity to control-relevant state variables than irrelevant noise. Linear probes also recover the true coordinates of a noisy maze with R² approaching 1, while downstream agents achieve substantially better control performance than baselines using noisy observations directly.

The deeper implication is that intelligent perception may emerge from intervention rather than prediction alone. This connects reinforcement learning with mutual information, causal representation learning, and control theory, suggesting a route toward task-independent world models.

However, the theoretical guarantees rely on specific assumptions and fully observable environments; learning controllable features does not establish recovery of the complete causal structure.

The next question is whether the same principle can help partially observable or language-model-based agents develop transferable causal representations through tool use and environmental interaction.

Perhaps building more capable agents requires teaching them not only to predict what happens next, but to discover which aspects of the future they can actually change.

https://t.co/i5t2BxKRmL

[Pyuyi @Pyuyi2333]

Standard RLVR maximizes expected correctness, but this objective can provide weak learning signals for problems the model rarely solves.

The authors from ICML26 oral show that expected-reward RL captures only the first-order term of a more fundamental objective: the log-likelihood of generating correct solutions.

Their proposed MaxRL bridges that gap by expressing maximum-likelihood gradients through a weighted combination of pass@k objectives. Surprisingly, the practical modification is almost trivial: normalize successful trajectory gradients by the number of successes rather than the total number of samples.

The resulting estimator is unbiased for a truncated likelihood objective and approaches maximum-likelihood optimization as sampling compute increases. In Qwen3 mathematical reasoning experiments, MaxRL improves solution coverage while maintaining competitive pass@1 performance, requiring up to approximately 19.2× fewer test-time samples than GRPO to reach comparable success rates in selected settings.

The deeper insight is that an optimization objective determines not just what a model learns, but which problems receive enough gradient signal to become learnable. This connects reinforcement learning with maximum likelihood, logarithmic scoring rules, and gradient geometry.

However, the method assumes reliable binary verification, cannot directly learn from prompts producing zero successful trajectories, and incurs additional training costs.

The next challenge is combining adaptive sampling, difficulty estimation, and likelihood-based optimization to help models learn from increasingly rare successes. Perhaps better reasoning RL begins with deciding how learning signals should be distributed—not simply how rewards should be maximized.
https://t.co/7S36dep7Mx

[Pyuyi @Pyuyi2333]

Most self-improving agents accumulate trajectories, retrieve past experiences and refine their behavior within a manually designed memory system.

MemEvolve from ICML26 challenges that assumption by introducing two coupled processes: evolving what an agent remembers and evolving how its memory works.

The authors organize 12 representative memory systems into four components—encoding, storage, retrieval, and management—then use trajectory analysis, candidate generation, and iterative evaluation to search for better architectures. Across four agent benchmarks, the method reports performance improvements of up to 17.06%, with evidence that some evolved memory architectures transfer across tasks and backbone models.

The deeper insight is that an agent's learning mechanism can itself become an optimization target. This connects agent memory with meta-learning, bilevel optimization and evolutionary search: one process accumulates useful experience, while another improves the machinery responsible for learning from it.

Unlike ordinary retrieval-augmented memory, the improvement operates at the level of system design. However, benchmark-specific architectural gains do not establish open-ended recursive self-improvement and evaluation overfitting, search costs, and long-term stability remain unresolved.

The next challenge is whether agents can jointly evolve memory, tool-use policies, and experience-transfer mechanisms while demonstrating measurable improvements in future learning efficiency. Perhaps scalable self-improvement requires not just learning more from experience, but learning better ways to learn.

https://t.co/2WlhwCDi8k

[Pyuyi @Pyuyi2333]

@zhengzhi20 Sure! I'll read it after a while

[Pyuyi @Pyuyi2333]

Conventional zeroth-order optimizers such as MeZO estimate gradients through random parameter perturbations, wasting evaluations on directions unrelated to meaningful updates.

AGZO from ICML26 exploits a simple but powerful geometric property: the weight gradient of a linear layer lies in the subspace spanned by its input activations. Rather than sampling perturbations across the entire parameter space, it extracts low-rank activation subspaces during forward computation and restricts gradient estimation to those directions.

Under spectral concentration assumptions, the authors prove better expected alignment with the true gradient than isotropic perturbations. On Qwen3-0.6B, AGZO achieves 89.2% accuracy on CB versus 80.3% for MeZO, while enabling Qwen3-4B fine-tuning in a hardware setting where full first-order training runs out of memory.

The deeper insight is that avoiding backpropagation does not require treating neural networks as structureless black boxes. Forward activations already reveal useful information about optimization geometry, connecting zeroth-order learning with randomized linear algebra and low-dimensional representation structure. However, the method remains substantially more iteration-intensive than first-order training, and its advantages have not been established at frontier pretraining scale.

The next question is whether adaptive activation subspaces can reduce both gradient-estimation noise and the number of forward evaluations required for effective learning. Perhaps the next generation of optimizers will use forward computation not merely to evaluate loss, but to determine where parameter updates should occur.

https://t.co/3MoXumbFpw

[Bojie Li @bojie_li]

非常赞同，我觉得 agent 开发就像 C 和 Python 一样，属于基本技能。

我甚至还有个暴论，编译原理、操作系统、计算机体系结构，这三门计算机本科最硬核的课，将逐渐让位于 agent、LLM、infra 三门核心课（这也是我翻译《图解大模型》和写 agent/infra 两本书的原因）。

传统软件，工程师接触的是编程语言，代码跑在编译器和运行时上；软件是在操作系统上跑的；操作系统又是基于体系结构。LLM 时代，工程师接触的是 token，agent 跑在 harness 上；agent 是在 LLM 上跑的；LLM 又是跑在 infra 上的。

[Pyuyi @Pyuyi2333]

@jacksunny18 Hope you like it🥰

[Pyuyi @Pyuyi2333]

While I was tinkering with a visual novel on my Mac, I suddenly had an idea that needed an experiment to test it.

The idea and discussion were all in my Mac’s Codex conversations, but the Windows machine was the one connected to the lab VPN and able to reach the server. So I wondered: could Dot on my Mac read the discussion, hand the task to Codex on Windows, and let Windows SSH into the server to run it?

It turned out that the available integrations couldn’t connect that whole workflow at the time: the Dots on the two computers couldn’t coordinate directly, and Dot couldn’t directly read the existing Codex Work／Chat conversations either…

So I built this little project. If you like it, consider leaving a little ⭐:

DotControl: One Dot. Multiple computers working together.

We increasingly work across multiple computers: a Mac for conversations and organizing ideas, a Windows machine for development environments, applications, and network access. When tasks and context are scattered across devices, we end up switching computers, copying instructions, and hunting down conversation history.

DotControl helps these devices work together: give Dot a task on one computer, let another computer execute it in its own environment, and bring the results back.

It connects each device to Dot through a private MCP plugin → OpenAI Secure MCP Tunnel → a local agent on the target computer, without requiring a self-hosted cloud server.

Execution happens on the target computer. Terminal tasks go to its official Codex installation, desktop actions use a controlled executor, and local conversations are returned through read-only access within the sharing scope configured by the user. Login credentials stay on the target device, permissions are configured by the user, and there are explicit controls to pause access, revoke it, or cancel tasks.

For computer use, the project has demonstrated calculator operations and screenshot return across Windows and Mac computers. It has also completed a workflow where Dot stays connected to the Mac, delegates a read-only terminal task to Windows Codex, and receives the result.

Installers are available for Windows, Apple Silicon Macs, and Intel Macs, with a console for managing connections, Codex, and conversation sharing. You can submit remote tasks, check progress, continue conversations, and choose which local chats Dot may read.

DotControl is making cross-device collaboration a natural part of the workflow: express your intent here, let the right computer execute it, and bring verifiable results back here.

https://t.co/mqQycmA2NV

[Pyuyi @Pyuyi2333]

TLDR:
DotControl is an open-source tool for cross-computer collaboration. Ask Dot on one computer to delegate tasks to Codex on another, read conversations you’ve chosen to share, and perform controlled desktop actions. It supports Windows and macOS without a self-hosted cloud server—connecting tasks, context, and results across devices.

[Pyuyi @Pyuyi233]

https://t.co/NEhmrvZMng

[Pyuyi @Pyuyi2333]

Most reinforcement learning with verifiable rewards (RLVR) reduces an entire reasoning attempt to a single success or failure signal, discarding information contained in compiler errors, failed tests and other environment feedback.

The authors introduce Self-Distillation Policy Optimization (SDPO) in ICML26, which lets a model reconsider its predictions after observing feedback, then distills those feedback-conditioned predictions into its original policy.

Instead of learning exclusively from sparse outcome rewards, the model receives dense token-level supervision from its own informed predictions. Across scientific reasoning, tool use and competitive programming, SDPO improves learning efficiency over strong RLVR baselines.

On difficult binary-reward problems, its test-time variant achieves comparable solution-discovery probabilities with 3× fewer attempts. The deeper insight is that a model's ability to recognize mistakes can become a mechanism for improving its future behavior.

This connects self-distillation with policy improvement, amortized inference, and continual learning, suggesting a promising direction for agents that learn from their interactions. But self-generated supervision is only useful when the underlying feedback is trustworthy and the model can interpret it correctly. Otherwise, distillation may reinforce existing errors.

The next question is whether tool feedback, environmental observations, and persistent memory can be converted into a unified learning signal that enables agents to accumulate genuinely transferable capabilities over time.

https://t.co/nY2vJyvuwo

[Pyuyi @Pyuyi2333]

The paper from ICML26 asks a fundamental question: can an AI reason causally about a world whose objects and relationships have changed?

Classical structural causal models typically assume a fixed set of variables, but real environments rarely preserve that structure. The authors prove that observations from familiar configurations may be insufficient even to identify the observational distribution of an unseen configuration.

Their solution, relational structural causal models (RSCMs), introduces graphical identification conditions that enable causal reasoning across changing object configurations under explicit assumptions, including certain forms of hidden confounding. In simulated traffic environments, their neural implementation reduces causal-effect estimation error by roughly 100× in some identifiable settings, successfully transferring knowledge between different arrangements of cars, pedestrians and traffic signals.

The deeper contribution is a mathematical bridge between causal inference and compositional generalization, connecting Pearl’s causal framework with relational learning, graph neural networks and object-centric world models. It suggests that generalizable intelligence may require learning not just representations of objects, but the causal mechanisms governing their interactions.

The limitations matter: the theoretical framework relies on assumptions including finite discrete variables, bounded relational neighborhoods and restrictions on cross-object hidden confounding, while experiments remain largely synthetic. The next challenge is learning such structures from high-dimensional observations and using them for planning and reinforcement learning.

Perhaps the most important thing a world model should generalize is not the world it has seen, but the causal rules that generate possible worlds.

https://t.co/EGRmGFLPPY

[Pyuyi @Pyuyi233]

可以水群、

然后可能我会放一些我知道的关于AI的workshop通知，大家可以自行选择性的报名参加一些，也可以稍微的讨论一起学习一下

还可以约游戏））

更多功能还在探索中qaq

[Pyuyi @Pyuyi2333]

@Fainterstar11 Recalibration might need be drift-triggered(? I guess a practical setup is to run lightweight held-out checks every short training window, with immediate recalibration when policy distribution, reward statistics, or verifier disagreement shifts materially.

[Pyuyi @Pyuyi2333]

The paper from ICML26 challenges a fundamental assumption in RL: why optimize the agent while treating its environment and reward function as fixed?

RLAnything replaces that separation with a joint optimization loop. The policy learns from both process-level and outcome feedback, the reward model improves through consistency signals and the environment adapts its training tasks using critic feedback.

Rather than simply adding more training examples, the system continuously changes what the agent learns, how its behavior is evaluated and which experiences it encounters next. The authors report improvements of 8.5% on OSWorld, 21.2% on AlfWorld, and 12.1% on LiveBench in their evaluated model settings, with training code publicly available.

The deeper contribution is a shift from policy optimization to learning-system optimization, connecting agentic RL with bilevel optimization, active learning, automated curriculum design and adaptive control.

This offers a concrete research direction for recursive self-improvement: an agent might improve not only its behavior but also the training experiences and feedback mechanisms responsible for future improvements. However, benchmark gains do not establish sustained self-improvement. Jointly evolving rewards and environments can create dangerous feedback loops in which biased evaluations reinforce increasingly misleading training tasks.

The next question is whether independent verification, principled improvement criteria and compute-aware environment generation can make this process stable and genuinely transferable. If so, the next frontier in agent training may be optimizing the entire policy–reward–environment ecosystem, rather than the neural network alone.

https://t.co/B2NIFGGCtk

[Pyuyi @Pyuyi2333]

Actually, from my point of view, nothing inherently prevents the reward model from drifting toward behaviors the current policy already performs well. That is a central failure mode of co-evolving RL systems: the policy generates more of its preferred trajectories, the reward model trains on them and both can enter a self-reinforcing feedback loop.

Preventing this requires independent anchors—held-out verifiers, frozen reference rewards, external labels or rules, out-of-distribution evaluation and periodic recalibration. Without them, rising reward may simply mean the system is getting better at satisfying its own evolving metric.

[Pyuyi @Pyuyi2333]

@DominickNoval It seems that the paper from Deepseek Harness using the Category Theory to describe the reversibility in Software Engeering can answer your doubts. Maybe it can inspire something.

[Pyuyi @Pyuyi2333]

The paper from ACL26 attacks a more important problem than simply extending context length: how should an agent learn what to remember, revise, forget and actually read later?

Memory-R1 turns external memory from a hand-engineered database into a learned control policy. A Memory Manager is RL-trained to choose among ADD, UPDATE, DELETE and NOOP, while a separate Answer Agent selects and distills relevant entries before reasoning and both are optimized with outcome-driven PPO/GRPO rather than fixed summarization or retention heuristics.

The striking part is its data efficiency: with only 152 training QA pairs, the system generalizes across LoCoMo, MSC, LongMemEval, and model scales from 3B to 14B. In the authors’ released implementation, the GRPO version on Llama-3.1-8B reaches 45.0 F1, 37.5 BLEU-1, and 62.7 LLM-as-a-Judge on LoCoMo, versus 30.4 / 22.2 / 45.7 for Mem0.

The deeper idea is that memory management can be treated as policy learning rather than infrastructure plumbing. That links agent memory to belief-state compression in POMDPs, long-term value estimation in cache replacement and the continual-learning question of which pieces of history remain sufficient statistics for future tasks. For production agents—research assistants, coding agents, customer-service systems, or personal assistants—the implication is that persistent memory may eventually be optimized for future utility instead of growing as an ever-larger transcript.

The current evidence is still narrow: controlled long-memory QA benchmarks do not establish robustness under open-ended tool trajectories, corrupted memories, or months of non-stationary interaction. The next step worth watching is joint memory-and-tool policy learning, where an agent learns when to query the world, when to write persistent state and when to forget while explicitly pricing tokens, latency, and error propagation.

If that scales, agent memory stops looking like an add-on RAG store and starts looking like the agent’s learned state-transition mechanism.

https://t.co/xFUZeKA3qK

[Pyuyi @Pyuyi2333]

@draupnir1244369 可以先看看这个https://t.co/Pb5egoGIDZ

[Pyuyi @Pyuyi2333]

Hi, there!

I’m Pyuyi, mmm as well (she/her). This account is for academic sharing.

What I mainly post here:
1. Paper notes and thoughts on AI interpretability, causal inference, math and learning theory.

2. Summaries and discussions of recent work from venues — including causal model reduction, RL interpretability, topology & replicability, latent reasoning as dynamical systems, etc.

3.Occasional updates on my own related work and open-source experiments.

If you’re interested in math, AI interpretability, or causal inference, follow@Pyuyi2333--would love to chat and exchange iddeas!

[Pyuyi @Pyuyi2333]

@draupnir1244369 （？大可不必说明后面一句……

你说的可重复性的工作是指的啥呢🤔