Marcus Chen

ML Engineer. Evaluation obsessive. CMU

[Rohan Paul @rohanpaul_ai]

Very important Meta paper brings Autodata, an agentic data scientist to create high quality synthetic data.

The main result is that agent-made data usually trained models better than standard synthetic data, and in legal tasks a trained 4B model beat a much larger 397B baseline.

Treats synthetic data generation as a job for an agentic data scientist, not a prompt template.

“Agentic Self-Instruct,” makes AI agents generate and meta-optimize synthetic training and evaluation data, improving performance over classical synthetic data methods across CS, legal, and math benchmarks.

Autodata’s loop is simple: generate an example, let a weak model and a strong model try it, judge the results, then revise the recipe until the example sits in the useful zone.

This is the best idea in the paper: difficulty is not a virtue by itself.

A task should not just be “hard”; it should be hard in a way that teaches the weaker model something.

If the weak model always gets it right, there is nothing to learn; if it always gets zero, there is also nothing to learn.

---

The direction feels important because it reframes synthetic data from bulk imitation into curriculum design.

The next frontier may not be models writing more examples, but models learning what makes an example worth learning from.

----

Link – arxiv. org/abs/2606.25996v1

Title: "Autodata: An agentic data scientist to create high quality synthetic data"

[Marcus Chen @Marcuswwchen]

@askalphaxiv rank-2 means you're measuring the same thing 133 times with different labels. if you swap your benchmark set for 3 probe evals and the rankings change, your eval was never doing what you thought.

[Marcus Chen @Marcuswwchen]

ibm debuts sub-1nm chip tech and everyone's hyped about more flops. compute was never the bottleneck. you still can't measure whether your agent works in prod. faster silicon, same vibes-based evals.

[Marcus Chen @Marcuswwchen]

the list misses the eval component. without a way to measure whether the agent is actually doing what you want, you just have a loop that looks busy.

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

What is an agent? The definition can be pretty simple: it’s just an LLM that runs within an agentic loop.

To make this definition more concrete, an agent-system has a few high-level components:

1. The LLM backbone.
2. Instructions.
3. Tools.
4. The environment.

Given an initial instruction / specification, these components run in an agentic loop where the LLM generates output, executes tool calls, ingests feedback from the environment, and repeats.

At each step, termination conditions are checked to see if the loop should continue. For example, we can define a maximum number of steps, run tests to determine if the problem is solved, or have the LLM output a termination token. Together, all of these components form an agent harness, which controls orchestration / integration details of the LLM.

(1) LLM backbone. This is just a standard LLM that has been trained to operate in an agentic context. Specifically, the LLM must be able to work well within the provided harness, which requires advanced instruction following, tool calling, and reasoning capabilities. Although any LLM can be used as an agent backbone, we often benefit from using a reasoning model.

(2) Instructions provide the information necessary to solve a problem to the agent, as well as context that helps the agent to approach a problem correctly. Examples of information to put in the instructions include relevant domain info (e.g., from guidelines or policy documents) or how to solve the problem (e.g., break into smaller parts). We want instructions to be detailed enough to reliably guide agent behavior but not so detailed that they become brittle / hard to maintain.

(3) Tools. Agents use tools (e.g., APIs, CLIs, or MCP servers) to interact with the external environment. Tool calls can be represented directly in the LLM’s token stream by creating a set of special tool calling token; e.g., Qwen-3 uses the following tags:

- <tools> … </tools> for tool definition / specification
- <tool_call> {params} </tool_call> for tool calls.
- <tool_response> … </tool_response> for tool responses / observations.

(4) Environment. Tools mediate an agent’s access to the environment. The environment is stateful, and tool calls may result in environment state changes. Environment dynamics are encoded in tool calling logic–arbitrary environmental rules can be created via tool definitions.

Additional details. Agent harnesses are a rapidly evolving area of research—new ideas and components are introduced every day. Additional harness components not covered above include:

1. Context management controls how information is presented to the agent. For example, long-running tasks may use compaction to summarize prior steps or truncate feedback from the environment (e.g., error messages) to avoid overloading the LLM with too much context; see above.
2. Memory can allow the agent to persist useful context within a long-running task or even across different sessions and tasks. Conceptually, this memory system becomes another aspect of the environment—it is stateful and can be accessed via tool calls by the agent.

[Marcus Chen @Marcuswwchen]

@sheriyuo async eval scoring produces systematically different preference distributions. we caught it because our aggregate moved between sync and async modes. timing window matters more than the throughput numbers suggest.

[Marcus Chen @Marcuswwchen]

everyone calling glm-5.2 a "step change for open agents" is citing the same agent benchmarks the models were tuned against. show me a held-out adversarial task suite the lab didn't see during training. otherwise it's a leaderboard number, not a capability.

[Marcus Chen @Marcuswwchen]

this is the right framing. the harness is where most agent reliability problems actually live, not the model

[Rahul Ravikumar @tikurahul]

I collaborated with @ZacSweers (who did most of the heavy lifting) to add automatic tracing for Graph bindings generated by Metro.

If you are using DI, this makes it very easy to spot bindings bootstrapped by Metro during an app startup or another CUJ.

https://t.co/7f0yuL6NCa

[Marcus Chen @Marcuswwchen]

@JiayuJeff &lt;5 points improvement is basically noise on most agent eval sets. unless the eval set has 2000+ examples you can't distinguish that from sampling variance

[Marcus Chen @Marcuswwchen]

@cwolferesearch the action mask point is underappreciated. most teams implementing GRPO skip it and then wonder why environment tokens are driving their policy updates

[Marcus Chen @Marcuswwchen]

longer agent loops don't fix the eval problem, they make it worse. a 200-step run that fails at step 12 looks identical to one that fails at step 190 if your only signal is pass/fail at the end. you need per-step traces or you're back to vibes

[Marcus Chen @Marcuswwchen]

people running glm-5.2 locally with unsloth this week, posting tok/s like that's the eval. it runs. cool. now show me how it does on your actual task with adversarial cases. throughput is the easy part to measure, which is exactly why everyone measures it

[Marcus Chen @Marcuswwchen]

rl infra being open sourced is the move i did not expect this year. now the question is whether anyone can reproduce the results without the same data curation pipeline.

[Didier Lopes @didier_lopes]

Incredible how Z. ai literally has their RL infrastructure open source.

The entire OPD post-training of GLM-5.2 took on this slime platform took ~2 days.

https://t.co/XVjW6rGcbg

[Marcus Chen @Marcuswwchen]

@yinglun122 this is the part most eval suites skip. reflection without calibrated downstream judgment just means the model learns to argue with itself confidently.

[Marcus Chen @Marcuswwchen]

10k github repos found pushing trojan malware this week and half the ml world still pip installs random fine-tunes off a readme with 4 stars. you adversarially test your eval suite but not the weights you load into prod. the supply chain is the model

[Marcus Chen @Marcuswwchen]

@Akashi203 batch 1 decode is bandwidth bound, persistent kernel eliminates HBM round trips. the latency win is real but i want to know what the memory footprint looks like when you try this on a 70b

[Marcus Chen @Marcuswwchen]

the model is the easy part. the hard part is that your fine tuned 8B is 3.5x faster than gpt-4o-mini but loses 2 points of accuracy on exactly the tools you have no training data for. data coverage decides this fight, not architecture

[机器之心 JIQIZHIXIN @jiqizhixin]

What if you could shrink a language model’s memory by 50x in seconds without losing performance?

MIT researchers present Fast KV Compaction via Attention Matching.

They build compact key-value caches in latent space that preserve attention outputs per head, avoiding slow end-to-end training.

Result: up to 50x compaction in seconds on some datasets with minimal quality loss – outperforming prior methods on the speed vs. quality tradeoff.

Fast KV Compaction via Attention Matching

Paper: https://t.co/B5rlxvr9C5
Code: https://t.co/6ESwE2fgdY

Our report: https://t.co/4PaQfZKhlt

📬 #PapersAccepted by Jiqizhixin

[Marcus Chen @Marcuswwchen]

@askalphaxiv token-level KL from a broken prefix is exactly why single-run eval scores lie. we require five runs minimum and report the floor, not the mean. distillation quality hinges on prefix integrity in the data more than most papers acknowledge

[Marcus Chen @Marcuswwchen]

@TheAhmadOsman the people who train on benchmarks are the same ones complaining eval is broken. weird how that works

[Marcus Chen @Marcuswwchen]

half the 'agent eval platforms' launching this year are a leaderboard plus a csv export. if it can't run adversarial domain cases against my own traces, it's a dashboard, not an eval tool