Jia

AI practitioner
Co-founder of @WizBoard
Microsoft for Startups Founders Hub
Apple Store:

[Jia @dolremi]

@HamelHusain @OpenAI Interesting interaction between human and AI

[Jia @dolremi]

Grab a cold coconut juice and sit by the lake to watch the floating clouds

[Jia @dolremi]

Learn to make a new simple dish by following online tutorials today

[Jia @dolremi]

Eat a sandwich filled with lettuce and ham for a quick lunch

[Harrison Chase @hwchase17]

Detecting issues in production agent traces is hard. You have to do it cheaply (because of volume) but also accurately (or too much noise)

We post-trained our own model for this. SOTA accuracy, at ~10-100x cheaper rates than frontier models

Try it out: https://t.co/xSK0Cd8fxt

[Quarq @quarqlabs]

https://t.co/Cs80smDjlJ

[elvis @omarsar0]

I consider this one of the most interesting research themes happening in AI today. Worth taking a look.

As I automate more with agents, I feel like there is all kinds of incredible opportunities to optimize multi-agent systems to do things like automated knowledge discovery or tuning advanced AI systems that gauge other AI agents at software engineering or AI engineering tasks. All kinds of new agent architectures, algorithms, prompting techniques, and data processing and synthesis techniques just waiting to be discovered.

[Jia @dolremi]

@karpathy Use Deepwiki MCP all the time with Claude Code, it has been embedded in plan mode.

[Manthan Gupta @manthanguptaa]

https://t.co/CzmfXDgPRP

[Jia @dolremi]

@bibryam @eugeneyan 's blog is always a good read

[elvis @omarsar0]

Weak-to-Strong GraphRAG

Interesting ICLR 2026 submission with some insights on improving GraphRAG systems and making them more feasible in production environments.

Graph-based RAG lets LLMs ground responses in structured knowledge graphs. But there's a fundamental mismatch between retrievers and the LLMs they serve. As knowledge graphs become central to RAG systems, aligning retrievers to LLM needs through LLM feedback offers a principled path to better multi-hop reasoning with lower costs.

The problem is twofold.

First, graph retrievers train on weak supervision like query-answer shortest paths. This misses key reasoning steps and introduces spurious connections.

Second, retrieved knowledge comes back unorganized. LLMs are sensitive to context ordering, and messy graph data adds unnecessary complexity.

This new research introduces ReG (Refined Graph-based RAG), a framework that uses LLM feedback to align weak retrievers with the LLMs they serve.

Graph-based RAG is essentially a black-box combinatorial search. Given a query, find the minimal sufficient subgraph for correct reasoning. The LLM acts as an evaluator. But exhaustively searching this space is computationally intractable.

ReG takes a simpler approach. Instead of optimizing over all possible subgraphs, it utilizes LLMs to select more effective reasoning chains from candidate chains extracted from the knowledge graph. The improved supervision trains better retrievers.

A structure-aware reorganization module then refactors retrieval results into logically coherent evidence chains. This aligns the presentation to how LLMs actually process information.

On CWQ-Sub with GPT-4o, ReG achieves 68.91% Macro-F1 versus SubgraphRAG's 66.48%. On WebQSP-Sub, 80.08% versus 79.4%. The gains hold across multiple LLM backbones.

The data efficiency is notable in the reported experimental results. ReG trained on just 5% of data, matches baselines trained on 80%. The refined supervision eliminates noise that larger datasets would otherwise compound.

When paired with reasoning LLMs like QwQ-32B, ReG reduces reasoning tokens by up to 30% while improving performance. The structure-aware reorganization prevents the "overthinking" problem where LRMs produce verbose traces in a noisy context.

Paper: https://t.co/mF9sLB63JN

[DAIR.AI @dair_ai]

First comprehensive framework for how AI agents actually improve through adaptation.

While there is a lot of hype about building bigger models, the research reveals a different lever: systematic adaptation of agents and their tools.

Researchers from many universities surveyed the rapidly expanding landscape of agentic AI adaptation.

What they found: a fragmented field with no unified understanding of how agents learn to use tools, when to adapt the agent versus the tool, and which strategies work for which scenarios.

These are all important for building production-ready AI agents.

Adaptation in agentic AI follows four distinct paradigms that most practitioners conflate or ignore entirely.

The framework organizes all adaptation strategies into two dimensions.

> Agent Adaptation (A1, A2): modifying the agent's parameters, representations, or policies.
> Tool Adaptation (T1, T2): optimizing external components like retrievers, planners, and memory modules while keeping the agent frozen.

Let's discuss each in more detail:

A1: Tool Execution Signaled Agent Adaptation.
The agent learns from verifiable outcomes produced by tools it invokes. This involves code sandbox results, retrieval relevance scores, and API call outcomes. Methods like Toolformer, ToolLLM, and DeepRetrieval also fall here. The signal comes from whether the tool execution succeeded, not whether the final answer was correct.

A2: Agent Output Signaled Agent Adaptation.
The agent optimizes based on evaluations of its own final outputs. This includes both tool-free reasoning (DeepSeek-R1, Kimi-1.5) and tool-augmented adaptation (ReTool, Search-R1). The signal comes from answer correctness or preference scores, not intermediate tool calls.

T1: Agent-Agnostic Tool Adaptation.
This involves tools trained independently of any specific agent, including HuggingGPT, ViperGPT, and classic ML tools that serve as plug-and-play modules. These tools generalize well across different agents but may not be optimized for any particular one.

T2: Agent-Supervised Tool Adaptation. Tools adapted using signals from a frozen agent's outputs. Includes reward-driven retriever tuning, adaptive search subagents, and memory-update modules like Reflexion and Memento. The agent stays fixed while tools learn to better support its reasoning.

The trade-offs between paradigms are explicit. Cost and flexibility: A1/A2 require substantial compute for training billion-parameter models but offer maximal flexibility. T1/T2 optimize external components at a lower cost but may hit ceilings set by the frozen agent's capabilities.

Generalization patterns differ significantly. T1 tools trained on broad distributions generalize well across agents and tasks. A1 methods risk overfitting to specific environments unless carefully regularized. T2 approaches enable independent tool upgrades without agent retraining, facilitating continuous improvement.

The researchers identify when each paradigm fits. A1 suits scenarios with verifiable tool outputs like code execution or database queries. A2 works when only the final answer quality matters. T1 applies when tools must serve multiple agents. T2 excels when the agent is fixed, but tool performance is the bottleneck.

State-of-the-art systems increasingly combine paradigms. A deep research system might use T1-style pretrained retrievers, T2-style adaptive search agents trained via frozen LLM feedback, and A1-style reasoning agents fine-tuned with execution feedback in a cascaded architecture.

Four open challenges remain unsolved:
- Co-adaptation: jointly optimizing agents and tools remains underexplored.
- Continual adaptation: enabling lifelong learning without catastrophic forgetting.
- Safe adaptation: preventing harmful behaviors during optimization.
- Efficient adaptation: reducing computational costs while maintaining performance.

The choice of adaptation paradigm fundamentally shapes what an agentic system can learn, how fast it improves, and whether improvements transfer across tasks. Teams building production agents need a principled framework for these decisions, not ad-hoc choices.

Report: https://t.co/o2KPQLLQsZ

Learn to build effective AI agents in our academy: https://t.co/g1Ijo0S5AA

[DAIR.AI @dair_ai]

First large-scale study of AI agents actually running in production.

The hype says agents are transforming everything. The data tells a different story.

Researchers surveyed 306 practitioners and conducted 20 in-depth case studies across 26 domains. What they found challenges common assumptions about how production agents are built.

The reality: production agents are deliberately simple and tightly constrained.

1) Patterns & Reliability

- 68% execute at most 10 steps before requiring human intervention.
- 47% complete fewer than 5 steps.
- 70% rely on prompting off-the-shelf models without any fine-tuning.
- 74% depend primarily on human evaluation.

Teams intentionally trade autonomy for reliability.

Why the constraints? Reliability remains the top unsolved challenge. Practitioners can't verify agent correctness at scale. Public benchmarks rarely apply to domain-specific production tasks. 75% of interviewed teams evaluate without formal benchmarks, relying on A/B testing and direct user feedback instead.

2) Model Selection

The model selection pattern surprised researchers. 17 of 20 case studies use closed-source frontier models like Claude Sonnet 4, Claude Opus 4.1, and GPT o3. Open-source adoption is rare and driven by specific constraints: high-volume workloads where inference costs become prohibitive, or regulatory requirements preventing data sharing with external providers. For most teams, runtime costs are negligible compared to the human experts the agent augments.

3) Agent Frameworks

Framework adoption shows a striking divergence. 61% of survey respondents use third-party frameworks like LangChain/LangGraph. But 85% of interviewed teams with production deployments build custom implementations from scratch. The reason: core agent loops are straightforward to implement with direct API calls. Teams prefer minimal, purpose-built scaffolds over dependency bloat and abstraction layers.

4) Agent Control Flow

Production architectures favor predefined static workflows over open-ended autonomy. 80% of case studies use structured control flow. Agents operate within well-scoped action spaces rather than freely exploring environments. Only one case allowed unconstrained exploration, and that system runs exclusively in sandboxed environments with rigorous CI/CD verification.

5) Agent Adoption

What drives agent adoption? It's simply the productivity gains. 73% deploy agents primarily to increase efficiency and reduce time on manual tasks. Organizations tolerate agents taking minutes to respond because that still outperforms human baselines by 10x or more. 66% allow response times of minutes or longer.

6) Agent Evaluation

The evaluation challenge runs deeper than expected. Agent behavior breaks traditional software testing. Three case study teams report attempting but struggling to integrate agents into existing CI/CD pipelines.

The challenge: nondeterminism and the difficulty of judging outputs programmatically. Creating benchmarks from scratch took one team six months to reach roughly 100 examples.

7) Human-in-the-loop

Human-in-the-loop evaluation dominates at 74%. LLM-as-a-judge follows at 52%, but every interviewed team using LLM judges also employs human verification. The pattern: LLM judges assess confidence on every response, automatically accepting high-confidence outputs while routing uncertain cases to human experts. Teams also sample 5% of production runs even when the judge expresses high confidence.

In summary, production agents succeed through deliberate simplicity, not sophisticated autonomy. Teams constrain agent behavior, rely on human oversight, and prioritize controllability over capability. The gap between research prototypes and production deployments reveals where the field actually stands.

Paper: https://t.co/AaNbPYDFt5

Learn design patterns and how to build real-world AI agents in our academy: https://t.co/zQXQt0PMbG

[Jia @dolremi]

洛水滔滔去，问鼎意何深。在德不在器，千古帝王心。

[Jia @dolremi]

七仙女漫游蟠桃园-宋代文人画风，再加一首如梦令

[Jia @dolremi]

七仙女漫游蟠桃园-隋唐青绿山水画风格

[Jia @dolremi]

Looking forward Gemini 3.0

[Matt Shumer @mattshumer_]

Yeah, AI progress is totally, definitely stalling...

Look at MathArena Apex.

GPT-5.1 scored 1%.

Gemini 3 scored 23%.

That is a >20x jump on one of the hardest reasoning tasks we have.

But sure, keep your head in the sand...

[Jia @dolremi]

@vista8 进不了了

[major tom @tailwiinder]

Linus used to speak to engineers in 2012 the way I speak to LLMs now.

[Jo Kristian Bergum @jobergum]

https://t.co/eOp5DbkXd7