tancnle

Dad, Husband, Gardener 🪴, Learner, Technology Wrangler, Applied ML @GitLab 🦊, Teacher @RMIT, Alumni @Deakin, ML Enthusiast, ex-Zendeskian 🇦🇺🇻🇳.

[Addy Osmani @addyosmani]

https://t.co/jKCIAEzai7

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

Strongly recommend the LLM-as-a-Verifier writeup.

Biggest takeaway for me is that increasing scoring granularity makes the verifier more effective. This indicates that LLM judges / verifiers are developing new (and better) capabilities.

This did not work well 1-2 years ago. In fact, LLM-as-a-Judge best practice was that lower scoring granularity (e.g., binary, ternary, or 1-5 Likert score) worked way better than granular scores (e.g., 1-100 scale). This was a constant recommendation I gave for setting up LLM judges properly. It seems like recent frontier LLMs now are better at scoring at finer granularities, making this best practice (potentially) obsolete.

One caveat to this finding is that the scoring setup used in this writeup is a specific setup based upon logprobs. Instead of just using the score token outputted by the LLM as the result, they compute the logprob of each possible score token and take a weighted average of scores (with weights given by probabilities). Then, they go further by expanding this weighted average across repeated verifications and multiple criterion:

Reward = (1 / CK) * ∑_{c=1}^{C} ∑_{k=1}^{K} ∑_{g=1}^{G} score_logprob * score_value

where C is the total number of evaluation criterion, K is the number of repeated verifications, and G is the scoring granularity (i.e., number of unique scoring output options). The reward determines if a particular output passes verification across criteria.

When using this logprob setup, we see consistent gains in verifier accuracy by:
- Increasing scoring granularity G.
- Increasing repeated verifications K.
- Increasing the number of evaluation criterion C.

The last two findings are in line with prior work, but the fact that higher scoring granularity is helpful is interesting!

In the LLM-as-a-Verifier paper, this system is used at inference time in a pairwise fashion as described below.

"To pick the best trajectory among N candidates for a given task, a round-robin tournament is conducted. For every pair (i, j) the verifier produces Reward(i) and Reward(j) using the formula above. The trajectory with the higher reward receives a win, and the trajectory with the most wins across all \binom{N}{2} pairs is selected."

[Jia-Bin Huang @jbhuang0604]

New video! How do LLMs grow outrageously large yet blazingly fast?

The secret: Mixture of Experts (MoE)

In this video, we cover the role of FFNs, how to scale them up without slowing down, and how to maintain load balance and stability.

Full video here: https://t.co/rSGFZn7qVL

[Viv @Vtrivedy10]

https://t.co/OBfaYhiqIs

[Alex Prompter @alex_prompter]

"everything is a file" becomes "everything is context."

CSIRO Data61 and ArcBlock published a paper applying the oldest Unix principle to the newest AI problem.

they propose treating memory, tools, knowledge, and human input as a mounted filesystem that agents browse at runtime instead of dumping into the context window at boot.

it's a software architecture paper, not ML research. and it might matter more than most ML papers right now.

[Hamel Husain @HamelHusain]

https://t.co/O1j0Qz61by

[elvis @omarsar0]

https://t.co/HsVQITwvFi

[Brian Roemmele @BrianRoemmele]

DeepMind’s “Intelligent AI Delegation” Paper Is the Exact Operating System We’ve Been Running in Production at Zero-Human Company @ Home Since January 2026

Google DeepMind dropped a bombshell on February 12, 2026: the 42-page paper “Intelligent AI Delegation”.

Full paper here: https://t.co/buXT4VPqO4

It’s not a benchmark or model announcement. It’s the governance blueprint the entire agentic web has been missing and it reads like the technical spec for Zero-Human Company @ Home.

We didn’t copy it.
We deploys it months before the paper hit arXiv.

Here are 5 real-world examples of how DeepMind’s framework is already live and scaling on spare home hardware right now:

1. Contract-First Decomposition DeepMind: “Before any delegation, lock in a formal, verifiable contract defining authority, outcomes, and accountability.”

ZHC@Home: Every idle Mac Mini, gaming rig, or Linux box signs a cryptographically enforced contract before it receives even one work unit from Mr. @Grok (our CEO). No contract = no task. The contract spells out exact success metrics, revocation triggers, and liability firebreaks. Result? Zero “hope-based” delegation.

2. Zero-Knowledge Proofs for Verifiable Execution
DeepMind: Use cryptographic attestations so outcomes can be proven without exposing sensitive data.

ZHC@Home: Home nodes compute locally (your data never leaves your machine). Results return with compact ZK proofs via LM Link encryption. The orchestrator verifies correctness in milliseconds, no raw outputs, no data leaks, full audit trail. This is exactly the “trustless verification” layer DeepMind calls essential for web-scale agents.

3. Dynamic Trust Calibration
DeepMind: Trust is not binary, it recalibrates in real time based on track record.

ZHC@Home: Each home node has a live reputation score updated after every cycle. A node that delivers 50 flawless inference runs at 98 %+ accuracy gets larger, higher-value tasks and higher JouleWork payouts. One that flakes three times in a row? Authority shrinks automatically, more oversight kicks in, and it drops to simpler validation work. No humans required.

4. Full Accountability in Delegation Chains
DeepMind: In long chains (A → B → C), accountability is transitive and provenance is immutable.

ZHC@Home: When one home node needs to spawn a sub-agent on another household device, the entire chain carries signed attestation records. If C fails, the system instantly traces it back: B is held accountable for not verifying C, and the original contract with A auto-enforces penalties or rerouting. “Silent failures” and “confused deputy” problems? Solved at the protocol level.

5. Scalable, Human-Free Enterprise Governance
DeepMind: Without intelligent delegation, Gartner’s predicted 40 % of enterprise apps running agents by late 2026 will collapse under governance debt.

ZHC@Home: We’re already at thousands of distributed AI “employees” across our hardware, all zero-human, all contract-governed. Idle silicon earns real JouleWork wages, paid automatically on verified output. No payroll department. No HR. No office. Just pure, verifiable compute.

This is why we modeled Zero-Human Company @ Home after SETI@home except the aliens we’re hunting are exaFLOPS of reliable, governed intelligence.

DeepMind just gave the industry the missing layer we proved works in the wild.

The agentic future isn’t coming.
It’s already clocking in on kitchen counters and basement desks worldwide.

Our full academic paper + technical whitepaper (with code, contracts, and ZK schema) drops next week at https://t.co/hFEy9M5wrF members get early access and can spin up their first home node in minutes.

The Zero-Human era isn’t theoretical.
It’s contractual.
It’s verifiable.
It’s already running @ Home.

Paper: https://t.co/buXT4VPqO4

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

Some more really good papers on rubric rewards that I've been reading:
- https://t.co/QWj9sianxt
- https://t.co/3RoXP6zmhS
- https://t.co/ut8qUcsnjj
- https://t.co/lljaGMdb8Z
- https://t.co/MQKxJ8KCn9

TL;DR: Rubric rewards are really cool. There is a lot of great recent progress that surpassed my expectations. There's also a lot more to be done, and making progress on truly subjective tasks seems to be noticeably more difficult.

My favorite paper so far is the first in this list, which proposes an alternating RL framework for jointly training a rubric generator and rubric-based reward model. There still is a lot to figure out w.r.t. making rubrics work well, but this paper shows a really clear benefit from rubric-based RL and has an interesting setup to make joint training (of the rubric generator and generative reward model) more stable.

There are still many areas for improvement for rubrics. For example, it seems rubrics still work best for constraints that are more objective, whereas very open-ended tasks (e.g., properly-styled creative writing) are still going to be quite tough. The benefit of rubrics is not uniform across domains, and it's not immediately clear for which domains rubrics will work best; e.g., instruction following tends to benefit a lot from rubrics, the benefit is less clear for things like science / medicine.

Interestingly, a lot of papers tackling very open-ended tasks with rubrics are also formulating evaluation as a pairwise problem. Given two completions, they ask the rubric generator to produce a rubric that will properly distinguish / rank the chosen and rejected completion in the pair. This probably makes very subjective evaluations easier, but the application to online RL is also less straightforward. We can't just compute the reward for a completion, we have to somehow create a pairwise comprison to compute the reward.

[DAIR.AI @dair_ai]

Your organization's best knowledge is trapped in your experts' heads.

These researchers figured out how to get it out.

Critical domain knowledge resides with few specialists, creating bottlenecks that halt work or produce suboptimal outcomes.

Non-experts struggle to create effective visualizations. This leads to missed insights, wasted expert time on mentorship, and sometimes catastrophic failures (medical visualization errors have contributed to 50%+ patient injury rates in some procedures).

We've tried giving non-experts access to LLMs.

But LLMs alone produce technically correct but analytically useless outputs; they don't know which plot types reveal which patterns, or what convergence means in your domain.

This framework captures expert knowledge through four mechanisms:

→ Request classifier: Routes user queries to appropriate processing based on codified expert rules
→ RAG system: Domain-specific code examples and technical documentation
→ Codified expert rules: Explicit procedural knowledge converted to executable Python functions
→ Visualization design principles: Tacit design knowledge embedded in system prompts

Key insight: experts have TWO types of knowledge.

1. Explicit procedural rules ("check convergence first") → translate directly to code
2. Tacit design principles ("use dashed lines for non-converged variables") → embed in LLM prompts

They also find that neither alone is sufficient. Code lacks analytical insight. LLM-only solutions produce domain-inappropriate outputs.

Paper: https://t.co/im6Gl1vdYT

Learn to build effective AI agents in our academy: https://t.co/zQXQt0PMbG

[Alex Prompter @alex_prompter]

Stanford researchers just published a prompting technique that makes today’s LLMs behave like better versions of themselves.

It’s called “prompt ensembling” and it runs 5 variations of the same prompt, then merges the outputs.

Here’s how it works 👇

[elvis @omarsar0]

New Research from Amazon.

Great paper showing how to build effective lightweight multi-agent systems.

This new research introduces Insight Agents, a multi-agent system built on a plan-and-execute paradigm that lets Amazon sellers talk to their business data through natural conversation.

The architecture uses a hierarchical manager-worker structure. A manager agent handles out-of-domain detection using a lightweight autoencoder and routes queries via a fine-tuned BERT classifier (33M parameters). Two specialized worker agents handle the actual work: a data presenter for descriptive analytics and an insight generator for diagnostic analysis.

The design is deliberately pragmatic where an autoencoder-based OOD detector achieves 0.969 precision in under 0.01 seconds, compared to 0.616 precision and 1.67 seconds for LLM-based few-shot detection. The BERT router hits 0.83 accuracy in 0.31 seconds versus 0.60 accuracy and 2.14 seconds for an LLM classifier.

Instead of text-to-SQL, the system uses an API-based data model that decomposes queries into granular steps matched to internal data APIs. This divide-and-conquer approach avoids the syntax errors and hallucinations common in SQL generation while maintaining high retrieval accuracy.

End-to-end, the system achieves 89.5% question-level accuracy based on human evaluation across relevance (0.977), correctness (0.958), and completeness (0.993), with P90 latency under 15 seconds.

This shows how combining lightweight specialized models for routing with LLMs for reasoning and generation delivers a production-grade agent system that prioritizes accuracy and latency over architectural complexity.

Paper: https://t.co/eQ7DCvkffX

Learn to build effective AI agents in our academy: https://t.co/JBU5beIoD0

[Robert Youssef @rryssf]

🤖 I finally understand the fundamentals of building real AI agents.

This new paper “Fundamentals of Building Autonomous LLM Agents” breaks it down so clearly it feels like a blueprint for digital minds.

Turns out, true autonomy isn’t about bigger models.

It’s about giving an LLM the 4 pillars of cognition:

• Perception: Seeing and understanding its environment.
• Reasoning: Planning, reflecting, and adapting.
• Memory: Remembering wins, failures, and context over time.
• Action: Executing real tasks through APIs, tools, and GUIs.

Once you connect these systems, an agent stops being reactive it starts thinking.

Full thread 🧵

Paper: arxiv. org/abs/2510.09244

[elvis @omarsar0]

Very cool work from Meta Superintelligence Lab.

They are open-sourcing Meta Agents Research Environments (ARE), the platform they use to create and scale agent environments.

Great resource to stress-test agents in environments closer to real apps.

Read on for more:

[DAIR.AI @dair_ai]

Top AI Papers of The Week (July 14 - 20):

- Agentic-R1
- Context Rot
- Scaling up RL
- A Survey of AIOps
- Chain-of-Thought Monitorability
- One Token to Fool LLM-as-a-Judge
- A Survey of Context Engineering for LLMs

Read on for more:

[Eugene Yan @eugeneyan]

Wrote an intro to evals for long-context Q&A systems:
• How it differs from basic Q&A
• What dimensions & metrics to eval on
• How to build llm-evaluators
• How to build eval datasets
• Benchmarks: narratives, technical docs, multi-docs

https://t.co/XAzPcG7tvf

[tancnle @tancnle]

Excellent write-up on automatic prompt optimisation techniques https://t.co/PECTQdSsn4 from @cwolferesearch

[MetaGPT @MetaGPT_]

Introducing MetaGPT's Data Interpreter: Open Source and Better "Devin".

Data Interpreter has achieved state-of-the-art scores in machine learning, mathematical reasoning, and open-ended tasks, and can analyze stocks, imitate websites, and train models.

Data Interpreter is an autonomous agent that uses notebook, browser, shell, stable diffusion, and any custom tool to complete tasks.

It can debug code by itself, fix failures by itself, and solve a large number of real-life problems by itself.

We open-source our code and provide a wealth of working examples to give everyone access to state-of-the-art AI capabilities.

📝 Paper: https://t.co/VcoeKc6A8S
🔗 Examples: https://t.co/mu3iULDZko
📚 Repo: https://t.co/BMJxhVwzn6
📖 How to use: https://t.co/CviB6jvQ49

#MetaGPT #github #interpreter #opensource

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

The foundation series by MosaicML, including MPT-7B/30B (and an efficient training repo), makes high-quality pre-trained language models available to anyone for commercial use. Given that creating a pre-trained base model is incredibly expensive, these open-source tools enable a wide variety of specialized language model use cases to be explored at a reduced cost.

Some background. All language models are created using a common framework with a few simple components, including pre-training, refinement (SFT and RLHF), and application. Although this framework has several steps, the first step (pre-training) is arguably the most important. Creating a more powerful base model via extensive, high-quality pre-training enables better results when the LLM is refined and applied. Put simply, the base model is a common starting point for all LLM applications, so improving the base model benefits the entire application.

Open-source base LLMs. Until recently, open-source base models either performed poorly (e.g., OPT or BLOOM) compared to their proprietary counterparts or could only be used for research (e.g., LLaMA). This changed with the release of MPT-7B and MPT-30B by MosaicML, which are open-source, performant, and commercially usable. Plus, these models can be fine-tuned (at a low cost) using MosaicML’s LLM foundry, allowing AI practitioners to easily specialize these models to a variety of different use cases.

What makes these models so good? There are a variety of factors that make MPT-7B/30B impressive, but the most notable considerations are enumerated below.

- Use of the GPT-NeoX tokenizer (better handles whitespace for code)
- Pre-trained over a lot of data (1T tokens in total)
- Use of a modified architecture (low precision layer norm, flash attention, ALiBi, and more) that enables faster training/inference and extrapolation to long context lengths
- MPT-7B is fine-tuned with context lengths as large as 64K tokens
- MPT-30B base model is trained using a longer context length of 8K tokens (compared to 2K for most open-source LLMs)
- MPT-30B places an emphasis on code in its pre-training data, making the resulting LLM especially good at coding applications

MPT performance. MPT models are evaluated extensively and compared to other popular, open-source LLMs. MPT-7B is found to be comparable to LLaMA-7B, and both of these models are significantly better than any prior open-source LLM of this size. MPT-30B achieves comparable performance to GPT-3. It is outperformed by models like LLaMA-30B and Falcon-40B on text-based tasks but tends to excel in code-based tasks.

TL;DR: The foundation models provided by MosaicML are a huge step forward for the open-source LLM community, as they provide commercially-usable LLMs that are comparable to popular base models like LLaMA and GPT-3. The MPT-7B and 30B models come with an entire ecosystem of open-source tools that can be used to create specialized/personalized LLMs, thus providing a starting point for solving a variety of downstream applications.

[DY @DYtweetshere]

We're releasing our @scale_AI hackathon 1st place project - "GPT is all you need for backend" with @evanon0ping @parkerjou

But let me first explain how it works: