nic lane

i invent & study efficient and scalable ML systems @camlsys; @raengnews chair prof. @cambridge_cl, fellow @stjohnscam, co-founder & CSO @flwrlabs (YCW23)

[nic lane @niclane]

Yesterday, we introduced Lizzy 7B, a UK-built open-weight frontier language model. The first SOTA LLM by @flwrlabs, now available in preview from @huggingface.

For all the momentum in AI globally, there has been a surprising gap much closer to home. The UK has world-leading research, strong institutions, and deep technical talent, yet until now it has lacked a state-of-the-art, sovereign, open-weight language model built for its own ecosystem. That absence has limited how organizations can deploy AI in ways that are fully aligned with local context, regulation, and infrastructure. Lizzy is designed to address that gap directly.

Developed entirely in the United Kingdom, Lizzy 7B has been built from the ground up to operate within UK-specific environments. From training through evaluation, it reflects local language, institutions, and domain-specific use cases. This results in a model that is not only capable in general terms, but meaningfully aligned with how AI is actually used across sectors such as financial services, public infrastructure, and government systems.

In benchmarking, Lizzy already demonstrates top performance among European open-weight models in its class, matching or exceeding the previous best European sovereign models such as Apertus 8B and EuroLLM 9B. It also introduces targeted evaluations focused on factual accuracy and stylistic alignment within UK contexts, combining strong general capability with local relevance.

Lizzy model weights are available, enabling immediate testing, integration, and deployment. It is compatible with modern inference frameworks and supports a wide range of infrastructure environments, allowing organizations to retain full control over their data, compute, and compliance requirements.

Lizzy builds on years of research across the full AI stack, including work published at ICLR, NeurIPS, and MLSys. It draws from advances in model architectures, optimization, and distributed training across heterogeneous systems, enabling scalable development at a lower cost.

This preview marks the beginning of a broader effort at Flower Labs to advance sovereign AI capabilities, starting in the UK and expanding across Europe and beyond. The aim is to enable organizations to build and deploy high-performance AI that is not only powerful, but aligned with their own environments and needs.

Lizzy 7B marks a milestone for UK AI sovereignty and for Flower Labs, as its first state-of-the-art frontier model made widely available.

[Andrej Jovanović @itsmaddox_j]

Excited to share LoRDO is accepted at ICML 2026! 🇰🇷

LoRDO unifies low-rank optimization with local updates, achieving near-parity with DDP while reducing communication overhead.

Details & collaborators in the thread below 👇 See you there!

[Chong Shen Ng, PhD @chongshenng]

Evals for agentic workflows are challenging in enterprise deployments, and existing open benchmarks lack domain-specificity. Introducing FlowerBench, our product to address this for enterprises and ensures evals are grounded in enterprise-relevant context.

[nic lane @niclane]

It took me only 18 years, but I’ve finally secured the handle @niclane on X.

[nic lane @niclane]

@liquidated0x Cheers!

[nic lane @niclane]

Building a Stargate for Agent Evals: The Flower Enterprise Evaluation Network

The next step in agent evals is not another public simulation of enterprise work. It is safe agent evaluation inside a network of private environments where the work actually happens.

Over the last six months, we have been working with a small group of opt-in Flower users to build the Flower Enterprise Evaluation Network. The goal was to test a different approach to enterprise agent evals, one that measures agents on real long-horizon enterprise tasks, inside the environments where that work already happens.

This is where I think the field needs to move. Most agent evals are still too far from the work enterprises actually care about. They are useful, but they often make tasks portable by removing the things that make them valuable. Proprietary data disappears. Internal tools become simplified interfaces. Domain rules become generic instructions. Human handoffs are removed. Private files are replaced with public substitutes. Strict deliverable requirements become softer and easier to grade. The result is a benchmark that scales, but the task no longer feels like the work.

Enterprise workflows are different. They are embedded in the organization. The benchmark problem is therefore also a privacy and IP protection problem. The most realistic tasks depend on proprietary data, internal tools, domain rules, human handoffs, and strict deliverable requirements, which makes them both valuable and hard to share.

This is why we have developed FlowerBench that summarizes results from our live evaluation network. It represents a completely different path. Instead of moving sensitive work to the eval, the eval runs where the work already lives. Private files, datasets, tools, and internal context stay in place. Only sanitized results are shared outside the organization. Those results then contribute to a broader enterprise evaluation network. This follows the same principle behind Flower more broadly, which is to move computation to the data rather than data to the computation.

WHAT WE LEARNED
Perhaps not surprisingly, the first thing we learned when building the Flower Enterprise Evaluation Network is that realistic long-horizon enterprise work is still hard. Looking across a wide range of agent/model combinations deployed, the average completion score hovers at 0.44. Many capable agents are still completing less than half of the benchmarked work correctly when the task is long, private, multi-step, and strictly verified. Beyond completion, FlowerBench tracks also runtime, token usage, and cost across real enterprise workflows.

Our results also show why a single score is not enough. Claude Code with Claude Opus 4.8 scores 0.66 in 68m 01s at a reported cost of $15.30. Claude Code with Claude Opus 4.7 scores 0.56, but uses more than 40M tokens. OpenCode with Qwen3.6 Plus scores 0.49 at $1.19. Gemini CLI scores 0.45, Kimi CLI scores 0.44, and several Qwen-, Kimi-, and Nemotron-based systems cluster below that. The interesting signal is not only the ordering. It is the trade-off surface across score, runtime, tokens, and cost.

Higher-scoring systems are not always the fastest or cheapest. Claude Code can produce polished end-to-end work when aligned, but may stop early or miss verifier requirements. Gemini CLI, Co-Pilot, and Terminus2 with Nemotron expose different speed and cost profiles, but often lose points on later-stage validation or completeness. Qwen-based approaches show that lower-cost systems can be competitive, but reliability depends heavily on the agent harness around the model. Kimi-based systems can be cost-efficient and strong on individual domains, while still being slower and less consistent end to end.

One lesson is that we should stop treating agent evaluation as only a model-ranking problem. For agents, the system is the unit of evaluation. The model matters, but so do the harness, tool interface, context policy, memory design, retry strategy, execution environment, verifier, and cost envelope. A weaker model in a better scaffold can outperform a stronger model in a brittle one. A cheap run that fails final validation may not be cheap if a human has to audit and repair the whole chain. A more expensive run may be economically sensible if it completes a high-value workflow with less supervision.

Another lesson is that failure location matters. It is not enough to know that the agent failed. We need to know whether it misunderstood the task, used the wrong context, called the wrong tool, produced a broken intermediate artifact, lost a constraint, stopped early, or failed a strict verifier at the end. Those are different failure modes, and they imply different interventions. Some are model capability problems. Some are harness problems. Some are environment problems. Some are eval design problems.

WHAT IS DIFFERENT ABOUT FLOWERBENCH
FlowerBench is built around the idea that the environment, broadly defined, is the critical part of the benchmark. A conventional benchmark usually has to make the task portable. FlowerBench keeps the task close to the original workflow. The public benchmark page describes long-horizon tasks that measure agent performance on real multi-step enterprise workflows. These tasks can at times even run inside private organization environments with access to proprietary context, internal knowledge, and tools, while only performance metrics are shared externally. The benchmark spans domains including finance, healthcare, insurance, operations, MLOps, legal, and marketing.

The Flower Enterprise Evaluation Network is the infrastructure layer that makes this possible. It is a privacy-preserving, opt-in layer for connecting private task environments from different organizations into a shared evaluation network. Organizations keep proprietary work inside their own environments, while still producing measurable and comparable results on real workflows under real constraints.And this important signal is something they can then also hill climb on towards better internal agents.

That approach suggests a different standard for the field. We need to improve access within eval to safely offer signals from private context and real tools where possible. We should report scores together with runtime, token usage, cost, and failure modes. We should distinguish model capability from agent-system quality. We should make privacy a first-class design constraint, because the most valuable enterprise tasks are often the least shareable. This is the broader lesson from building the Flower Enterprise Evaluation Network. Agents will not move forward only through larger models. They will move forward through better data, better evals, and better environments. The eval should look more like the work itself.

Links to more details about FlowerBench and the Flower Enterprise Evaluation Network in the thread below.

[nic lane @niclane]

@Timur_Yessenov Yes you're right; although so far we have found the most important thing is knowing if real improvement is happening -- even if this comes at the expense of ease of cross org comparisons.

[nic lane @niclane]

Full details about the The Flower Enterprise Evaluation Network infrastructure https://t.co/rBXWHdWS4b

[nic lane @niclane]

Launching this week -- Flower Bench https://t.co/tGeT16MAqa

[Zachary Lipton @zacharylipton]

AI companies complaining about distillation is the single greatest act of hypocrisy in the history of humanity.

[Andrej Jovanović @itsmaddox_j]

🚨 Announcing the Red Queen Gödel Machine (RQGM)!

Building on prior self-improving agent frameworks, we let the agents and the evaluators that score them evolve together.

TL;DR: higher-quality coding agents for a substantially reduced token budget.

(1/n) 🧵

[nic lane @niclane]

@rohanpaul_ai AMA folks about RQGM

[Rohan Paul @rohanpaul_ai]

New paper from Cambridge Univ+NVIDIA and other top labs teaches AI agents and AI judges to improve together, so neither side gets stuck.

Moves self-improving AI away from fixed benchmarks and toward a loop where the thing doing the judging can also get better.

The problem is that most self-improving agents train against a fixed benchmark or fixed evaluator, so the score can become stale, too easy, or easy to game.

The paper’s idea is to let the evaluator improve too, but only at safe handoff points, so each training stretch still has a stable judge.

During each stretch, agents are tested by the current frozen evaluator, while possible better evaluators are tested separately against held-out human or objective answers.

The authors try this on coding, paper writing, paper reviewing, proof writing, and proof grading, where some tasks have clear answers and others need learned judgment.

On coding, the system beats the earlier best self-improving coding agent while using 1.35× to 1.72× fewer tokens, because a cheap code reviewer adds useful feedback.

On paper writing, the co-evolved writer gets about 1.86X higher average acceptance from a reviewer panel than the fixed-evaluator baseline.

The big point is that stronger AI systems may need stronger judges growing with them, because fixed tests can stop giving useful pressure.

----

Link – arxiv. org/abs/2606.26294

Title: "The Red Queen Gödel Machine: Co-Evolving Agents and Their Evaluators"

[Alex Iacob @Alex__Iacob]

Excited to share that our work, LoRDO, has been accepted at ICML 2026.

It shows how foundation model training can work with much less synchronization and lower optimizer-state memory. 📄 Paper link at the end.

[nic lane @niclane]

@badguyty @itsmaddox_j @Alex__Iacob @williamfshen @DBBurkhardt @meghdadkurmanji @lorenzosani97 @NickVenanzi @AmbroiseOdonnat @__cao_jamie__ @xinchiqiu @CaMLSys @nvidia @flwrlabs @mbzuai @Inria It’s coming

[nic lane @niclane]

LoRDO paper, catch it at @icmlconf!
https://t.co/G4i9LNpoz6

[nic lane @niclane]

Mixed-generation wide-area GPU training just got one big step closer to being practical.

New @icmlconf paper led by @camlsys and @flwrlabs, with @Cambridge_Uni, @RedHat_AI, ISTAustria and @LancasterUni.

LoRDO shows how foundation model training can work with much less synchronization and lower/mixed optimizer-state memory. 🧵

[nic lane @niclane]

Come see it at ICML this Wednesday. Andrej Jovanović, @itsmaddox_j, is presenting

🗓️ Wednesday, 8th July
Poster Session 3
10:30 AM – 12:15 PM KST
Hall A #3705

LoRDO Team
@itsmaddox_j @Alex__Iacob Mher Safaryan Ionut-Vlad Modoranu @lorenzosani97 @williamfshen @xinchiqiu @DAlistarh @niclane7