Munim Thahmid

[Tianyin Xu @tianyin_xu]

Thanks for the shoutout to TLA+ @bcherny !

As suggested by Markus Kuppe, let me showcase the brilliant work done by colleagues at Specula and the TLA+ community. While Specula (https://t.co/hoV6UJIybf) can effectively model system code in TLA+ (and then use model checkers to find hundreds of deep bugs), we can further prove the correctness of system specs using TLAPS. The TLAPS-Bench (https://t.co/hrYHc4DSow) project aims to offer a proof service for any TLA+ spec of important systems/protocols. Basically, Specula writes the spec; TLAS-Bench generates the proof.

AI writes amazing proofs! We had to retire all Proof-Completion tasks in TLAPS-Bench as they're too easy for frontier AI. We also proved classic protocols and algorithms like 2PC, Paxos, TCP, etc.

However, proving low-level specs of real-world system code _from scratch_ is still non-trivial. @qiancheng9788 ran a subset of TLAPS-Bench using Claude Code w/ Opus 5. The success rate is merely 51.4%. @Muse is great, but Muse Spark 1.3 can only achieve 16.7%.

Ruize has a bag of tricks to push AI agents. Once he pushed Codex to prove one of the nine invariants for the ZooKeeper implementation. Codex struggled 5 days, burned $2000, and did it correctly. We ran out of money to continue. And, @xu_dong_sun shows us that liveness proofs are still difficult for AI without guidance. We have an exciting avenue, but still a lot of work to be done.

If anyone wants to verify their protocols/systems, send us some tokens -- we can do it in TLAPS-Bench. @HacksonClark tells me that people are excited about long-horizon tasks -- TLAPS-Bench may be a perfect benchmark for that.

@bcherny Consider further tuning Claude for TLA+, you may find more bugs and/or prove the correctness of SDK code more efficiently.

[Jackson Clark @HacksonClark]

Super excited to be presenting https://t.co/HdYZGstPVj at @SnorkelAI's Frontier Data Summit, Oct 8 in SF. Can AI really solve production issues? Come find the poster. Request an invite: https://t.co/drAHbW8nOU

[Jackson Clark @HacksonClark]

See you all in SF! I will be presenting https://t.co/R721iEN3kt :)

[Michael Black @Michael_J_Black]

What is the role of academic computer vision research in the age of increasingly powerful large models? Is GPT-6 Astra a step change? How can a researcher have an impact today in academia?

These are the questions I ask myself as I head off to ECCV 2026, a conference I’ve attended since 1992. One of my papers this year is VIGA, a method that takes an image as input and outputs a 3D Blender scene that represents that image. This is a classical inverse-graphics task and VIGA was the first method to solve it using an agentic approach.

The idea is now several years old and the first version of the paper was rejected. This delayed publication significantly. After it was accepted at ECCV, it was quickly surpassed by people using Claude Code for the same purpose. Today GPT-6 Astra blows away all previous results. But we still head off to ECCV to tell the community about our invention that is now fully out of date.

The way academic work often progresses is that one reads recent papers, notices that they have limitations, comes up with a new idea, explores this, publishes it, etc. Any published paper I read today is based on ideas that are at least a year old. And those ideas were based on the literature of the time, which was also a year old. That means that any paper I see at ECCV is likely two years out of date. In AI today, two years means your work is likely irrelevant.

At CVPR this summer I noticed that many authors have not gotten the message. They continue to work on “old” problems that have a long history. This history is based on assumptions about how the “vision problem” will be “solved”. The truth is that it is being solved in a very different way and many of these problems are no longer relevant. Another group of papers focuses on very niche problems where large models likely fail because of insufficient data or lack of business interest. The impactful papers were largely from industry and had long author lists and massive data+compute behind them. These papers were also out of data, describing systems that had been released months before, but at least they served to provide the community with more complete documentation and analysis of commercial systems.

So what should academics do? First, we need to put aside the tools we’ve used for years and start from scratch. Every project should start by trying really hard to solve the problem with existing tools. I would like to see every paper begin with a detailed experimental analysis of how existing models perform and why they fail (if they do). This gives the kind of insight we need today. Then, assuming current models fail, the solution should provide some fundamental insight that will outlive the next release of such models.

Reviewers today still focus on technical novelty. This pushes people to focus on tweaking architectures rather than clearly moving the field forward. Papers need to be judged based on their novel insight and not their novel technical contribution. This is a real shift in thinking but it focuses us on what matters - progress of the field.

If we want there to be a “field” of computer vision, then it can’t become a marginal backwater, focusing on esoteric problems. If you haven’t tried using Astra (or whatever comes next) to solve your problem, then you have not done your homework. This omission should be seen as negatively as not having a previous work section.

Concretely, I think papers should include a new section analogous to “Related Work” where that related work is current models and how they perform on the task. Reviewers should start asking for this and expecting authors to be able to articulate their insights about the limitations of existing large models.

I'm interested in your thoughts.

[OpenAI @OpenAI]

We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.

The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.

The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.

[Tianyin Xu @tianyin_xu]

Yeah, great time for reliability research! The scope of reliability is significantly expanded from model reliability, to agent reliability, to (vibe) code reliability, and their interactions in end-to-end systems, not to mention the opportunities of leveraging AI to attack some of of the decades-long reliability challenges in large-scale systems.

What's interesting about AI is that reliability is no longer about edge cases, but a foundation to have functional, stable systems.

[Tianyin Xu @tianyin_xu]

Glad to see more and more folks like @cerebral_system are using #SREGym to evaluate SRE/Prod agents. It's a great sigh that folks are more serious in building useful agents than claiming fake victories.

Results on low-fidelity benchmarks that do trivial fault injections (e.g., flipping a feature flag in AS) is honestly like the emperor's new clothes. We all know that prod issues are never alike.

Fidelity is arguably the hardest problem in SRE benchmarks. #SREGym doesn't have a perfect solution (there's no free lunch), but pushes very hard on it while balancing cost and reproducibility.

Kudos to @HacksonClark @Yiming_Su3 @SaadMRP @lilygn6 @MunimThahm81566 and many other students who treat eval as a way of understanding the problem, not playing the game.

[Tianyin Xu @tianyin_xu]

We released a beta version of Specula, a tool that automatically checks system code using TLA+ based formal methods. Specula instructs coding agents to fully automate formal specification (model + invariants) and mode-code conformance, which were major barriers to adopt formal methods in practice. We find Specula rather useful in finding deep bugs that require formal reasoning. (Specula found hundreds.)

If you need automatic formal reasoning of your code, give it a try. (The tool is push-button.)

Code: https://t.co/gcqFUHYGL6
Paper: https://t.co/4W2p1yX5ia

(While I'm posting it, the students, especially @qiancheng9788 and @SaadMRP, and Ruize Tang did all the hard technical work.)

CC @disalg_spec @Yiming_Su3

[Andrej Karpathy @karpathy]

I packaged up the "autoresearch" project into a new self-contained minimal repo if people would like to play over the weekend. It's basically nanochat LLM training core stripped down to a single-GPU, one file version of ~630 lines of code, then:

- the human iterates on the prompt (.md)
- the AI agent iterates on the training code (.py)

The goal is to engineer your agents to make the fastest research progress indefinitely and without any of your own involvement. In the image, every dot is a complete LLM training run that lasts exactly 5 minutes. The agent works in an autonomous loop on a git feature branch and accumulates git commits to the training script as it finds better settings (of lower validation loss by the end) of the neural network architecture, the optimizer, all the hyperparameters, etc. You can imagine comparing the research progress of different prompts, different agents, etc.

https://t.co/YCvOwwjOzF
Part code, part sci-fi, and a pinch of psychosis :)