Denis Sevostyanov

[German Magai @MetatrolN]

AI4Math research focuses on Lean autoformalization and theorem proving. But how do we come up with new conjectures?

In our work with @fin_presented, we study how well can LLM agents use computation to create hypotheses and solve research-level mathematical problems?

arXiv: https://t.co/9xeEVSyFkW

We evaluate 15 LLMs in zero-shot and SageMath-augmented agentic setup on RealMath 133 problems extracted from arXiv mathematical papers. Key findings:
- Tool access improves every model, by 9.7 pp on average. Open-weight models gain 15.3 pp, compared with 6.5 pp for closed frontier models, narrowing the gap between them.
- Most intriguingly, a CAS-augmented agent reproduced a computational mathematician’s workflow: computing intermediate objects, finding patterns, forming conjectures, recovering from errors, and validating formulas across parameters (see the details in the case study).
- Gains vary: #Qwen 3.7-Max rises from 42.1% to 69.9%, a gain of 27.8 pp that brings it close to frontier performance. #Kimi 2.7 gains only 1.5 pp.
- Tool-use behavior is strongly bimodal. Strong agents usually finish in 3-4 tool turns, while weaker agents often exhaust all tool budgets.
- The largest gains are in combinatorics (+18.7 pp) and rings and algebras (+10.7 pp), while algebraic topology and group theory remain difficult.
- Recovery after a failed tool call ranges from 16% (#Sonnet-5) to 77% (GPT-5.5) across models. The ability to revise a strategy after receiving computational feedback separates effective agents more clearly than the raw number of errors.

Interesting observations about some models:
- #GPT 5.5 leads in both solve rate and efficiency, reaching a 75.2% accuracy with the lowest token usage among tool-enabled agents.
- #MiniMax M3 is the least efficient, using the most tokens per problem and achieving substantially lower accuracy.
- #Opus 4.8 exceeds Opus 4.7 by only one solved problem.
- #Grok 4.3 shows one of the worst results and produces 248/336 SyntaxErrors🥲.
- #Fugu Ultra shows the smallest increase in token usage with tool access, at 4.5×, averaging 70k tokens per problem.

Today we are presenting our poster at the @ai4mathworkshop at @icmlconf. Come by to discuss our work.

#ICML2026 #AI4Math #AI #Agentic #LLM #Mathematics

[Denis Sevostyanov @newmrdenis]

@FedEx, today I am going to the police. Because your workers have stolen my parcel. That is it. Period.

[Denis Sevostyanov @newmrdenis]

@FedExHelp Human, did you read what I wrote? YOUR LINK IS NOT WORKING!

[Denis Sevostyanov @newmrdenis]

@FedExHelp I have been chatting with you for three weeks. You have all info in DM with the account from where you wrote this reply. YOUR LINK IS NOT WORKING. ARE YOU TROLLING ME !??

[Denis Sevostyanov @newmrdenis]

@FedExHelp @FedExHelpEU @FedEx, are you kidding me? Is there any way to obtain my shipment? I wrote thousands of times to all your customer support services no answer anywhere! I have even DM'd @FedExHelpEU. Are you gonna do your job?

[Denis Sevostyanov @newmrdenis]

@FedExHelpEU, your Georgian branch is absolutely awful. Please, help me. So, according to my list, I cannot obtain my shipment due to nothing. I need to pay a border tax, but I cannot contact them -- all numbers are either down or being hung up on me. No answer via email too