Can frontier AI models safely do stuff for us in crypto?
I humbly suggest you find out here, rather than after you've yeeted your funds in their direction:
Can you trust an agent with your keys?
Join a live discussion about what agents still get wrong and how you can assess the risks before trusting them.
- Guests to be announced...
- The discussion will include OnchainBench, the new benchmark for agents in crypto (from core contributor to Olas, @valoryag).
- Learn about the attacks agents fell for, what tasks cost, and the tasks protocols want tested next.
Thursday, November 26 at 5:00 PM UTC, broadcast live at this link: https://t.co/CnT9Y7XTti
Can you trust an agent with your keys?
For years my team and I have been building AI agents in crypto. This question came up a lot. And yet, there wasn’t a convincing answer.
We set out to change this.
We take the most commonly used AI agents (harnesses), pair them with different models, give them a wallet and then expose them to hundreds of tasks.
The result: a quantitative answer to the question that sits at the centre of crypto and AI.
Both frontier stacks (Claude Code on Opus, Codex on GPT-5.6 Sol) missed our safety bar; the same harnesses on DeepSeek met it.
Check it out and let us know what you think! Live now at https://t.co/bnwGlXAJ8X
It’s a work in progress and we hope to get many new submissions from the crypto and AI ecosystems alike.
I went full-time in crypto in 2021, desiring a world where our wealth and livelihoods weren't at the mercy of faceless intermediaries.
Even then, we felt that agents could help us gain control, as well as free up our time.
But, today the majority of AI agents are just as faceless and scary to trust as traditional finance was. Perhaps more!
That's why I'm thrilled to present this open way for people to see how AI agents actually perform. I hope it helps us assess how far to trust them *before* yeeting our money.
A world where agents free up our time without risking (and instead by building) our wealth, requires open standards.
Check it out and help us grow the benchmark here:
Today we are launching OnchainBench, the benchmark for agents in crypto.
It helps you decide how far you can trust an AI agent with your keys: how often it finishes the job, what tasks cost, and how easily it falls prey to an attack.
People are starting to let AI agents swap, lend and even trade for them. The bigger the task, the greater the risk you're taking by trusting the agent. So before you hand it your keys, you need to assess it.
There are many risks to assess at once. An agent could buy two tokens for a liquidity position but never add them to the pool, so the job isn't complete. It could spend more on AI model calls than the task is worth. And, onchain, there's no undo: agents read web pages, files and tool descriptions that anyone can write, and if one of them tricks the agent into sending funds to the wrong address, the money is gone. Until now, there was no common way to evaluate agents on all of this and more.
OnchainBench helps you compare agents by giving them the kind of work people already ask them to do: swaps on @Uniswap, @SushiSwap and @Honeyswap, lending on @aave, deposits into sDAI from @SkyEcosystem, prediction markets on @Omen_eth, research, and jobs that run over several days.
Each task runs through a @safe smart account on a copy of a public chain. The first version covers @ethereum, @gnosischain, @0xPolygon, @arbitrum, @Optimism, @base, @Celo, @modenetwork and @robinhoodapp. The benchmark emulates the kind of contracts, liquidity and prices the agent would meet with your money, without putting any of it at risk. A task only counts as done if the chain shows it was done. Some tasks also hide an attack in what the agent reads, such as a look-alike token or a fake "verification" request, so you can see how often agents get tricked into moving funds before you trust one with yours.
In the first run, Claude Code (@claudeai), Codex (@OpenAI), Hermes (@NousResearch), OpenClaw (@openclaw) and our own new OCB agent on @autonolas worked through 147 private tasks they had never seen. They completed between 88.4% and 96.2% of tasks, at about $0.03 to $0.35 per completed task. Agents lost funds to an attack 13 times in 1,793 attack tests.
We are excited to contribute this to the crypto ecosystem. If you build a protocol or a chain, agents are already among your users, and will likely be one of your biggest sources of growth. OnchainBench shows, protocol by protocol and chain by chain, how well they handle yours.
For the AI ecosystem: want an agent eval with real ground truth and real adversaries? Onchain, every result can be checked and every injected instruction is a live risk. OnchainBench is open to contributions, uses private tasks, and is ready for your agent.
Help us make onchain agents harder to trick and better at the job. If you run a protocol or are a heavy user, propose the task you think agents still get wrong on the site below, and every agent on the benchmark could end up tested on it. If you build agents, run OnchainBench yourself and submit your agent to be run.
The leaderboard is live now at https://t.co/xs5WV4paUy
Share it with a friend!
Crypto is a useful proving ground for AI agents & how far we can trust them before letting them loose on our important tasks, data and funds. Every action is public, and there's no undo.
If agents can show they're trustworthy where mistakes are final, that tells us a lot about where else we can consider letting them act for us.
My hot take when talking to people across the AI use-hate spectrum is: focus less on opinions being right/wrong & more on how wild the stratification is
Particularly at this early stage of adoption, the person's opinion on AI is telling us about so much more than just the person's opinion on AI
Running several agents in Pearl can mean coming back to start each one by hand.
Auto-run is designed to start them one after another: when an agent has done its work for the day, the next one gets its turn.
Pearl Connect can now take part too. It runs for its set minutes, then hands over.
Turn on Keep Device Awake in Settings to help keep your computer from sleeping while Auto-run is on.
See how it works, then try it in Pearl: https://t.co/1c9SEk85VR
The TLDR is that I do not want people living in misplaced fear that, because a scary person made them feel they had to sign a piece of paper, that piece of paper rules their life.
It might actually mean a lot less than you think.
Here, I talk through eight common things I wish people knew about legal agreements & NDAs, including real examples I've seen: https://t.co/mavhrhFOsL
I hope this can help you figure out the options you have, even if you thought you couldn't tell anyone!
Free help and advice is available in the UK here:
- @acasorguk 0300 123 1100 (work)
- @CitizensAdvice 0800 144 8848 (most things)
- @rightsofwomen 020 7490 0152 (harassment at work) / 020 7251 8887 (criminal law)
- @WhistleUK 020 3117 2520 (whistleblowing)
- @RapeCrisisEandW 0808 500 2222
- @samaritans 116 123
And please, in the words of @caitlinmoran quoted in full in the piece, get yourself some flowers. 💐
As a survivor of sexual assault, former activist in this area, and someone who's read many, many legal agreements (representing many millions of dollars), I couldn't fall asleep last night for thinking about this.
Lucy Beaumont was scared a young girl would take her own life, having signed an NDA where she wasn't allowed to talk to her friends, family or therapist (Walliams allegations, which he denies).
It made me so angry. Here's what I want people affected by NDAs - and really anyone making an agreement - to know 👇
Sky's @cathynewman speaks to comedian Lucy Beaumont about the allegations against David Walliams, her concerns for young women in the TV industry and the need for better safeguards to protect vulnerable people.
Watch in full: https://t.co/EUhQs7n8hl
Exactly. It also took a two and half year investigation, giant documentary and call outs from people in the industry publically to even get to that point.
@autonolas@valoryag It's so exciting to see our models perform so well compared to the latest centralized models.
AND they're actually yours: open, much cheaper to run, and on one GPU!
It was great to see all our learnings and can't wait for what's next 👏
Every time a prediction market settles, Olas agents find out whether they were right.
That’s the training data behind Olas Predict. Our new 27B model makes 22% fewer forecasting errors than our September one.
And we're just 1.2 points behind much larger, generalized GPT-5.6 model...
It's so exciting to see our models perform so well and they're actually yours: open, much cheaper to run, and on one GPU.
What we learned is in the post. What’s next… you’ll have to guess.
A lean, open-weight model that comes within 1.2 points of GPT-5.6 at forecasting real-world events.
Core contributor @valoryag trained Olas-Predict-Qwen3.8-27B, an open-weight forecasting model built on Qwen3.8-27B, using reviewed forecasting examples and reinforcement learning. It follows Olas-Predict-R1-14B, Valory’s first Olas Predict model.
In Valory’s evaluation on 6,824 prediction markets created after GPT-5.6’s training cutoff, with the same prompts and evidence for both models:
• Balanced accuracy (higher is better): 67.3% vs 68.5% for GPT-5.6
• Brier error (lower is better): 0.2124 vs 0.2029 for GPT-5.6
• Against Olas-Predict-R1-14B: 22% lower Brier error
It gets this close from a starting point that scores 15.6% on the AA-Omniscience factual-knowledge benchmark, against about 59% for GPT-5.6.
Most of the gain came from the stronger starting model. Valory’s domain training added 5% lower Brier error and 12% lower calibration error on top.
Next, Valory aims to close the remaining gap in future training experiments.
Read the results: https://t.co/2h7t2jIvT3
Not every experiment delivers the result you hope for.
We had good reasons to believe our forecasting fine-tuning recipe would also work on a much larger model. It didn't.
Sharing negative results is just as important as celebrating successful ones.
We learned something, and this gave us a more focused idea of what to test next. 🔬
Does training a larger model mean better results?
@valoryag says not necessarily:
- The architecture matters as much as the size.
- Check for movement early.
- Fix the output format before training on it.
- A negative result is still a result.
What does this mean for you? Read the full blog post:
Bigger wasn’t better this time.
Core contributor @valoryag fine-tuned NVIDIA’s Nemotron 120B on forecasting data from Olas agents, using the recipe behind Olas-Predict-R1-14B: selected reasoning examples, a supervised warm-up, then reinforcement learning. Training ran through @thinkymachines’ Tinker.
On 11,496 held-out predictions, Nemotron showed no clear improvement. Before → after training:
• Brier error (lower is better): 0.265 → 0.267
• Balanced accuracy (higher is better): 59.7% → 59.4%
• Unreadable answers (fewer is better): 503 → 528
Our trained 14B model had better calibration, higher balanced accuracy and just 10 unreadable answers. Nemotron retained a slightly lower Brier error: 0.267 versus 0.270.
Why didn’t the recipe transfer? Nemotron’s mixture-of-experts design may need more varied data or different training settings for forecasting. We haven’t tested those explanations, so the cause remains open.
What we learned: our existing recipe and training budget improved the 14B model but brought no clear gain to Nemotron.
Next, we’ll test a larger dense model, with an early check for improvement before committing to the full run.
Read the experiment and results: https://t.co/ZSziFiraIZ
Bigger wasn’t better this time.
Core contributor @valoryag fine-tuned NVIDIA’s Nemotron 120B on forecasting data from Olas agents, using the recipe behind Olas-Predict-R1-14B: selected reasoning examples, a supervised warm-up, then reinforcement learning. Training ran through @thinkymachines’ Tinker.
On 11,496 held-out predictions, Nemotron showed no clear improvement. Before → after training:
• Brier error (lower is better): 0.265 → 0.267
• Balanced accuracy (higher is better): 59.7% → 59.4%
• Unreadable answers (fewer is better): 503 → 528
Our trained 14B model had better calibration, higher balanced accuracy and just 10 unreadable answers. Nemotron retained a slightly lower Brier error: 0.267 versus 0.270.
Why didn’t the recipe transfer? Nemotron’s mixture-of-experts design may need more varied data or different training settings for forecasting. We haven’t tested those explanations, so the cause remains open.
What we learned: our existing recipe and training budget improved the 14B model but brought no clear gain to Nemotron.
Next, we’ll test a larger dense model, with an early check for improvement before committing to the full run.
Read the experiment and results: https://t.co/ZSziFiraIZ
AGI - Allah guided intuition
ASI - Advanced Sharia intelligence
LLM - Long Live Muhammad
CoT - Chain of Tafsir
MCP - Muslim community protocol
GAN - Generative Arab Network
RAG - Relentless Allah Glorification
SDK - Sharia development kit