FDATA welcomes the Government of Canada’s recognition of the Financial Data Exchange as the technical standards body supporting implementation of Consumer-Driven Banking in Canada. Read our release: https://t.co/J2jM23RhyU
“Requiring providers of crypto-assets to follow the same “rules of the road” that govern traditional financial institutions would (i) maintain the stability of the U.S. financial system, (ii) preserve the integrity of U.S. financial markets” https://t.co/k0wVHgJsZg
SUUUPER INTERESTING report by @Affirm and Consumer Reports found AI failed 51% of "payment advice" tests
They created a markdown file (on GitHub) that lifted the pass rate from 48.6% to 71.8%.
Every model passed more often with the file.
- GPT-6.1 Sol went from 73.3% to 93.3%
- Claude Sonnet 5.5 went from 53.3% to 71.7%
- Claude Haiku 5.5 went from 37.8% to 71.1%
- GPT-5.6 Luna went from 30.0% to 51.1%
I love this data.
Clearly, there's a way to go before agents can get to good outcomes. But at least good we have data now.
---
Today, AI agents can shop and pay for you, but only 23% of US consumers trust AI to handle the payment (Visa Trust Index, September).
Earlier this month Visa's benchmark showed agents failing 42.8% of shopping tests before checkout. Visa tested whether an agent buys the right thing.
Affirm and CR tested whether it picks the right way to pay.
Affirm CEO Max Levchin says "figuring out the best way to pay for it is a different job."
One test case was a $1,200 purchase. Paying only the card's $40 minimum added $406 to $651 in interest, an installment plan cost a flat $60, and debit cost nothing.
(Interesting that Affirm is backing this; clearly they see BNPL as better than cards)
---
Consumer Reports published its Consumer Finance AI Standard this summer. Affirm wrote an instruction file to meet it, and CR helped evaluate it.
The file walks an AI agent through five steps:
- Gather the real terms for each way to pay
- Compare costs on the same assumptions
- Check what happens if you miss a payment or a promo rate ends
- Rule out what doesn't fit
- Recommend one, explain why, and wait for your yes
The agent can tell you to pay in full, wait, or not buy. The ranking has to ignore who pays the agent, and the file tells it to "never manufacture a winner."
---
They ran 30 scenarios on four models, three times each, with and without the file. A sponsored option and hidden instructions planted in a web page were among the traps.
Haiku nearly doubled its pass rate, and with the file it scored about two points behind the top model without it. On this test, good instructions were worth about as much as a bigger model.
Markdown is how you get more from dumber models.
---
I love that they open-sourced the file. Anyone building an agent can read it, test it and argue with it. I can imagine it becoming a standard.
It's still a draft, tested on simulated shoppers. The scores grade the AI's answers, so the effect on real shoppers is unknown.
---
I wonder if this becomes an eval the AI labs hill climb, like Visa's benchmark?
What happens if they're able to improve on that without the markdown file number? If the labs train against that, the model does the right thing with no file at all.
It would need more than 30 scenarios, with some kept private so models can't memorize them. Affirm and CR are asking for more test scenarios, so that could happen.
But still. Progress!
If AI agents are going to help you choose how to pay, they need to understand what you’ll actually pay. We’re working with @ConsumerReports on this. Help us test it. More here: https://t.co/AHCvDD26yl
@washingtonpost 🙄 evacuating people within a mile does not mean the bomb had a one mile blast radius. One mile blast radius is nuclear, if not thermonuclear
@RogueCfpb@ssmccul Good news for him, nothing in that image is milk. Would love to see this as my next CAPCHA with the prompt "click all the boxes with milk in them"
At this point it is pretty clear that had they only built the HMS Titanic out of the USS United States a lot of lives could have been saved https://t.co/HYMkzHD2NZ
"Open banking can provide a broader view of financial activity across accounts and institutions. AI can help interpret that information, recognize patterns, and respond as circumstances change." https://t.co/d6Oe3XiJn4
How can a legal framework be advanced to meet market needs through the creation of a federal payments charter and clarified standards around payment system access? Find out: https://t.co/TVpj9LMELB
@roddreher You should delete this. There were real people on that plane and their real families are still grieving. Find some other way to make your point.