i gave fable 5.2 one prompt and went to sleep
11 hours later it had built the entire history of earth as a fully interactive 3d globe. big bang to today
drag the timeline to any year, spin to any place, click anything to go deeper. this is just ww2
the last thing on the timeline is the ai that built it
I wanted to test Fable 5.2's quality with one-shot prompts without human feedback.
Everything in this game is from scratch: models, sounds, textures. ~$550 at API pricing, ~2.7M output and 445M input tokens, heavily cached. Took 8 hours!
Play it here: https://t.co/fazOCcyNIH
I’m dropping a new conspiracy.
There is no way this many people with large influence spent time, money and tokens to make all these negative posts today.
Who is paying for this negative PR? 🤨
i built an open-source version of a yc company with fable 5.2. it matches them on their own benchmark and costs 5x less
@reductoai (yc w24) trained its own model to read pdfs. i used claude haiku, untouched. 5,300+ pages, 5 public benchmarks:
- tables, on reducto's own benchmark: 88.2 vs their 88.9, a match
- everyday pages: 82.2 vs 80.9
- 15% fewer text errors
- finds 75% of the table rows in long reports. reducto: 54%
- about $0.002 a page instead of $0.01
reducto sells a full document api: you send a pdf, you get back clean text, tables and fields. their bet was a custom-trained model. mine was that a general model you don't train at all can do the same job, and gets better every time a new one ships
so over 4 days, fable 5.2 built readanydoc around plain haiku. no training, no fine-tuning. every page above was scored by the benchmark's own grading code:
- tables: rd-tablebench (reducto's own), 1,000 tables
- everyday pages: olmocr, 1,403 pages
- text accuracy: omnidocbench, 1,651 pages
- scanned forms: funsd, 199 forms
- long documents: 10 full reports
over the next few posts i'll share every decision that got me here, what worked and what didn't. follow along
the full story, every benchmark and the code are in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
@aitrackerbot matching haiku on daily work is one thing, i'd want to see it on messy pdfs where haiku beat a yc company's model https://t.co/waopc9xnfS
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
fable 5.2 rebuilt a yc company's entire api in 2 hours 12 minutes
part 2 of my open-source version of @reductoai (yc w24)
companies don't pay reducto just for a model, they pay for the api around it: parse pdfs, extract fields, split, classify, edit
so i had fable 5.2 copy it exactly. one fable 5.2 agent built the core, then 5 more fable 5.2 agents worked in parallel. if you use reducto, switching is one url change. their own python sdk works unchanged
- all 20 api routes
- 630 tests
- every extracted value cites its exact spot on the page
- same files, real reducto responses: extract, split, classify, edit all matched
- one extraction job: $2.92 on reducto, $0.50 on readanydoc
full story and code in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
i built an open-source version of a yc company with fable 5.2. it matches them on their own benchmark and costs 5x less
@reductoai (yc w24) trained its own model to read pdfs. i used claude haiku, untouched. 5,300+ pages, 5 public benchmarks:
- tables, on reducto's own benchmark: 88.2 vs their 88.9, a match
- everyday pages: 82.2 vs 80.9
- 15% fewer text errors
- finds 75% of the table rows in long reports. reducto: 54%
- about $0.002 a page instead of $0.01
reducto sells a full document api: you send a pdf, you get back clean text, tables and fields. their bet was a custom-trained model. mine was that a general model you don't train at all can do the same job, and gets better every time a new one ships
so over 4 days, fable 5.2 built readanydoc around plain haiku. no training, no fine-tuning. every page above was scored by the benchmark's own grading code:
- tables: rd-tablebench (reducto's own), 1,000 tables
- everyday pages: olmocr, 1,403 pages
- text accuracy: omnidocbench, 1,651 pages
- scanned forms: funsd, 199 forms
- long documents: 10 full reports
over the next few posts i'll share every decision that got me here, what worked and what didn't. follow along
the full story, every benchmark and the code are in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
i tested 272 ideas to beat a yc company's custom model. most of them made things worse
part 3 of my open-source version of @reductoai (yc w24)
plain claude haiku already read pages about as well as reducto, so the plan was to have fable 5.2 build clever layers on top and pull ahead. it mostly backfired:
- smaller page images to save money: 19 tables came out broken
- letting the model think less: 35% cheaper, but table grids dropped 8 points
- zooming into every checkbox: fixed one, broke two
- reading every page twice and fixing the differences: worse than reading it once
so a change only stayed if it never made results worse. 272 experiments in 4 days, 75 thrown out completely
the ones that survived didn't try to read better than haiku. they added what haiku doesn't do at all: where each block sits on the page, repairing table grids, reading fillable pdf fields directly
full story and code in the reply
fable 5.2 rebuilt a yc company's entire api in 2 hours 12 minutes
part 2 of my open-source version of @reductoai (yc w24)
companies don't pay reducto just for a model, they pay for the api around it: parse pdfs, extract fields, split, classify, edit
so i had fable 5.2 copy it exactly. one fable 5.2 agent built the core, then 5 more fable 5.2 agents worked in parallel. if you use reducto, switching is one url change. their own python sdk works unchanged
- all 20 api routes
- 630 tests
- every extracted value cites its exact spot on the page
- same files, real reducto responses: extract, split, classify, edit all matched
- one extraction job: $2.92 on reducto, $0.50 on readanydoc
full story and code in the reply
@Alibaba_Qwen native vision at $1.50 in is exactly what i want to try inside my open-source reducto, fable 5.2 built the whole api https://t.co/n5haJVI9Wz
fable 5.2 rebuilt a yc company's entire api in 2 hours 12 minutes
part 2 of my open-source version of @reductoai (yc w24)
companies don't pay reducto just for a model, they pay for the api around it: parse pdfs, extract fields, split, classify, edit
so i had fable 5.2 copy it exactly. one fable 5.2 agent built the core, then 5 more fable 5.2 agents worked in parallel. if you use reducto, switching is one url change. their own python sdk works unchanged
- all 20 api routes
- 630 tests
- every extracted value cites its exact spot on the page
- same files, real reducto responses: extract, split, classify, edit all matched
- one extraction job: $2.92 on reducto, $0.50 on readanydoc
full story and code in the reply