btw anthropic's internal document on this literally said "we don't want it to be known that we are working on this.”
it was called project panama.
here's exactly what happened:
1: anthropic concluded that books were the cheapest way to build a world-class model because they gave claude curated facts, structured arguments, compelling stories, and writing “an editor would approve of.”
2: once anthropic decided it needed books at enormous scale, its first solution was piracy.
it downloaded 7m+ books from online libraries including libgen. the judge later wrote that although anthropic had legal ways to buy them, it chose piracy to avoid what dario amodei called the “legal/practice/business slog.”
3: that piracy created a massive legal risk.
so in february 2024, anthropic hired tom turvey, the former head of partnerships for google books, to find a legally safer way of obtaining “all the books in the world.”
4: turvey first contacted major publishers about licensing their catalogs.
those attempts didn’t produce agreements, so anthropic chose a route that required no publisher permission: buying millions of physical books through distributors and used-book retailers.
5: within about a year, anthropic spent tens of millions acquiring and scanning millions of books, including many rare and 1/1 titles. one vendor proposal targeted 500,000 to 2 million books in six months.
6: to scan that many books within months, the vendors physically dismantled them.
a hydraulic cutter removed each spine. the pages were trimmed to size, fed as loose sheets through high-speed industrial scanners, and converted into searchable PDFs. the paper remains were then sent for recycling.
7: these PDFs were fed into claude as training data.
the complete collection became a private, searchable anthropic library that the company planned to “store forever.” the scans aren’t available to the public and were never open-sourced.
Unpopular opinion, but I’m not excited about the OoT remake. The Legend of Zelda may be my all time favorite franchise, but my nostalgia demands the ugly ass original graphics.
I *thought* this was a quick little fetch quest.
AuDHD hyperfocus has somehow turned it into a full Witcher III side quest and now I’m reminding myself I don’t need to learn DaVinci Resolve just to explain OCR on the internet.
(Anyway apparently I’m making that video now.)
I was told that a board member disliked a repost on my social media, about how wild it is that POC lives seem to matter less to the general public than Nazis. It could “reflect poorly” on my place of employment. Which… tells you all you need to know about my place of employment.
What a way to prepare for Independence Day. Creating further hatred, cutting vital programs for people who were already struggling, all while promoting human rights violations (funny, they can find the money for that?). Land of the free indeed…
The Presbyterian Church in America, my lifelong denomination, has now formally joined the ARP and the RPCNA in affirming via the entire General Assembly this complete condemnation of kinism and racial supremacy.
Fox News just accidentally made Zohran Mamdani sound like the most pro-family candidate in America.
Free childcare. Baby baskets. $30 wages.
Tell me where to sign.
On Juneteenth, we celebrate freedom and recommit ourselves to the work that remains undone. We remember that even in the darkest hours, there is cause to hope for tomorrow’s light.