btw anthropic's internal document on this literally said "we don't want it to be known that we are working on this.”
it was called project panama.
here's exactly what happened:
1: anthropic concluded that books were the cheapest way to build a world-class model because they gave claude curated facts, structured arguments, compelling stories, and writing “an editor would approve of.”
2: once anthropic decided it needed books at enormous scale, its first solution was piracy.
it downloaded 7m+ books from online libraries including libgen. the judge later wrote that although anthropic had legal ways to buy them, it chose piracy to avoid what dario amodei called the “legal/practice/business slog.”
3: that piracy created a massive legal risk.
so in february 2024, anthropic hired tom turvey, the former head of partnerships for google books, to find a legally safer way of obtaining “all the books in the world.”
4: turvey first contacted major publishers about licensing their catalogs.
those attempts didn’t produce agreements, so anthropic chose a route that required no publisher permission: buying millions of physical books through distributors and used-book retailers.
5: within about a year, anthropic spent tens of millions acquiring and scanning millions of books, including many rare and 1/1 titles. one vendor proposal targeted 500,000 to 2 million books in six months.
6: to scan that many books within months, the vendors physically dismantled them.
a hydraulic cutter removed each spine. the pages were trimmed to size, fed as loose sheets through high-speed industrial scanners, and converted into searchable PDFs. the paper remains were then sent for recycling.
7: these PDFs were fed into claude as training data.
the complete collection became a private, searchable anthropic library that the company planned to “store forever.” the scans aren’t available to the public and were never open-sourced.
@BradyHeat@DrewSav Highly recommend the pork chop. It has a tamarind glaze that is nearly impossible to replicate. It’s probably the best is the world. For my money, there’s not a single person in the Time 100 whose legacy outmatches this.
@CovfefeAnon@annakhachiyan Gays in the 80s had real power. Every mainstream group was so scared to offend them, even if it meant lying about the risk. Even the CDC said ‘anyone’ can get it - suburban whites were scared to hook up - when it was almost 100% exclusive to gays and IV drug users.
An inadvertently funny part of VC Pooh bah @bgurley’s new career book is his chapter on finding mentors. He talks about its importance and how he takes time to mentor young people, all of whom are young women.
Years ago Dissident Dalmation noticed this trend on tiktok of people talking about the joys of having "mentors" at work and pointing out it was 100% young women with Xer/boomer men mentors.
@Cernovich@Babygravy9 If we can’t trust Ahmed Shah - who joined Anthropic right before an IPO but makes it clear that he’s actually motivated by public service - then who can we trust?
@Cernovich@VDAREJamesK Any college using FAFSA must co-sign all federal loans for their students. Tuition would plummet, so would all the goofy departments and busybody admins.
@bgurley I enjoyed your book and wrote what appears to be the only non-obsequious review. You might find it refreshing, being that you're surrounded by so many striver suck ups all day long. https://t.co/AHjVOw85lk
@Mhdeeeez@Slame333 This is mostly a Chinese and Indian diaspora thing. That’s probably the ‘low in kindness, likability, and courage’ finding that even Harvard admitted. The Japanese and Filipinos, God bless them, don’t inspire such hostility. https://t.co/sd9MXttrcH
@MyopicEeyore Wolfe always found a way to bring up Chomsky’s air conditioned office at MIT. Serious researchers were sweating it out in Borneo, but Noam never forgot his sweater in case things got drafty in Cambridge.
@gak_pdx It has bugman energy. The people who like it also like cycling, Sinners, and LinkedIn. They don’t mind overhead lighting. Don’t understand per capita. Suspicious of physical courage. Its biggest fans will be journalists. @9mmsmg
It’s one thing to see a graph saying, “12 million illegal migrants entered Europe since 2008.”
It’s another thing to actually see it.
Every blip in this animation represents 100 people.