小野寺　直彦 @topololabo - Twitter Profile

6 days ago

ChatGPT has now a big problem. Researchers at Oxford and Cambridge exposed a massive threat to large language models.” They call it “model collapse." Internet ecosystem is rapidly changing, and generative AI will soon contribute much of the text found online. This forces us to consider what happens to future iterations like gpt-n when they are trained on data scraped from the web that was already generated by an llm. According to the research, indiscriminately using model-generated content in training causes "irreversible defects" in the resulting ai. the model loses the "tails of the original content distribution." in other words, it begins to forget the creative, fringe, and unique nuances of actual human writing, collapsing into a repetitive echo chamber. This isn't just a chatgpt issue.. the researchers built theoretical intuition showing this collapse is ubiquitous across learned generative models, occurring in large language models as well as in variational autoencoders and gaussian mixture models. Tech companies rely on scraping the internet for large-scale data to build smarter models. However, the paper warns that if we want to sustain the benefits of training on web data, model collapse must be taken seriously. Ultimate takeaway? data collected from genuine human interactions is going to become increasingly valuable in a web filled with ai content.

thesupermannx's tweet photo. ChatGPT has now a big problem.

Researchers at Oxford and Cambridge exposed a massive threat to large language models.”

They call it “model collapse."

Internet ecosystem is rapidly changing, and generative AI will soon contribute much of the text found online. This forces us to consider what happens to future iterations like gpt-n when they are trained on data scraped from the web that was already generated by an llm.

According to the research, indiscriminately using model-generated content in training causes "irreversible defects" in the resulting ai. the model loses the "tails of the original content distribution." in other words, it begins to forget the creative, fringe, and unique nuances of actual human writing, collapsing into a repetitive echo chamber.

This isn't just a chatgpt issue.. the researchers built theoretical intuition showing this collapse is ubiquitous across learned generative models, occurring in large language models as well as in variational autoencoders and gaussian mixture models.

Tech companies rely on scraping the internet for large-scale data to build smarter models. However, the paper warns that if we want to sustain the benefits of training on web data, model collapse must be taken seriously.

Ultimate takeaway? data collected from genuine human interactions is going to become increasingly valuable in a web filled with ai content.

412

4K

1K

3K

1M

小野寺　直彦 @topololabo

9 days ago

今月の読書。読みやすい字組のライトな体裁の書物だった。手に取り易くするためか。再読、再再読を促すためか。いずれにせよ自分は今の今まで読まなかった。ボケるまでに一読できて間に合った感。

0

4

0

59

topololabo retweeted

cat of the day @catoftoday

10 days ago

Apparently Caracal kittens sound like laser beams.

438

86K

9K

13K

4M

topololabo retweeted

Occupy Democrats

@OccupyDemocrats

10 days ago

BREAKING: BRAVO! The New York Times announces it will NOT publish or distribute ANY photos from the White House pool today in protest of Trump's assault on free speech. This fight has spiraled totally out of control for Donnie... “The New York Times will not publish or distribute any images taken as part of the White House still photography pool today, in conjunction with the other members of that pool and in a show of support for the news organizations being unfairly targeted by the president’s barring of journalists from the White House," New York Times spokesperson Charlie Stadtlander stated. "We remain committed to covering the president’s travel to New York City today. Americans deserve a free and independent media covering their government without intimidation, as well as comprehensive coverage of newsworthy events," he added. The Washington Post released a similar statement— “The Washington Post will not publish any photographs of today’s presidential events — including Trump’s departure from the White House and his events at the United Nations in New York — until midnight Eastern Time, joining a collective effort with photographers in other newsrooms covering the White House in a show of support of the banned outlets," said Post spokesperson Olivia Petersen. These new announcements come after major news outlets including CBS News, ABC News, CNN, Fox News, and NBC News suspended press pool coverage of Trump over his decision last Friday to ban CNN, Politico, and MS NOW from the White House over their alleged "fake news" stories. That collective pushback resulted in a particularly embarrassing incident for Trump, because there was no audio for his big White House helipad ribbon cutting ceremony today. With an even bigger week ahead as the United Nations convene in New York, Trump is going to be feeling the burn from his authoritarian power grab. He's drunk with his own power and in obvious cognitive decline, which is why he thought that he could shred the First Amendment without consequence. Thankfully, the media is fighting back — and winning bigly. Please ❤️ and share to thank the Times and Post!