#Commoncrawl - Twitter Hashtag

8 months ago

@Meta @ruoxijia @SanghaniCtrVT @newsha_a @m_elhoushi @ShangwenLi1 @CarolejeanWu 2/n) #ScalingLaws: #Pretraining on *pure* synthetic data is not superior to #CommonCrawl (CC). Mixtures of rephrased/natural texts *significantly* outperform pure synthetic types and CC; textbook-style “pure” synthetic data alone results in notably higher loss on many domains.

feiyang_ml's tweet photo. @Meta @ruoxijia @SanghaniCtrVT @newsha_a @m_elhoushi @ShangwenLi1 @CarolejeanWu 2/n) #ScalingLaws: #Pretraining on *pure* synthetic data is not superior to #CommonCrawl (CC). Mixtures of rephrased/natural texts *significantly* outperform pure synthetic types and CC; textbook-style “pure” synthetic data alone results in notably higher loss on many domains. https://t.co/z4u5lneZDK

0

3

0

79

Common Crawl Foundation

@CommonCrawl

9 months ago

#WMDQS #COLM_conf #CommonCrawl

Workshop on Multilingual Data Quality Signals @wmdqs

9 months ago

In collaboration with @CommonCrawl @MLCommons @AiEleuther, the first edition of WMDQS at @COLM_conf starts tomorrow in Room 520A! We have an updated schedule on our website, including a list of all accepted papers.

wmdqs's tweet photo. In collaboration with @CommonCrawl @MLCommons @AiEleuther, the first edition of WMDQS at @COLM_conf starts tomorrow in Room 520A! We have an updated schedule on our website, including a list of all accepted papers. https://t.co/NHmrtzRb7e

1

7

5

0

2K

0

2

0

363

niyesordunki @niyesordunki

12 months ago

$DAG’s partnership with Forward Edge-AI secures AI data for gov use. Their work with Common Crawl validates 9 petabytes of AI training data on $DAG’s network, ensuring trust for national security & fintech apps. AI + blockchain = future. 🧵 [7/10] #ForwardEdgeAI #CommonCrawl #AI

1

10

1

0

217

Miles

@Miles__Tweet

over 1 year ago

$DAG x #CommonCrawl is one of the most impressive AI / Big Data partnership in the entire industry. #CommonCrawl = 9 million gigabytes of internet data, used by 80% of big AI language models in the world like ChatGPT! You really need to realize the importance and the scope of this partnership. $DAG's Hypergraph will make all this data immutable, more secure, more transparent, more trustable and accessible.

Constellation²

@Conste11ation

over 1 year ago

We asked @grok to describe Constellation's partnership with @CommonCrawl. Constellation partnered with the Common Crawl Foundation, a non-profit dedicated to archiving the internet, to enhance the security, transparency, and utility of web-crawled data for artificial intelligence (AI) and data applications. Announced on October 24, 2024, this strategic collaboration leverages Constellation’s Hypergraph—a Directed Acyclic Graph (DAG)-based decentralized network—to integrate and validate Common Crawl’s massive dataset, which spans nearly 9 petabytes, includes over 250 billion crawled web pages (19 billion in 2024 alone), and is used by approximately 80% of large language models (LLMs) for AI training. The partnership focuses on addressing key challenges in AI development, such as data provenance, integrity, and ethical sourcing. By incorporating Constellation’s blockchain technology, specifically its Hypergraph Transfer Protocol (HGTP), the collaboration aims to create the industry’s first cryptographically secure, immutable archive of internet data. This ensures that the data used for AI training is auditable, transparent, and tamper-proof, tackling concerns about trust and authenticity in machine learning datasets. The initiative began with the launch of a customizable subnet, or “metagraph,” integrating a subset of Common Crawl’s data. This metagraph, initially deployed on Constellation’s test network, went live on the public Hypergraph mainnet by December 18, 2024. Key aspects of the partnership include: Data Validation and Security: DAG secures the archived internet crawls, providing end-to-end encryption and immutability. This allows developers to access verified historical data with confidence, enhancing traceability throughout the AI development lifecycle. Scalability and Accessibility: The Hypergraph’s scalable architecture supports the processing of Common Crawl’s vast dataset, aiming to make cryptographically validated data distribution a standard part of Common Crawl’s release process. This aligns with Constellation’s mission to democratize data and bridge traditional infrastructure with decentralized networks. Real-World Impact: The partnership has attracted attention from projects like TraceAI, backed by the National Science Foundation, which uses Constellation’s technology to add auditability and watermarking to AI models, further extending data origin tracking. It also builds on Constellation’s prior validation by the U.S. Department of Defense, reinforcing its credibility for secure data solutions. Future Development: Over the coming months, the two organizations plan to expand solution sets for AI developers, with ongoing integration efforts to refine how this authenticated data is delivered. Constellation’s partnership with Common Crawl combines blockchain’s strengths with one of the internet’s largest open datasets to revolutionize AI data management, offering a scalable, secure, and transparent foundation for the future of decentralized technology and responsible AI innovation.

Conste11ation's tweet photo. We asked @grok to describe Constellation's partnership with @CommonCrawl.

Constellation partnered with the Common Crawl Foundation, a non-profit dedicated to archiving the internet, to enhance the security, transparency, and utility of web-crawled data for artificial intelligence (AI) and data applications.

Announced on October 24, 2024, this strategic collaboration leverages Constellation’s Hypergraph—a Directed Acyclic Graph (DAG)-based decentralized network—to integrate and validate Common Crawl’s massive dataset, which spans nearly 9 petabytes, includes over 250 billion crawled web pages (19 billion in 2024 alone), and is used by approximately 80% of large language models (LLMs) for AI training.

The partnership focuses on addressing key challenges in AI development, such as data provenance, integrity, and ethical sourcing. By incorporating Constellation’s blockchain technology, specifically its Hypergraph Transfer Protocol (HGTP), the collaboration aims to create the industry’s first cryptographically secure, immutable archive of internet data. This ensures that the data used for AI training is auditable, transparent, and tamper-proof, tackling concerns about trust and authenticity in machine learning datasets.

The initiative began with the launch of a customizable subnet, or “metagraph,” integrating a subset of Common Crawl’s data. This metagraph, initially deployed on Constellation’s test network, went live on the public Hypergraph mainnet by December 18, 2024.

Key aspects of the partnership include:

Data Validation and Security:
DAG secures the archived internet crawls, providing end-to-end encryption and immutability. This allows developers to access verified historical data with confidence, enhancing traceability throughout the AI development lifecycle.

Scalability and Accessibility:
The Hypergraph’s scalable architecture supports the processing of Common Crawl’s vast dataset, aiming to make cryptographically validated data distribution a standard part of Common Crawl’s release process. This aligns with Constellation’s mission to democratize data and bridge traditional infrastructure with decentralized networks.

Real-World Impact:
The partnership has attracted attention from projects like TraceAI, backed by the National Science Foundation, which uses Constellation’s technology to add auditability and watermarking to AI models, further extending data origin tracking. It also builds on Constellation’s prior validation by the U.S. Department of Defense, reinforcing its credibility for secure data solutions.

Future Development:
Over the coming months, the two organizations plan to expand solution sets for AI developers, with ongoing integration efforts to refine how this authenticated data is delivered.

Constellation’s partnership with Common Crawl combines blockchain’s strengths with one of the internet’s largest open datasets to revolutionize AI data management, offering a scalable, secure, and transparent foundation for the future of decentralized technology and responsible AI innovation.

11

254

62

8

16K

0

3

1

374

CrazySuzie @crazy_suzie

over 1 year ago

@HGTP_Joe_Crypto @Conste11ation @stardustco11ect @intrana @AlkimiExchange @LatticeExchange @paulbarrontv @CryptosR_Us @cryptomanran @crypto_banter @BenJorgensen Totally Agree 👍 $DAG is built for now and into the future. Highly scalability, secure, Trusted , proven it works with big hitters like the DOD , #Panasonic, #Commoncrawl , #Mattersmost , #ForwardEdgeAI !

1

8

0

95

SOCRadar®

@socradar

over 1 year ago

🚨 12,000 live #credentials found in #LLM training dataset! New research reveals #API keys & passwords exposed in #CommonCrawl data, used in training platforms like #DeepSeek – one key appeared 57,000+ times! 🔑⚠️ 🔗 Read more: https://t.co/fvJ0aYkaxH

1

3

0

2

199

hgtp://theDAGisnow @shizcryp

over 1 year ago

DAG-powered AI is set to revolutionize data analytics, creating secure, verifiable datasets for industries like finance, logistics, and healthcare. With CommonCrawl’s metagraph integration, the future of trustworthy AI is here. $DAG #AmericasBlockchain #CommonCrawl

shizcryp's tweet photo. DAG-powered AI is set to revolutionize data analytics, creating secure, verifiable datasets for industries like finance, logistics, and healthcare. With CommonCrawl’s metagraph integration, the future of trustworthy AI is here. $DAG #AmericasBlockchain #CommonCrawl https://t.co/XTKeVk3u98

2

31

2

0

626

DONALD_DAG💎 @DONALD_DAG4547

over 1 year ago

🚀 Excited to share that Constellation Network is teaming up with Common Crawl! This partnership will enhance data accessibility and empower developers to leverage vast web resources. $DAG #DAG #AmericasBlockchain #CommonCrawl #DataInnovation #Partnership

DONALD_DAG4547's tweet photo. 🚀 Excited to share that Constellation Network is teaming up with Common Crawl! This partnership will enhance data accessibility and empower developers to leverage vast web resources. $DAG #DAG #AmericasBlockchain #CommonCrawl #DataInnovation #Partnership https://t.co/qehaIi1Sc5

Dagnum²

@Dagnum_PI

over 1 year ago

AI's future depends on trust in the data it learns from Common Crawl’s blockchain-powered metagraphs Ensuring tamper-proof & accountable datasets Redefining 'Truth & Safety' @elonmusk this the solution for clean AI Let’s make truth the AI standard

7

92

18

1

2K

0

3

2

0

136

DONALD_DAG💎 @DONALD_DAG4547

over 1 year ago

This collaboration marks a significant step towards a more connected and transparent digital ecosystem! 🌐🤝 $DAG #DAG #AmericasBlockchain #Blockchain #Innovations #ConstellationNetwork #CommonCrawl #Partnership #DataInnovation 💥💥💥

0

2

1

0

34

DONALD_DAG💎 @DONALD_DAG4547

over 1 year ago

🌟 Exciting news! @Conste11ation Network has partnered with Common Crawl to revolutionize web #data accessibility. This collaboration will enhance our #decentralized infrastructure, enabling more efficient data processing and insights. $DAG #DAG #AmericasBlockchain #CommonCrawl

DONALD_DAG4547's tweet photo. 🌟 Exciting news! @Conste11ation Network has partnered with Common Crawl to revolutionize web #data accessibility. This collaboration will enhance our #decentralized infrastructure, enabling more efficient data processing and insights. $DAG #DAG #AmericasBlockchain #CommonCrawl https://t.co/N530gMZjWY

0

10

1

0

229

CryptoAssetInvestor

@CryptoAssetView

over 1 year ago

@BenJorgensen We want to hear you. We are not tired. Happy to hear from you. Thank you for all the work you and your team put in. You all are appreciated 💎 $DAG #ConstellationNetwork #Commoncrawl #Digfoundry #Metagraph #HGTP 💎

0

3

0

114

DONALD_DAG💎 @DONALD_DAG4547

over 1 year ago

🚀 Exciting news! 🌐 @Conste11ation Network has partnered with Common Crawl to enhance data access and improve web transparency. Together, we're empowering developers and researchers by providing rich datasets for innovative solutions. $DAG #DAG #AmericasBlockchain #CommonCrawl

DONALD_DAG4547's tweet photo. 🚀 Exciting news! 🌐 @Conste11ation Network has partnered with Common Crawl to enhance data access and improve web transparency. Together, we're empowering developers and researchers by providing rich datasets for innovative solutions. $DAG #DAG #AmericasBlockchain #CommonCrawl https://t.co/l4oAoEfbK3

0

14

3

0

109

crypto_rick²

@Hgtp_cryptoRick

over 1 year ago

$dag @Conste11ation #commoncrawl

Rolf

@r0lluf

over 1 year ago

Get ready for a $DAG update Something is cooking 🔥 (I don’t know the details but it looks like @CommonCrawl is ready for mainnet)

13

134

37

1

7K

0

18

0

177

Vinh Le @VinhLe44918736

over 1 year ago

@r0lluf @CommonCrawl Good finding! Always excited to see $DAG is cooking something with #CommonCrawl in this Age of #AI

1

9

0

216

Vinh Le @VinhLe44918736

over 1 year ago

@Dagnum_PI @Forbes @CommonCrawl @Conste11ation We need a deep dive into the collaboration between $DAG and #CommonCrawl!!

0

2

0

137

over 1 year ago

@Conste11ation Network is #AmericasBlockchain 🇺🇸 ! $DAG $LTX EL $PACA $DOR #Metagraphs #NDF #DOD #CommonCrawl #DePIN #AI #PLATFORM1 #IRONSPYDR $LEET $BIOFI $AUTO

cromagin's tweet photo. @Conste11ation Network is #AmericasBlockchain 🇺🇸 ! $DAG $LTX EL $PACA $DOR #Metagraphs #NDF #DOD #CommonCrawl #DePIN #AI #PLATFORM1 #IRONSPYDR $LEET $BIOFI $AUTO https://t.co/JF06xFaT5X

0

10

1

0

221

Vinh Le @VinhLe44918736

over 1 year ago

@Dagnum_PI @tunguz @Conste11ation $DAG x #CommonCrawl will be one of the most exciting development in crypto and AI

0

2

0

47

CRYPTO HAZE @HipvanHout

over 1 year ago

@Dagnum_PI Awsome. Where can I find the amount of data going throug the $dag metagraph of #commoncrawl

2

5

0

603

ROGER_C64 @Roger_C64

over 1 year ago

Even though they’re still under the radar, they’re already working on #depin projects with the world’s biggest players. @Conste11ation #CommonCrawl #DataIntegrity #AI