Top Tweets for #finephrase

6 months ago

In the spirit of openness and upon community request we just released the full 120TB of checkpoints trained for the #finephrase. 1M+ files worth of trained models all hosted on @HuggingFace buckets.

joelniklaus's tweet photo. In the spirit of openness and upon community request we just released the full 120TB of checkpoints trained for the #finephrase.

1M+ files worth of trained models all hosted on @HuggingFace buckets. https://t.co/EsEf2TpZsq

647

Joël Niklaus

@joelniklaus

7 months ago

Community asked, we delivered. 🚀 We just released almost 7TB of raw rephrased data from #FinePhrase to enable further experimentation and analysis. Code is also public for full transparency and reproducibility. We're using the new Hugging Face Buckets feature for this release. Unlike git-based repos, buckets provide S3-like object storage with content-addressable deduplication. Perfect for this use case because: - No version control overhead for massive files (7TB would be painful in git) - Fast, mutable storage for artifacts that don't need history tracking - Simple CLI and Python API for syncing, filtering, and browsing - Server-side file copying without re-uploading @ratishsp ran a detailed quality analysis on the dataset and found some interesting patterns. Format compliance on tables was rough, hallucination rates were high across splits. Makes sense since the rephrasing was done by a 1.7B model. Weird thing: models pretrained on this data still hit decent benchmark scores after 20B and 100B tokens, even with the quality issues. We covered some of this in the original blog post, but there's clearly more to understand here. That's exactly why we're releasing everything. If you want to dig into: - How synthetic data quality actually impacts pretraining - Why benchmark performance doesn't always track with perceived quality - Better filtering or multi-round rephrasing approaches - The counterintuitive relationship between data quality and model performance Now you can. Data: https://t.co/DNNKKZEO3H Code: https://t.co/WuqFvYobh9 Curious what you find. Hit me up if you do something interesting with it!

joelniklaus's tweet photo. Community asked, we delivered. 🚀

We just released almost 7TB of raw rephrased data from #FinePhrase to enable further experimentation and analysis. Code is also public for full transparency and reproducibility.

We're using the new Hugging Face Buckets feature for this release. Unlike git-based repos, buckets provide S3-like object storage with content-addressable deduplication. Perfect for this use case because:
- No version control overhead for massive files (7TB would be painful in git)
- Fast, mutable storage for artifacts that don't need history tracking
- Simple CLI and Python API for syncing, filtering, and browsing
- Server-side file copying without re-uploading

@ratishsp ran a detailed quality analysis on the dataset and found some interesting patterns. Format compliance on tables was rough, hallucination rates were high across splits. Makes sense since the rephrasing was done by a 1.7B model.

Weird thing: models pretrained on this data still hit decent benchmark scores after 20B and 100B tokens, even with the quality issues. We covered some of this in the original blog post, but there's clearly more to understand here.

That's exactly why we're releasing everything. If you want to dig into:
- How synthetic data quality actually impacts pretraining
- Why benchmark performance doesn't always track with perceived quality
- Better filtering or multi-round rephrasing approaches
- The counterintuitive relationship between data quality and model performance

Now you can.

Data: https://t.co/DNNKKZEO3H
Code: https://t.co/WuqFvYobh9

Curious what you find. Hit me up if you do something interesting with it!

11K

Top Tweets for #finephrase

Last Seen Hashtags on Sotwe

Trends for you

Most Popular Users