Pretraining eval contamination is not just “remove near-duplicates.”
I wrote 3 posts on:
- what contamination actually means
- why embedding similarity is not enough
- how to validate pretraining ideas across model scale before spending serious compute
https://t.co/vPIE0NuLjl