HojiChar v0.18.0 is out!
MinHash LSH is a key building block for deduplicating LLM training corpora. Now, we’re putting it in the fast lane.
Generate document fingerprints at over 11K documents per second.
That’s 4.6× faster than the previous version and 19× faster than Hugging Face DataTrove⚡️⚡️
https://t.co/3bb0iefrJn
We had a team of agents rebuild SQLite from its 835-page manual.
It created a replica in Rust which passed 100% of a held-out test suite.
Interestingly, cost varied 15x depending on which model mix we used.