Why don't SSD controllers just compress with Zstandard or zlib?
In an SSD controller, power, throughput and latency matter more than the last few percent of compression. Our engine at Tenafe comes within 5% of gzip -9's ratio on the Silesia corpus in 4 KB blocks, in 3 to 5 times less silicon area than a conventional hardware gzip design, and decompresses about 4 bytes per clock cycle. To get there, we didn't put zlib or Zstandard into silicon. Both are excellent compressors, but they fit a controller badly. Eight reasons:
THE WORKLOAD
1. 4 KB blocks. A drive compresses each 4 KB sector on its own. On the Silesia corpus, that cuts Zstandard level 1's ratio from 2.89 to 2.32 and its decode speed from 1,629 to 811 MB/s.
2. Flash sets the pace. A Gen5 drive reads at about 14 GB/s. At 811 MB/s, keeping up would take about 17 CPU cores. My own WZIP, which compresses 4 KB blocks 5% more, would need about 24.
3. Every read has a deadline. Decompression sits on the read path, so its worst case must be bounded, not just its average.
4. Much of the data is already encrypted by the host. The controller must spot it cheaply, store it as is, and never make it larger.
THE ALGORITHMS
5. zlib's hash-chain search is serial: each step's address depends on the previous read. On-chip SRAM gives one random read per port per cycle, and extra ports or copies cost area.
6. Deflate's Huffman code has up to 286 symbols and codes up to 15 bits long. Building the table needs sorting, which is slow in hardware. And because any valid stream may use 15-bit codes, the decoder must handle them, which holds down its clock rate.
7. Zstandard's FSE coder is harder still. Besides building its tables, it encodes in reverse order, so the encoder must buffer the whole block before it emits the first bit. Hardware wants to stream, first in, first out.
THE WAY OUT
8. An SSD is a closed system. The same controller compresses and decompresses, and the host only ever sees the original data, so the compressed format never has to follow a standard. Hardware Deflate already exists, in Intel QAT and in IBM's on-chip accelerators, but it must produce standard streams and is built for server budgets. An SSD controller has to keep pace with flash on far less power and chip area, and its closed format is what frees us to design the algorithm around the silicon.
That's the problem we work on. What constraint would you add?
Software figures: one core of an AMD EPYC 9334, from my WZIP benchmark tables:
https://t.co/Pe7s405YJE
Here is a puzzle that hides an NP-hard problem.
You have n sets of things, and the sets may overlap. Pick as many things as you can, with one rule: no two picked things may come from the same set. A thing that belongs to three sets uses up all three.
For example, people and clubs: how large a team can you form if no two members may share a club?
If the sets did not overlap, the answer would simply be n, one thing from each set. Overlap is what makes it hard: when every set has two things, this is exactly the maximum independent set problem on a graph, one of the classic NP-hard problems. (In hypergraph terms, it is the maximum strong independent set.)
When the exact answer is out of reach, the next best thing is a bound it can never exceed. In our paper, Jason Cong (UCLA) and I start from one that fits in a line:
answer ≤ the sum, over all sets, of 1 / (the fewest sets that any member of that set belongs to)
Why it holds: let each picked thing split one point evenly among the sets it belongs to. A set receives points from at most one picked thing, and never more than that 1 / (fewest) share. Adding over the sets gives the bound.
It takes one pass over the data and no solver, and it is exact when the sets do not overlap. It also sharpens by cutting: removing sets can only make the true answer larger, yet it can make the bound smaller, so the smaller value still holds.
The same quantity drives our greedy algorithm. It first takes, from each set that has one, a thing that belongs to no other set, which is always safe. It then goes through the remaining sets from the most constrained (the smallest "fewest sets" number) up, picks from each the member that rules out the fewest competing candidates, and drops everything that shares a set with it. It works on the sets directly and never lists the conflicting pairs (one set of a million things alone would make half a trillion), so its memory grows only linearly with the input.
The picture works through a small case: 9 people, 8 clubs. The shares add up to 4.17, and cutting one club brings the sum to 3.83. Team sizes are whole numbers, so no team of 4 exists, and the greedy's team of 3 is the best possible.
The paper, submitted to the Journal of the ACM, also proves exact reductions and a covering bound. On web-scale deduplication data (1.6 and 4 billion documents, each set a bucket of likely duplicates), the greedy result comes within 0.6% of the sharpened bound, and so within 0.6% of the best possible answer:
https://t.co/oISvoE4H2U
The most detailed open LLM data release yet: K2 Horizon's pre-training data and post-training data-generation pipelines are now open!
13 trillion pre-training tokens across 5 datasets and 24 subsets. Over 10 trillion tokens of synthetic data spanning mathematics, reasoning, coding, planning, and problem-solving.
Beyond the scale, four highlights for model builders (details in the threads):
- Scaling synthetic data without losing diversity. Our generation methods produce diverse training examples at scale, nearly matching natural web text in measured diversity.
- More verification isn't always better. Strict verification can throw away useful training signal.
- Small-scale wins can mislead. Our chosen data mix actually lost in smaller experiments.
- Beyond teacher imitation. Our post-training data generation methods produce training trajectories the teacher couldn't generate on its own.
Open pre-training datasets, post-training data-generation pipelines, and processing tools for researchers to inspect, reuse, and build on.
Pre-training data: https://t.co/3Lb28JhyG9
Post-training data-generation code & tools: https://t.co/YWgJpOnzVe
If identical bytes don't matter, zlib-ng and libdeflate are faster; if you can leave deflate, zstd or my WZIP compress smaller and decode faster. zlib-turbo is for systems where the bytes must not change.
Code, numbers and techniques: https://t.co/y9Hh5SOs13
zlib sits inside git, PNG, Python, Java and countless .gz tools. How much faster can it get without changing a single byte of its output?
zlib-turbo 1.0.1: decompression up to 86% faster on one core, compression 5.7× faster on 16 threads. Byte-identical output. 🧵
Also: levels 10–12 (smaller than libdeflate's level 12), prepared dictionaries (4.1× for small messages), optional x86 SIMD checksums.
Checked against unmodified zlib: ~80M differential fuzzing runs, ~35K damaged streams, 6 CPU architectures. Now in lzbench and TurboBench.
Update: wzip 1.0 is now on Matt Mahoney's Large Text Compression Benchmark. At -13 it compresses enwik9 to 210.1 MB and decompresses it in 2.6 s (one core, EPYC 9334), which LTCB marks as the fastest among all 110 entries with a smaller total. https://t.co/zIEFPGPIKM
LZ77, a 1977 breakthrough, still powers most compressed files you download. I found a way to improve it: give each match length its own window.
The result: two open-source compressors, now in lzbench and TurboBench, one beating Zstandard's top level on Silesia. How it works ↓
Every back-reference ("copy 12 bytes from 3,000 bytes ago") costs bits, and the farther back it points, the more it costs. Yet LZ4 and Zstandard give every match the same reach: a 4-byte match, which saves almost nothing, may point as far back as a 300-byte one, and the distance code must allow for that.
So in my compressors, short matches may only point to nearby data, where distances are cheap to write, and long matches may reach up to 128 MB back. The decoder reads the length first, so it knows the window and no extra bits are sent.
The rest of the design builds on that rule. Each group of lengths gets its own distance code; one symbol carries the length, the literal run before the match and any reuse of a recent distance; and the parser prices every candidate under these codes to find the cheapest parse.
The numbers:
• WLZ4 (LZ4 class): files 14% smaller than LZ4HC at its highest level, at 78% of its decompression speed. It reaches Zstandard level 3's ratio while decompressing 2.5× faster, but compresses far more slowly: it's built for compress-once, read-many data.
• WZIP (Zstandard class): on the Silesia corpus, a 3.5% better ratio than Zstandard at its maximum level (1.2% without WZIP's filters for executables and tables), at 88% of its decompression speed.
Beyond lzbench and TurboBench, Matt Mahoney's Large Text Compression Benchmark lists WZIP: no compressor ranked above it there decompresses faster.
Its theorical framework is submitted to IEEE Trans. Information Theory (arXiv 2610.06530). It proves that the windows keep LZ77's long-run optimality while cutting search time and memory. It also isolates why the length code matters: on purely random data, the original 1977 format's overhead stays on the order of log(B)/B whatever longest match it allows, where B is the window size in bits, while a match cap that grows with the window, plus a cheap code for lengths, brings it down to log(log B)/B.
Code, benchmarks and papers in the first reply ↓
I tested GPT‑5.2‑think on a state‑of‑the‑art FFT benchmark: describing the split‑radix decimate‑in‑time FFT in a non‑recursive form. This time, it appeared overly confident—it skipped the web search and produced a completely incorrect answer, as shown here:
https://t.co/uEPX2TUx38
I’ve been benchmarking GPT against state-of-the-art topics in algebraic coding theory. Three months ago, GPT-5 Thinking correctly described the Berlekamp–Massey algorithm, but stumbled on the Berlekamp algorithm. I was surprised to see that GPT-5.2 Thinking can now properly describe the Berlekamp algorithm:
https://t.co/kRkGLtCFnR
It even covered the lesser-known Koetter–Horiguchi formula:
https://t.co/6CA3fn5iuy
One important caveat: this success relied on web search and verification, rather than deriving directly from first principles.