New paper w/ Yihe Dong, @DimitrisPapail & @ShivamGarg91462: Perturbing Inputs Improves Pretraining Distillation.
In pretraining distillation, a small student LM learns to match a larger teacher’s next-token predictions on the training text. We change one thing: for the first part of training, both models read a perturbed version of the text (for example, with noise added to the token inputs), and the student matches the teacher’s predictions on that. Then we switch back to clean text.
With the same steps and token budget, this beats standard distillation, and it also works well in data-constrained, multi-epoch settings. 🧵
I am working on in-browser PDF -> markdown models.
1. A yolo11n finetuned on PDF boxes
2. A docling-258M model pruned + distilled into a quantized 100M model
It'll be cute if this thing can run on a browser for free.
Tencent just released the most interesting home-lab model of the month.
Hy4 flash is 224B total, 12B active, 1M context, natively multimodal. For local the key number is inside that 224B: 64B of it is Engram, the lookup-table memory DeepSeek introduced.
Engram gets read a few rows per token, so it can stay in system RAM or even on NVMe without hurting decode. That leaves about 160B of real weights:
- 4-bit: ~80 GB, fits fully in 96 GB of VRAM (4x 3090) with room for context
- ~3.2 bpw: ~64 GB, 3x 3090
- a 128 GB Mac holds the 4-bit weights in unified memory
DeepSeek V4.1 Flash 1017 has ~290 GB of FP4 experts before you even count Engram, so at home it streams experts over PCIe on every prefill and prefill crawls because of it. Hy4 flash needs no expert offload at all.
Hybrid attention plus LatentMoE cut KV cache to 1/6 of Hy4 and 1/24 of Hy3, so 1M context won’t eat your VRAM either.
Tencent’s own tests put it level with or above DeepSeek V4.1 Flash 1017 on agent tasks and 2.5% over GLM-5.3-Flash on chat. The catch: long-horizon coding is weaker, 19.2 on Terminal-Bench 4.0 vs 44.4 for full Hy4.
@TencentHunyuan
open this gem to the people (I genuinely believe in this one)!
@uctive Structured output is the dream case for speculative decoding—deterministic JSON keys and brackets push draft acceptance lengths through the roof.
Introducing 𝚡𝚎𝚎𝚝: https://t.co/YaLEMbmKwf.
A hermetic, pixel-perfect X post renderer in safe Rust / wasm. Zero browser, network, filesystem, system fonts. Renders a post to PNG in ~1ms (and 0.4 ms for @jack's "setting up my twttr" 😁).
X is the global town square, and the medium is the message. Being able to preview what things will look like before posting is great for both humans and agents.
Rendering with xeet is is 100x-1000x faster than booting up a browser and taking a screenshot, yet it's just as accurate.
It reproduces Blink & HarfBuzz text shaping, 1/64px line breaking, RTL bidi, 28 Chirp font subsets, CJK/Math/Emoji fallbacks, "Show more" rules, etc. It's the kind of thing that would be wildly impractical, if not impossible, for a human engineer to pull off.
Except 𝚡𝚎𝚎𝚝 is structured as an "autorepo." An oracle boots up Chrome, and verifies tens of thousands of text layout rules. A CI canary then watches the evolution of the original, and files signed PRs for agents to act on. It was conceived from the ground up around 'verification engineering', and the result speaks for itself.
Enjoy!
@mert_gulsun Padding is tempting until it breaks ds_read_b128 alignment and eats your occupancy, so XOR swizzling is almost always worth the extra VALU ops.