Geon Park

PhD Student @ KAIST MLAI

[Arthur Douillard @Ar_Douillard]

InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU

what: proposes to extend context to 1M tokens without additional training [1]

1. pruning

* do a three pruning stages
* in each group contiguous tokens in chunk. in each chunk, select a representative token using [2]
* query attends to each representative token, and then select chunks with highest score
* 32k tokens after the first pruning stage, 8k and 2k after the second and third stages
* they keep initial tokens (nicknamed "sink") and latest tokens (nicknamed "stream") without pruning
* this chunk-based pruning work because of attention locality where nearby tokens have similar scores

2. RoPE:

they modify RoPE [3,4] in two ways:
* for the first three layers, which are more reliant on positional info, they do a RoPE where each chunk is given an ID instead of per token
* for the remaining layers, they do a relative-style RoPE based on the distance to the stream tokens

3. offloading

* they offload most pruned keys and values on CPU RAM but allow keeping some w/ LRU
* they only refresh the pruning patterns every few steps, because of temporal locality where new queries are somewhat similar to the few previous ones
* they use a Paged Block Sparse Attention [5]

in the end, they get competitive results on long- and infinite-bench benchmarks despite having a "smaller" window size

[yukke @yukke_]

例えば、沢山の論文や法律系の文章や判例、QAシステムの過去のやりとり、物語や映画やシナリオを丸ごと食わして読解させたり、それをもとにした対話システムを安価で構築できるということなんだよな。凄いことだなあ。

[Geon Park @geonpark_ai]

@_akhaliq Thanks for sharing our paper!
We have a working SGLang integration at https://t.co/rZVzcT768T.
Try our Live Demo with DeepSeek 14B at https://t.co/jBI8JvylOK!

[AK @_akhaliq]

InfiniteHiP

Extending Language Model Context Up to 3 Million Tokens on a Single GPU

[Turing Post @TheTuringPost]

A very interesting study from Korean @kaist_ai and DeepAuto AI introduced InfiniteHiP.

It's an inference framework that handles long context up to 3M tokens on a single GPU with ~19x speed boost.

But how? The key improvements are:

• Offloading memory — it moves less-used data from the GPU to regular memory.

• Smarter hierarchical context pruning

• Dynamically adjusted RoPE to handle longer texts without retraining.

Here are the details:

[Geon Park @geonpark_ai]

HiP Attention got its first citation!
https://t.co/OnEgG1gs9c

[Heejun Lee @ainl0310]

📢😎Introducing our novel attention mechanism, HiP!

Enjoy 3x+ large-scale, long-context LLM decoding throughput just by replacing a single function⚡

HiP is a sub-quadratic, training-free sparse attention mechanism with hierarchical attention pruning.

📝https://t.co/aA21rUxSCI

[Gabriel Peyré @gabrielpeyre]

Approximation vs quantization: two ways to compress. https://t.co/R83Nru23GR https://t.co/1RETz6CI1h