Top Tweets for #streamingllm

over 2 years ago

#StreamingLLM is available in #TensorRT LLM. Deploy LLMs for long text streams without running out of memory. StreamingLLM finds the "attention sink" phenomenon: pinning the initial tokens in the KV cache can restore window attention's performance.

NVIDIA AI Developer

@NVIDIAAIDev

over 2 years ago

💬 Are long chats with your #LLM app running out of memory? Our distinguished engineer and MIT professor Song Han developed and incorporated StreamingLLM into #TensorRT LLM v0.8. ✨Click to code StreamingLLM with Mistral 7B on a GPU-backed notebook. ➡️ https://t.co/L771lHySWx

NVIDIAAIDev's tweet photo. 💬 Are long chats with your #LLM app running out of memory?

Our distinguished engineer and MIT professor Song Han developed and incorporated StreamingLLM into #TensorRT LLM v0.8.

✨Click to code StreamingLLM with Mistral 7B on a GPU-backed notebook.

➡️ https://t.co/L771lHySWx https://t.co/TGtvvRNsZ8

178

37K

12K

VITA Group @VITAGroupUT

almost 3 years ago

Kudos to our incredible @KyriectionZhang and collaborators: Our H2O fast #LLM #inference has surpassed (the great!) #StreamingLLM, on various benchmarks - supporting extended generation up to 4 #million tokens! Join H2O presentation at #NeurIPS2023 next Tue (Poster session 1)✨

Zhenyu (Allen) Zhang

@KyriectionZhang

almost 3 years ago

Our H2O approach now can handle infinite stream of input and is better than StreamingLLM! Check out our NeurIPS 2023 work: H2O: Heavy-Hitter Oracle (https://t.co/GNL21hSQpC), to empower LLMs for efficient and effective generation inference.

25K

811

AITopTools

@aitoptools

about 3 years ago

#AI models now run more smoothly than ever with #StreamingLLM, a new solution that keeps performance up despite data overload. Check it out: https://t.co/YkVhASagQi #ML #DataScience

Top Tweets for #streamingllm

Last Seen Hashtags on Sotwe

Trends for you

Most Popular Users