You can now run 70B LLMs on a 4GB GPU.
AirLLM just made massive models usable on low-memory hardware.
๐ช๐ต๐ฎ๐ ๐ท๐๐๐ ๐ต๐ฎ๐ฝ๐ฝ๐ฒ๐ป๐ฒ๐ฑ
AirLLM released memory-optimized inference for large language models.
It runs 70B models on 4GB VRAM.
It can even run 405B Llama 3.1 on 8GB VRAM.
๐๐ผ๐ ๐ถ๐ ๐๐ผ๐ฟ๐ธ๐
AirLLM loads models one layer at a time.
Instead of loading everything:
โ Load a layer
โ Run computation
โ Free memory
โ Load the next layer
This keeps GPU memory usage extremely low.
๐๐ฒ๐ ๐ฑ๐ฒ๐๐ฎ๐ถ๐น๐
โข No quantization required by default
โข Optional 4-bit or 8-bit weight compression
โข Same API as Hugging Face Transformers
โข Supports CPU and GPU inference
โข Works on Linux and macOS Apple Silicon
๐ช๐ต๐ฎ๐ ๐๐ผ๐ ๐ฐ๐ฎ๐ป ๐ฑ๐ผ
โข Run Llama, Qwen, Mistral, Mixtral locally
โข Test large models without cloud GPUs
โข Prototype agents on cheap hardware
@AskSynchrony Unfortunately after several hours of waiting on the phone for a 'banking specialist'. They have outlined how 'their hands are tied' that I have to wait another 2-3 business days. Where I am not guaranteed my account will become unrestricted. What gives?
Back when in-person conferences were a thing, I often found myself answering questions about how to get started in #opensource in a large company. This article touches on how we were able to successfully go about it, as well as other large companies.
Itโs #HBOMax day and weโre filled with Euphoria โจ๐Grab your Friends โ๐ญ๐๏ธ and get Looney ๐ฐ๐ฅas you explore Parts Unknown, dive into the DC Universe, follow the Yellow Brick Road to Sesame Street, & more!
Everything is Awesome! ๐๏ธ๐ฅ Get Schwifty because it's Adventure Time!
Highlight: "We met a lot of resistance in retail in wanting to put refrigerators on the baby food aisle," @LittleSpoon co-Founder Michelle Muller says about her baby food business with Lisa Barnett. Full interview: