People are just starting to realize the power of Local LLMs
Especially with new Apple chips. It's a game-changer
Let me show you:
Falcon is 180B LLM. It is the size of GPT3.
How fast would it run on your Mac?
Apple M3 Max: 3.5 tokens/sec
We can compare it to 2 x A100 (2 x 30k$)
2x A100 80GB: 7 tokens/sec
Now add to this the recent speed of @NousResearch & @teknium with models like openHermes
And the fact that soon they will add Vision functionality to send images
https://t.co/qBSVHPGTH5
and we get real competition to commercial models from OpenAI and Anthropic
More of independent companies start to publish their model weights
Take https://t.co/HJtAdLy9LO team from China. It is currently leading the OSS leaderboard in quality & context size
01 team has 200k context window model.
It's the 7 copies of Shakespeare's Hamlet which can fit onto a single prompt
GPT4 turbo has only 128K
7B models like Mistral are already running at 40+ tok/s!
This is insanely fast! I have run a test on my old M1 a gif below to give you a feeling of the speed of it.
Absolutely stunning work is done by @ggerganov. Georgi & llama.cpp community have literally shaped the progress of open source models with llama.cpp project. It allows you to run any LLM on your computer. You can even run llama on your ios with it
I am incredibly bullish on Apple metal & local computations.
There are many interesting projects rising around it, one that came to my mind is @opentensor – decentralized commodity of computing resources like selling your GPU to someone
And I haven't even started on the power of webGPU & Web LLM
What other interesting stuff have you seen? share in comments
Cada semana recibo más y más emails para externalizar servicios de desarrollo web o marketing.
El filtro antispam de Gmail se ha ablandando con los años...👴📩
My power’s really low, so this may be the last image I can send. Don’t worry about me though: my time here has been both productive and serene. If I can keep talking to my mission team, I will – but I’ll be signing off here soon. Thanks for staying with me.
El número de despidos en tech en lo que va de 2022 (120.000) ya se asimila al del estallido de la burbuja puntocom.
Meta ⟶ 11.000 (13%)
Booking .com ⟶ 4.375 (25%)
Uber ⟶ 3.700 (14%)
Twitter ⟶ 3.700 (50%)
Groupon ⟶ 2.800 (44%)
https://t.co/tj6wPRmh5Y
¡Se acabó la espera!
Aquí va el Capítulo 2 de mi mega-hilo-relato ambientado en el universo Lovecraft con imágenes que he creado con arte generativo.
De nuevo muchas horas escribiendo y sobre todo, ilustrando, esta historia. ¡Disfrutadla!
Hilo largo 🍿 (70 tuits) 🧵👇