Michael Zhang

Writing on the world of AI at @ArtisanaAI

[Michael Zhang @mztwo]

@iandanforth Great way to put it. Thanks!

[Hacker News 50 @betterhn50]

Meta's plan to offer free commercial AI models puts pressure on Google, OpenAI https://t.co/GvNNyBqzmG (https://t.co/3aonqkaJAY)

[Vivek Ramachandran @vivekramac]

I am loving the AI model wars! https://t.co/U4yrywjtjz

[Michael Zhang @mztwo]

For more on this, see my full breakdown: https://t.co/evSIfOXLqy

Other links:
Paper: https://t.co/rgJOHjnp3E
Code+Demo: https://t.co/Gv5E0WqgKf
Samples: https://t.co/04CDVakORV
Colab: https://t.co/xnCKJrXSph

[Michael Zhang @mztwo]

Mobile devices could start to fine-tune LLMs soon. This opens up so many options for data privacy, personalized LLMs, and more.

[Michael Zhang @mztwo]

Open-source is emerging as an even bigger threat to closed-source. Many of these closed-source models haven't even considered using LoRA fine-tuning, and instead prefer to train from scratch. The pace is real.

[Michael Zhang @mztwo]

Frequent fine-tuning enables models to incorporate real-time info. By bringing cost down, this is more possible.

[Michael Zhang @mztwo]

What does this mean for the future of AI? Here's where things get interesting.

[Michael Zhang @mztwo]

Producing highly capable, state of the art models no longer requires expensive compute for fine-tuning. You can do it with minimal commercial resources or on a RTX 3090 now. Everyone can be their own mad scientist.

[Michael Zhang @mztwo]

The results are noteworthy: The 65B and 33B Guanaco variants consistently matched ChatGPT-3.5's performance. While the Vicuna benchmarking is imperfect (the researchers note that extensively), it's nonetheless significant and newsworthy.

[Michael Zhang @mztwo]

They quantize the quantization constants. This is akin to compressing their compression formula as well. And memory spikes typical in fine-tuning are optimized, which reduces max memory load required.

[Michael Zhang @mztwo]

A special 4-bit NormalFloat data type is efficient at being precise, versus the 16-bit floats and integers which are memory-intensive. Best way to think about this is that it's like compression (but not exactly the same).

[Michael Zhang @mztwo]

How does the science work here? Three things to know:

[Michael Zhang @mztwo]

This is open-sourced and available now. Huggingface already enables you to use it. Things are moving at 1000 mph here.

[Michael Zhang @mztwo]

A commercial GPU with 48GB of memory is now able to produce the same fine-tuned results as the same 16-bit tuning requiring 780GB of memory. This is a massive decrease in resources.

[Michael Zhang @mztwo]

It's so efficient that researchers were able to fine-tune a 33B parameter model on a 24GB consumer GPU (RTX 3090, etc.) in 12 hours, which scored 97.8% in a benchmark against GPT-3.5.

[Michael Zhang @mztwo]

QLoRA is a novel approach that enables 4-bit finetuning of LLMs. Let's talk about why this is a milestone moment for open-source LLMs. 👇

[Michael Zhang @mztwo]

QLoRA is an even more efficient way of fine-tuning which truly democratizes access to fine-tuning (no longer requiring expensive GPU power). It's a 4-bit approach that generates performance similar to 16-bit finetuning methods.

[Michael Zhang @mztwo]

Fine-tuning an existing model is already a popular and cost-effective way to enhance an existing LLMs capabilities. It's cheaper than training from scratch, but it can still cost $$$.