Supergoofy

@superg00fy

Joined March 2010

230 Following

25 Followers

158 Posts

Supergoofy @superg00fy

about 1 month ago

@shiri_shh @justvugg or soon use Colibrí 😉 https://t.co/ztuwom88qG

superg00fy retweeted

Michele Sciabarra

@msciab

about 1 month ago

#colibri is a project by the Italian developer @JustVugg, whose real name is Vincenzo. It recently attracted attention because it does something that sounds almost impossible: it runs the open GLM 5.2 model, a frontier-class model with performance comparable to high-end closed models, on completely ordinary hardware: a PC with around 25 GB of RAM and no GPU. What makes this remarkable is that GLM 5.2 is a roughly 750B-parameter beast. Hugging Face lists it as a 753B-parameter model, while some news sources round that figure to 744B. In its original 16-bit BF16/FP16 format, the weights alone take up about 1.5 TB. Even in INT4, the model is still hundreds of gigabytes, far beyond the RAM available in a normal consumer PC. It is no surprise, then, that colibri has recently been widely discussed on X, LinkedIn, and in several news outlets, including Tom's Hardware, and that it quickly gained thousands of stars on GitHub. Of course, do not expect record-breaking performance. On the reference hardware it runs at around 0.05-0.1 tokens per second, or one token every 10-20 seconds. I tested colibri on my DGX Spark and got 0.25 tokens per second. The project's README includes several benchmarks; the best result at the moment appears to be 1.23 tok/s. The point is clearly not to build a production system, but to explore alternative ways of running state-of-the-art models on consumer hardware. Personally, I am firmly convinced that, with approaches like this and low-cost custom hardware support, it will soon be possible to run these models without spending hundreds of thousands of euros on specialized data center hardware. The technical approach used by colibri is very interesting: it does not load the entire model into memory. Instead, it loads only the parts it needs, when it needs them. GLM 5.2 is an MoE model, a mixture of experts. This means that, when the next token is generated, the model does not use the entire roughly 750B-parameter neural network. At several points inside the model there are multiple specialized subnetworks, called experts. A small component, called a router or gate, decides which experts to activate for that token. In practice, the model has a huge number of total parameters, but only a fraction of them is actually used at each generation step. This is the key difference between a so-called dense model and an MoE model: in a dense model, each token always passes through almost all the parameters; in an MoE model, each token passes only through a few experts selected by the router; the model can therefore have enormous overall capacity while keeping the per-token compute cost much lower than that of a dense model with the same total size. This does not mean the model becomes "small": the experts still exist, and they still need to be available somewhere, either on disk or in memory. But this property makes a strategy like colibri possible. Instead of loading the whole model into RAM, it progressively loads only the parts it needs, based on the experts requested during generation. An intuitive way to picture this is to imagine a large lab full of specialists. For each word to generate, you do not call in every specialist. You call only the two or three who are most useful at that moment. If the text is about code, the router will tend to activate experts useful for syntax, debugging, and programming. If the text is about mathematics, it will tend to activate experts better suited to symbolic reasoning. This specialization is not rigid or directly visible from the outside, but the operating principle is the same: many experts are available, only a few are active for each token. For colibri, this property is decisive. If the model were fully dense, running a roughly 750B-parameter model on a consumer PC would be much harder, because every step would require accessing an enormous mass of weights. With MoE, by contrast, the system can treat the model as a collection of blocks that can be loaded on demand. In practice, it works as shown in the figure. During generation, the system identifies the expert it needs, loads it into memory, and keeps it cached for as long as necessary using an LRU policy. Code analysis I analyzed the project's source code to better understand how it is structured. The core is glm.c, which implements the engine in a single C file. The file includes several backends for running matrix multiplication on different GPUs, such as CUDA for Nvidia and Metal for macOS, but the core remains glm.c. It is worth noting that glm.c is strictly limited to the engine: it reads input from standard input and writes output to standard output. This keeps the engine logic concentrated in one file. The file is fairly complex, but it does nothing else, which means it can be read and understood as a whole. I spent a few hours reading the source and eventually understood its structure. That matters, because it means the project can be extended and adapted to other architectures. The engine is strictly GLM-only, although its library functions can presumably be reused with other models. It is a C translation of the modeling_glm_moe_dsa.py file from Hugging Face's Transformers library. A model, after all, is essentially a collection of large matrices being multiplied together. colibri uses a model quantized to INT4, meaning 4 bits per weight. If the original model is BF16, that means 2 bytes per weight; INT4 uses 0.5 bytes per weight. The quantized model therefore takes up about 25% of the original space, reducing storage for the quantized weights alone by roughly 75%. colibri also supports Multi Token Prediction, but those layers were kept in INT8 because 4-bit quantization is not stable enough there. The model has to be downloaded separately, because it is stored in a format specifically designed for colibri. On top of this engine there is currently a Python wrapper. It launches the engine, implements an OpenAI-compatible API, provides a text chat interface, and, interestingly, parses the chat format. I find this design choice very appealing because it simplifies development and makes it possible to work on the engine without getting distracted by details that are not relevant to the core code. Personally, I already have a few contributions in mind for the project. For now, though, I am starting by sharing what I have learned so far.

$msciab's tweet photo. #colibri is a project by the Italian developer @JustVugg, whose real name is Vincenzo. It recently attracted attention because it does something that sounds almost impossible: it runs the open GLM 5.2 model, a frontier-class model with performance comparable to high-end closed models, on completely ordinary hardware: a PC with around 25 GB of RAM and no GPU. What makes this remarkable is that GLM 5.2 is a roughly 750B-parameter beast. Hugging Face lists it as a 753B-parameter model, while some news sources round that figure to 744B. In its original 16-bit BF16/FP16 format, the weights alone take up about 1.5 TB. Even in INT4, the model is still hundreds of gigabytes, far beyond the RAM available in a normal consumer PC. It is no surprise, then, that colibri has recently been widely discussed on X, LinkedIn, and in several news outlets, including Tom's Hardware, and that it quickly gained thousands of stars on GitHub. Of course, do not expect record-breaking performance. On the reference hardware it runs at around 0.05-0.1 tokens per second, or one token every 10-20 seconds. I tested colibri on my DGX Spark and got 0.25 tokens per second. The project's README includes several benchmarks; the best result at the moment appears to be 1.23 tok/s. The point is clearly not to build a production system, but to explore alternative ways of running state-of-the-art models on consumer hardware. Personally, I am firmly convinced that, with approaches like this and low-cost custom hardware support, it will soon be possible to run these models without spending hundreds of thousands of euros on specialized data center hardware. The technical approach used by colibri is very interesting: it does not load the entire model into memory. Instead, it loads only the parts it needs, when it needs them. GLM 5.2 is an MoE model, a mixture of experts. This means that, when the next token is generated, the model does not use the entire roughly 750B-parameter neural network. At several points inside the model there are multiple specialized subnetworks, called experts. A small component, called a router or gate, decides which experts to activate for that token. In practice, the model has a huge number of total parameters, but only a fraction of them is actually used at each generation step. This is the key difference between a so-called dense model and an MoE model: in a dense model, each token always passes through almost all the parameters; in an MoE model, each token passes only through a few experts selected by the router; the model can therefore have enormous overall capacity while keeping the per-token compute cost much lower than that of a dense model with the same total size. This does not mean the model becomes "small": the experts still exist, and they still need to be available somewhere, either on disk or in memory. But this property makes a strategy like colibri possible. Instead of loading the whole model into RAM, it progressively loads only the parts it needs, based on the experts requested during generation. An intuitive way to picture this is to imagine a large lab full of specialists. For each word to generate, you do not call in every specialist. You call only the two or three who are most useful at that moment. If the text is about code, the router will tend to activate experts useful for syntax, debugging, and programming. If the text is about mathematics, it will tend to activate experts better suited to symbolic reasoning. This specialization is not rigid or directly visible from the outside, but the operating principle is the same: many experts are available, only a few are active for each token. For colibri, this property is decisive. If the model were fully dense, running a roughly 750B-parameter model on a consumer PC would be much harder, because every step would require accessing an enormous mass of weights. With MoE, by contrast, the system can treat the model as a collection of blocks that can be loaded on demand. In practice, it works as shown in the figure. During generation, the system identifies the expert it needs, loads it into memory, and keeps it cached for as long as necessary using an LRU policy. Code analysis I analyzed the project's source code to better understand how it is structured. The core is glm.c, which implements the engine in a single C file. The file includes several backends for running matrix multiplication on different GPUs, such as CUDA for Nvidia and Metal for macOS, but the core remains glm.c. It is worth noting that glm.c is strictly limited to the engine: it reads input from standard input and writes output to standard output. This keeps the engine logic concentrated in one file. The file is fairly complex, but it does nothing else, which means it can be read and understood as a whole. I spent a few hours reading the source and eventually understood its structure. That matters, because it means the project can be extended and adapted to other architectures. The engine is strictly GLM-only, although its library functions can presumably be reused with other models. It is a C translation of the modeling_glm_moe_dsa.py file from Hugging Face's Transformers library. A model, after all, is essentially a collection of large matrices being multiplied together. colibri uses a model quantized to INT4, meaning 4 bits per weight. If the original model is BF16, that means 2 bytes per weight; INT4 uses 0.5 bytes per weight. The quantized model therefore takes up about 25% of the original space, reducing storage for the quantized weights alone by roughly 75%. colibri also supports Multi Token Prediction, but those layers were kept in INT8 because 4-bit quantization is not stable enough there. The model has to be downloaded separately, because it is stored in a format specifically designed for colibri. On top of this engine there is currently a Python wrapper. It launches the engine, implements an OpenAI-compatible API, provides a text chat interface, and, interestingly, parses the chat format. I find this design choice very appealing because it simplifies development and makes it possible to work on the engine without getting distracted by details that are not relevant to the core code. Personally, I already have a few contributions in mind for the project. For now, though, I am starting by sharing what I have learned so far.$

Supergoofy @superg00fy

almost 5 years ago

@thetitanborn https://t.co/lAvDLS2bVp

Supergoofy @superg00fy

almost 5 years ago

@stickytoffita https://t.co/lAvDLS2bVp

Who to follow

Marcia Cunha 🚩

@Sucubus

'Eppur si muove’! Psicóloga clinicando em Botafogo. Adendo: Flamenguista, Mangueirense, apaixonada pelo Baiana System e filha de Ogum. Salve!

Sò_Norman

@NormanUncle

Padre di tuo cugggino, famoso educatore moderno... HA STATO IL VACCINOH !

Ramūnas Blavaščiūnas

@ramunasbl

#Photography, #Mac, #coffee, and #technology lover :) (AKA Munas)

Supergoofy @superg00fy

over 5 years ago

Fractal

FRACTAL ID @Fractal_ID

over 5 years ago

📣 $FCL Whitelist Announcement! The Whitelist lottery will be open Today Monday February 15th at 2PM UTC and will close on February 19th at the same time. Follow 3 easy steps mentioned in the article to qualify! 😍 $POLS https://t.co/MPDLLdUd73