Disty

[Disty @Disty00]

42

[Disty @Disty00]

Finally got SDNQ INT8 MM to work on Intel NPUs.
Generation time with Z-Image Turbo SDNQ INT8 model drops from 10 Mins with pure CPU to 5 Mins with NPU + CPU.

Here is an image generated with NPU running the nn.Linear layers and CPU running the rest:

[Disty @Disty00]

SDNQ runs on a phone

[Disty @Disty00]

An experimental model with no text encoder that still can do full natural language just fine: https://t.co/TQxAyYltSo

[Disty @Disty00]

This model has no text encoder.

Prompt: "This illustration consists of two girls on the image. The woman on the right side has red hair, purple eyes and orange shirt. The woman on the left side has brown hair, blue eyes and white shirt. sfw rating, 2girls, white background"

[Disty @Disty00]

A simple diffusion transformer model trained at home in 2 weeks with 0.75 kWp power budget.

Uses Flux2 VAE.
No text encoder, diffusion transformer itself handles text tokens.
Diffusion Transformer and its nn.Embedding is randomly initialized.

[Disty @Disty00]

A simple transformer diffusion / rectified flow model trained on 2x Intel ARC B60 GPUs for 5 days.
It doesn't use any text encoder, it uses direct tokens from the tokenizer instead.
It uses Flux2 VAE.

[Disty @Disty00]

@Holianum Sapphire Pulse with stock bios. Just increased the power limit from 303W default to 402W with LACT and applied -20mV undervolt.

[Disty @Disty00]

optimized int8 mm kernel pulling 500w on a 400w gpu

[Disty @Disty00]

Added a whole new stack of custom floating point types to SDNQ

[Disty @Disty00]

@Kmllg @Ali_TongyiLab 1024x1024.
Code is in the model readme.

[Disty @Disty00]

Z-Image Turbo with INT8 quantization: https://t.co/H7xROj3FBa

With an RTX 5090 (on Runpod):
BF16 MM: 2.6 it/s
INT8 MM: 3.82 it/s
INT8 MM + torch.compile: 4.84 it/s

[Disty @Disty00]

@pakar_ai https://t.co/TNvokIYD9q

[Disty @Disty00]

Z-Image on Google Colab T4 with SDNQ UINT4 SVD

Normal matmul vs SDNQ INT8 matmul
18 s/it vs 6 s/it
3x faster with SDNQ INT8 matmul!

[Disty @Disty00]

And the quantized model: https://t.co/DmSQxPCANs

[Disty @Disty00]

Flux Dev
BF16 (23.8 GB) vs 4 bit SDNQ quant (6.8 GB)

[Disty @Disty00]

@main_horse torch.randn / normal dist with 0 mean and 1 std.

[Disty @Disty00]

FP16 vs BF16 vs INT8 (quantized, sym) vs UINT8 (quantized, asym) vs FP8 E4 (quantized, sym) after 1000000 steps of x.mul_(0.99).add_(y)
using quantization with group_size 32 and fp32 scales / zero_points

[Disty @Disty00]

Open sourced my quantization engine SDNQ: https://t.co/53veBayMyK

[Disty @Disty00]

Full training of an 8B model with Muon optimizer fully fits into 24 GB without any offloading with INT8