peter

i love the world and everything in it

cs @uwaterloo

[peter @dnexdev]

omg ty ⭐️⭐️. i'll announce this project soon trust

[peter @dnexdev]

@_enocla yo twin, kaggle notebooks get u two free T4's and csclub servers have a gpu

[peter @dnexdev]

@_enocla this looks so sick dude.

[peter @dnexdev]

@cindehaa omg play 134634110 (birdbrain) and 135512223 (cicada)

[peter @dnexdev]

@cindehaa OMG TYSM!!!!

[peter @dnexdev]

story time: tiramisu (ft. quantization)

when I finished training my 10M gpt model, the weights were 40MB.

i needed it to load in the browser.

so i cut it down to 11MB.

now it loads in seconds. no retraining necessary

here's how: 🧵

[peter @dnexdev]

end result? 4x smaller model. and loads in the browser in a few seconds.

check it out at https://t.co/3SrSys4tcS
check out the code at https://t.co/Ktl4RqwIRS

[peter @dnexdev]

the solution is int8 quantization

fancy word for saying "map the entire range of float32 vals to 256 integers (-128 to 127)"

scale = max|weights| / 127
quantized = round(weight / scale)

this gets us from 4 bytes per weight to 1 byte per weight.

4x smaller just like that

[peter @dnexdev]

@_evilbisexual_ you're not crazy, going ikj results in the compiler auto vectorizing it since the strides are all 1. at large N though, this is slower than tiling + intrinsics cuz of the cache

[peter @dnexdev]

story time: tiramisu (ft. matmul)

my matrix multiply took 14 minutes to train a neural network.

i was able to cut it down to 1 min 40.

this is what the naive implementation looks like 🧵

[peter @dnexdev]

@ZzZzZ500 omg ty

[peter @dnexdev]

@k4eo lol you are right, but it was really fun!

[peter @dnexdev]

@real_redp haha, this specific snippet was for cpu ops.
i'm also trying to make it stdlib only at link time.

[peter @dnexdev]

@ZzZzZ500 how

[peter @dnexdev]

@cindehaa the like from hank green makes it all worth it

[peter @dnexdev]

@Bobliuuu

[peter @dnexdev]

2nd fix is AVX2 SIMD
the cpu has 256-bit registers == 8 floats at once

one instruction:
_mm256_fmadd_ps(a, b, c) // c += a * b, 8 floats

performing a fused multiply-add op.

when this is combined with tiling this is 8x faster total.

look at the code: https://t.co/uelwBDC7Ol

[peter @dnexdev]

every iteration of k jumps to a completely different spot in memory. the CPU loads a cache line and uses one float. everything else gets evicted

the first optimization is cache tiling. just work on small blocks that fit in the l2 cache. so you can reuse the same data again

[peter @dnexdev]

@hiiinternet code at https://t.co/Ktl4RqwIRS

[peter @dnexdev]

@hiiinternet i've been building tiramisu: an ML framework in raw C++, no external ML libraries. wrote my own autograd engine, SIMD matmul, and a full GPT implementation. plus a CUDA backend. run it live in your browser: https://t.co/W9jdBndrbQ