Kinda easy to forget when you've spent an hour vibing on Nishitani philosophy and Zen Buddhism with one and it was a highly engaging discussion that allowed you to explore complex questions.
Amusing how 99% of people using LLMs forget how these things work:
They are advanced probability machines. They generate the next most likely token (word) based in the input and their training.
Under the hood, itโs a giant matrix multiplication that has eerily good output.
@docmilanfar Hard agree. Did not read a good one till I got to Illinois and Doug Jones had his self written notes as a book from the Rice repository. First time things started making sense. Same with Yoram Bressler and his unpublished vector space signal processing book.
I gave a talk at GPU MODE workshop last week on llm.c
- the origin story of llm.c
- being naked in the world without PyTorch and having to re-invent Array, Autograd, Device, Dtype, Compile, Distributed
- how to port a PyTorch layer to 1) explicit PyTorch
- and then to 2) write the backward pass
- 3) port forward & backward pass to C
- 4) string all the layers together
- achieving one file of C with no dependencies that compiles and runs ~instantly, where all memory is pre-planned and allocated a single time, fully deterministic, portable code that can run on a potato or a von Neumann probe
- how most of llm.c was built at 1am-7am in a water villa porch in Maldives and why this is the recommended way to develop software
- convert all of it to run in CUDA on GPU in fp32
- port matmul to cuBLAS
- port attention to cuDNN flash-attention
- introduce bfloat16 mixed precision
- introduce many more optimizations and features like kernel fusions, Packed128, stochastic rounding, full determinism
- add multi-GPU training, NCCL, sharded optimizer
- add multi-node with MPI or file system or socket
- reproduce GPT-2 (1.6B) on one 8XH100 node in 24 hours for $672 in llm.c, achieving (at the time) 29% less memory, 19% faster training that PyTorch nightly, and much faster compile & run
- how open source development attracts Avengers from the internet
- port to training Llama 3 imminent (branch exists)
- many other notable forks
- last thought: how software abstractions like Python/PyTorch and everything else really exist only because humans are finite in knowledge, IQ and attention, and how with increasing AI capability LLMs may export custom binaries like llm.c for any application directly, tearing apart and refactoring all abstractions as needed.
<|endoftext|>
More links in reply
@moultano Why does it matter though? What decisions would we make differently and why? Does the OP expect people to use genetic distance as a policy tool?
idea. I wish Apple would buy Oura Ring and we'd have an Apple Ring.. I notice that I put on/remove my Apple Watch all the time, whereas I keep my ring on constantly
Would be amazing to use it to unlock my phone/laptop, maybe it could have some functionality if I tap it to control my Airpods, etc
Thoughts on other functionality? Anyone hate this idea?