ReJ 𓀨 Renaldas Zioma

AI, GPU & Neuromorphic chips 🚀 Let’s steal ideas from biology 🧠 My code runs on billion devices - brought Unity to iOS. PBR & ML to Unity 🎮 ex @Unity ex @EA

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

Under-the-hood look at #GPU architecture and performance that Machine Learning people might find interesting.
I originally gave this talk at Unity's internal ML Workshop in Seattle.

https://t.co/KnabqJcLLA
Please share, if you find it useful.

[Nils Cremer @nilscmr]

CPUs suck. We're building a new general-purpose chip that scales to thousands of cores while being more energy-efficient.

We're hiring hardware design engineers, consider joining us https://t.co/8GCN0bN0xN

What we do differently ...

[mrdoob @mrdoob]

TIL Salvador Dalí invented Light Probes in 1952

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@bookazoid_ Nice to see those behaviours nevertheless!

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

Boarding ticket. New service levels unlocked! On the way to Revision2026 @revision_party

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

“Over-engineering”

Maritime museum in Lisbon.

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@phillip_isola Doesn’t that sound a little bit like what Diffusion Models learn - vector field pulling towards the manifold?

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@PixelCNinja @renderwonk I love that game! Such a low latency and fluent experience!

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

AI is the new OS… and everyone is back typing in the command line!

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@AxcanNathan Doesn’t that mean they are operating at DISK speed? If there is no cache and reuse it would constantly be waiting on KV cache data from disk.

Doesn’t matter, if you prefetch that data ahead, you still are bound by the slowest part in your pipeline.

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@BioAlessandro @__tinygrad__ Interesting! The optimisation space however should be humongous. Ideally need some form of a backpropagation through the synthesis to guide the design choices.

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@wholyv Pretty much exactly the same speed or perhaps even slower. Slower because of worse iteration time.

Think like that - what are the main performance bottlenecks in ML? Are they even related to Python code?

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@ssskryl @zzznah @tinytapeout This looks sweet almost realtime. What is the performance you are getting vs real hardware?

[Alex Mordvintsev @zzznah]

Working on the new simulator. I just wanted to see what Atari2600 fetching data from ROM looks like at CMOS FET level
(@tinytapeout TT09 Atari circuit by @__ReJ__)

[ReJ 𓀨 Renaldas Zioma @__ReJ__]

@Kaizou_Bhuvi_17 @samsoniuk The bottleneck for LLM (and most modern AI models) is not compute, but memory bandwidth. It doesn’t really matter how you arrange compute - you either have to put tens of billions weight parameters on chip OR have really wide bus to HBM/DRAM.

[NB @Noahbolanowski]

1968: "This flat pattern graphic was generated by applying a periodically repeating order to a Markov chain of unit color planes (5 colors).

The computer analyzed three existing artworks as data, captured the connective relationships of four unit color planes within them as a transition probability matrix, and generated random numbers reflecting this probabilistic formal grammar to draw a fourth-order approximation image. (Created using the OKITAC 5090A)."

[NB @Noahbolanowski]

1968: "...the computer shatters the everyday boundaries of traditional art, generating imaginative visions of free, newly possible worlds."

Quote from The 1st Japanese Computer Art Contest [第1回日本コンピュータ・アートコンテスト] in Computopia [コンピュートピア], April 1968.

[NB @Noahbolanowski]

Hiroshi Kawano in 1968: "...computer art will unravel the secrets of artistic creation, clarify artistic thought and theory, and awaken artists to the true role of the human in artistic creation."

Quote from Computopia [コンピュートピア], April 1968.

[Philosophy Of Physics @PhilosophyOfPhy]

What learning actually looks like at level of Neurons.