Kiambuthi Kinuthia

25% • 🇰🇪

[Firefly Aerospace @FireflySpace]

Watch Firefly land on the Moon! After identifying surface hazards and selecting a safe landing site, #BlueGhost landed directly over the target in Mare Crisium. A historic moment on March 2 we'll never forget. We have Moon dust on our boots! #BGM1

[Scott Manley @DJSnM]

We've never seen anything like this recording of a landing on another world. @fireflyspace

[Dwarkesh Patel @dwarkesh_sp]

.@satyanadella on:

- why he doesn’t believe in AGI but does believe in 10% economic growth

- Microsoft’s new topological qubit breakthrough and gaming world models

- whether Office commoditizes LLMs or the other way around

Links below. Enjoy!

Timestamps
0:00:00 - Intro
0:05:48 - AI won't be winner-take-all
0:16:02 - World economy growing by 10%
0:22:23 - Decreasing price of intelligence
0:31:03 - Microsoft's Quantum breakthrough
0:43:35 - Microsoft's gaming world model
0:50:35 - Legal barriers to AI
0:56:30 - Getting AGI safety right
1:05:43 - 34 years at Microsoft
1:11:31 - Does Satya Nadella believe in AGI?

[Dwarkesh Patel @dwarkesh_sp]

The @JeffDean & @NoamShazeer episode.

We talk about 25 years at Google, from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and soon to ASI.

My favorite part was Jeff's vision for AGI as one giant MoE that is grown in bits and pieces over time like a forest, rather than trained all at once.

Specialization, distillation, inference time scaling all emerge organically rather than by design.

Noam bites every bullet: 100x world GDP soon; let’s get a million automated researchers running in the Google datacenter; living to see the year 3000.

Links below. Enjoy!

Timestamps
0:00:00 - Intro
0:03:29 - Joining Google in 1999
0:06:20 - Future of Moore's Law
0:11:04 - Future TPUs
0:13:56 - Jeff’s undergrad thesis: parallel backprop
0:15:54 - LLMs in 2007
0:25:09 - “Holy shit” moments
0:27:28 - AI fulfills Google’s original mission
0:32:00 - Doing Search in-context
0:36:12 - The internal coding model
0:37:29 - What will 2027 models do?
0:43:20 - A new architecture every day?
0:49:10 - Automated chips and intelligence explosion
0:53:07 - Future of inference scaling
1:02:38 - Already doing multi-datacenter runs
1:08:15 - Debugging at scale
1:12:41 - Fast takeoff and superalignment
1:20:51 - A million evil Jeff Deans
1:24:22 - Fun times at Google
1:27:51 - World compute demand in 2030
1:34:37 - Getting back to modularity
1:44:48 - Keeping a giga-MoE in-memory
1:49:35 - All of Google in one model
1:57:59 - What’s missing from distillation
2:03:10 - Open research, pros and cons
2:09:58 - Going the distance

[Andrej Karpathy @karpathy]

New 3h31m video on YouTube:
"Deep Dive into LLMs like ChatGPT"

This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their "psychology", and how to get the best use them in practical applications.

We cover all the major stages:
1. pretraining: data, tokenization, Transformer neural network I/O and internals, inference, GPT-2 training example, Llama 3.1 base inference examples
2. supervised finetuning: conversations data, "LLM Psychology": hallucinations, tool use, knowledge/working memory, knowledge of self, models need tokens to think, spelling, jagged intelligence
3. reinforcement learning: practice makes perfect, DeepSeek-R1, AlphaGo, RLHF.

I designed this video for the "general audience" track of my videos, which I believe are accessible to most people, even without technical background. It should give you an intuitive understanding of the full training pipeline of LLMs like ChatGPT, with many examples along the way, and maybe some ways of thinking around current capabilities, where we are, and what's coming.

(Also, I have one "Intro to LLMs" video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version of this topic. They can still be combined, as the talk goes a lot deeper into other topics, e.g. LLM OS and LLM Security)

Hope it's fun & useful!
https://t.co/75mXcUBI8L

[Amjad Masad @amasad]

Whatever you need… make an app for that.

Now on your phone. For everyone. Free.

[Andrej Karpathy @karpathy]

There's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like "decrease the padding on the sidebar by half" because I'm too lazy to find it. I "Accept All" always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or webapp, but it's not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.

[Reinbō ⛩️ @Reinbo_game]

Browser version is now online! Any early feedback on our buggy mess would be greatly appreciated.
https://t.co/3DPwNXuBMc

#indiedev #gamedevelopment #pixelart #animation

[Nod Ulus @nodp53]

@elder_plinius OMG! @elder_plinius !!!

[ぬあー@箱٩( 'ω' )و @Nuaah_Box]

#三連休はフォロワーさんが増えるらしい

[Portalgraph @portalgraph]

Today is your last chance to see a tiny person dancing on a table. Come see us at Hall G, Level 1, booth #62801 #23. #CES2024 #portalgraph

[Dan Carlin @HardcoreHistory]

Happy Hardcore History Day everyone. I was really hoping to get this out before 2024 ended but it wasn't quite ready. But it is now.
I hope you like it!

https://t.co/IgjVqCagtL

[SpaceX @SpaceX]

Starship’s sixth flight test aimed to push the vehicle to its limits as we iterate towards a rapidly reusable rocket.

Up next: a new ship, new year, and new limits. Excitement still guaranteed → https://t.co/QNCSPTewLA

[Andrej Karpathy @karpathy]

DeepSeek (Chinese AI co) making it look easy today with an open weights release of a frontier-grade LLM trained on a joke of a budget (2048 GPUs for 2 months, $6M).

For reference, this level of capability is supposed to require clusters of closer to 16K GPUs, the ones being brought up today are more around 100K GPUs. E.g. Llama 3 405B used 30.8M GPU-hours, while DeepSeek-V3 looks to be a stronger model at only 2.8M GPU-hours (~11X less compute). If the model also passes vibe checks (e.g. LLM arena rankings are ongoing, my few quick tests went well so far) it will be a highly impressive display of research and engineering under resource constraints.

Does this mean you don't need large GPU clusters for frontier LLMs? No but you have to ensure that you're not wasteful with what you have, and this looks like a nice demonstration that there's still a lot to get through with both data and algorithms.

Very nice & detailed tech report too, reading through.

[Xuan Luo @XuanLuo14]

Excited to share our latest paper: Quark – Real-time, high-resolution, and generalized neural view synthesis! 🚀

🎥 1080p @ 30fps
🌍 Generalizes across diverse scenes & datasets
🏆 Achieves real-time quality rivaling even offline methods
🔗 Project page: https://t.co/no48RZMWcV

[irmgard @gardirms]

Declassified nuclear test footage enhanced with AI using techniques such as slow motion, frame interpolation, upscaling, and colourization.

[Berke Gökce @brkgkc16]

Apollo 16 launch footage taken from the atop Vehicle Assembly Building (VAB).

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

Diffusion Models Are Real-Time Game Engines

abs: https://t.co/ILLlxR9dj3
project page: https://t.co/x8pGRvfD9k

Google presents GameNGen, the first game engine powered entirely by a neural model that enables real-time interaction with a complex environment over long trajectories at high quality. GameNGen can interactively simulate the classic game DOOM at over 20 frames per second on a single TPU. Next frame prediction achieves a PSNR of 29.4, comparable to lossy JPEG compression. Human raters are only slightly better than random chance at distinguishing short clips of the game from clips of the simulation. GameNGen is trained in two phases: (1) an RL-agent learns to play the game and the training sessions are recorded, and (2) a diffusion model is trained to produce the next frame, conditioned on the sequence of past frames and actions.

[Kiambuthi Kinuthia @kyambvthi]

https://t.co/nRxAA6zrij

[Kiambuthi Kinuthia @kyambvthi]

the sounds from halfway here

[Kiambuthi Kinuthia @kyambvthi]

https://t.co/2CXHUbDEvM

[Andrej Karpathy @karpathy]

Be good. Future AIs are watching.