Gokulakrishnan

Deep learning practitioner.

Building bootstrapped autonomous machines @ podcasts, books, dota

[Gokulakrishnan @GokulEpiphany]

In every factory that I visit in India, I see lots of people just moving things around. Why aren't machines (AMRs) doing these jobs?
A look at how things are today and what we are focusing on.

[Gokulakrishnan @GokulEpiphany]

@LinzhanMou Great work! btw, is automated rigging solved? any idea?

[Gokulakrishnan @GokulEpiphany]

Help us find my parents and remaining 34 people.
Exact lat/lang where this happened - from survivor in bus 2 @CMOTamilnadu @mkstalin @Udhaystalin @imrajmohan @MEAIndia @DrSJaishankar @PMOIndia
https://t.co/lAyBa2Ychu

[Gokulakrishnan @GokulEpiphany]

We are trying to pass this info to nepal army. A group of people from nepal are helping us in this tuff times. Full video on what happened to 3 buses. @CMOTamilnadu @mkstalin @Udhaystalin @imrajmohan @MEAIndia @DrSJaishankar @PMOIndia

[Gokulakrishnan @GokulEpiphany]

My parents are missing for 3 days. We are in Kathmandu for last two days to trace them. Here is what we know so far. We are requesting for search in and around location of bus 2
@CMOTamilnadu
@mkstalin
@Udhaystalin
@imrajmohan
@MEAIndia
@DrSJaishankar

[Gokulakrishnan @GokulEpiphany]

@godaharuki @SakanaAILabs My pleasure! the reports are so in depth! Been following Sakana labs for a long time, congratulations on the release and good luck!

[Gokulakrishnan @GokulEpiphany]

hey @SakanaAILabs just tried Sakana Marlin for couple of researches. Got back presentation for both, but unable to access reports -> the website redirects to onboarding plan. please fix.

[Gokulakrishnan @GokulEpiphany]

@godaharuki @SakanaAILabs Yes, it's fixed. Thank you 😊

[Gokulakrishnan @GokulEpiphany]

@realmrfakename @huggingface Hey, were the results good? How was the experience?

[Gokulakrishnan @GokulEpiphany]

@kwindla Is this tavus? Or something else?

[Gokulakrishnan @GokulEpiphany]

@realmrfakename Yea, if someone says, they are the deepseek movement, they mostly won't be. Love echo TTS. But going unnoticed

[Gokulakrishnan @GokulEpiphany]

@kwindla Beyond moshi, has there been any progress at all in building these end to end model? Even the team behind moshi, moved to unmute to make their systems modular. I'm wondering if there is any one working at this at all.

[Gokulakrishnan @GokulEpiphany]

@ezyang This is the content that makes me use x a lot :D thanks!

[Gokulakrishnan @GokulEpiphany]

Thank you @karpathy for being active on goodreads. As a noob divine dota player, I was always curious about what goes into building a game such as WoW or Dota 2. I was not aware of this book.

[Gokulakrishnan @GokulEpiphany]

haha!

[Tricky @trickylabyrinth]

2 viewer streamer

[Andrej Karpathy @karpathy]

⚡️ Excited to share that I am starting an AI+Education company called Eureka Labs.
The announcement:

---
We are Eureka Labs and we are building a new kind of school that is AI native.

How can we approach an ideal experience for learning something new? For example, in the case of physics one could imagine working through very high quality course materials together with Feynman, who is there to guide you every step of the way. Unfortunately, subject matter experts who are deeply passionate, great at teaching, infinitely patient and fluent in all of the world's languages are also very scarce and cannot personally tutor all 8 billion of us on demand.

However, with recent progress in generative AI, this learning experience feels tractable. The teacher still designs the course materials, but they are supported, leveraged and scaled with an AI Teaching Assistant who is optimized to help guide the students through them. This Teacher + AI symbiosis could run an entire curriculum of courses on a common platform. If we are successful, it will be easy for anyone to learn anything, expanding education in both reach (a large number of people learning something) and extent (any one person learning a large amount of subjects, beyond what may be possible today unassisted).

Our first product will be the world's obviously best AI course, LLM101n. This is an undergraduate-level class that guides the student through training their own AI, very similar to a smaller version of the AI Teaching Assistant itself. The course materials will be available online, but we also plan to run both digital and physical cohorts of people going through it together.

Today, we are heads down building LLM101n, but we look forward to a future where AI is a key technology for increasing human potential. What would you like to learn?
---

@EurekaLabsAI is the culmination of my passion in both AI and education over ~2 decades. My interest in education took me from YouTube tutorials on Rubik's cubes to starting CS231n at Stanford, to my more recent Zero-to-Hero AI series. While my work in AI took me from academic research at Stanford to real-world products at Tesla and AGI research at OpenAI. All of my work combining the two so far has only been part-time, as side quests to my "real job", so I am quite excited to dive in and build something great, professionally and full-time.

It's still early days but I wanted to announce the company so that I can build publicly instead of keeping a secret that isn't. Outbound links with a bit more info in the reply!

[Gokulakrishnan @GokulEpiphany]

Ggwp, longest video till date. Tysm https://t.co/MhBbdISyfY

[Gokulakrishnan @GokulEpiphany]

@realmrfakename Nope :( This was for a client!

[Daniel Han @danielhanchen]

LLAMA-3 IS OUT!
https://t.co/VfulJWwl8M

[Andrej Karpathy @karpathy]

# explaining llm.c in layman terms

Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity.

For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very general Tensor abstraction (a way to arrange and manipulate arrays of numbers that hold the parameters and activations of the neural network), a very general Autograd engine for backpropagation (the algorithm that trains the neural network parameters), and a large collection of deep learning layers you may wish to use in your neural network. The PyTorch project is 3,327,184 lines of code in 11,449 files.

On top of that, PyTorch is written in Python, which is itself a very high-level language. You have to run the Python interpreter to translate your training code into low-level computer instructions. For example the cPython project that does this translation is 2,437,955 lines of code across 4,306 files.

I am deleting all of this complexity and boiling the LLM training down to its bare essentials, speaking directly to the computer in a very low-level language (C), and with no other library dependencies. The only abstraction below this is the assembly code itself. I think people find it surprising that, by comparison to the above, training an LLM like GPT-2 is actually only a ~1000 lines of code in C in a single file. I am achieving this compression by implementing the neural network training algorithm for GPT-2 directly in C. This is difficult because you have to understand the training algorithm in detail, be able to derive all the forward and backward pass of backpropagation for all the layers, and implement all the array indexing calculations very carefully because you don’t have the PyTorch tensor abstraction available. So it’s a very brittle thing to arrange, but once you do, and you verify the correctness by checking agains PyTorch, you’re left with something very simple, small and imo quite beautiful.

Okay so why don’t people do this all the time?

Number 1: you are giving up a large amount of flexibility. If you want to change your neural network around, in PyTorch you’d be changing maybe one line of code. In llm.c, the change would most likely touch a lot more code, may be a lot more difficult, and require more expertise. E.g. if it’s a new operation, you may have to do some calculus, and write both its forward pass and backward pass for backpropagation, and make sure it is mathematically correct.

Number 2: you are giving up speed, at least initially. There is no fully free lunch - you shouldn’t expect state of the art speed in just 1,000 lines. PyTorch does a lot of work in the background to make sure that the neural network is very efficient. Not only do all the Tensor operations very carefully call the most efficient CUDA kernels, but also there is for example torch.compile, which further analyzes and optimizes your neural network and how it could run on your computer most efficiently. Now, in principle, llm.c should be able to call all the same kernels and do it directly. But this requires some more work and attention, and just like in (1), if you change anything about your neural network or the computer you’re running on, you may have to call different kernels, with different parameters, and you may have to make more changes manually.

So TLDR: llm.c is a direct implementation of training GPT-2. This implementation turns out to be surprisingly short. No other neural network is supported, only GPT-2, and if you want to change anything about the network, it requires expertise. Luckily, all state of the art LLMs are actually not a very large departure from GPT-2 at all, so this is not as strong of a constraint as you might think. And llm.c has to be additionally tuned and refined, but in principle I think it should be able to almost match (or even outperform, because we get rid of all the overhead?) PyTorch, with not too much more code than where it is today, for most modern LLMs.

And why I am working on it? Because it’s fun. It’s also educational, because those 1,000 lines of very simple C are all that is needed, nothing else. It's just a few arrays of numbers and some simple math operations over their elements like + and *. And it might even turn out to be practically useful with some more work that is ongoing.