Grayson Hieb

[Grayson Hieb @LowBand21]

@itsfoss Rust dev here, they're great tools. But primarily due to functional changes, not language. Rust enables perf and safety guarantees, but the C based tools are already often as fast or faster, and commonly very safe after receiving patches for decades. Built in rust isn't enough

[Andrej Karpathy @karpathy]

Seeding my Bear ʕ•ᴥ•ʔ blog with more random posts, e.g. here's something I had on backlog for a while:

# The append-and-review note

An approach to note taking that I stumbled on and has worked for me quite well for many years. I find that it strikes a good balance of being super simple and easy to use but it also captures the majority of day-to-day note taking use cases.

Data structure. I maintain one single text note in the Apple Notes app just called "notes". Maintaining more than one note and managing and sorting them into folders and recursive substructures costs way too much cognitive bloat. A single note means CTRL+F is simple and trivial. Apple does a good job of optional offline editing, syncing between devices, and backup.

Append. Any time any idea or any todo or anything else comes to mind, I append it to the note on top, simply as text. Either when I'm on my computer when working, or my iPhone when on the go. I don't find that tagging these notes with any other structured metadata (dates, links, concepts, tags) is that useful and I don't do it by default. The only exception is that I use tags like "watch:", "listen:", or "read:", so they are easy to CTRL+F for when I'm looking for something to watch late at night, listen to during a run/walk, or read during a flight, etc.

Review. As things get added to the top, everything else starts to sink towards the bottom, almost as if under gravity. Every now and then, I fish through the notes by scrolling downwards and skimming. If I find anything that deserves to not leave my attention, I rescue it towards the top by simply copy pasting. Sometimes I merge, process, group or modify notes when they seem related. I delete a note only rarely. Notes that repeatedly don't deserve attention will naturally continue to sink. They are never lost, they just don't deserve the top of mind.

Example usage:

- Totally random idea springs to mind but I'm on the go and can't think about it, so I add it to the note, to get back around to later.
- Someone at a party mentions a movie I should watch.
- I see a glowing review of a book while doom scrolling through X.
- I sit down in the morning and write a small TODO list for what I'd like to achieve that day.
- I just need some writing surface for something I'm thinking about.
- I was going to post a tweet but I think it needs a bit more thought. Copy paste into notes to think through a bit more later.
- I find an interesting quote and I want to be reminded of it now and then.
- My future self should really think about this thing more.
- I'm reading a paper and I want to note some interesting numbers down.
- I'm working on something random and I just need a temporary surface to CTRL+C and CTRL+V a few things around.
- I keep forgetting that shell command that lists all Python files recursively so now I keep it in the note.
- I'm running a hyperparameter sweep of my neural network and I record the commands I ran and the eventual outcome of the experiment.
- I feel stressed that there are too many things on my mind and I worry that I'll lose them, so I just sit down and quickly dump them into a bullet point list.
- I realize while I'm re-ordering some of my notes that I've actually thought about the same thing a lot but from different perspectives. I process it a bit more, merge some of the notes into one. I feel additional insight.

When I note something down, I feel that I can immediately move on, wipe my working memory, and focus fully on something else at that time. I have confidence that I'll be able to revisit that idea later during review and process it when I have more time.

My note has grown quite giant over the last few years. It feels nice to scroll through some of the old things/thoughts that occupied me a long time ago. Sometimes ideas don't stand the repeated scrutiny of a review and they just sink deeper down. Sometimes I'm surprised that I've thought about something for so long. And sometimes an idea from a while ago is suddenly relevant in a new light.

One text note ftw.

[Noam Brown @polynoamial]

@OpenAI @rao2z @OpenAI's o1 thinks for seconds, but we aim for future versions to think for hours, days, even weeks. Inference costs will be higher, but what cost would you pay for a new cancer drug? For breakthrough batteries? For a proof of the Riemann Hypothesis? AI can be more than chatbots

[Andrej Karpathy @karpathy]

⚡️ Excited to share that I am starting an AI+Education company called Eureka Labs.
The announcement:

---
We are Eureka Labs and we are building a new kind of school that is AI native.

How can we approach an ideal experience for learning something new? For example, in the case of physics one could imagine working through very high quality course materials together with Feynman, who is there to guide you every step of the way. Unfortunately, subject matter experts who are deeply passionate, great at teaching, infinitely patient and fluent in all of the world's languages are also very scarce and cannot personally tutor all 8 billion of us on demand.

However, with recent progress in generative AI, this learning experience feels tractable. The teacher still designs the course materials, but they are supported, leveraged and scaled with an AI Teaching Assistant who is optimized to help guide the students through them. This Teacher + AI symbiosis could run an entire curriculum of courses on a common platform. If we are successful, it will be easy for anyone to learn anything, expanding education in both reach (a large number of people learning something) and extent (any one person learning a large amount of subjects, beyond what may be possible today unassisted).

Our first product will be the world's obviously best AI course, LLM101n. This is an undergraduate-level class that guides the student through training their own AI, very similar to a smaller version of the AI Teaching Assistant itself. The course materials will be available online, but we also plan to run both digital and physical cohorts of people going through it together.

Today, we are heads down building LLM101n, but we look forward to a future where AI is a key technology for increasing human potential. What would you like to learn?
---

@EurekaLabsAI is the culmination of my passion in both AI and education over ~2 decades. My interest in education took me from YouTube tutorials on Rubik's cubes to starting CS231n at Stanford, to my more recent Zero-to-Hero AI series. While my work in AI took me from academic research at Stanford to real-world products at Tesla and AGI research at OpenAI. All of my work combining the two so far has only been part-time, as side quests to my "real job", so I am quite excited to dive in and build something great, professionally and full-time.

It's still early days but I wanted to announce the company so that I can build publicly instead of keeping a secret that isn't. Outbound links with a bit more info in the reply!

[Alfaiz Ali @AlfaizAliX]

Less than 24 hours since Anthropic released Claude 3.5 Sonnet, and it surpassed GPT-4o.

Here are 10 wild examples you don't want to miss:

[Grayson Hieb @LowBand21]

@notch Given your familiarity and experience with windows something like Linux Mint would be perfect to try before writing Linux off. Trying in a VM would be pretty easy. You'll likely find mint stable, relatively familiar, and highly capable. Checkout flatpaks for package management.

[Andrej Karpathy @karpathy]

# explaining llm.c in layman terms

Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity.

For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very general Tensor abstraction (a way to arrange and manipulate arrays of numbers that hold the parameters and activations of the neural network), a very general Autograd engine for backpropagation (the algorithm that trains the neural network parameters), and a large collection of deep learning layers you may wish to use in your neural network. The PyTorch project is 3,327,184 lines of code in 11,449 files.

On top of that, PyTorch is written in Python, which is itself a very high-level language. You have to run the Python interpreter to translate your training code into low-level computer instructions. For example the cPython project that does this translation is 2,437,955 lines of code across 4,306 files.

I am deleting all of this complexity and boiling the LLM training down to its bare essentials, speaking directly to the computer in a very low-level language (C), and with no other library dependencies. The only abstraction below this is the assembly code itself. I think people find it surprising that, by comparison to the above, training an LLM like GPT-2 is actually only a ~1000 lines of code in C in a single file. I am achieving this compression by implementing the neural network training algorithm for GPT-2 directly in C. This is difficult because you have to understand the training algorithm in detail, be able to derive all the forward and backward pass of backpropagation for all the layers, and implement all the array indexing calculations very carefully because you don’t have the PyTorch tensor abstraction available. So it’s a very brittle thing to arrange, but once you do, and you verify the correctness by checking agains PyTorch, you’re left with something very simple, small and imo quite beautiful.

Okay so why don’t people do this all the time?

Number 1: you are giving up a large amount of flexibility. If you want to change your neural network around, in PyTorch you’d be changing maybe one line of code. In llm.c, the change would most likely touch a lot more code, may be a lot more difficult, and require more expertise. E.g. if it’s a new operation, you may have to do some calculus, and write both its forward pass and backward pass for backpropagation, and make sure it is mathematically correct.

Number 2: you are giving up speed, at least initially. There is no fully free lunch - you shouldn’t expect state of the art speed in just 1,000 lines. PyTorch does a lot of work in the background to make sure that the neural network is very efficient. Not only do all the Tensor operations very carefully call the most efficient CUDA kernels, but also there is for example torch.compile, which further analyzes and optimizes your neural network and how it could run on your computer most efficiently. Now, in principle, llm.c should be able to call all the same kernels and do it directly. But this requires some more work and attention, and just like in (1), if you change anything about your neural network or the computer you’re running on, you may have to call different kernels, with different parameters, and you may have to make more changes manually.

So TLDR: llm.c is a direct implementation of training GPT-2. This implementation turns out to be surprisingly short. No other neural network is supported, only GPT-2, and if you want to change anything about the network, it requires expertise. Luckily, all state of the art LLMs are actually not a very large departure from GPT-2 at all, so this is not as strong of a constraint as you might think. And llm.c has to be additionally tuned and refined, but in principle I think it should be able to almost match (or even outperform, because we get rid of all the overhead?) PyTorch, with not too much more code than where it is today, for most modern LLMs.

And why I am working on it? Because it’s fun. It’s also educational, because those 1,000 lines of very simple C are all that is needed, nothing else. It's just a few arrays of numbers and some simple math operations over their elements like + and *. And it might even turn out to be practically useful with some more work that is ongoing.

[ryandcrypto @ryandcrypto]

@MKBHD Ah the normies are back

[Andrej Karpathy @karpathy]

I usually just go down the list of a few posts and cheery-pick, e.g.:
https://t.co/2PqHMDRLz7
https://t.co/u1xrhfgQzq
https://t.co/0APzMi8czG
for this round I think the major deviation is that I'm going to give @warpdotdev a shot as my Terminal. It looks nice only they are sketching me out a bit with their telemetry, and for some reason needing a login and a connection to the internet.

[The Bitcoin Historian @pete_rizzo_]

NEW: Satoshi Nakamoto's earliest collaborator Martii 'Sirius' Malmi just released their entire email history.

At 120 pages, its the most significant addition to the archives of #Bitcoin's unknown inventor.

Here are the most important new findings ✨

[Gabor Gurbacs @gaborgurbacs]

@elonmusk “A computer is like air conditioning – it becomes useless when you open Windows.” -Linus Torvalds

[Dr. Rhonda Patrick @foundmyfitness]

Establishing a solid foundation of muscle mass before the age of 50 is crucial. We know muscle mass peaks in our 20s and 30s, then declines at about 8% per decade, speeding up to 15% per decade after we hit 70.

This means by the time we're in our 70s and 80s, we're left with just 60-80% of the muscle mass from our 30s.

The pillars of muscle growth are resistance training and adequate protein intake. However, as we age, we face anabolic resistance, where our muscles don't respond to amino acids as effectively.

This phenomenon compounds muscle atrophy, highlighting the critical importance of proactively building and maintaining muscle mass early in life to ensure longevity and enhance the quality of life in our later years.

[Marques Brownlee @MKBHD]

Having a hard time processing this... generative fill... for videos... built into Premiere

[fudge @fudgeposting]

the male urge to get a giant warehouse and put a squat rack next to the air mattress with one desk, a projector, massive speakers, a cnc machine and a 3d printer

[Elon Musk @elonmusk]

@karpathy “Prompt engineering” is natural language programming

[Charly Wargnier ♨️ @DataChaz]

This is moving way too fast!

@OpenAI has just announced a new set of plugins for uploading and processing video clips! 🤯

And just like that, they made so many SaaS obsolete!

[Elon Musk @elonmusk]

Accounts engaging in repeated, egregious weaponization of DMCA on Twitter or encouraging weaponization of DMCA will receive temporary suspensions.

That said, reasonable media takedown requests are, of course, appropriate and will always be supported.

[Alohe @alemalohe]

@Snowden Out of all the sci-fi movies out there, it's sad that this is the most probable

[Mass Effect @masseffect]

This sounds familiar...