Tegan Snyder

Corporate eCommerce by day, Cofounder by night, dad and husband in-between :)

[Tegan Snyder @tegansnyder]

I think we found the family daredevil. #funny

[Tegan Snyder @tegansnyder]

@KingAmadDiallo Sheep, especially domesticated breeds, have been selectively bred for traits beneficial to humans, such as high wool production. Unlike their wild counterparts, these domesticated sheep have lost many of their natural survival traits, including the ability to shed excess wool.

[Andrej Karpathy @karpathy]

# explaining llm.c in layman terms

Training Large Language Models (LLMs), like ChatGPT, involves a large amount of code and complexity.

For example, a typical LLM training project might use the PyTorch deep learning library. PyTorch is quite complex because it implements a very general Tensor abstraction (a way to arrange and manipulate arrays of numbers that hold the parameters and activations of the neural network), a very general Autograd engine for backpropagation (the algorithm that trains the neural network parameters), and a large collection of deep learning layers you may wish to use in your neural network. The PyTorch project is 3,327,184 lines of code in 11,449 files.

On top of that, PyTorch is written in Python, which is itself a very high-level language. You have to run the Python interpreter to translate your training code into low-level computer instructions. For example the cPython project that does this translation is 2,437,955 lines of code across 4,306 files.

I am deleting all of this complexity and boiling the LLM training down to its bare essentials, speaking directly to the computer in a very low-level language (C), and with no other library dependencies. The only abstraction below this is the assembly code itself. I think people find it surprising that, by comparison to the above, training an LLM like GPT-2 is actually only a ~1000 lines of code in C in a single file. I am achieving this compression by implementing the neural network training algorithm for GPT-2 directly in C. This is difficult because you have to understand the training algorithm in detail, be able to derive all the forward and backward pass of backpropagation for all the layers, and implement all the array indexing calculations very carefully because you don’t have the PyTorch tensor abstraction available. So it’s a very brittle thing to arrange, but once you do, and you verify the correctness by checking agains PyTorch, you’re left with something very simple, small and imo quite beautiful.

Okay so why don’t people do this all the time?

Number 1: you are giving up a large amount of flexibility. If you want to change your neural network around, in PyTorch you’d be changing maybe one line of code. In llm.c, the change would most likely touch a lot more code, may be a lot more difficult, and require more expertise. E.g. if it’s a new operation, you may have to do some calculus, and write both its forward pass and backward pass for backpropagation, and make sure it is mathematically correct.

Number 2: you are giving up speed, at least initially. There is no fully free lunch - you shouldn’t expect state of the art speed in just 1,000 lines. PyTorch does a lot of work in the background to make sure that the neural network is very efficient. Not only do all the Tensor operations very carefully call the most efficient CUDA kernels, but also there is for example torch.compile, which further analyzes and optimizes your neural network and how it could run on your computer most efficiently. Now, in principle, llm.c should be able to call all the same kernels and do it directly. But this requires some more work and attention, and just like in (1), if you change anything about your neural network or the computer you’re running on, you may have to call different kernels, with different parameters, and you may have to make more changes manually.

So TLDR: llm.c is a direct implementation of training GPT-2. This implementation turns out to be surprisingly short. No other neural network is supported, only GPT-2, and if you want to change anything about the network, it requires expertise. Luckily, all state of the art LLMs are actually not a very large departure from GPT-2 at all, so this is not as strong of a constraint as you might think. And llm.c has to be additionally tuned and refined, but in principle I think it should be able to almost match (or even outperform, because we get rid of all the overhead?) PyTorch, with not too much more code than where it is today, for most modern LLMs.

And why I am working on it? Because it’s fun. It’s also educational, because those 1,000 lines of very simple C are all that is needed, nothing else. It's just a few arrays of numbers and some simple math operations over their elements like + and *. And it might even turn out to be practically useful with some more work that is ongoing.

[Andrej Karpathy @karpathy]

Btw writing the llm.c training code would imo be a very interesting, impressive, self-contained and very meta challenge for LLM agents. The prompt is:

Take the PyTorch code train_gpt2.py
And write, compile and unit test a single .c file that reproduces the training: train_gpt2.c

The current models are not there, but we can check back in a year or two or so. If that worked...

[👩‍💻 Paige Bailey @DynamicWebPaige]

🔫 Badass! A team at the @MistralAI hackathon in SF trained the 7B open-source model to play DOOM, based on an ASCII representation of the current frame in the game. 🤯 @ID_AA_Carmack

[Tegan Snyder @tegansnyder]

@sm I’ve also seen this. Some larger enterprises have strict financial standards at play that require any vendor that is netting fees to be able to ensure those fees are accountable back to the payment vendor contractual terms. Happy to share more if Stripe wants to reach out.

[Tegan Snyder @tegansnyder]

@ericlaw Upstream injection?

[David Cramer @zeeg]

Sentry is giving $500k+ to open source it relies on this year, >2x last year. We didn’t start with a high $ amount - and honestly this is still small potatoes - so what’s stopping your company from doing it?

[Tegan Snyder @tegansnyder]

@housecor Wish this would just be a one liner baked into the W3C spec.

[Lenny Rachitsky @lennysan]

How @Linear builds product:

1. No product managers, just a head of product. PM duties are distributed across engineering and design.

2. No durable cross-functional teams. Teams assemble around a project and disperse once the project is done.

3. No metrics-based goals. Just a North Star company-level metric goal.

4. No A/B tests. Decisions are based on taste and opinions.

5. Job candidates go through a paid work trial. They join the team for 1-5 days and work on a real project with the team.

6. The team is completely remote. And always has been.

Also, they have more cash in the bank than they’ve ever raised in VC funding, they’ve been profitable for 2+ years (before it was cool), and have spent a grand total of $35K on paid marketing in the history of the company. Only two people have ever left the company, and their CEO, Karri Saarinen, is a designer (who rarely gives interviews).

To get an in-depth glimpse into how Linear builds product, see link below for my interview with @karrisaarinen

[Tegan Snyder @tegansnyder]

Just want to say that the product and team at @roboflow is amazing. Shout to Jacob Witt for some excellent advise on a computer vision project I’m working on.

[Tegan Snyder @tegansnyder]

@jaredpalmer Whats the verdict on your sleep now that you have this?

[Andrew White 🐦‍⬛ @andrewwhite01]

We report a model that can go from natural language instructions, to robot actions, to synthesized molecule with an LLM. We synthesized catalysts, a novel dye, and insect repellent from 1-2 sentence instructions. This has been a seemingly unreachable goal for years! 1/3

[Tegan Snyder @tegansnyder]

@theprincessxena @mikejulian One thing that “might” have saved you some time is using the simple query syntax. That allows your uses to query using the built in DSL “fieldName: val Something”. https://t.co/fn5cphfRI9 Overall though a nice article that walks through the struggle.

[Tegan Snyder @tegansnyder]

@GergelyOrosz @auchenberg Wonder if we can get any clarification if this will come to USA?

[Bill D'Alessandro @BillDA]

We spend over $100,000 per year to be sales tax compliant @ElementsBrands. That’s software, filing fees, and labor.

This screenshot is what proper sales tax compliance looks like for ONE order.

You must file a separate return for each jurisdiction and every state…quarterly.

[Tegan Snyder @tegansnyder]

@noclador W

[LlamaIndex 🦙 @llama_index]

Users have mentioned that a key pain point of applying LLM’s to SQL db's is: too many tables+columns, can’t fit context in one prompt!

LlamaIndex 0.4.12 helps solve that pain point: you can pass table context to our SQL index…and this context can be another index! 🧠👇

[Tegan Snyder @tegansnyder]

@daveholtz @BerkeleyHaas Sent you a DM, if your interested in adding a guest lecturer.

[Tegan Snyder @tegansnyder]

@JMGrange Now combine it with this https://t.co/CHIBSmEGcw and your in business :)

[Tegan Snyder @tegansnyder]

@philwinkle @brianjlange Congrats Phil this is great. You really are passionate about this space and I’m happy to see you invest more into Future Commerce. Looking forward to what’s ahead.