These 94 lines of code are everything that is needed to train a neural network. Everything else is just efficiency.
This is my earlier project Micrograd. It implements a scalar-valued auto-grad engine. You start with some numbers at the leafs (usually the input data and the neural network parameters), build up a computational graph with operations like + and * that mix them, and the graph ends with a single value at the very end (the loss). You then go backwards through the graph applying chain rule at each node to calculate the gradients. The gradients tell you how to nudge your parameters to decrease the loss (and hence improve your network).
Sometimes when things get too complicated, I come back to this code and just breathe a little. But ok ok you also do have to know what the computational graph should be (e.g. MLP -> Transformer), what the loss function should be (e.g. autoregressive/diffusion), how to best use the gradients for a parameter update (e.g. SGD -> AdamW) etc etc. But it is the core of what is mostly happening.
The 1986 paper from Rumelhart, Hinton, Williams that popularized and used this algorithm (backpropagation) for training neural nets:
https://t.co/f52IcDNitR
micrograd on Github: https://t.co/GaTd16jRnB
and my (now somewhat old) YouTube video where I very slowly build and explain:
https://t.co/EPGG6kd5Yz
Do you have ADHD or trouble staying focused while reading a book?
Here's a simple hack I created that changed my life.
It's called a "Distraction Catcher":
How to do hard stuff:
write down a 1 page doc with your key assumptions, the hypothesis/goal you want to test, and your plan to do it
now read it with a maximally critical eye…does it make sense? are there gaps? iterate until you can’t find any
now go execute like crazy
When you don't take venture capital:
- You can be profitable and proud
- You are debt free
- You can sell your business for $5M and make life changing money
- You don't have to care about valuation
- You decide if and when to grow
- You make all the decisions not a Silicon Valley VC who spends 1 hour a week on your business
- You reward yourself and your team through dividends
- You don't need to send updates to investors
- You have a stable business for your team and customers
- You put customers first, always
- You can change your business without asking anyone
- You choose when or if you want to sell without any approvals
- You can pass on the business to your family
- You keep your company's culture the way you like it
- You aim for a long-lasting business without being rushed
- You operate without the weight of the exit goalpost, allowing the business's essence to evolve naturally
- You have no bosses
- You might make less, but it's your baby
- You can do what you think is right and that's everything
- You don't have to be feel pressured to spend your capital
- You're building without strings attached
- You're chasing fundamentals never funding
The next generation of entrepreneurs, solopreneurs and multipreneurs (@MultipreneurGuy) value freedom and sustainable growth over quick hits of VC dollars
This fires me up... you?
People think they struggle to grow because they are not good at learning, they are actually not good at unlearning.
Their old identity is in the way, their inner insecure voice keeps telling them what they cannot do.
They must learn to ignore it in order to rebuild themselves.
“give yourself a lot of shots to get lucky” is even better advice than it appears on the surface.
luck isn’t an independent variable but increases super-linearly with more surface area—you meet more people, make more connections between new ideas, learn patterns, etc.
I built many projects as a developer. I made $0.
Then I learned marketing.
Now I own a $350k+ a year SaaS business.
Here is how you can do the same if you're a developer 🧵👇
You are sabotaging your own personal and professional growth if you're not finding time to read.
These are the books that changed my life, and what I learned from them:
most of the madness you see on the internet can be explained by the simple fact that an enormous percentage of jobs are absolutely pointless roleplaying activities