Beautiful work / attention to detail trying to get Gemma to finetune correctly. There are so many foot guns here to be super careful with. All of these issues don't throw any errors, they silently make your network worse.
A great example of what I wrote about in my "A Recipe for Training Neural Networks":
"""The "fast and furious" approach to training neural networks does not work and only leads to suffering. Now, suffering is a perfectly natural part of getting a neural network to work well, but it can be mitigated by being thorough, defensive, paranoid, and obsessed with visualizations of basically every possible thing. The qualities that in my experience correlate most strongly to success in deep learning are patience and attention to detail."""
And why I so emphasize the need for understanding all the parts of the deep learning stack in great detail. I exist in a perpetually terrified state of the remaining 20 silent bugs that certainly remain in my code.
# on technical accessibility
One interesting observation I think back to often:
- when I first published the micrograd repo, it got some traction on GitHub but then somewhat stagnated and it didn't seem that people cared much.
- then I made the video building it from scratch, and the repo immediately went through hockey stick growth and became a verty often cited reference for people learning backpropagation.
This was interesting because the micrograd code itself didn't change at all and it was up on GitHub for many months before, stagnating. The code made sense to me (because I wrote it), it was only ~200 lines of code, it was extensively commented in the .py files and in the Readme, so I thought surely it was clear and/or self-explanatory. I was very happy with myself about how minimal the code was for explaining backprop - it strips away a ton of complexity and just gets to the very heart of an autograd engine on one page of code. But others didn't seem to think so, so I just kind of brushed it off and moved on.
Except it turned out that what stood in its way was "just" a matter of accessibility. When I made the video that built it and walked through it, it suddenly almost 100X'd the overall interest and engagement with that exact same piece of code. Not only from beginners in the field who needed the full intro and explanation, but even from more technical/expert friends, who I think could have understood it if they looked at it long enough, but were deterred by a barrier to entry.
I think as technical people we have a strong bias to put up code or papers or the final thing and feel like things are mostly self-explanatory. It's there, and also it's commented, there is a Readme, so all is well, and if people don't engage then it's just because the thing is not good enough. But the reality is that there is still a large barrier to engage with your thing (even for other experts who might not feel like spending time/effort!), and you might be leaving somewhere 10-100X of the potential of that exact same piece of work on the table just because you haven't made it sufficiently accessible.
TLDR: Step 1 build the thing. Step 2 build the ramp. 📈
Some voice in your head will tell you that this is not necessary, but it is wrong.
The "centralized anything is evil by default, use defi and self-custody" ethos did very well this week, but remember that it too has risks: bugs in smart contract code.
Important to guard against it:
* Keep code simple
* Audits, formal verification, etc
* Defense in depth
This evening I learned about the magic number 0.4323320871 8590286890… Drop everything left of the decimal place. Take the reciprocal of this fractional part. Keep doing that. This makes a sequence of numbers. What do you notice about the integers left of the decimal place?
The SOLUNA Telenovela😱
—
Now you know #TAParmy why I never pushed, nor bought #SOL & #LUNA for myself
Call me an old fogey, but I have laser sharp instincts from years of combat and battlefield experience
Often you feel it, that is when something is too fast or too soon
Made a little CLI that just pipes my programming questions to GPT-3, so I now can ask it stuff when I'm in the command line!
LLMs are better than Stack Overflow now — I just ask it, and it gives me a comprehensive answer in one shot, right there in my terminal, in a couple secs.
#catecoin ‘s fiat PRICE is down because #BNB is down a lot
This is nothing unusual. #BNB flies later on, #CATE will surely fly
Buying #CATE is like owning #BNB but with a higher upside on a Bull Cycle