@david_bonilla@raulcotrina No la veas hasta que acabes los tres porque aunque solo empieza la historia, pesca de los tres libros haciendo una mezcla un poco rara. Mi preferido es el segundo.
* Language is low bandwidth: less than 12 bytes/second. A person can read 270 words/minutes, or 4.5 words/second, which is 12 bytes/s (assuming 2 bytes per token and 0.75 words per token). A modern LLM is typically trained with 1x10^13 two-byte tokens, which is 2x10^13 bytes. This would take about 100,000 years for a person to read (at 12 hours a day).
* Vision is much higher bandwidth: about 20MB/s. Each of the two optical nerves has 1 million nerve fibers, each carrying about 10 bytes per second. A 4 year-old child has been awake a total 16,000 hours, which translates into 1x10^15 bytes.
In other words:
- The data bandwidth of visual perception is roughly 16 million times higher than the data bandwidth of written (or spoken) language.
- In a mere 4 years, a child has seen 50 times more data than the biggest LLMs trained on all the text publicly available on the internet.
This tells us three things:
1. Yes, text is redundant, and visual signals in the optical nerves are even more redundant (despite being 100x compressed versions of the photoreceptor outputs in the retina). But redundancy in data is *precisely* what we need for Self-Supervised Learning to capture the structure of the data. The more redundancy, the better for SSL.
2. Most of human knowledge (and almost all of animal knowledge) comes from our sensory experience of the physical world. Language is the icing on the cake. We need the cake to support the icing.
3. There is *absolutely no way in hell* we will ever reach human-level AI without getting machines to learn from high-bandwidth sensory inputs, such as vision.
Yes, humans can get smart without vision, even pretty smart without vision and audition. But not without touch. Touch is pretty high bandwidth, too.
To celebrate the 30th birthday of DOOM, here's a thread of everything that I've found that DOOM can run on. Some are real, some... maybe not. It's up to you to figure out which 😉
Let's begin with:
DOOM... on an ultrasound scanner.
Naval Ravikant shares the most valuable advice he’s received
When asked about the best advice he’s ever received, @naval says he first ignored it.
“It's the same [piece of advice] that everyone who's been in the business for a long time and has a lot of gray hairs tells you… whether it's Warren Buffett telling you about public investing or whether it's an experienced venture capitalist telling you about private investing… They'll all tell you, ‘It’s the people, stupid.’”
There are three qualities to look for in a partner: intelligence, energy, and integrity.
“You need all three. You can't compromise on any one of them. Otherwise, you'll end up with either someone who's not smart (which does you no good) or someone who's not hardworking (which also does you no good) or the worst case is you end up with a smart, hardworking crook who ends up working against your interests…”