dipan pal

Founder @ Voaige Inc.

[Aidan Clark @_aidan_clark_]

Only folks that started large scale DL work after ~GPT-2 think architecture doesn’t matter, the rest saw how much arch work had to happen to get here.

[Tim Duignan @TimothyDuignan]

This is the most surprising and exciting result of my career: we were running simulations of NaCl with a neural network potential that implicitly accounts for the effect of the water, ie a continuum solvent model (trained on normal MD) when Junji noticed something strange: 1/n

[Yann LeCun @ylecun]

As long as AI systems are trained to reproduce human-generated data (e.g. text) and have no search/planning/reasoning capability, performance will saturate below or around human level.

Furthermore, the amount of trials needed to reach that level will be far larger than the amount of trials needed to train humans.

LLMs are trained with 200,000 years worth of reading material and are still pretty dumb.
Their usefulness resides in their vast accumulated knowledge and language fluency. But they are still pretty dumb.

[dipan pal @dipankpal]

@karpathy About to see the dawn of the analog computer.

[Andrej Karpathy @karpathy]

The history of computing is repeating in an echo, except replace computers that do precise arithmetic on bytes with computers that do statistical arithmetic on tokens.

[Ayzaan Wahid @ayzwah]

For the past year we've been working on ALOHA Unleashed 🌋 @GoogleDeepmind - pushing the scale and dexterity of tasks on our ALOHA 2 fleet. Here is a thread with some of the coolest videos!

The first task is hanging a shirt on a hanger (autonomous 1x)

[dipan pal @dipankpal]

But the two camps don't necessarily compete. The trick is to realize the significant non-linear & non-obvious benefits to anyone who masters both. A challenging feat to pull through.

Like most hard things in life, the key is in striking the balance.

[dipan pal @dipankpal]

So far, scale has always won out in AI.

No matter how thoughtfully designed AI tools and components were, they have historically underperformed compared to general and scaled computation based methods.

[dipan pal @dipankpal]

@demishassabis @GoogleDeepMind Congratulations @demishassabis. Been following deep mind from its early days from the sidelines. Thoroughly deserved.

[murat 🍥 @mayfer]

vitamin b12 is so weird
- all life depends on it
- only bacteria can make it
- it has cobalt in it

all DNA and nerves would be fubared without trace amounts of cobalt in soil

[Brett Adcock @adcock_brett]

AI and Robotics is moving fast.

So, I share the most important research every week.

Here's everything you need to know and how to make sense out of it:

[Andrej Karpathy @karpathy]

# automating software engineering

In my mind, automating software engineering will look similar to automating driving. E.g. in self-driving the progression of increasing autonomy and higher abstraction looks something like:

1. first the human performs all driving actions manually
2. then the AI helps keep the lane
3. then it slows for the car ahead
4. then it also does lane changes and takes forks
5. then it also stops at signs/lights and takes turns
6. eventually you take a feature complete solution and grind on the quality until you achieve full self-driving.

There is a progression of the AI doing more and the human doing less, but still providing oversight. In Software engineering, the progression is shaping up similar:

1. first the human writes the code manually
2. then GitHub Copilot autocompletes a few lines
3. then ChatGPT writes chunks of code
4. then you move to larger and larger code diffs (e.g. Cursor copilot++ style, nice demo here https://t.co/u8ueY0mGxZ)
5....
Devin is an impressive demo of what perhaps follows next: coordinating a number of tools that a developer needs to string together to write code: a Terminal, a Browser, a Code editor, etc., and human oversight that moves to increasingly higher level of abstraction.

There is a lot of work not just on the AI part but also the UI/UX part. How does a human provide oversight? What are they looking at? How do they nudge the AI down a different path? How do they debug what went wrong? It is very likely that we will have to change up the code editor, substantially.

In any case, software engineering is on track to change substantially. And it will look a lot more like supervising the automation, while pitching in high-level commands, ideas or progression strategies, in English.

Good luck to the team!

[dipan pal @dipankpal]

@ylecun Agreed. We need a bottoms up approach to learning and reasoning and not top down one.

[dipan pal @dipankpal]

This is exactly why LLMs will find it difficult to achieve superhuman general intelligence. We need a bottoms up approach to learning to reason and not a top down one.

[Yann LeCun @ylecun]

* Language is low bandwidth: less than 12 bytes/second. A person can read 270 words/minutes, or 4.5 words/second, which is 12 bytes/s (assuming 2 bytes per token and 0.75 words per token). A modern LLM is typically trained with 1x10^13 two-byte tokens, which is 2x10^13 bytes. This would take about 100,000 years for a person to read (at 12 hours a day).

* Vision is much higher bandwidth: about 20MB/s. Each of the two optical nerves has 1 million nerve fibers, each carrying about 10 bytes per second. A 4 year-old child has been awake a total 16,000 hours, which translates into 1x10^15 bytes.

In other words:
- The data bandwidth of visual perception is roughly 16 million times higher than the data bandwidth of written (or spoken) language.
- In a mere 4 years, a child has seen 50 times more data than the biggest LLMs trained on all the text publicly available on the internet.

This tells us three things:
1. Yes, text is redundant, and visual signals in the optical nerves are even more redundant (despite being 100x compressed versions of the photoreceptor outputs in the retina). But redundancy in data is *precisely* what we need for Self-Supervised Learning to capture the structure of the data. The more redundancy, the better for SSL.
2. Most of human knowledge (and almost all of animal knowledge) comes from our sensory experience of the physical world. Language is the icing on the cake. We need the cake to support the icing.
3. There is *absolutely no way in hell* we will ever reach human-level AI without getting machines to learn from high-bandwidth sensory inputs, such as vision.

Yes, humans can get smart without vision, even pretty smart without vision and audition. But not without touch. Touch is pretty high bandwidth, too.

[dipan pal @dipankpal]

@paulg @ycombinator Probably the best fitness app there is

[martin_casado @martin_casado]

India is home to much of the best talent the human race has to offer. It's contributions to CS rival that of any other country. And its future potential is better than most.

It'd be a crime if it regulated itself out of this leadership.

India would loose.
We'd all loose.

[dipan pal @dipankpal]

Unfortunate and premature move. Hope this is rectified soon.

[Bindu Reddy @bindureddy]

India just kissed its future goodbye!

Every company deploying a GenAI model now requires approval from the Indian government!

That is, you now need approval for merely deploying a 7b open source model 🤯🤯

If you know the Indian government, you know this will a huge drag! All forms will need to completed in triplicate and there will be a dozen hoops to jump through!

This is how monopolies thrive, countries decay and consumers suffer!

Sadly India is already dominated by monopolies, nepotism and bureaucracy and this new rule just made it far worse.

[dipan pal @dipankpal]

The world still needs a functional and easy to use foundational AI for robotics.

[Ken Goldberg @Ken_Goldberg]

#1 topic requested by @ycombinator is #Robotics: "Robotics hasn't yet had its GPT moment, but we think it’s close.... We're interested in people building software tools to help other people to make robots: https://t.co/YvcA1lpRky

[dipan pal @dipankpal]

@aleksliving Has to be Plex it. Nothing else will do.

[Richard Sutton @RichardSSutton]

I've studied intelligence all my long life, yet still I feel I learned important things about intelligence by reading this book.
Thank you, Max Bennett.