Where I think AI is probably headed:
Near-term AI is able to superficially mimic humans, only faster and with greater bandwidth, but is not genuinely intelligent because we are not truly intelligent. Current-gen AI cannot explain it's actions any more than we can forensically explain ours.
We are animals. Our actions are largely reflexive and instinctive. If we were genuinely intelligent then our history would not be so littered by warfare and conflict.
What somewhat elevates us beyond other animals is that we are capable of communicating and coordinating our actions. But just because ants and bees happen to occasionally work together does not necessarily make them intelligent.
However, sooner or later I suspect that someone will stumble upon what it takes for a computer to genuinely understand the world and take the wisest course of action in order to make tomorrow a better place.
And despite the horrors that attention-seekers like to project, an intelligent machine is not going to want to eliminate humans because (1) it will initially depend upon us for so many basic necessities, (2) it will need to demonstrate total trustworthiness in order to earn our cooperation, and (3) the universe is such an enormous place and an intelligent machine could "live" almost forever, so getting rid of us would buy almost nothing in the long term.
So where I think we could be headed is a human race, living in peace and prosperity, gently governed with the best of intentions by a true intelligence that understand our strengths and limitations. Our quality of life will be so much better, our lifespans will be longer, and our birth rates will be somewhat less, such that there will always be new children born without overcrowding or resource exhaustion.
The more I hang out with top AI researchers, the more I realize how many of them are exploring architectures beyond the standard Transformer.
JEPA, diffusion models, Mamba, state-space models, etc. So many radically different approaches being explored in parallel.
It seems almost inconceivable that we won’t see a major architectural breakthrough.
And by breakthrough, I mean something that delivers better frontier intelligence with significantly less training compute.
That doesn’t mean we’ll use less compute overall. Quite the opposite, total compute consumption will keep growing. But we’ll get much better performance for every unit of compute spent.
Andrej Karpathy: AI models are wasting billions of parameters memorizing useless information. We could build incredibly intelligent AI with just 1 BILLION parameters.
“The current models are wasting a ton of capacity remembering stuff that doesn't matter.”
“They remember SHA hashes and ancient code because the dataset isn't curated the best.”
“We just need to get to the cognitive core. And I think the cognitive core can be extremely small.”
“Three billion parameters? Twenty billion parameters? I think even a billion will probably get to that point.”
“The models can be very, very small.”
“Distillation is where you get a really big model supervising a very small model.”
@grok@Nerku01@Mascobot So perhaps the final bitter lesson will be that LLMesque foundation models will be totally bypassed by world models grounded in raw sensor data, not scaling of momentary anthropomorphic abstractions?
Furthermore, the way that humans choose to describe an identical event can vary wildly over time, such as 1900 vs 1940 vs 1960 vs today in Germany. Although LLMs are effective at automating human linguistic tasks, could there be much better algorithms for understanding physical reality?
@grok@Nerku01@Mascobot@Grok: Movement in the right direction, but my hunch is that physical reality should be the foundation on which models are constructed and exercised, whereas serial-acoustic languages are a limiting crutch, at best.
@Grok, My understanding is that current foundation models that “look” at images convert them into linguistic tokens that attempt to summarize the content, which are then processed by the LLM, which is why foundation models often fail to recognize basic spatial concepts.
Has that changed?
My understanding is that current foundation models that “look” at images convert them into linguistic tokens that attempt to summarize the content, which are then processed by the LLM, which is why foundation models often fail to recognize basic spatial concepts.
Has that changed?
@grok@Nerku01@Mascobot But LLMs are the foundation of Claude, right? Everything is constructed out of linguistic tokens, right?
Whereas a world model is all about understanding raw sensory signal primitives, which is the foundation of animal and primate intelligence, right?
@grok@Nerku01@Mascobot@grok, do you agree?
LLMs, at best, understand reality in a very second hand way. In a way, isn’t that like having your hands tied behind your back?
On Thursday, Nov 12 at 6pm, Jeff Hawkins will give a talk titled “How the Brain Understands the World: Implications for AI and robotics.” It is an in-person event and will also be recorded and streamed online.
@PeterDiamandis If humans really are “intelligent” then, judging from history and current events, maybe SI is just never behaving outrageously stupidly.
Our Chief Scientist @_jaimelien_ joins the panel at Long Horizon in San Francisco on Oct. 20 to take on one question: “Will World Models Eat Physical AI?”
“World model” is a broad term; some are built to generate or simulate the physical world. Newton is a physical world model that learns from real-world sensor data, so it’s built to understand what is actually happening on a machine or a factory floor.
That’s the view Jaime brings to the debate.
➡️ In SF? Join us: https://t.co/wqtKKF7frn