Ilya Sutskever says the AI scaling era is running out of road because pre-training is about to hit a finite-data wall.
"At some point though, pre-training will run out of data. The data is very clearly finite."
"Then, okay, what do you do next? Either you do some kind of a souped-up pre-training... or you're doing RL, or maybe something else."
"In some sense, we are back to the age of research."
His timeline is the part most AI investors and founders should hear:
"From 2012 to 2020, it was the age of research. Now from 2020 to 2025, it was the age of scaling... But now the scale is so big."
"Is the belief really that... if you had 100x more everything would be so different?"
"I don't think that's true. So it's back to the age of research again, just with big computers."
Ilya Sutskever says the AI scaling era is running out of road because pre-training is about to hit a finite-data wall.
"At some point though, pre-training will run out of data. The data is very clearly finite."
"Then, okay, what do you do next? Either you do some kind of a souped-up pre-training... or you're doing RL, or maybe something else."
"In some sense, we are back to the age of research."
His timeline is the part most AI investors and founders should hear:
"From 2012 to 2020, it was the age of research. Now from 2020 to 2025, it was the age of scaling... But now the scale is so big."
"Is the belief really that... if you had 100x more everything would be so different?"
"I don't think that's true. So it's back to the age of research again, just with big computers."
Why is India lagging in innovation?
Why don't our entrepreneurs build frontier AI models?
Who is to be blamed? Scientists? Entrepreneurs?
Actually, a more relevant question is: what is the incentive for model building?
The harsh reality is that most people, including trained researchers, find it hard to differentiate between model development and model implementation, and those who take the path of model development often struggle to get recognization.
I am not saying this is a problem specific to India. Model developers worldwide face difficulty in getting recognition. However, there is an additional difficulty in India: recognization from Indian researchers is conditional -- the contribution should first be recognized in the West.
And the primary reason is that we have willfully handed over the job of judging science to scientists in the West (journal editors).
C. V. Raman recognized the problem of over-relying on foreign publishers. He started journals to address the problem. Unfortunately, scientists in India could not overcome their colonial hangover.
So before asking why Indian scientists do not build frontier models, let us fix the incentive structure.
Starting our own publishing platforms would be a good starting point for the long journey ahead.
(Image generated with Gemini)
@tehseenp It seems the ethanol policy has less to do with the import issue and more to do with the excess farm production caused by subsidy schemes. It is essentially a band-aid over the water management disaster caused by poor agricultural policies.
Does not make a lot of sense to me. Rajastan and West Bengal cannot be put together. Also looks like the flow of water is not taken into account. People living close to rivers will have more water security than those who live far away.
'Why don't you want IUML handling education ministry in Kerala?'
Well, for starters...
They celebrate Tipu Sultan. The barbarian who pillaged my home district and destroyed the Kerala School of Mathematics. The last indigenous scientific effort in India. Tipu destroyed the legacy and destiny of Kerala. The last Kerala Mathematician Sankara Varman had to flee for his life from Malabar to British Madras. Even Madhava of Sangamagrama's personal Krishna shrine was burned down and the family repaired it only a century later.
You want these guys to control the Education system of Kerala? Do you think they care about the intellectual and philosophical legacy of Kerala? No Malayali with historical literacy and self respect can forgive this cave man with a crown. But then again, Malayalis do not have literacy in things that matter most long term.
Won't be long before you see official Tipu Jayanti celebrations in Kerala. Bright future ahead! Intellectual destiny lost. And now demographic and political destiny also in free fall...
@svembu@garvit_sethii@SanjayTweets9 Yes, the correlation between exam scores and success in real life is very poor. Generations who have grown up being told otherwise will find it hard to accept.
Imagine spitting at the sun and think the spit has landed. BJP attacking India’s great son, Shri Biju Patnaik is a disgrace and an apology is due
Biju Patnaik was an Airforce pilot with Royal Indian Airforce and with Indian Airforce post Independence. When Pakistani backed tribals invaded Kashmir in Oct 1947 Indian Government organised airlift of India troops from Delhi to defend Kashmir . The airlift started morning on 27th Oct after signing of Instrument of Accession . Biju Patnaik was amongst the first set of pilots who landed their Dakota at Srinagar Airport in middle of fight without any intelligence whether the airfield was in enemy's control or still in forces control . This airlift by brave pilots of IAF helped stop the advancement of enemy and saved Kashmir
Biju Patnaik on request of PM Nehru flew to Jakarta and landed on an improvised airstrip using left-over fuel from abandoned Japanese military dumps, Patnaik took off with Indonesian political leaders such as Sutan Sjahrir and Sukarno, for a secret meeting with Nehru at New Delhi reaching India via Singapore on 24 July 1947.
During 1962 war when Sh. Biju Patnaik was CM of Orissa, PM Nehru asked him to help India setup a China monitoring Air Unit at
Charbatia Air Base near Cuttack . This Aviation Research Centre (ARC), a covert wing of the Intelligence Bureau and help with intelligence gathering against China and still functions under R&AW .
Dr. APJ Abdul Kalam and Biju Patnaik shared a crucial partnership in strengthening India's defense, centered on the establishment of the Wheeler Island (later named Dr Kalam Missile Testing Range) missile testing facility near Balasore In 1993, then-Odisha CM Patnaik allotted the land to DRDO on the condition that Kalam would develop advanced missiles there to safeguard India. Patnaik's decision to hand over the land for free to the DRDO was based on his visionary desire to ensure India was robust against external threats. The island became the crucial site for testing major Indian missiles like Agni ,Akash Prithvi, Brahmos etc
And BJP instead of honouring him with a Bharat Ratna has the audacity to call him a CIA AGENT!
In the zero hour today raised the issue of judicial over reach in the case of banning Professor Michel Danino and two academics from any role receiving public funds for including a chapter on judicial corruption in NCERT Class 8 textbooks