Paul Weatherilt

Aviator based in Hong Kong

[Paul Weatherilt @PaulWeatherilt]

If you want to understand what has happened @cathaypacific, start here, @tculpan @opinion

"Turns Out, Pilots Are Crucial to Running an Airline"
https://t.co/hTd8EXfjAE

[Paul Weatherilt @PaulWeatherilt]

@PeteApps @threadreaderapp unroll

[Will Bryk @WilliamBryk]

Thoughts on the eve of AGI

I talked to several friends about o3 this week. Their summarized response is basically "holy crap is this actually happening?"

Yes, this is actually happening. The next few years are going to be insane. This is historic stuff, galactic even.

What's ridiculous is that there's no sophisticated discussion about what's happening. AI labs can't talk about it. The news barely touches it. The government doesn't understand it.

The fact that a social media meme app newsfeed is how we discuss the future of humanity feels like some absurdist sitcom, but here we are.

Below is a bunch of my thoughts about what's happening -- my contribution to the X idea abyss.

Note, these thoughts are HALF-BAKED and FUN SPECULATION. I haven't had enough time to think through / research all of them and I'll be wrong about many. But I do hope these are interesting to some people out there who are trying to process what's happening.

Enjoy.

------------------------------------------------------

- o3 should not have been shocking. OpenAI showed us the test-time scaling graph 2 months ago, and the history of computers teaches us to believe the trendlines no matter how unbelievable. What should be shocking is that it happened in 2 months. That's how quickly we went from college-level AI to phd-level AI. For humans, change is exciting, but rapid change is shocking.

- It's pretty obvious what's going to happen next. The o3 class models are reeeaally good at optimizing for anything you can define a reward function for. Math and coding are pretty easy to design a reward function for. Fiction writing is harder. So that means in the short term (1 year), we're going to get spiky models. They're going to be basically AGI-level at math and coding and general reasoning but write generic fiction. While better reasoning will make the models feel smarter across the board, they're still gonna fail in stupid ways they weren't RL'd for -- ie, not in their training data. Overtime in the longer term (1-3 years), we'll keep adding new domains to RL them with (emotion data, sensory data, etc.) until the blind spots are patched up, and then these models will clearly be AGI to anyone who's not Gary Marcus.

- Agents really are coming in 2025. There's no way o3-like models won't be able to navigate the browser/apps and take actions. That stuff is easy to design reward models for. It's also a huge market -- automating computer work -- so there's big incentives for the labs that need to justify their big spend. I'd guess by December 2025 you'll be able to tell your computer to do any sort of workflow that involves navigating webpages/apps and moving data around.

- Of all the intellectuals who are most "cooked", it's gotta be the mathematicians. Mathematicians work in symbolic space. Their work has little contact with the physical world and therefore is not bottlenecked by it. LLMs are the kings of symbolic space. Math isn't actually hard, primates are just bad at it. Same with regex.

A big question is how hard it will be to make research-caliber synthetic data. I'd guess not that hard. Phd-level math and researcher-level math look qualitatively different to us, but might look the same in kind to an AI, just requiring a couple more magnitudes of RL. I give mathematicians 700 days. (That sounds crazy, but o6 not beating mathematicians sounds equally crazy, so I'm more than 50/50 on this prediction, like all the other predictions in this post). That's 700 days until humans are no longer the top dogs at math in the known universe.

- What about us software engineers? In the short-term it's going to be heaven. Every SWE just got a promotion to tech lead, nicely done. For those who fully adopt LLMs, coding by end of 2025 will feel more like orchestrating a bunch of small jobs that little agents go and perform. Any PR that has very clear specification should be doable by an o4 system with an error rate that's small enough to be acceptable. One problem here could be context windows too small to contain a codebase, but leaders like Sam are well aware of this.

Will AI automate all software engineers away soon? No. Software engineering is more than making PRs based on hyper clear prompts. Unlike mathematicians, software engineers constantly interface with the physical world, namely other humans. Engineers have to work with customers to understand their needs and with teammates to understand their needs. When engineers are designing an architecture or writing the code, they're doing it with a ton of organizational context. o4 won't be able to do that. But o4 will help the engineers who do have the context move 10x faster.

If software engineers are 10x faster then maybe we need fewer? Well, if you take a specific company then yes they might need fewer software engineers bc they can achieve the same output with a leaner team. However, the whole world's need for software engineers might go up bc the world can def use 10x more quality software. So I think we'll see a golden age of applications from leaner companies. Personalized microapps for every person and business.

- In the longer term (>2 years is considered long term lol), software engineering will be completely different, hard to say how. How could it not, when o6 systems exist and are fully integrated into our applications? Roles like frontend engineer might not exist in 3 years. Is that weird? Not really -- the frontend engineer role didn't exist 30 years ago either.

We should take a step back and recognize that software turns itself on its head every generation. Software has and always will be about converting needs into pure logic. That conversion process has risen in the abstraction levels from binary to python. The difference now is that it's rising to english.

Moving to english opens up coding to the non-technical. But the best builders will still always be the ones who can move up and down abstraction levels.

In short, bc software engineering is really all about understanding and fixing organization's needs through code, the day software engineering is fully automated is the day all organizations are.

- We've talked about some knowledge workers, but what about the physical workers? AI is coming for you too, but slower bc it has to deal with gravity and friction. But the o-class of models will not help robotics as much bc a model that takes an hour doesn't help a robot on a factory line. The base-model getting smarter does help, and o-class models will help train those, but I don't think that fixes the biggest bottleneck to robotics progress. I'd guess the biggest bottlenecks are hardware improvements and fast/reliable models for perception+action. Those will both take longer to improve (ie several more years). Crazy fast progress in robotics will only happen once robots start building robots and AI starts doing AI research. That could come from o-class models, but I think that's a couple years away.

- I keep talking in units of years, but maybe we should really talk in units of compute. Time determines human output but compute determines AI output, and AI output will increasingly be the most important at research orgs. That's why the rat race is on to build superclusters -- Meta's 2GW cluster, Xai's additional 100k H100s, etc.

All the labs will quickly follow OpenAI with test-time compute models and some can make up for worse algorithms initially with more compute. They'll play catch up just like they did with GPT-4. To make these models there's a mix of common knowledge and each lab's secret sauce. Unclear how much secret sauce OpenAI has with the o-class models, but their rate of improvement suggests it's an algorithmic advance (which is easier to replicate) and not some unique mix of data (harder to replicate).

In the age of test-time compute, it's not clear to me whether having more compute or better models is more important. On the one hand, you can make up for a worse model by throwing more test-time compute at it. On the other hand a slightly better model might save an exponential amount of compute.

It would be kindof funny if Xai catches up to OpenAI because they're simply better at spinning up massive clusters.

Regardless, there's not going to be a model moat that lasts longer than a year, bc labs swap researchers like baseball cards, and, perhaps more importantly, the researchers between labs party and sleep with each other. Plus I think researchers are too idealistic to not share information if things got out of hand.

Kindof crazy situation we have here. The AI race is like the nuclear race, but where the Americans and Soviets party together in Los Alamos on weekends and bait each other on twitter with "bet you're not gonna have the biggest nuke in 2025 lols :)"

The AI race will continue to feel hippy and fun-loving until the government steps in and/or something really bad happens.

- o-class models change the dynamics of the compute scale up in a few interesting ways.

o-class models incentivize massive buildout bc they have clear gains with every order of magnitude more compute. Compute providers couldn't have asked for a better scaling law. I'm guessing this law is what Sam saw when he wanted a multi-trillion dollar compute cluster.

This might not actually be great for Nvidia. o-class models make inference more important than training. I think super optimized inference chips are easier to build than training chips, so Nvidia doesn't have as much of a moat there.

Very speculative: what if o-class models unlock the aggregated compute from the whole world to train the best models? Like how cool would it be if opensource beats closed souce bc we band together our macbook pros into an inference gigacluster.

- Another new exponential in the mix now beyond compute is the code itself. If one lab has unique/privileged access to the smartest model and so their software engineers get 2x more productive than other labs, then they get closer to the next doubling of productivity faster. Unless code speed maxes out and there's a long queue of experiments to run, and the lab is once again bottlenecked by compute. (Idk, the dynamics are hard. Would be super cool to see how labs model how much they should spend on compute vs people.)

- As crazy as all this compute buildout and knowledge work automation sounds, it only starts to get really crazy when the scientists start feeling the AGI. I'm thinking of you physicists, chemists, biologists.

It'll start with anything theoretical in the name. Theoretical physics is up first. If math is actually solved (sounds ridiculous even writing this, but that doesn't make it not likely), then theoretical physics can't be that far behind. It too lives in the symbolic realm at which LLMs will be superhuman.

What happens when we have a million AI von neumann's working day and night in the fields of Lousiana (Meta's upcoming datacenter)? How quickly will they read every physics paper written by the thousands over the past century and immediately spit out more correct tokens?

Obviously this is the part of the story that is hard to predict. Theoretical physics, chemistry, biology -- what if these are a joke to an LLM trained with RL? What reasonable argument at this point do we have that it won't be? Yes we haven't seen true innovation from these models yet, but they've been mostly at high school / college level and those age groups don't invent new physics. We're at phd-level now so we might start seeing some inventiveness.

- Once AI starts churning out new scientific theories, the bottleneck to progress will be testing and experimentation in the physical world. The bottlenecks there are labor and materials. By that point it would be surprising if there aren't robots that can build more robots. So labor is solved. And then materials can be mined by the robots. The timelines here will be slow because building/shipping physical stuff takes a long time, but it's years not decades.

- Everything I've said above assumes no new bottlenecks are introduced to AI + robotics research/development, and that the models are allowed to learn as they please. That is almost certainly not going to happen. The biggest bottleneck to AI progress will be humans. By that I mean regulation, terrorism, and societal collapse.

Governments are not going to sit back and let the Earth be mined by automated robots run by a couple SF companies (regulation). And if the governments are too incompetent to stop them, then angry jobless people might resort to violence (terrorism). Unless people are so brain rotten from AI enhanced media that we can't function as a society (societal collapse).

If war happens, I think it won't be a bottleneck, rather an accelerant.

Things are gonna get serious. 2025 might be the last year where AI is this wild thing SF tech twitter memes about, before the normies in suits get involved, so let's enjoy roon and sama while we can.

- Is this gonna kill everybody? I'm more scared of humans using AI badly than the AI going rogue.

We have 5000 years of evidence of humans using the latest technology to kill each other. The post-WW2 peace is an anomaly that could fall apart the second the US missteps or when an adversary thinks a first-strike is necessary to stop the AI acceleration. When the weapons get more lethal, more autonomous, the stakes get higher.

The other big risk is AI causing societal chaos. AI generated media could cause mass confusion, mass hysteria, mass brain rot. An authoritarian country could win the AI race and use the new tech to deprive us all of freedom for 1000s of years.

Another risk is that the AI goes rogue. Meaning it causes something extinction level that we didn't predict. Especially with RL being back in the game, AI is now discovering its own optimizations instead of trying to match human data (matching humans is safer). But so far the underlying brain of these models is still an LLM and LLMs have show to just understand people. Like if you include in the prompt "make sure not to do anything that could kill us", burden is on you at this point to claim that it's still likely to kill us. Of course I haven't considered all the arguments here, but when I have nightmares about an AI dystopia, I see Chinese and Russian flags, not OpenAI's logo.

- I'm definitely more excited than scared though.

The science-fiction world I've always wanted is coming. It's coming a bit faster than expected -- hence the fear -- but of all the possible paths to get there, I'm not sure how much better the best path would be. This is a pretty great timeline.

Top of mind things that I hope are coming within a decade:
- some crazy cool physics discoveries
- Mars and Moon bases initially built by robots
- perfect tutor/advice for everything (nearly here, needs good retrieval, memory, and more personality)
- biology enhancing drugs with zero side effects
- getting flown around in super optimized drones
- super clean energy across the board with fusion, geothermal, and lots of solar
- the unexpected: AI astronomer discovers alien signals in telescope data? AI chemist easily designs room temperature superconductor? AI physicist unifies some theories? AI mathematician solves Riemann Hypothesis?

These don't seem like science fiction anymore, they feel like nearby science reality.

- So where is this all going? Well eventually we get superintelligence and that means we get whatever the laws of physics allow for. I'd like immortality and to see other star systems. I'd also expect to upgrade our meat bodies to something way better. But by far I'm most excited to learn where the universe comes from. 10 years ago I started journalling about how much I want to know that answer and how AI will get us there, and now it could actually be happening, which is insane.

- We're now living in a world where this all sounds plausible. Every new AI development makes a larger percentage of people realize that, o3 being the latest.

The only way the future isn't spectacular now is if we the people mess it up. Like our public opinion, our downstream policies, our societal stability, our international cooperation -- these are the roadblocks that could prevent this spectacular future.

- People think the people at AI labs are controlling our future. I disagree. Their work is already determined. They're merely executing on model architectures that are going to happen in one lab or another.

But our public opinion, our downstream policies, our societal stability, our international cooperation -- this is completely uncertain. That means we collectively are the custodians of the future.

It falls upon each of us to help our world navigate these wild times ahead so that we get a great future and not a horrible one.

- There are lots of ways to help out. Help build products that somehow make society more stable or that make people smarter (ex: an app that helps people regulate social media). Help inform people of what's going on (more high quality commentary on social media, a really good search engine, etc). Help clean up our streets so that the city asking to bring us all into utopia doesn't look like a dystopia (getting involved in local politics).

- Almost everyone I've talked to is scared of losing meaning in an AI world, and you might be too. To you I say, isn't it the total opposite? You're living at the most important time in history and you have the ability to influence it. Helping to save the world should be enough meaning no? You want to go back to a time where the only thing progressing was your career and not the world?

Perhaps the transition people need to make is from getting meaning through individual success to getting meaning through collective success. Many of our current jobs will be automated soon. We'll have to adapt. If you derive meaning from a specific skill, yes that skill might no longer be necessary in 5 years and you're out of luck. But if you can derive meaning from helping the world however you can, well that isn't ever going away.

- To all the new grads being given advice bc of o3, here's my advice: learn how to be 1) a high agency problem solver and 2) a great team player. Your specific skills you learn along the way won't matter bc the world will change so fast. But jumping to solve problems and working well with a team will matter for a long time.

You also might need to accept an unstable life in an unstable world. It's gonna get weird. You're prob not gonna have two kids and a dog in the suburbs. You might have two cyborg kids and an AI dog on an interstellar ark.

We're living on the eve of AGI, and on this Christmas eve I ask that you help make the AGI transition go well, so that I can say hi to you on Christmas eve 3024 AD, on a planet four light years away orbiting Altman Centauri.

[Paul Weatherilt @PaulWeatherilt]

@blinkchargingUK is there a UK number to call for customer support. Or even an email address?

[Faisal Islam @faisalislam]

Google have invented Willow, a quantum computer so powerful & quick that they slip out mid way through the blog post “It lends credence to the notion that quantum computation occurs in many parallel universes, in line with the idea that we live in a multiverse”

So there you go.

[Laura Westbrook @LauraWestbrook]

Last year IATA head Willie Walsh told me Russia’s closure of its airspace will be the “single biggest factor” in the restoration of flights to China.

Now European airlines are pulling out of China. Great article by @skift on the myriad of reasons why.

https://t.co/6LoIW9MW92

[Paul Weatherilt @PaulWeatherilt]

@CannixYau in @SCMPNews

The pandemic was hard but @cathaypacific hasn't been a good partner. Company leaders should have worked with staff and government to protect HK aviation and the company.

https://t.co/mmxd5MsIln

This is the result:

[ParalympicsGB @ParalympicsGB]

Dear Prime Minister,

It’s time for #EqualPlay

#ParalympicsGB 🟰

[Paul Weatherilt @PaulWeatherilt]

Initially @cathaypacific and HK aviation's recovery was swift, reactivating the pilots that remained.

Now it's about recruitment, retention and a mountain of pilot training.

[HKAOA @HKAOA]

The impact of 2020 @cathaypacific restructuring.

Cathay left trailing in the wake of competitors that kept their frontline teams together.

HK Aviation ❌

[Saul Staniforth @SaulStaniforth]

A former adviser to Boris Johnson says its all very well for trade unions to secure better pay rises through disputes and industrial action, but what about the millions of people who are not part of a union?

@schneiderhome: "I agree... the answer to that is join a trade union"

[Matt Green @mattgreencomedy]

Elon Musk buys a pub

[HKAOA @HKAOA]

@CannixYau asks in @SCMPNews
"Are Hong Kong’s aviation hub ambitions hampered by @cathaypacific's pilot shortage...?"

Here to let you know that yes, CX is short of pilots and their recovery has been stalled for months

HK Aviation Hub ❌❌❌

https://t.co/4A6N4AZzyS

[Aaron Busch @tripperhead]

🇭🇰🏊🥉

Siobhán Haughey finishes third in the 200m freestyle final to take the bronze!

Haughey is now Hong Kong's most decorated Olympian with three medals.

[Aaron Busch @tripperhead]

🇭🇰🥇🤺

Vivian Kong secures Hong Kong's first gold medal of the Paris Olympics!

Kong defeated local favourite Auriane Mallo-Breton 13-12 in the women’s epee final.

[European Pilots @eu_cockpit]

Aircraft manufacturers and airlines are betting on passengers getting comfortable with just one pilot in the cockpit instead of two. But let's face it, neither passengers nor the piloting community are on board with this risky idea. We believe it’s a gamble with safety, and that's why we’ve set a new mission: to raise awareness about the push for single-pilot flights.

In the coming days, travelers passing through Brussels-Zaventem Airport will spot our new ads in the bathrooms, highlighting the safety risks of reducing pilots in the flight deck during flight.

Click below and read the full article about our new campaign.

https://t.co/54tBmQoRMP

[Massimo @Rainmaker1973]

Hong Kong International Airport timelape.
https://t.co/yj6RX18XzC

[Paul Weatherilt @PaulWeatherilt]

This from @SCMPNews is more about capacity than service quality.

@cathaypacific sidelined unions, fired thousands of staff, reduced pay for those that remained and stood by as thousands more resigned.

CX stalled for months now at 60% of pre-covid capacity. 70% including UO.

[HKAOA @HKAOA]

@SCMPNews

"HK gov required to implement policies to maintain the city’s status as a global and regional aviation hub".

Can't happen without:

✈️ Airport Workers
✈️ Flight Attendants
✈️ Pilots

Time for a gov inquiry into the actions of Swire Pacific managers @cathaypacific

[Paul Weatherilt @PaulWeatherilt]

@kjoules There is no doubt that the pandemic was a factor in pilots leaving @cathaypacific. However, management's decision to impose permanent reductions to pay and conditions also drove many pilots away.

[HKAOA @HKAOA]

It doesn't really matter which metric you use.... @cathaypacific rebuild has been stalled for 7 months at between 60% and 70% of pre-covid capacity.

#notenoughpilots

[Air Line Pilots Association @ALPAPilots]

Safety starts with two: Two wings, two engines, and two sets of eyes focused on keeping air travel the safest mode of transportation in the U.S. Learn more: https://t.co/KeCs5yoNgU #SafetyStartsWith2 #NationalSafetyMonth

[Air Line Pilots Association @ALPAPilots]

Safey starts with two. Two pilots are as essential as two wings. Keep air travel the safest mode of transportation in the U.S. Learn more about how pilots are committed to maintaining safe air travel. https://t.co/jp3OFY3k67 #SafetyStartsWith2 @EASA @Airbus