3. Computational resources: Amount of processing power (GPU/TPU hours) required for training and deployment.
4. Training time: Duration of the training process.
5. Performance metrics: Evaluation scores on benchmarks such as BLEU, ROUGE, and perplexity.
Metrics to measure the scale of large models like GPT-4 include:
1. Model size: Number of parameters, layers, and neurons in the model.
2. Training data size: Volume of text or data used for training the model.
It's odd that OpenAI's executive team isn't represented on the board. Adding seemingly unrelated individuals to the board for the sake of independence may lead to potential efforts to undermine OpenAI. Introducing a supervisory board is necessary.#OpenAI
In fact, these are all common business knowledge, but these supposed scholars or scientists seem to be unaware of it. Without sufficient resources, OpenAI will quickly fade away. Eventually, their ideals will be reduced to mere dreams. #BusinessInsights#OpenAI
It sounds like Sam Altman will not be returning to OpenAI.
It turns out that Mira was actually “Team Sam” and tried to get him back. So the board at OpenAI installed a co-founder of Twitch, Emmett Shear, as CEO instead. They seem to be holding strong to their decision.
This is wilder than any tv series I’ve watched lately!
The narrative that I believe is that Sam was a bit too enterprising (for the board) and the company was growing too fast. But it was growing in a direction that Ilya and the board didn’t like. Ilya, an academic and scientist, wanted progress on the research and to reach AGI… but without all the public pressures of the whole world paying attention and easy access for anyone who wanted it.
Sam, on the other hand, while not overly accelerationist, saw the need for more capital to achieve their mission. The capital would come through raising and from subscriptions and API fees. By putting a system in place to have the necessary capital, he also put the tool in the hands of the whole world. It also landed the company squarely in the public eye and much more on the radar of governments and policy makers.
The board saw this as getting in the way of their original non-profit mission, focused on research and breakthroughs.
Then there’s the issue of compute. It’s a finite resource. A research lab would want as much compute as possible dedicated towards the actual training and advancement of the tech. However, the more ChatGPT and Dalle grow, the more compute needs to be dedicated to the actual commercial side of the business, taking resources from the research side.
With each new announcement Sam made about larger context windows for users, building GPTs, everything being available in MSFT tools, etc., Ilya and the board were probably getting more and more frustrated because both compute and employees were being dedicated towards the commercial aspects instead of the research aspects.
I think it helps to try to see both sides of the coin. I think both Ilya and Sam were both doing what they thought was best for the company. They just didn’t align on what they believe is best for the company.
This is all speculation of course but, the more that comes out, and the more I try to play armchair detective, this is the narrative that I buy.
I don’t think this ends well for OpenAI though. Sam and Greg will start a new company. They’ll raise money in a heartbeat. Employees from OpenAI will follow them over, taking what they’ve learned at OpenAI with them.
As users of tools like ChatGPT, we’ll likely see a degradation in service. They’ll shift what resources they have left back to research. The tools we use will either stay where they’re at, with no new improvements or they’ll get worse because budget will be shifted away.
Ilya and the board will be happy that they’re back on the research path and close to AGI without as much public pressure. But I think they’ll hit snags when they struggle to raise more. I think Microsoft will begin to deepen their relationship with Meta and/or Sam’s new company. OpenAI will struggle to pay and maintain staff as users leave their services.
They may figure it out, they may not but, moving forward, I bet ChatGPT and the tools we get access to from OpenAI will not be what they once were.
Again… just armchair detectiving here. I don’t actually know what I’m talking about. :)