Mark Huang

Scientific SuperIntelligence @periodiclabs

[Mark Huang @_markkh]

@mdgarratt @chriszeoli @preet1rathi https://t.co/iHwWwrDCPX

[Mark Huang @_markkh]

@agarwl_ is insanely cracked

[Rishabh Agarwal @agarwl_]

Not sure if there's any recording for my CoLM talk but you can find the slides here. https://t.co/bvoVszoLO6

Posting couple of slides which look directionally good (to me)

[Chris Barber @chrisbarber]

I asked people which neolabs under $10B have the highest talent density:

32 votes: Core Automation (@MillionInt, @_arohan_)

26 votes: Periodic Labs (@LiamFedus, @ekindogus)

20 votes: Flapping Airplanes (@spectorb, @amspector100, @aidanmantine)

18 votes: Standard Intelligence (@G413N, @devanshpandey)

8 votes: Ricursive (@annadgoldie, @Azaliamirh)

5 votes each:
- Ineffable (David Silver)
- Mirendil (@bneyshabur, @HarshMeh1a, @shayan_, @tararezaeikh)
- Recursive (@RichardSocher, @_rockt, @timshi_ai, @josh_tobin_, @CaimingXiong, @jeffclune, @tydsh, Alexey Dosovitskiy)

3 votes each:
- Applied Compute (@ypatil125, @rhythmrg, @lindensli)
- Isara (@ezhang7423, @hegasz)
- Prime Intellect (@vincentweisser, @johannes_hage)

2 votes each:
- Elorian (@AndrewDai, @yinfeiy, @SethInternet)
- Goodfire (@eric_ho, @DanJBalsam, @banburismus_)
- Inherent (@tantumscollins, @edwardfhughes, @LouisKirschAI, @kallyaleksiev)
- Trajectory (@rronak_, @QuantumArjun, @MichaelElabd)
- World Labs (@drfeifei, @jcjohnss, @BenMildenhall, @chlassner)

People couldn't pick a company they work at or founded. Disclosure: I'm a small investor in Applied Compute, Factory, Standard Intelligence, Trajectory and Wafer. I didn't vote.

Continued:

[Mark Huang @_markkh]

The model just wants to learn!

@periodiclabs is building the flywheel starting from the closed data loop.

[Deedy @deedydas]

The economics of a Neolab.

A neolab is loosely defined as a startup of AI researchers who raises a lot of money pre-production to be able to finance GPU compute to take on a large AI problem.

To buy 1000 GB300s or ~14 NVL72 racks will set you back $125-150M for 3yrs with 15-30% upfront. That’s about ~2-2.5MW.

Thats about enough to do 10^25 flops a quarter and get to a GPT-4 level model which is 1-2 OOMs off frontier for pretraining.

If you post-train on a great open source model, you have a better chance of getting to frontier. The risks are a) you need to spend millions on RL environments too and b) being lapped by another model release while being tied to a base model.

For this to payback, you need to give your customers a better and ideally cheaper inference service than a base model and serve them for long enough to recoup your large investment. Even at 50% margin on inference, to recoup $10M in training means serving ~10T tokens (!) if you price like Fable / Astra given a standard cache read / input / output split ($2/M blended). And you have to justify being better than a release like Opus 5.5 which is even cheaper. Often, you end up charging your customers a huge premium in terms of platform fees and compute fees on top of pure inference.

Meanwhile, every hour you’re not utilizing your GPUs you are burning money so you typically resell this compute back to a broker or run inference for open models / resell spot instances. At below a ~60% utilization on spot, you will still lose money.

Add to that insane cost of talent.

So what can you do with the compute?
- Not play the model game at all.
- Play an entirely different model game (Jev, World Labs) that if big labs played, would either a) cannibalize their business or b) be incrementally not significant revenue c) would cause too much distraction from the main main thing
- Acquire a proprietary data set (Peridodic Labs) in enough volume in a domain of usefulness to eclipse frontier quality. Often happens in robotics, biology, chemistry.

If you do overcome the challenge of building a model that is useful and well priced beyond big labs models, given the huge price of compute, you still need to play in an area where the revenue / compute ratio is signficant and market demand is large enough to payback your compute spend.

It is a difficult game.

[Mark Huang @_markkh]

@deedydas Did you mean @periodiclabs ?

[Zijie Yan @zijie_y]

Neon’s scientific traces are long and vary widely in length. Training on them shouldn’t mean GPUs chewing through padding or waiting for each other.

Here’s how we pack sequences, balance the work, and distribute long contexts across GPUs at @periodiclabs 👇
https://t.co/bUN5HK3l6V

[Rohan Pandey @khoomeik]

Another Neon infra tidbit: @periodiclabs accelerated checkpoint conversion ~30x down to just 1 minute, helping us deploy faster to get rapid feedback from our labs.

Our models train in Megatron, but SGLang inference consumes weights in HuggingFace format. Traditionally, conversion ran serially:
1. rebuild the entire checkpoint from individual weight tensors
2. convert the checkpoint
3. reshard it to be inference-ready

This could take 30 min for a trillion parameter model!

Noticing that conversion shouldn't require materializing the full checkpoint, @hsu_byron introduced Fast Resharding:
1. parallelize conversion across Ray actors
2. convert expert chunks directly instead of assembling the full expert tensor in memory

We upstreamed Fast Resharding to Miles in PR #1371. Now even you can deploy trained models to production in a matter of minutes.

[Mark Huang @_markkh]

At Periodic Labs we build labs that learn. Our team recognizes that the transition from digital environments to the physical world is highly complex and unpredictable.
The next breakthroughs in science are deeply coupled with intelligence that is given the opportunity to explore through experimentation.

I've been thrilled and honored to help contribute to the product surface areas that help theorists, experimentalists and industry practitioners in the physical sciences accelerate their goals.

Here's a preview of the scientific and technological progress we've made in closing the experimental loop for XRD analysis using Neon our post-trained 1T model.

[Periodic Labs @periodiclabs]

We’ve spent the past year building high-throughput labs and infrastructure for large-scale physics experiments in our search for better superconductors and magnets.

We ran our first experiments by hand. Now, experiments run continuously in the labs. We’re building AI to make sense of what they produce and to determine which experiments to run next.

[Liam Fedus @LiamFedus]

We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.

Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.

This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.

Read our blog posts below.

[Mark Huang @_markkh]

@swyx Glad you’re here man. Been a pleasure to see you succeed throughout it all these years.

[Mark Huang @_markkh]

@TheNoahHein @charlierguo @ronaldmannak @aiDotEngineer @karinanguyen @charles_irl Time flies!

[Mark Huang @_markkh]

@minilek @AnthropicAI wow what a hire

[Mark Huang @_markkh]

@_arohan_ HRT vibes on maintaining talent density

[Mark Huang @_markkh]

@saurabh_shah2 @datadoghq @LightningAI Sad LightningAI really did help the community early on

[Mark Huang @_markkh]

@BrianInCrypto Yep

[Mark Huang @_markkh]

@Hacubu @jxnlco Marketplace deals like this happen all the time snowflake has hundreds of vendors that are DBs on their own marketplace. What will really matter is how they are positioned and the partner marketing spend allocated, ie put your money where your mouth is

[Mark Huang @_markkh]

@GenAI_is_real @radixark Wow RL support already

[Mark Huang @_markkh]

@jamesgreen2016 @levie Lmao

[Mark Huang @_markkh]

@MillionInt Efficient markets…

[Mark Huang @_markkh]

@colemurray Thanks for heads up

[Mark Huang @_markkh]

@bilaltwovec You have to link to the Imbue reference?