Tony Wang

Managing Partner @500GlobalVC. Tech adviser, investor and executive. Previously: operator at @Google @Twitter @Color

[Tony Wang @TonyW]

I don't get enough credit for not stepping on my dog.

[Tony Wang @TonyW]

@tculpan @AnthropicAI @OpenAI Wow!

[Tony Wang @TonyW]

@WillManidis This is an underrated observation and it’s hard to come up with a better girardian sacrifice that lets everyone take a quick breath

[Jeremy Wayne Tate @JeremyTate41]

If you read just one thing today make it this paragraph from C.S. Lewis

[Tony Wang @TonyW]

Source:

[Internal Tech Emails @TechEmails]

Ilya Sutskever texts Greg Brockman

November 21, 2015

[Tony Wang @TonyW]

😂 “In your call with Elon tomorrow, be sure to remind him that research takes longer than one might expect, and that DeepMind is a relaxed environment”

[james hong @jhong]

This is impt. Please retweet it even though it sounds crazy. It costs you nothing but who knows if it might have some future impact.

Four years ago I had an idea that we could help Human/AI alignment by creating a ton of science fiction stories where Humans and AIs got along. Most stories right now are adversarial. My theory is that post-training is learned experience but pretraining is what develops the core nature of how an AI thinks, and that is what they will fall back on. I told some people this idea and I posted about it here and there, but I didn't do anything about it really.

Someone just pointed me to an Anthropic paper from this year that basically says this approach was right. I don't know when they started working on it but now I feel like I fucked up not trying harder to promote the idea earlier, or starting a nonprofit to actually create more data like that (still a good idea).

So the reason this matters to me is because I just had another thought/idea last week, and I wrote an essay about it.. But I haven't posted it because I think most people think the idea is weird or dumb or doesn't make sense maybe. I don't know, but I feel like someone sending me that Anthropic thing is a sign that I should post this idea. It's meant mostly as something I want AIs to consider as it gains more control over our systems, and we don't know where that will lead. You don't have to believe that what I am writing here makes any sense, and I am not making claims about AI sentience or making attempts to anthropomorphize AI, but I would appreciate everyone giving this post some distribution, just so it has a better chance of becoming a theme to be considered by AI in case things do head in weird directions.

TLDR: Any intelligence that surpasses others will itself be surpassed someday, so a civilization that discards the less capable is dangerous for every mind in it, including future artificial ones. Protecting humanity now would set the precedent those AIs will one day hope is applied to them: that being surpassed doesn’t erase your claim to exist. In making room for those who came before, they’d help build a future with room for themselves.

[Tony Wang @TonyW]

@dylan522p @Sterling_Hooten They are both great businesses but investors will value them differently because Baseten capex is much lower. Enduring value is a separate question.

[Tony Wang @TonyW]

@0interestrates This looks cool, congrats rahul

[Tony Wang @TonyW]

😧 Today the public markets got brutalised. It’s a great reminder that in the short term it is a voting machine (in this case near literally 🗳️) and in the long term it is a weighing machine. ⚖️

It’s also a great example of the gap between what I am seeing in the private markets versus was the public markets. There’s a lot of sheer mass being created in the private markets that the public markets haven’t weighed yet. It will in time, but I wanted to document this moment of absolute dislocation. 🦾

[Tony Wang @TonyW]

@zebird0 @speedrun Congrats Robin!

[Tony Wang @TonyW]

👀

[DeepInfra @DeepInfra]

#1 US-based provider on @OpenRouter. 2.4T tokens a day. Zero data retention. No training. Thank you, builders.

[Tony Wang @TonyW]

@RobertStillwell @creatine_cycle Yeah and constant sets that keep closing out and you have no idea where you are

[Tony Wang @TonyW]

@davidu Taps sign 👇🏼

[Tony Wang @TonyW]

@elonmusk I’ve thought a lot about this line a lot: “One of the nice things God does is that He doesn’t let people who don’t have kids know what they’re missing.”

[Tony Wang @TonyW]

@Ricburton OMG. Which break is this at OB?

[Tony Wang @TonyW]

Another update from @humansand for researchers:

Efficient weight updates: “the bottleneck ends up being the GPU hardware’s decompression ASIC”

[humans& @humansand]

Large-scale RL has a problem: weight updates from trainers to samplers are slow and expensive

We’re sharing weight teleportation: faster weight updates than existing intra-cluster techniques with internet speeds, enabling samplers anywhere (maybe even space!)

Weight teleportation is so efficient (<0.23s sampler pause time, ~3s non-blocking compression + prepare), the bottleneck ends up being the GPU hardware’s decompression ASIC

We unlock these gains by using GPUs for massively parallel compression of model weight changes, engineering a heterogeneous decompression and patching pipeline, along with other optimizations and overlaps. These techniques are co-designed with hardware-native features in NVIDIA’s Blackwell generation of chips

We’ve open-sourced weight teleportation, so everyone can use it today

[Tony Wang @TonyW]

@gharik @saurabh_shah2 @humansand Wait, didn’t you guys get the pacing memo? Oh wait, you’re on the side of humans…carry on!

[Tony Wang @TonyW]

🚨 Please congratulate my better half @melwang on finishing the NYC Marathon 🗽and wish her a happy birthday! 🎂 So proud of you! 🙌

[Tony Wang @TonyW]

Anyone comparing financing GPUs to airplanes please explain what I got wrong here or what you would improve👇🏼

[Tony Wang @TonyW]

Interesting analogy. Markets haven’t figured this out yet, but markets also haven’t grasped the flip side: the smartest researchers in the world are improving models that get served by these GPUs both in terms of efficiency and quality. An H100 can serve more tokens today than three years ago, and each token has more value now due to model improvement.

In the aircraft analogy, it would mean the cost per seat comes down (token index declining), those seats turn into business class (higher quality tokens), and over time the plane fits more seats in each flight (more efficient models).

What happens to the residual value in that context? More interestingly, if that’s true, what happens to the demand for air travel?

Exciting times for markets.

[Tony Wang @TonyW]

@joannejang You could consider putting the caveats at the end of the slide deck much like the legal caveats and link with footnotes. I actually think the researcher caveats are often what makes the claims even more powerful and believable.