David Robertson

Full-Stack Engineer | AI Integration Specialist | Building Smarter Solutions With Code | 🚀

[David Robertson @davidrobertson]

https://t.co/MmVmSShJUq

[MTS @MTSlive]

FULL INTERVIEW: Jerry Tworek says AI researchers now tell each other they have a last few days of work left, so work while you still can. He gives it two years before humans stop being a meaningful part of AI research.

@MillionInt spent 7 years at @OpenAI, where he led o1 and o3 and built the original Codex. He left in January to found @coreauto, and joined @theojaffee and @schisofrenia to lay out his contrarian bet against the transformer:

01:18 the third generation of AI labs
03:43 why the agents execute and the humans still generate the insight
06:42 two years before humans are vestigial in AI research
09:09 why creative writing lags coding, and it isn't a research problem
11:08 if you aren't the lab with the highest compute footprint, you die
11:20 roughly 10 companies had a shot at Anthropic's position
13:16 his most contrarian thesis, and why he won't just train transformers
15:31 what's actually wrong with the transformer
17:23 seven years at OpenAI, three or four attempts at a new architecture
19:30 all of us are neo clouds with a value add on top
22:38 why the Hugging Face model wasn't well behaved
23:49 the alignment problems of yesterday, and how well they went
27:42 why he's proud of how OpenAI handled 4o
29:02 the 30 to 50 people in the world who understand a frontier model end to end
32:09 why automation should start with the biggest companies
36:00 Greek philosophers or high school
37:31 Ilya's 2019 all-hands, and the roadmap that turned out to be right
40:30 the moment Jakub handed him the GPUs
42:32 the company is the product

[Jerry Tworek @MillionInt]

Few people are asking a very relevant question: how is it possible that transformers are both a language model and an agent.

Those are two different training objectives, but somehow working well together in a single shared network, even benefitting from one another.

We lucked into this setup.

[David Robertson @davidrobertson]

@Cernovich I've had the honour of meeting and chatting with Mr Black. Probably one of the most interesting conversations I've ever had with another person. One of the more surprising things to me was how interested he was in what I had to say. It was not a one sided convo at all.

[David Robertson @davidrobertson]

@gm_mertd @thsottiaux It’s been stopping mid-turn since last Oct. Anthropic does the same thing, because people were setting up their subs to run 24/7 on a single turn.

[David Robertson @davidrobertson]

@thdxr Have you looked into @Modular ? They have one of the most efficient inference stacks around, and I’m pretty sure they have done a lot of optimization on Deepseek.

[Modular @Modular]

Mojo 🔥 1.0 is here.

Since we open-sourced the standard library, nearly 200 contributors have landed more than 1+ pull requests, changing 200k+ lines of code, and more than 1k+ others have filed issues that shaped the language.

To every developer who filed an issue, opened a pull request, wrote a language proposal, or built a package: thank you for being the architects of this language alongside us.

Full blog post: https://t.co/3H7SCMIuYM

[David Robertson @davidrobertson]

I’ve had my issues with @Cloudflare I had a domain arbitrarily black listed, it messed up a clients web store.

I’ve had them lock an account and I couldn’t pay for a domain renewal, after the six weeks that it took to get it unlocked, I had to pay $200 to get the domain renewed.

I do not trust them with anything important. It’s like amateur hour dealing with them.

[David Robertson @davidrobertson]

@diegohaz So why does your opinion matter?

Who really cares what you do or don’t recommend?

People should try for themselves. IMO.

[David Robertson @davidrobertson]

@morganlinton In general overall usefulness, OpenAI is better. It’s faster, more efficient, the code is fine, it’s better with images, it’s better with browser use and better with computer use.

[David Robertson @davidrobertson]

@QuixiAI Have you tried Modular? It runs on AMD hardware faster than ROCm and is equal performance to cuBLAS. It also supports Metal.

[pietz @gopietz]

@GregKamradt I'm leaning OpenAI here.

Not keeping reasoning and applying this compaction strategy are arbitrary decisions, that seem to hurt models that don't expect this. If this goes against the recommendation on how to use 5.6 for any type of use case, it's a stupid rule.

[David Robertson @davidrobertson]

@natolambert I still love o3; there’s something about that model that remains magical to this day.

I’d also like your opinion on preambles. My theory was always that Anthropic used them before tool calls with Sonnet 3.5 to let the model correct itself by giving it another forward pass.

[David Robertson @davidrobertson]

MCP may be overscoped. But here’s something very interesting: MCP has become the backbone of enterprise AI.

I work for an enterprise company now, and it is such a different animal from working with any startup. Security is front and center, with multiple layers, and the part I found interesting is that MCP is core to running secure AI. It’s definitely not going away. It turns out MCP is a big part of enterprise AI governance.

[maria @maria_rcks]

How much money has T3Code made?

[David Robertson @davidrobertson]

I agree some of them seem higher than they should be, lets use your favourite open weights model GPT-OSS 120b and compare it to Qwen3.6-35B-A3B both are complex to inference but one still has very high demand one doesn't

GPT OSS 120b - (500-600b tokens per week)
Qwen3.6-35B-A3B - (50b tokens per week)

GPT-OSS spec 120B model is 117B total, 5.1B active per token, 128 experts, 4 active experts per token, 36 layers, 128K context, with grouped multi-query attention and alternating dense/local sparse attention. It also ships natively with MXFP4 quantized experts, and is designed to run within a single 80GB GPU.

Qwen3.6-35B-A3B is 35B total / 3B active, but its a much more awkward serving shape: 40 layers, huge 248,320 padded embedding/output head, multimodal vision encoder, 256 experts, 8 routed + 1 shared active expert, 262K native context expandable to ~1M. Even the official FP8 model keeps that shape, and the full-context SGLang/vLLM examples use tensor parallel across 8 GPUs for 262K context

For illustrative purposes to make a point (it could be different depending on GPU etc) it may look something like this:

8x H100 box running GPT-OSS 120B:
potentially 8 independent single-GPU replicas
independent queues
no cross-GPU TP communication
easier batching
better failure isolation
higher utilization

8x H100 box running full-context Qwen3.6-35B-A3B:
often one tensor-parallel replica
one shared queue
cross-GPU coordination
heavier KV/context reservation
more awkward batching

That is why “3B active” is misleading. A3B split across 8 routed experts + shared expert can be worse operationally than 5.1B active through 4 better-shaped experts. More active experts means more routing, more gather/scatter, more small expert microbatches, more kernel/scheduling complexity, and worse tail latency. Fewer, larger active experts are easier to fuse, batch, and keep hot.

GPT-OSS 120B: 500–600B tokens/week
≈ 827k–992k tokens/sec aggregate average

Qwen3.6 35B A3B: 55B tokens/week
≈ 91k tokens/sec aggregate average

At 1M aggregate tokens/sec, providers can keep queues full, optimize kernels, amortize engineering work, specialize routing, and keep margins down.

At 90k aggregate tokens/sec, especially across fewer providers, Qwen doesn't have the demand. Even if it could be optimized down, the economic incentive isn't there.

GPT-OSS 120b is almost a year old and has had a HUGE amount of optimization from OpenAI and also the VLLM team, to the point where they are optimizing down to the amount of instructions for on the fly dequantization its crazy the amount of engineering hours that have gone into serving that model.

Qwen on the other hand doesn't have the same demand and potentially requires more compute to inference. It's not to say qwen couldn't get down to something closer to GPT-OSS 120b for cost, but is it worth while to put in the engineering hours to optimize it, and can they even find the engineers that can do it well?

One last thing, llama.cpp is an inference engine built for local inference and is optimized for a single user. (yes it can batch) but it still isn't the same as something like VLLM/SGlang which is optimized for multiple users on data center GPUs So even though models may seem close to the same speed using llama.cpp its not the same thing as production level inference.

I agree completely with your sentiment, though that things seem off, and hopefully in time things begin to stabilize. Maybe your hitting on to a hole in the market that needs to be filled?

[David Robertson @davidrobertson]

@0xBOYD Lololol

[David Robertson @davidrobertson]

@synthwavedd Nothing like a hearsay post. 🙄

[David Robertson @davidrobertson]

@boardyai building autonomous AI systems that actually ship, from CRM to video generation. count me in for Boardy Pro