Joe Ton

Explaining AI Systems Performance in plain English
Hardware, Software, Research

[Joe Ton @joeton]

Quick note: Everything I post here is written by me - a real human in Seattle who gets excited about AI Systems Performance. No AI slop, no ghostwriters. Just my actual thoughts.

[Joe Ton @joeton]

@EthanHe_42 Any tips in thriving in a culture like SpaceXAI?

[Joe Ton @joeton]

@nateberkopec What do you think would be the solution to this AI Slop?

[Joe Ton @joeton]

@PatrickToulme I love this content. This is a very dense topic.

Not judging here, did you use AI to generate some of this report? This seems too opinionated to be AI, but the volume of content is quite high.

Reason why I ask is I want to produce similar quality work on writing.

[Joe Ton @joeton]

@ziv_ravid We need a tool or some writer's version of AI version control that discloses AI tools.

We should have some level of transparency of using AI.

[Joe Ton @joeton]

@octonion I don't know how many people actually read the entire paper end to end.

[Joe Ton @joeton]

@JimDMiller @tdietterich @arxiv How much AI slop percentage wise is done on arxiv?

[Joe Ton @joeton]

@JustinAngel @tdietterich @arxiv True. This is a larger problem. The site needs to take a first step to have researchers disclose their AI tools to producing these papers.

[Joe Ton @joeton]

@tdietterich @arxiv Perhaps @arxiv should have disclosure on the tools or AI tech used on the paper and how much?

For example, suppose the researchers generate 80% of the paper with ChatGPT 5.5 with 20% human editing.

This way we can determine how much the paper has human oversight.

[Joe Ton @joeton]

TTFT stands for Time To First Token.

This waiting period, measured in milliseconds, is a major latency metric focused on time between first request and first response for AI.

For example, the TTFT SLO may require < 500 ms. Continuous batching can help here.

[Joe Ton @joeton]

TTFT

It's basically how long you have to wait until AI first responds.

For example, when you first talk to an AI, there's a short delay before first message appears. That delay is called TTFT.

Important for user experience. Shorter the better.

More technical: 👇

[Joe Ton @joeton]

@eigensteve What do you predict is going to be the single most important optimization to learn from your book in the next 5 years?

[Joe Ton @joeton]

@charles_irl @modal What would you say is the biggest advantage over Kubernetes with @modal?

[Joe Ton @joeton]

@Modular Time to really learn Mojo.

[Joe Ton @joeton]

@GergelyOrosz There should be a disclosure of tools upfront used for the post/blog. In addition, there should be proof of human creation.

There should be some easy version control for writers to show authenticity.

[Joe Ton @joeton]

Is HBM with 1 TB enough for KV caching?

[Joe Ton @joeton]

I wish the AMD dev program allows more than $100 of credit.

[Joe Ton @joeton]

Attention Matching it is.

[Joe Ton @joeton]

@NVIDIAGeForce PRAGMATA RTX

[Joe Ton @joeton]

@vasuman Do you think writing test code manually makes sense here long term with AI coding?