Sid Prasad

Research Assistant Professor at TTIC, CMU CS PhD

[Sid Prasad @sidpra5ad]

@tarunchitra Will the talk be streamed/recorded?

[TTIC @TTIC_Connect]

Friday, November 6th at 11:00AM: Research at TTIC presents Siddharth Prasad with a talk titled "Inference Auctions." Please join us in Room 530, 5th floor.

[Sid Prasad @sidpra5ad]

@_onionesque You might also find our recent work on Inference Auctions interesting/relevant:

https://t.co/PS5mJAwFdt

[Asher Trockman @ashertrockman]

@dair_ai You might also like our recent work on inference auctions: https://t.co/ih85Q04Mbk

[Eitan Borgnia @EBorgnia]

Separating API requests into different tiers of SLAs is a core problem of inference at scale.

It's also incredibly difficult to price. Using a standard fixed contract requires constructing some function of minimum TPS, maximum TTFT, workload shape --> input/cached input/output token rates.

It turns out this function is non-linear in all of these input variables.

Ideally, you'd like to have a single central deployment per model that can handle multiple customer requirements simultaneously.

Great work by @sidpra5ad @ashertrockman @keegan_w_harris and @nhaghtal on solving this by blending an auction directly into the inference engine.

[Sid Prasad @sidpra5ad]

We design an auction for allocating inference priority in LLM serving systems. It is carefully designed to maximize economic efficiency while retaining key performance metrics (cache hits and mean TTFT) of the underlying SGLang inference engine.

https://t.co/RwrsiruNAr

[Asher Trockman @ashertrockman]

When LLM demand exceeds capacity, which requests should go first?

Model providers already offer a few priority tiers. But even within a tier, requests compete for compute, and users value latency differently. Your coding agent might be blocked due to a request that belongs to an unattended background job.

Why not let the market decide?

In our new paper, Inference Auctions, users bid for request priority, and our auction allocates that priority to those who value it most.

Unfortunately, naively selling to the highest bidders destroys prefix-cache locality, increasing SGLang's average latency by up to 12x in our experiments.

Our fix is to run an auction over depth-first search traversals of the request prefix tree, ordering subtrees by average bid. This maximizes social welfare while retaining cache optimality, with VCG payments incentivizing truthful bidding, and the resulting request order preserves key performance metrics (cache hits and mean TTFT) of the underlying scheduler.

w/ @keegan_w_harris @sidpra5ad @nhaghtal and Michael Jordan

Paper: https://t.co/CJWn21oYy8
Project page: https://t.co/l2Dg2jUtiS

[Keegan Harris @keegan_w_harris]

Check out our new paper on Inference Auctions! We also design inference auction autobidders, which allow a user to specify an inference budget and dynamically adjusts its bids over time to maximize user utility subject to the budget constraint.

[Asher Trockman @ashertrockman]

When LLM demand exceeds capacity, which requests should go first?

Model providers already offer a few priority tiers. But even within a tier, requests compete for compute, and users value latency differently. Your coding agent might be blocked due to a request that belongs to an unattended background job.

Why not let the market decide?

In our new paper, Inference Auctions, users bid for request priority, and our auction allocates that priority to those who value it most.

Unfortunately, naively selling to the highest bidders destroys prefix-cache locality, increasing SGLang's average latency by up to 12x in our experiments.

Our fix is to run an auction over depth-first search traversals of the request prefix tree, ordering subtrees by average bid. This maximizes social welfare while retaining cache optimality, with VCG payments incentivizing truthful bidding, and the resulting request order preserves key performance metrics (cache hits and mean TTFT) of the underlying scheduler.

w/ @keegan_w_harris @sidpra5ad @nhaghtal and Michael Jordan

Paper: https://t.co/CJWn21oYy8
Project page: https://t.co/l2Dg2jUtiS

[TTIC @TTIC_Connect]

How should #AI systems fairly credit creators whose work shapes their outputs? Siddharth Prasad (TTIC) presents "In-Context Credit Assignment via the Core," applying cooperative game theory to distribute value among content creators. Now streaming: https://t.co/ibyzjGReQK