Rules that fall out of this:
- static first, dynamic last
- append-only history
- don't swap tools or models mid-session
- check cache_read_input_tokens / cached_tokens
Sources for every number are in the description.
two API calls. same 50K-token prompt and same question!
one costs $0.25. the other cost $0.01.
the only difference: the current time on line 1 of the system prompt.
Why prompt caching breaks on a single changed token: https://t.co/KchMrh56TC
Why it has to be an exact prefix: every token's K/V depends on every token before it.
vLLM chains block hashes (each block's hash includes its parent's), so one changed token at the top invalidates every block after it.
i typed https://t.co/rkNDm6HYuH and the HTML came back in ~300 ms.
in that time the request got redirected, encrypted with a post-quantum key exchange, load balanced, and handed to servers that fan out to thousands of machines.
link: https://t.co/ZTjQ9vsguP
some cool thingies:
- "13 root servers" are really 2,049 machines (anycast)
- my root DNS query answered in 18 ms from India, not America (OBV)
- https://t.co/rkNDm6HYuH's TLS key exchange is X25519 + ML-KEM, post-quantum, today
- the cert lasts 84 days
a 70B model is 70.55 billion numbers, at 16 bits that's 141 GBs and an H100 has 80 GBs
Quantization rounds each weight onto just 16 values and the model fits in ~36 GBs.
Why that barely hurts, where it does hurt, and FP4 vs INT4 vs NVFP4, in 17 min:
https://t.co/tXPibkzJnr
perplexity also hides things.. in one long-context study, 8-bit lost ~0.8% on average, but 4-bit dropped up to 59% on 64K+ token inputs, depending on model and method.
in Red Hat's 500,000+ evals, FP8 was effectively lossless, SO test on your task
distributed: one token bucket (100 tokens), N gateways, 1000 requests.
read, compute, write as separate steps: 20 gateways let all 1000 through.
one redis lua script: exactly 100, in all 30 runs.
video: https://t.co/4Ts5d8X0hr
code: https://t.co/LRElGKqvWl
i set a rate limit of 100 requests/min on a real redis.
then i sent 200 requests in 0.1 seconds at the minute mark.
all 200 got through.
nothing was broken. fixed window resets its counter at the boundary.
i ran the same attack on 5 algorithms ๐งต
leaky bucket means two things.
as a meter: same decisions as a token bucket. 199 in a minute.
as a queue: never over 100/min, the only one that smooths traffic. the price: the last request waited 60s.
Integer a = 127, b = 127;
a == b // true
Integer c = 128, d = 128;
c == d // false
i made a free, open-source book that draws java all the way down, so things like this finally make sense.
192 chapters, 300+ figures and every snippet checked on a real jdk 25.