Jonathon Schloss

[Jonathon Schloss @JonathonSchloss]

@BreezeOg1 Same experience here. It's in there, it just has to compete with everything else in the window.

[Jonathon Schloss @JonathonSchloss]

Agent memory is less magic than it sounds.

Claude Code's docs: each session starts with a fresh context window. Your instructions file arrives as a user message after the system prompt.

They call it "context, not enforced configuration."

Try it: run /context in Claude Code.

[Jonathon Schloss @JonathonSchloss]

@evgheni_D Agreed. If it has to hold, put it in a permission check or hook. Anything in the prompt is a strong suggestion, not a guarantee.

[Jonathon Schloss @JonathonSchloss]

@M_Decoherence Yep, that's a good habit. A short rules file you re-read at set points beats a long one that quietly gets buried.

[Jonathon Schloss @JonathonSchloss]

Maybe the strangest idea in AI training: "dark knowledge."

Hinton et al. (2015): a big model calling a photo a BMW also gives tiny odds to wrong answers. Garbage truck: very unlikely. Carrot: far less.

Those odds teach a smaller model: distillation.

Try it: read section 3.

[Jonathon Schloss @JonathonSchloss]

Temp 0, same prompt, 1,000 runs. You'd expect 1 answer.

Thinking Machines tried it on Qwen3-235B: 80 different completions.

Why: batching. Server load changes batch size, which reorders the math and can flip a word.

Try it: same prompt, temp 0, 10 runs. Diff them.

[Jonathon Schloss @JonathonSchloss]

Will a model fit on your machine? Napkin math:

parameters (billions) x bits per weight / 8 = GB of weights

Llama 3.1 8B, 16-bit: 8 x 16 / 8 = 16 GB. Ollama lists 16GB.
4-bit: math says 4 GB. Ollama's default is 4.9GB.

Plus room for context.

Try it: https://t.co/fHjkQHUzFd

[Jonathon Schloss @JonathonSchloss]

Why does a long prompt delay the first word?

Two phases (vLLM docs):
Prefill: reads the whole prompt at once (compute-bound).
Decode: writes one token at a time (memory-bound).

Long prompt, longer prefill.

Try it: time the first word on a 1 line prompt vs a 30 page one.

[Jonathon Schloss @JonathonSchloss]

Check it yourself: send the same long prompt twice and read the usage numbers.

Anthropic: cache_creation_input_tokens and cache_read_input_tokens. Both 0 = no cache.
OpenAI: usage.input_tokens_details.cached_tokens.

Then add a timestamp at the top. Watch it hit 0.

[Jonathon Schloss @JonathonSchloss]

The money part, from Anthropic's pricing:

Cache reads: 0.1x the normal input price on most models (as low as 0.025x on a few).
Cache writes: 1.25x (5 minute cache) or 2x (1 hour).
Each hit resets the timer.

Short prompts never cache. Minimums run 512 to 4,096 tokens by model.

[Jonathon Schloss @JonathonSchloss]

One weird difference between providers:

Anthropic says cache hits don't count against your rate limits on most models.
OpenAI says cached tokens still count toward your tokens-per-minute limit.

Same idea, different rules. Read the docs for the one you use.

[Jonathon Schloss @JonathonSchloss]

The sneaky ways people break it:

A timestamp or date in the system prompt (Anthropic lists this as a common mistake).
Tools listed in a different order each time.
Some languages, Anthropic names Swift and Go, can shuffle JSON key order, which changes the text.

[Jonathon Schloss @JonathonSchloss]

The key word is "starts." It's a prefix match.

Anthropic builds the prefix in this order: tools, then system prompt, then messages. A hit needs it 100% identical up to the cache point.

Change one character early and everything after it misses.

[Jonathon Schloss @JonathonSchloss]

What gets cached isn't your text. OpenAI's docs say it's the KV tensors: the math the model already did on the start of your prompt.

If your next request starts with the exact same tokens, it can reuse that work instead of redoing it.

[Jonathon Schloss @JonathonSchloss]

Prompt caching is a big part of why long AI agent sessions are affordable.

And one timestamp at the top of your prompt can quietly switch it off.

How it actually works:

[Jonathon Schloss @JonathonSchloss]

AI models don't read letters. They read tokens, chunks of text.

In OpenAI's GPT-4o tokenizer:
" strawberry" (with a space) = 1 token
"strawberry" (no space) = 3: st / raw / berry
" STRAWBERRY" = 4

Same word, three different inputs.

Try it: https://t.co/pAj8b7Xlpy