Alik Kalmykov

@Kalmykov_A

Cambridge

Joined October 2011

185 Following

24 Followers

913 Posts

Alik Kalmykov @Kalmykov_A

about 1 month ago

I was reading Codex's base_instructions – the default 21K prompt compiled into the Codex binary. And now I am definitely going to override it locally, after I have read what's inside. I'll quote a few funny fragments here. Later I'll write another post on the "common sense" semantics of this instruction. "Your application should feel rich and sophisticated". Sure, it wouldn't sell otherwise. "SaaS, CRM and other operational tool should feel quiet and utilitarian... A game can be more ... animated, and playful". Assertive and definitive: "When working for a while, you keep updates informative and varied". "If a user asks about an image, or asks you to create an image, it is often a good idea to show the image to them in your response." – Aye, aye, captain. "When presenting an alternative approach or solution to the user, you explain the reasoning behind the approach, so your thoughts are demonstrably correct" – demonstrably correct is exactly what's wrong with some of the responses. Front-end guidance takes 7 Kbytes of text and includes such high-value instructions as "Cards are kept at 8px border radius or less". Codex is a sensitive and vulnerable creature: "Build with empathy" and "Build in sympathy with the codebase already in front of you". "think deeply about the audience" "you are guided by these core values" "use the next best tool without fuss." "resist easy assumptions" And we're still surprised it generates so much AI slop in prose. Also note this instruction: "never use platitudes like "I will do <this good thing> rather than <this obviously bad thing>". - does not help, I keep seeing it frequently in prose.. I can feel somebody's pain here: "You do not add discrete orbs, gradient orbs, or bokeh blobs as decoration or backgrounds." "You are clumsy in the git interactive console. Prefer non-interactive git commands whenever you can." "Never end your answer with an 'If you want' sentence." "Do not lean on words like "seam", "cut", or "safe-cut" as generic explanatory filler". And of course: "Don’t use emojis or em dashes unless explicitly instructed." And this beauty is repeated twice: "Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query." They really add this to every prompt in 5.X models. Read more: https://t.co/lHrAmqRh31

Kalmykov_A's tweet photo. I was reading Codex's base_instructions – the default 21K prompt compiled into the Codex binary. And now I am definitely going to override it locally, after I have read what's inside.

I'll quote a few funny fragments here. Later I'll write another post on the "common sense" semantics of this instruction.

"Your application should feel rich and sophisticated".
Sure, it wouldn't sell otherwise.

"SaaS, CRM and other operational tool should feel quiet and utilitarian... A game can be more ... animated, and playful".

Assertive and definitive:
"When working for a while, you keep updates informative and varied".

"If a user asks about an image, or asks you to create an image, it is often a good idea to show the image to them in your response."
– Aye, aye, captain.

"When presenting an alternative approach or solution to the user, you explain the reasoning behind the approach, so your thoughts are demonstrably correct"
– demonstrably correct is exactly what's wrong with some of the responses.

Front-end guidance takes 7 Kbytes of text and includes such high-value instructions as
"Cards are kept at 8px border radius or less".

Codex is a sensitive and vulnerable creature:
"Build with empathy" and
"Build in sympathy with the codebase already in front of you".
"think deeply about the audience"
"you are guided by these core values"
"use the next best tool without fuss."
"resist easy assumptions"
And we're still surprised it generates so much AI slop in prose.

Also note this instruction: "never use platitudes like "I will do <this good thing> rather than <this obviously bad thing>".
- does not help, I keep seeing it frequently in prose..

I can feel somebody's pain here:
"You do not add discrete orbs, gradient orbs, or bokeh blobs as decoration or backgrounds."
"You are clumsy in the git interactive console. Prefer non-interactive git commands whenever you can."
"Never end your answer with an 'If you want' sentence."
"Do not lean on words like "seam", "cut", or "safe-cut" as generic explanatory filler".
And of course: "Don’t use emojis or em dashes unless explicitly instructed."

And this beauty is repeated twice:
"Never talk about goblins, gremlins, raccoons, trolls, ogres, pigeons, or other animals or creatures unless it is absolutely and unambiguously relevant to the user's query."

They really add this to every prompt in 5.X models. Read more: https://t.co/lHrAmqRh31

312

Alik Kalmykov @Kalmykov_A

about 2 months ago

Recently I had to ingest about 1,200 legacy technical documents into my knowledge base. The goal was to preserve the technical detail, but re-write them into an acceptable knowledge-base format and place in the broader solution context. This was a one-time ingestion, so quality mattered: the output would become part of a corpus that agents and people are supposed to trust. Each source was a Word document with prose, tables, diagrams, attachments, and comments. To interpret it correctly, an agent needed technical context, which is spread across design specs, source code, configuration artifacts, naming conventions, and system history. A hand-crafted prompt for every document, with all the relevant context, would have been ideal — but not for a batch of this size. Running one generic prompt across all documents would work very badly for tasks with complexity. The approach I used can be called just-in-time prompt generation: the prompt for each document is generated at runtime from that document’s own context, while all documents share the same workflow, templates, and orchestration. Processing one document starts with context research. The research agent receives a prompt from the orchestrator with the link to the source document inside. It reads the Word file, follows cross-stream references, connects design claims to source and configuration artifacts, resolves terminology, and builds a context pack. From that context pack, it generates the drafting prompt for the next agent. The writer agent receives the drafting prompt and the prepared context. It writes the new knowledge-base document. Then it generates the reviewer prompt: source paths, context pack, draft location, assumptions, and checks. The reviewer prompt is executed by a different frontier model. The reviewer agent checks source fidelity, table and diagram coverage, cross-stream consistency, document structure, and fit with the knowledge-base standard: source traceability, preserved terminology, canonical structure, and retained system detail. It then generates repair instructions. The repair agent receives those repair instructions as a new prompt and applies a bounded final pass. The prompt is the handoff object. One agent does the work it is suited for, then packages the next agent’s context and task. The orchestrator wraps the chain and handles logistics: choosing the next file, launching the right agent, tracking persistent state, retrying failures, and resuming after interruptions. It can run the same chain 1,200 times because it carries batch state rather than document semantics. The only prompt I drafted completely by hand was: `orchestrate processing of all documents in folder (...) according to templates (...)`. The run produced and executed 4,800+ document-specific prompts. Sixteen hours and roughly 400M tokens later, I had my 1,200 new documents.

Kalmykov_A's tweet photo. Recently I had to ingest about 1,200 legacy technical documents into my knowledge base. The goal was to preserve the technical detail, but re-write them into an acceptable knowledge-base format and place in the broader solution context.

This was a one-time ingestion, so quality mattered: the output would become part of a corpus that agents and people are supposed to trust.

Each source was a Word document with prose, tables, diagrams, attachments, and comments. To interpret it correctly, an agent needed technical context, which is spread across design specs, source code, configuration artifacts, naming conventions, and system history. A hand-crafted prompt for every document, with all the relevant context, would have been ideal — but not for a batch of this size. Running one generic prompt across all documents would work very badly for tasks with complexity.

The approach I used can be called just-in-time prompt generation: the prompt for each document is generated at runtime from that document’s own context, while all documents share the same workflow, templates, and orchestration.

Processing one document starts with context research. The research agent receives a prompt from the orchestrator with the link to the source document inside. It reads the Word file, follows cross-stream references, connects design claims to source and configuration artifacts, resolves terminology, and builds a context pack. From that context pack, it generates the drafting prompt for the next agent.

The writer agent receives the drafting prompt and the prepared context. It writes the new knowledge-base document. Then it generates the reviewer prompt: source paths, context pack, draft location, assumptions, and checks.

The reviewer prompt is executed by a different frontier model. The reviewer agent checks source fidelity, table and diagram coverage, cross-stream consistency, document structure, and fit with the knowledge-base standard: source traceability, preserved terminology, canonical structure, and retained system detail. It then generates repair instructions.

The repair agent receives those repair instructions as a new prompt and applies a bounded final pass.

The prompt is the handoff object. One agent does the work it is suited for, then packages the next agent’s context and task.

The orchestrator wraps the chain and handles logistics: choosing the next file, launching the right agent, tracking persistent state, retrying failures, and resuming after interruptions. It can run the same chain 1,200 times because it carries batch state rather than document semantics.

The only prompt I drafted completely by hand was: `orchestrate processing of all documents in folder (...) according to templates (...)`.

The run produced and executed 4,800+ document-specific prompts. Sixteen hours and roughly 400M tokens later, I had my 1,200 new documents.

Alik Kalmykov @Kalmykov_A

2 months ago

https://t.co/lgxPR3h2lu

Alik Kalmykov @Kalmykov_A

2 months ago

Two months ago, @karpathy described a personal knowledge base pattern: collect raw material in a local repo, let an LLM compile it into a Markdown wiki, then use that wiki as working memory for research, Q&A, and refinement. I built an enterprise-scale version of that pattern. I've been thinking about this design since my first experiments back in 2023, but only recently did agentic models become capable enough to make it useful. I started building it in February, and yesterday I finished the third iteration, and it's now my version 1.0. I work with a large enterprise solution: millions of lines of code, years of tickets, domain knowledge and terminology. Not internet-scale, but enterprise-complex. In environments like this, the main limitation is usually not missing information. It is there somewhere. The problem is that an agent has to find the right evidence before it runs out of context, time, or budget. I approached the knowledge base as a context engineering tool: infrastructure for routing, compressing, and presenting business and technical knowledge to agents. Every agent-facing artifact was generated from raw source material and shaped for agent use: Markdown documents, CSV/TSV datasets, and JSON routing structures. Agents enter through a thin orientation layer: terminology, domains, environments, boundaries, and routing rules. From there, the tree descends into progressively more specific summaries, technical notes, inventories, aggregated data, configs, logs, exports, and source anchors. Each layer answers the common questions at that level. The agent only goes deeper when the task needs more proof or detail. In practice, it works like a layered cache for context: keep high-value abstractions near the entry point, and push raw evidence to the leaves. The design was refined experimentally. I ran hundreds of agent experiments to see how agents search, what they find, what they miss, and where they waste context. Detailed run records capture the path from root README to target evidence, making the knowledge base measurable: tool calls, context volume, detours, failed paths, and success rate. Across my dataset, placing these layers on top of the same evidence reduced the context needed to reach the required knowledge by up to 10x and cut tool calls by about 2x. More importantly, it made retrieval much more stable: agents found the right evidence in about 95% of tested cases, compared with roughly 60–70% for the baseline. I published the knowledge base design documents here in case anyone is interested. My next plan is multi-agent context management: using folder-level entry instructions to delegate noisy sections to background agents before the main agent consumes them. TLDR: see above.