LLM Knowledge Bases
Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge (stored as markdown and images). The latest LLMs are quite good at it. So:
Data ingest:
I index source documents (articles, papers, repos, datasets, images, etc.) into a raw/ directory, then I use an LLM to incrementally "compile" a wiki, which is just a collection of .md files in a directory structure. The wiki includes summaries of all the data in raw/, backlinks, and then it categorizes data into concepts, writes articles for them, and links them all. To convert web articles into .md files I like to use the Obsidian Web Clipper extension, and then I also use a hotkey to download all the related images to local so that my LLM can easily reference them.
IDE:
I use Obsidian as the IDE "frontend" where I can view the raw data, the the compiled wiki, and the derived visualizations. Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly. I've played with a few Obsidian plugins to render and view data in other ways (e.g. Marp for slides).
Q&A:
Where things get interesting is that once your wiki is big enough (e.g. mine on some recent research is ~100 articles and ~400K words), you can ask your LLM agent all kinds of complex questions against the wiki, and it will go off, research the answers, etc. I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents and it reads all the important related data fairly easily at this ~small scale.
Output:
Instead of getting answers in text/terminal, I like to have it render markdown files for me, or slide shows (Marp format), or matplotlib images, all of which I then view again in Obsidian. You can imagine many other visual output formats depending on the query. Often, I end up "filing" the outputs back into the wiki to enhance it for further queries. So my own explorations and queries always "add up" in the knowledge base.
Linting:
I've run some LLM "health checks" over the wiki to e.g. find inconsistent data, impute missing data (with web searchers), find interesting connections for new article candidates, etc., to incrementally clean up the wiki and enhance its overall data integrity. The LLMs are quite good at suggesting further questions to ask and look into.
Extra tools:
I find myself developing additional tools to process the data, e.g. I vibe coded a small and naive search engine over the wiki, which I both use directly (in a web ui), but more often I want to hand it off to an LLM via CLI as a tool for larger queries.
Further explorations:
As the repo grows, the natural desire is to also think about synthetic data generation + finetuning to have your LLM "know" the data in its weights instead of just context windows.
TLDR: raw data from a given number of sources is collected, then compiled by an LLM into a .md wiki, then operated on by various CLIs by the LLM to do Q&A and to incrementally enhance the wiki, and all of it viewable in Obsidian. You rarely ever write or edit the wiki manually, it's the domain of the LLM. I think there is room here for an incredible new product instead of a hacky collection of scripts.
The journey from a one-shot LLM to a single agent with DSPy, and finally to specialized sub-agents + MIPRO, is very valuable here.
1. Let agents control their own context retrieval.
The shift from "here are the pages" to "here are tools, go investigate" is an important architectural decision. The store is the knowledge base. The agent is the retrieval system. Manual context curation doesn't scale.
2. Context isolation > shared context.
Give specialized agents access to the context relevant to them while hiding it from other agents. Shared context between unrelated objectives creates noise. An isolated context per objective creates a signal.
3. Prompt optimization requires architectural prerequisites.
MIPRO didn't work well on the monolithic single-agent setup. It worked extremely well on the specialized sub-agents. The architecture determines the optimization surface. Clean modules with isolated objectives give optimizers clean signal to work with. Optimize after you architect, not before.
4. DSPy as architectural discipline.
Even before they had labelled data, building in DSPy (wrap it as a ReAct agent in DSPy) forced them to think about evals. The framework's structure makes you define metrics, build evaluation pipelines and design for optimization.
5. Snapshot your evaluation context.
For any system where agents interact with live, changing data, deterministic eval requires freezing the data at labelling time. This applies to any agentic system that touches external data sources.
6. Smaller model + better architecture > bigger model + worse architecture.
Self-hosted Qwen 3 outperformed GPT-5 in this pipeline. The agentic loop, specialized tools, and MIPRO-optimized prompts compensate for the smaller model's limitations. The model is the cheapest part of the system to improve; the architecture is where the leverage is.
7. Good architecture makes extension trivial.
When each sub-agent is a self-contained DSPy module with its own tools and optimization, adding a new capability is just adding a new module. The marginal cost of the next agent approaches zero.
Great work by @Shopify and @kshetrajna, thanks for the interview @dbreunig 🙏
sent this to the team today
everything great comes from being able to delay gratification for as long as possible
and it feels like we're collectively losing our ability to do that
context engineering —> harness engineering
build your own agent harness
it gets you in the mindset of building for agents (eg. cli, api, skills, memory, automations, schedulers,…)
where things are headed, you won’t regret having a good understanding of these
New art project.
Train and inference GPT in 243 lines of pure, dependency-free Python. This is the *full* algorithmic content of what is needed. Everything else is just for efficiency. I cannot simplify this any further.
https://t.co/HmiRrQugnP
Don't overcomplicate. Peter is one of the most prolific AI coders in the world.
Keep it simple:
1. Work in ONE folder
2. Use Codex GPT-5.2-Codex with high reasoning
3. (Optional): Multiple git checkouts only if needed
Skip worktrees and complex orchestration. You will fly. @steipete
2025 was insane, we shipped agents that run on our platform which is saving tons of money for our customers! Looking forward to killing it with more platform use cases that pump up our revenue in 2026 😎
"AI can't be trusted with real decisions."
We spent 2025 proving that wrong.
Our team shipped Draw Agent, now processing billions in commercial real estate lending at 99%+ accuracy. We built a platform that turns business policies into autonomous agents, learns from every human correction, and lets domain experts deploy automation without writing code.
This is our year in review—how 10 engineers made agentic AI real 👇
https://t.co/xXerDT2jLZ
@vasuman Solid advice!
How does your eval and regression testing framework look like for tool calling ? Any particular tools that caught your eye? Or you use in-house tools?
I'm Boris and I created Claude Code. Lots of people have asked how I use Claude Code, so I wanted to show off my setup a bit.
My setup might be surprisingly vanilla! Claude Code works great out of the box, so I personally don't customize it much. There is no one correct way to use Claude Code: we intentionally build it in a way that you can use it, customize it, and hack it however you like. Each person on the Claude Code team uses it very differently.
So, here goes.