If your AI agents like Hermes generate hundreds of Markdown files a day, traditional folders won't scale.
The common workaround is treating Obsidian like a database: dump machine files into a hidden machine/YYYY/MM/DD/ folder, enforce strict YAML frontmatter (type, source, project, created, status), and navigate everything using Dataview or Bases (e.g., Today, Review, and Project views).
Why It Breaks at Scale
At ~100,000 files a year, using Obsidian as a database hits a wall:
Performance tanks: Obsidian must parse every file, slowing down startup, sync, and mobile. Dataview noticeably lags around 10,000โ30,000 files and becomes nearly unusable past 50,000.
No data integrity: Frontmatter isn't a strict schemaโagents will eventually break the YAML, and concurrent writes aren't safe without transactions.
Slow queries: Dataview scans files without proper indexing, whereas SQLite (with FTS5) searches millions of rows in milliseconds.
The Solution: SQLite + Obsidian Hybrid
Split raw storage from your reading experience:
Log everything to SQLite: Have agents write the metadata fields and body text directly to SQLite.
Browse raw logs with Datasette: Use Datasette for a zero-config web UI when you need to inspect raw data.
Export only what you read: Run a scheduled script to export daily digests, project summaries, and flagged reports into Obsidian.
Your vault stays at a few thousand curated files, keeping Obsidian fast while SQLite handles the heavy lifting.
Suppose I get hundreds of new Markdown files every day
what is the best way to organize them? I don't think Obsidian's folders or categories work very well for this
Suppose I get hundreds of new Markdown files every day
what is the best way to organize them? I don't think Obsidian's folders or categories work very well for this