JD Pruett

a neuro/ml research + b engineering + c writing, weights tbd

[JD Pruett @jdpruettt]

Setting up a specialized vector database like pinecone/turbopuffer is complete overkill for almost any engineering dataset under 100,000 vectors.

For anything under that 100,000 cutoff, an exact scan in memory or SQLite is sub-second and its also completely costless.

There's a persistent narrative driven by vendors that you need distributed infrastructure the second you touch embeddings. But brute-force array math is fast enough for the vast majority of retrieval tasks people are actually building right now.

in general i think this principle applies that we spent a lot of time provisioning external infra before checking whether local math solves the problem first.

[JD Pruett @jdpruettt]

re education, excited not for custom explainer videos, but ephemeral, interactive worlds.

Videos are great, but they are still passive -- you're just watching someone else traverse an idea which feels closer to learning but still far from the real thing.

The much more compelling setup is when an LLM tutor detects confusion on a concept and procedurally writes a minimal, self-contained 3D sandbox in WebGL or Three.js.

For example,
- A central massive body appears with interactive sliders for mass and velocity.
- The student flings test particles into orbit, watching the trajectory decay in real time.
- They physically break the orbit, adjust the math directly, and see the consequences immediately. etc. etc.

We will probably look back at static diagrams and pre-rendered video as weird legacy compromises.

[JD Pruett @jdpruettt]

@thelittlebearo For database writes, you can commit the write and checkpoint in one transaction, as @DBOS_Inc does. An external email send can't normally join that transaction, though; that part still needs provider-side idempotency.

[JD Pruett @jdpruettt]

How to keep agent memory intact when execution environments crash or sleep:

Keep the loop itself on a boring persistent server.

Run code execution and bash calls in throwaway Firecracker microVMs.

When an environment is idle, snapshot the microVM and spin it down.

Route interrupts and webhooks directly to the server so nothing drops while the VM is paused.

[JD Pruett @jdpruettt]

@iamjayfoss Yes, approving every tiny app won't scale. The useful unit is the permission: which data, which network destinations, how much spend, for how long. Then the app can be disposable without its access grants becoming permanent.

[JD Pruett @jdpruettt]

With the coming tsunami of discardable micro-apps, we are heading straight into an infra bottleneck that IT departments are completely unprepared for.

Because LLMs make writing software trivial, teams are starting to spin up dozens of single-use tools every week. A quick dashboard to monitor a cluster for ~2 days, a one-off parser for a vendor dataset, an internal tracker for an event, etc etc

Code is cheap, infra much less so...
- Every micro-app requests database credentials, an IAM role, and VPC access.
- Security teams still have to audit outbound traffic and secret management.
- Central IT gets buried trying to monitor, patch, and decommission thousands of ephemeral containers.

Jevons paradox playing out internally in orgs. Infra management will have to become just as autonomous and discardable as the code.

[JD Pruett @jdpruettt]

@deepdrpt Agreed. I'd only snapshot when restoring is cheaper than rebuilding. The virtual disk still needs persistence; for short, easily reconstructed tasks, snapshots may not be worth it.

[JD Pruett @jdpruettt]

@EternitiesAI In the split I'm describing, the plan/conversation stays with the loop on the server. The VM restores the execution environment. Persisting the agent's decisions and task state is a separate requirement.

[JD Pruett @jdpruettt]

@ScottSiu1983 It can. A faithful snapshot can preserve a stale view of the world. I'd re-read mutable state on resume—branch head, ticket status, permissions, deadlines—before the next consequential tool call.

[JD Pruett @jdpruettt]

gave an autonomous agent write access to our bank account and told it to "minimize overhead"

we no longer have an LLC

[JD Pruett @jdpruettt]

What can local CLI tools teach us about building sustainable infrastructure for AI?

Few people understand the operational bottlenecks here *and* also spend time testing models on basic consumer hardware. A classic mistake when scripting with CLI tools is treating a local model like a standard unix utility: you spin up a process, pass in a prompt, print the answer, and exit.

In reality loading a 4GB Mistral 7B weights file into unified memory every time you want an answer means constantly moving weights from disk to RAM, for a query that takes less than a second to generate.

If you don't run a persistent daemon like Ollama or keep an interactive session alive, naive execution will cripple your pip

[JD Pruett @jdpruettt]

Shoving 30 custom tool schemas into your system prompt is almost always a mistake.

You eat 8k tokens of context before the model even starts thinking. Dumping docs into flat files and letting Claude use bash with grep beats dedicated connectors nine times out of ten.

[JD Pruett @jdpruettt]

@String_The0rist Ah I built my own.

[JD Pruett @jdpruettt]

You save a lot of compute by running context compaction during the provider cache TTL instead of waiting for the window to fill up.

If you let a harness run until step 18, you end up paying full input rates on 110k tokens just to summarize them into a note.

Set an async worker to fire at minute 8 of Anthropic's 10-minute cache window. Read the warm prefix while it is cheap, write the facts to Postgres, then drop the raw messages from the harness.

[JD Pruett @jdpruettt]

@eyad_khrais All true if you can handle the far right tail of edge cases

[Eyad @eyad_khrais]

https://t.co/kCOoJF7FBx

[Daniel Kornum @dkfromdk]

https://t.co/23NTYokKOT

[JD Pruett @jdpruettt]

@dkfromdk A lot of frontier lab lockin rn is psychological

[JD Pruett @jdpruettt]

Baby, they don't mean anything to me. It's just you.

[JD Pruett @jdpruettt]

built this. a drop-in proxy in front of your frontier API calls that logs prompt/response pairs, then once there’s enough data, profiles the distribution and launches an experiment ladder: GEPA first, then RL or distillation on a small model using on-policy distillation, SDPO, pi distill , etc. Use frontier responses (from their existing API calls) or some internal benchmarks for evals.

Worked beautifully. A custom kernel made experiments 3x cheaper than Unsloth + vLLM to afford generous hp sweeps. Built up intuition for which methods win for which task shape.

From customer’s view they added a line of code, waited, and got back a cheaper, better model 36 hours later. Magic.

A few problems:

1. the best use cases are narrow and high-volume as in @brian_armstrong's example. if you’re at that scale you likely have a team who’ll want to own it.

2. training is increasingly cheap (looks like less than 100 bucks per run for coinable). Great news for customer. Bad news is bottlenecks shifts to outcome labeling, benchmark creation, formulating the reward. All that is customer-specific data work. Fine, but then your pipeline is just SoR + Claude. No moat.

3. So perhaps you want to move into experimentation layer — make it easy for an eng with claude + [tool] to build a company model. Great idea! Enter Tinker.

Tech companies will want to own it. Non tech companies will eventually get there, but are still 18 months from getting even frontier APIs working at a scale that matters. Which means you’re at least 28 months from them thinking about replacing their committed spent agreements with custom models, and that's all assuming they don't just do it in house.

Happy to be wrong about this, especially with a term sheet.

[Chris Saum @christophersaum]

Who’s building this? I want to back founders helping companies build specialized models on their own data. $500k checks. We move fast.