Juan Diego Caballero

Learner. Physicist. Engineer.
- Founder

[dearlordylord @ohdearlordylord]

Instead of .md specs, try this: make the system a state machine (or rather, a reducer). Ask your slopus to make an interactive diagram with real events from your flow and the real reducer. Step through them, asking the agent "whys" this and that.

[Juan Diego Caballero @jdcaballerov]

Qué problema es el @BancodeBogota no sirve la banca en línea para descargar documentos

[Juan Diego Caballero @jdcaballerov]

Who designed this future, South Park writers?

[Nate Berkopec @nateberkopec]

remember how people used to spend years learning vim or emacs? haha take a big fat L buddy, I used Sublime Text until December and now I just talk to Claude through this gimp mask

[Juan Diego Caballero @jdcaballerov]

@AryShriv You can easily setup proactive agents using @memseek with the different triggers

[Juan Diego Caballero @jdcaballerov]

The business lesson:

Public data teaches a model the world. Private data teaches it your customers: what they quit, what brings them back, what's new today.

Models are commodities. Proprietary behavioral data compounds.

Paper: https://t.co/iCRxMLTOzg

[Juan Diego Caballero @jdcaballerov]

Netflix just showed an open-source LLM beating their years-tuned recommendation ranker.

The model is available to anyone. What made it work isn't: Netflix's private behavioral data.

OWN YOUR DATA

Sharing a set of slides I made to understand it better

[Juan Diego Caballero @jdcaballerov]

Step 3: this is where private data does the work.

Off-the-shelf LLM → baseline
Adapted on Netflix data → +10–20%
Post-trained on engagement → +35–50% more, ~80% two weeks later

The base model never changed. The data did.

[Juan Diego Caballero @jdcaballerov]

@DhruvWalia16 @memseek

[Juan Diego Caballero @jdcaballerov]

Does an AI browser agent get faster at a job it has done before?
We ran 180 live scraping runs to find out.

After two practice runs, an agent with @memseek memory needed 7 model turns instead of 16.5. It finished in half the time and at half the cost.

[Juan Diego Caballero @jdcaballerov]

@DhruvWalia16 @memseek For these concrete runs, the agent helpers file was never edited, so the gain was due to the playbook.

[Juan Diego Caballero @jdcaballerov]

@alexatallah IMO the key is to allow further decomposition down into sub planners. This is beautifully shown by cursor . It is pretty much the same conclusion from Cybernetics Viable system concept.

https://t.co/zt7vAs0G1a

https://t.co/jUn8Mo3oFj

[Juan Diego Caballero @jdcaballerov]

We’ve been experimenting with self-learning playbooks for scraping skills.

Added the terrifyingly dangerous benchmark: https://t.co/dVcGG1cHlO. Literally a toy site full of Goodreads quotes.

Agent gets 2 pages in and repeatedly crashes on content_filter.

We still don’t know which terrifying sentence did it. Maybe Einstein. Maybe Elie Wiesel. Maybe a misattributed Marilyn Monroe quote.

We want agents doing real work, but they can’t reliably survive scraping inspirational quotes.
Amazing.

[Juan Diego Caballero @jdcaballerov]

@memseek For terminal 1, dont forget to set the memseek API key returned after make quickstart

export MEMSEEK_API_KEY=

[Juan Diego Caballero @jdcaballerov]

The limits of this test:
• one site, one small model
• "native" saved nothing new, so its ups and downs are noise
• only 2 of 20 playbooks found the site's hidden JSON endpoint. Most learned "click each year, wait 2.5s."

Even that halved the work.

[Juan Diego Caballero @jdcaballerov]

Set env vars for al terminals:

- OPENAI_API_KEY=
- DATABASE_URL=postgresql://postgres:[email protected]:55432/memseek_test
-

Terminal 1:
$ make quickstart # to setup the db

Terminal 2:
$ run uvicorn memseek.api:app --host 127.0.0.1 --port 8000uv # API SERVER
Terminal 3:
$ uv run memseek worker # Worker

Terminal 1:
$ uv run memseek eval skill-learning --trials 1 --k-train 1 --arms native,playbookuvrun memseek eval skill-learning --suiteevals/scrape_suite.yaml

[Juan Diego Caballero @jdcaballerov]

@hthieblot @memseek a self learning context engine

[Juan Diego Caballero @jdcaballerov]

Where did the turns go? A cold agent rediscovers the page every time: open it, screenshot it, read the HTML, try a selector, try again.

Cold spent 11 of its 16.5 turns exploring. Playbook spent 2.6 of 7.

Median tool errors fell from 3 to 0.

[Juan Diego Caballero @jdcaballerov]

One surprise: browser-harness can keep its own notes, a helpers file and per-site tips. We turned both on and kept them between runs.

With its current skill prompt, the agent never edited them. Zero new helpers in 50 runs.

[Juan Diego Caballero @jdcaballerov]

After 2 practice runs, playbook vs cold:
• 9.5 fewer turns (95% CI: 5.4–13.4)
• ~95k tokens vs ~317k
• ~$0.07 vs ~$0.15 per run
• ~1.7 min vs ~3.6 min

Accuracy was unchanged: both passed the grader 60% of the time. It did the same work with less effort.