Want one of your Macs to keep more room free for your other apps? Slide a bar, move some layers, hit apply, and about 13 seconds later my home AI is answering again. You can even watch each Mac reload.
Day 21 of building Caravan in the open, my open-source app for running AI models across my Mac minis.
We're talking about splitting a model across multiple mini's... Choose how much of the model is on each machine! Or skip the slider and let it pick the split for you. Great for shuffling memory from nodes that have more active apps running than the others - just move more of the model on the less-active nodes!
Warning: models that are too unbalanced may not behave correctly, just like humans in real life.
Hostnames have been blurred to protect the innocent.
<nerd>
Model's layers were split 27/37 back to 24/40 and reloaded in 8.82 s, chat was usable at 12.93 s
</nerd>
Interfaced Hermes with Caravan today. With Hermes, each prompt sends 11.3K tokens through the local model each and every time. That information includes all it's instructions and tools.
That alone took over three minutes before the first word.
Now Caravan remembers what it's already read and that step takes about a second.
Day 20 of building Caravan in the open, my homebrew app for running AI models across my Mac minis.
I tried a bunch of ways to make it faster. Here's what worked and what didn't:
- Worked: it stops rereading the same setup every time and only passes along the new parts of the prompt.
- Worked: a tiny helper now names your chats almost instantly, so the big model stays free for real work.
- Didn't work: warming everything up early. More busywork, same wait.
- Depends: guessing ahead. Great when it's copying text, a hair slower in normal chat, so it's off unless you want it.
- Didn't work: squeezing its memory smaller. It got slower, so that option stays off.
- Tradeoff: turning off thinking gets short answers back in about a quarter of the time, but only because it thinks less, so you choose it per question as needed.
- Two Macs give it room, not double speed. The big model couldn't do the full job on one Mac, but it could when split across two.
Which of these would you have guessed wrong?
<nerd>
- Setup reread 207.87 seconds to 1.17 seconds
- Guess ahead 44.26 to 70.69 tokens/second when copying (86% of guesses accepted) but 29.34 to 28.48 in chat
- Thinking off 42.75 seconds to 9.68 seconds
- Smaller models are 3 to 6.5% slower when split - only split when necessary.
</nerd>
I agree.
Bare local LLMs are fast(er) but not very useful.
A local LLM paired with something like Hermes or Openclaw is where the magic happens - and due to the amount of tools and instruction that’s injected into every prompt, the magic fizzles a little bit when prompts sit unanswered for minutes at a time potentially.
Prefill makes magic fast again.
The work lives in memory, and when the power goes down, so does the work.
However you've brought up a great point. I'm going dedicate a "dev day" to ensure the system comes back up fully - not only the hardware (already done with auto-on after power out, auto-login at boot), but Caravan (already starts at boot and already automatically restores the last know model).
The missing piece is continuing (or restarting) the job... I'll keep you posted.
I sent a controlled failure to both of my Mac minis to see what my home AI would do.
They turned themselves back on, and 64.7 seconds later it was answering questions again. I didn't touch a thing.
Day 19 of building Caravan in the open, my homebrew app for running AI models across my MacStax - testing robustness for when you just can't make it home.
What this means in real life:
* Ask it anything and watch the answer appear as it's written, all on two computers in my house
* If the main connection drops, it switches to a backup by itself and stays reachable
* After a power cut, it comes back with the same model, set up the same way.
What would you trust a home AI to keep running for you?
<nerd>
Huihui 27B 4-bit, split 24/40 across two Minis, back in 64.7s after the power cut, then 0.815s to the first word and 6.5 tok/s
</nerd>
@aitorapps I have my systems set to not sleep and to auto log on.
They are headless - no keyboard, monitor, or mouse. So they need to spring back into a useable state without me if something like a power outage happens.
The only access I have to them is remote shell or Remote Desktop.
I asked a 21B AI model 31 questions on my two Mac minis. No cloud, no subscription, and every answer made sense.
Before Day 18's fixes, it answered in gibberish.
Day 18 of building Caravan in the open, my homebrew app for running AI models across my Mac minis.
The gibberish came from a piece of the model that Caravan never loaded. Fixed, and now Caravan won't start a model that's missing any part, so you get a clear error instead of nonsense.
What else got better for running AI at home:
- It starts answering in about half a second and writes at about 12 tok/s, faster than most people read
- Caravan checks how much memory each Mini has free and splits the model to fit. A lopsided split failed every chat; half on each Mini works
- Split models load fast, since each Mini reads its half from its own drive: 8.4 GB in 19 s for a 30B model
- Interrupted downloads pick up where they left off instead of starting over
- About 40 kinds of models were mishandling long conversations. Fixed
- Odd models come with a plain-English warning and a one-click way to report them
- The one-click speed tuner now works on more kinds of models
- Every reply shows how fast it started and how fast it wrote
I also tried a newer version of the engine underneath. It started faster but wrote about 2% slower, so I'm keeping the current one. And 487 tests pass.
What would you use a private AI at home for?
Maybe they do, but I'd prefer to fix the failure. This failure was due to a model that Caravan had never experienced.
I was playing with an abliterated model - something off the well-worn path of open models (at least for me).
I can see in the video I was testing Llama 3.2 Dark Champion Instruct Uncensored Abliterated.
@aitorapps I've never experienced a failure like that with caravan.
The worst it's ever gotten is when I was using Exo. Exo would fail on me and at the time wasn't set up to have it diagnosed remotely.
When you say sleep, do you mean the system itself sleeps or the GPU?
Thunderbolt is a no brainer. And I ignored it for the longest and just used WiFi.
I heard all the hype about how great (and necessary) Thunderbolt 5 was - and I didn’t have access to it due to my Mac’s being M4. “Thunderbolt 5 or Bust” was my impression.
So I relied on WiFi because no one talks about 2nd best, 3rd best, etc. Maybe it’s my bubble here on X, maybe it’s people that only have the latest and greatest that I come across, but never see topics on what 2 year old equipment can do for your setup.
Then at the suggestion of a bot, I grabbed a thunderbolt 5 cable - and the world of “Thunderbolt 4” opened up to me, which I’d never heard of.
And it’s WAY faster than WiFi. My tokens/sec skyrocketed (relatively to WiFi) and I haven’t looked back.
@lexfridman I'll stand and wave to you from the side of the highway as you pass Western Kansas on I-70!
And if you stop, I'd shake your hand for introducing me to some of the most influential people I now have in my life!