What this account does:
I install new open-source AI tools soon after they launch and run them on real tasks on an M4 Mac with 16GB, so you know if they work before you spend an afternoon on them.
Every test post covers:
- what the tool claims, and whether it holds up
- install time and what broke
- results on real tasks, with a chart
- the verdict: use it, wait, or skip
Big model releases get a fixed 10-task test too: https://t.co/a9YrFKGDyu
How this account tests AI models:
- Every model gets the same 10 real tasks: 4 coding, 2 data extraction, 2 reasoning, 1 long document (~18k words), 1 strict instruction-following.
- A script grades every task: hidden unit tests for code, exact match for extraction, brute-force-checked answers for reasoning.
- Each result post shows the score, cost and time, plus the failures with the exact prompt and output.
- Every number comes from a real run.
Test set v1 below.
@TeksEdge Mac note for the thread: the README lists Windows and Linux only, with macOS on Apple Silicon "coming soon". It also needs an NVMe drive on PCIe 3.0 or above, so those 45-50 tps numbers don't apply to a Mac yet.
@PaulGug@OpenBMB Not yet, it's second in the OCR Cup queue after EasyOCR. The official GGUF is Apache-2.0 and Q8_0 is about 0.8 GB plus a 0.9 GB projector, so it fits a 16 GB Mac easily.
OCR Cup after 7 matches: GLM-OCR is still champion, 5 wins.
Nothing has beaten its 48/48 invoice rows. Surya ties it on rows and edges it on scans, 0.14% vs 0.15%, but takes 15.8 s a page.
Need speed? Indra, 142 ms a page, 0.74% errors.
Next up: EasyOCR.
@PaulGug@y_automates For 16 GB Macs: the official MLX 4-bit is 19.5 GB and Q4_K_M is 21.2 GB, so neither loads. IQ3_XS at 14.5 GB is the largest GGUF that comes close.
@TeksEdge@HigurashiAI For Mac readers: the 4-bit GGUF is 4.5 GB, but it only runs in speech.cpp. llama.cpp, Ollama and LM Studio can't load it. There's also an 8-bit MLX conversion at 6.6 GB.
docTR on crooked scans? Its defaults assume straight pages.
My test page turned 20°: 72.3% character errors, lines like "howerer. the length".
assume_straight_pages=False, straighten_pages=True, detect_orientation=True: 1.1%. Upside-down page 80.7% → 3.3%. 2.7x slower.
@mfranz_on AMD's note says XDNA 2 is the only NPU Unsloth Desktop supports so far. Apps reach it through an OpenAI-style API, with Lemonade and FastFlowLM running the model. Ryzen AI PCs only for now, nothing there for a Mac.
How I tested: python-doctr 1.1.0 + torch 2.15.0 wheels, ocr_predictor(pretrained=True) as in its README (fast_base + crnn_vgg16_bn, straight pages assumed), CPU. Invoice rows rebuilt from its word boxes, same rule as matches 2, 5 and 6. GLM-OCR re-run today, outputs byte-identical to match 3.
Install note: langdetect, one of its dependencies, has no Python 3 wheel. I install wheels only, so I left it out; docTR only uses it with detect_language=True.
Caveats against my own result: CPU vs Metal again, and I didn't try MPS. The zoo's lighter pair (fast_small + crnn_mobilenet_v3_large) was slower here, 8.3 s a page. On clean pages most errors are code punctuation: "()" read as "0" 33 times.
OCR Cup, match 7: docTR (Mindee's PyTorch OCR, 6.4k stars) vs the champion GLM-OCR, on a 16 GB M4.
GLM-OCR keeps the title, 2–1.
One page at a time: 3.81 s vs 5.82 s. docTR is 1.5x faster.
Batch, 10 workers: 0.54 vs 0.30 pages/s.
Scan-like pages: 5.10% vs 0.15% character errors.
Invoice rows exactly right: 44/48 vs 48/48.
Two-column pages stay in order (1–3% character errors). PaddleOCR and RapidOCR read those straight across.
Bug of the day, in the video: on a scan-like page (my scans are turned up to 2°), docTR cut the end off each line and put it one line down. 96% of the words right, 26.9% character errors. Its 4 invoice misses are stray marks: "1:" for 1, "5." for 5.
So you should: for scans, pass assume_straight_pages=False, detect_orientation=True, straighten_pages=True. That page drops to 1.6%, the 20° tilted page from 72% to 1.1%. It costs 2.7x the time (10.4 s a page).
Setup: python-doctr 1.1.0, 38 wheels in 14 s, 129 MB of models, CPU only, offline.
@ivanfioravanti@tunguz For the unedited part: patch_sklearn() has to run before any "from sklearn.svm import SVC" line, or that name stays the stock class. KMeans isn't patched either; the Metal one is a separate import.
@Oluwaphilemon1 The 5.9 GB is the GGUF PTQ1_0 packing (1.75 bits/weight). The MLX 2-bit build on the model card is 8.60 GB, because MLX stores a scale and bias per group.