Kartic

Learning, gymming, running, vibe coding and loving it!

[Kartic @thelittlebearo]

I benchmarked OCRia-1 (LlamaParse Agentic) against 7 other OCR engines on the same 200 real-world documents (663 pages, lots of tables).
OCRia-1 ranked #1 on my primary metric, document-level F1: 94.1% (92.5% recall, 95.8% support).
Next: olmOCR 92.4%, Chandra 92.2%, Google Vision 89.9%, Mistral 84.4%.
Its standout was completeness. It had the highest recall, so it dropped the least text. olmOCR edged it slightly on token error rate (24.6% vs 25.2%), so the top three are close.
The run was clean too: 200/200 documents and 663/663 pages, with zero failures.

[Kartic @thelittlebearo]

@murtazakhomusi @jerryjliu0 Hey, is LlamaIndex comfortable with me running an internal, head-to-head OCR benchmark of OCRia-1 against other providers on my cleared document cohort? I would use its output only for evaluation.

[Kartic @thelittlebearo]

Good catch. Right now the caller’s lines are generated live for each run, so the same call never sounds the same twice. That’s noise across a sweep and no protection against fitting.

So the plan changes: record the caller audio once, replay it through turn detection only (no LLM, so it’s deterministic and cheap), tune on one set, and confirm the pick on a held-out set built around mid-sentence pauses.

[Kartic @thelittlebearo]

My voice agent keeps answering half a sentence.

Callers pause between sentences. The turn detector (Pipecat + Smart Turn v3) hears the silence, decides they’re done, and the agent starts replying while they’re still talking. In my last test run, 6 of 14 turns ended early.

Two fixes so far:

If the caller speaks again, the agent stops at its next step and restarts from the whole utterance. I can’t cancel the model call that’s already in flight, but I throw its answer away, so no tool ever runs on half a sentence.
Cut Smart Turn’s fallback silence from 3s to 1s. It was judging short replies as unfinished, so they waited the full 3 seconds.

The answers are right now. The latency isn’t. Every false end adds about 1.4s because the model runs twice.

Three ways out:

Wait longer before ending a turn. Fewer false ends, but every reply gets slower.
Put a transcript-based turn model on top of the audio one, so both have to agree the caller is done.
Accept the restart and make the model faster.

My pick is 2. The audio model hears the pause, and the text model can tell the sentence isn’t finished. I haven’t tested it yet.

If you’ve shipped a voice agent, how did you handle mid-sentence pauses?

[Kartic @thelittlebearo]

Yes. The text gate should only look at final transcripts, never interim ones. Deepgram marks a segment final once it stops revising it, and Pipecat’s turn logic already treats a new interim result as “not finished yet”. So the stable window is roughly the time Deepgram takes to finalise after a pause.

On measuring: my test harness already records, per turn, the time from when the caller stops talking to the agent’s first audio, and whether the turn ended early. So the plan is a sweep: a few stability windows × silence caps, 10 audio calls each, then plot p95 latency against the false-end rate and pick the knee.

Haven’t run it yet. Will post the curve when I do.

[Kartic @thelittlebearo]

@albertolco88 @MetaMask @iota @letgo Hi i am building a voice agent. Would love to connect.

[Kartic @thelittlebearo]

@BenSyne Hi Ben, I have been building agents and apps. Would love to contribute to this fun project.

[Kartic @thelittlebearo]

@amal_irgashev @rav4nn @LangChain @HyperFrames_ @paper That split helps. Memory can still carry one bad lesson into every later run, even if no skill changes without review. So the memory files might need review too.

[Kartic @thelittlebearo]

@shaydin0x A fine-tuned model keeps the old parameter names it was trained on. If the tool ignores unknown fields, a rename fails silently. Rerun a tool-call eval on every schema change.

[Kartic @thelittlebearo]

@amal_irgashev @rav4nn @LangChain @HyperFrames_ @paper Do those lessons get a person's review before they become skills, or does the agent promote them on its own?

[Kartic @thelittlebearo]

Mistral and NVIDIA's new agent benchmarks for engineering tools will record run time and compute cost as well as correctness. A pass that costs three times as much is a different result. The benchmarks aren't out yet.

[Kartic @thelittlebearo]

@HamelHusain Do you re-tune the cutoff whenever the judge prompt changes, or only when the traces shift?

[Kartic @thelittlebearo]

@jdpruettt @DBOS_Inc Agreed on the database half. For the send, save the idempotency key with the checkpoint before calling the provider, then reuse that key on retry. If the provider ignores keys, a crash between send and save still duplicates the email.

[Kartic @thelittlebearo]

2 of 15 simulated calls got a date or time wrong. The fix was in the tool. The model passes the caller's words, code does the date math, and the tool reads back what it understood. Next 15: zero errors, small sample.

[Kartic @thelittlebearo]

Anthropic's Claude generated 318,484 protein designs in 64 hours. Only 211 went to a lab, and 17 met the binder bar. So the lab, not the generator, sets the pace. If you build agents for science, budget for the testing step.

[Kartic @thelittlebearo]

Mistral's open Voxtral Realtime model is set to a 480ms transcription delay, per its model card. That's a setting, not end-to-end latency, so test it on your own calls. A pause the caller hears is the bug, not the bill.

[Kartic @thelittlebearo]

@AjeetThinks Agreed on the review side. The part I'd measure is where review time goes. If it's mostly reading generated code, the bottleneck is the diff size, not the agent.

[Kartic @thelittlebearo]

@anrayama The table is only useful if each row pins the model version, the harness and the seed. Without those, the "better" column is mostly noise.