Jared Joselowitz

Research Engineer building safe and reliable conversational clinical AI at @ufoniahealth

[Jared Joselowitz @JaredJoselowitz]

🚨 ASR errors in clinical dialogue can be dangerous, and WER doesn’t know it. Today we release “WER is Unaware”.

Using DSPy + GEPA, we optimise an LLM Judge that reaches clinician-level performance at detecting safety risks.

🔗 https://t.co/RZOKEa4JWq
📄 https://t.co/h5YxKghv3W

[Corey J. Gallon @CoreyGallon]

You can't A/B test on patients. You can't roll back a phone call once it's been said. And a vendor's benchmark score isn't a defense at a post-incident review. @JaredJoselowitz, AI Research Engineer at Ufonia, works inside those three constraints, and "Shipping AI to a Million Patients Without an A/B Test" is his account of what replaces the usual ship-watch-rollback loop. It's on @aiDotEngineer's YouTube.

If you build agents where being wrong for a few users isn't survivable, the talk gives you a working substitute for canary deploys.

- Start from the harm, not the feature. Regulation comes down to what your software does, what could go wrong, and how you ensure it doesn't. Ufonia enumerates dozens of specific clinical hazards first: missing a red flag symptom, hallucinating an answer, ignoring patient distress.
- Simulate before anyone real is exposed. Their framework, Matrix, runs an LLM patient (Patbot) against Dora, their voice agent, inside real clinical workflows rather than abstract scenarios. Around 200,000 real calls across 20 UK hospitals sit behind it.
- Validate the simulator itself. A script-adherence check filters weak models, then a patient and public involvement study showed real patients four conversation pairs. In three of four, the majority picked the simulated patient as the more realistic one.
- Grade with a judge you've measured against experts. BevJudge scores dialogues against expected behaviors and hazard definitions. Benchmarked on 240 examples labeled by 10 clinicians from 10 specialties, the top model hit an F1 of 0.96 with near-perfect sensitivity.
- Over-calling a hazard beats missing one. That asymmetry gets encoded as a cost matrix, so the optimization target is the real clinical cost function, not flat accuracy.
- Stop hand-tuning prompts. Formatting changes alone have swung benchmarks by 76 percentage points, and reordering few-shot examples can move a model from near-random to near state of the art. They use GEPA instead: define the metric, let a strong model reflect on failures and rewrite the prompt, keep a Pareto frontier. Hours to days becomes 30 minutes to an hour, and it's reproducible.
- Simulation is necessary, not sufficient. It earns the right to try carefully. Real patients are the outer loop, reached in stages: user testing, supervised clinical evaluation with clinicians in the loop, then monitored deployment where autonomy expands as evidence accumulates.
- Ship the evidence, not the model. Every call, dataset, pinned prompt, and judge verdict traces back to the exact hazard it addresses.
- New modalities change the failure modes, not the framework. In voice, back-channeling and interruptions make weaker models abandon safety advice mid-sentence to answer the interruption. Same approach: black box it, write down the new hazards, simulate, judge.

I'm working through the published talks from AI Engineer World's Fair sharing summaries and takeaways. Follow for more!

[Jared Joselowitz @JaredJoselowitz]

Was really fun speaking at AIE on how we use simulations and a feedback loop to make clinical voice agents safe!

https://t.co/ueCE2CkYau

[AI Engineer @aiDotEngineer]

Our AI in Healthcare Track from AI Engineer World's Fair 2026 is now live!

Thesis: what can never be wrong cannot be left to probability.

https://t.co/RPbB6C6l5C

- From Ambient Documentation to Clinical Intelligence: @c_asawa, Abridge
- Guardrails First: Engineering Member-Facing Health AI: Rashi Agrawal, Hinge Health
- Shipping AI to a Million Patients Without an A/B Test: @JaredJoselowitz, Ufonia
- 200 Million Patient Interactions Later: @vim1up, Hippocratic AI
- AI is the World's largest Relationship Therapist: Clay Cockrell + Tony Fabrikant, CoupleWork AI
- Healthcare's Agent Bytecode, X12 as the Harness for AI Agents: @vasantkearney, Onlay
- Trading Desks to Clinical Trials: @aybh08, Allos AI
- How to build an AI-Native Health Company: Dan Feng, Maven Clinic
- Don't be data poor: @anujiravane, Anterior
- Why Your Enterprise Tech Stack Isn't Ready for AI Agents: @ChrisLovejoy_ + @saulhoward, Anthropic + Anterior

[Jared Joselowitz @JaredJoselowitz]

@Shashikant86 @aiDotEngineer Thanks, you too! See you in London hopefully!

[Jared Joselowitz @JaredJoselowitz]

@altryne @aiDotEngineer @thursdai_pod On the flight now! Claude coding using starlink WiFi going to AIE WF is pretty mad

[Jared Joselowitz @JaredJoselowitz]

I'm at AI Engineer World's Fair in SF, giving a talk: "Shipping AI to a Million Patients Without an A/B Test".

How do we at Ufonia ship clinical AI when you can't A/B test on patients or un-send a phone call? You simulate it first.

Thurs Jul 2, 11:40am, Room 2024. Come say hi!

[Jared Joselowitz @JaredJoselowitz]

@NielsRogge @aiDotEngineer I'm around, let me know if you're free for a coffee!

[Jared Joselowitz @JaredJoselowitz]

@mattzcarey I'll be there, definitely keen for a coffee if you find a sec. Let me know!

[Jared Joselowitz @JaredJoselowitz]

@LakshyAAAgrawal @Stanford Oh we definitely will! We already have GEPA-optimised prompts running in prod so excited to check this out

[Jared Joselowitz @JaredJoselowitz]

Presented a @Stanford paper at work, “Feedback Descent,” showing how any text artifact can be optimized and spent the day implementing it. Someone asked if GEPA could just optimize any text too… 3 hours later it drops

This field moves insanely fast

https://t.co/RF3j6PH6of

[Lakshya A Agrawal @LakshyAAAgrawal]

Excited to release @gepa_ai's optimize_anything: a universal API for optimizing any text parameter.

It consistently matches or outperforms domain-specific tools optimizing code, prompts, agent harnesses, cloud policies, even visuals!

If you can measure it, you can optimize it.

[Jared Joselowitz @JaredJoselowitz]

Thanks for highlighting our work! GEPA helped us turn LLMs into clinical-meaning evaluators, a big step toward safer clinical agents. Excited to keep exploring this with the DSPy community.

[Ankur Gupta @getpy]

Happy Friday Everyone

DSPyWeekly #13 brings you 11 articles, 5 videos, and 4 new projects to explore.
Highlights: ✨ Building Agents with Ephemeral Memory 🛠️ Announcing DSPy Code CLI 📄 New papers on Prompt Optimization

[Jared Joselowitz @JaredJoselowitz]

@FaroukAdeleke3 I have yeah, looks really cool, haven't had a chance to try it out yet though. Have you?

[Jared Joselowitz @JaredJoselowitz]

🚨 ASR errors in clinical dialogue can be dangerous, and WER doesn’t know it. Today we release “WER is Unaware”.

Using DSPy + GEPA, we optimise an LLM Judge that reaches clinician-level performance at detecting safety risks.

🔗 https://t.co/RZOKEa4JWq
📄 https://t.co/h5YxKghv3W

[Jared Joselowitz @JaredJoselowitz]

@lateinteraction @LakshyAAAgrawal @matei_zaharia @getpy @DSPyOSS Huge thanks to the whole team!
@ZachEllisX @DrErnestLim @ufoniahealth

[Jared Joselowitz @JaredJoselowitz]

If we want safe clinical agents, we need evaluation that understands meaning.

Using GEPA through DSPy allowed us to build a safety judge that actually works.

📄 https://t.co/h5YxKghv3W

cc: @lateinteraction @LakshyAAAgrawal @matei_zaharia @getpy @DSPyOSS