Ryan Tsang

building harmonize health
currently @spc
previously microsoft azure openai, ucla

unfortunate chargers fan

[Ryan Tsang @rvtsang]

First time speaking publicly about what we're building for independent doctors at Harmonize.

We are powering the resurgence of private practice. I believe physicians should remain at the heart of care. AI is an amplifier and a superpower for them, not a replacement.

I'm fully expecting a long, tough road with lots of mistakes. If you're a doctor that resonates with this mission I'd love to work together on this.

Thanks to my good friends @vatsalbajaj and @jakekang for the fun chat. Let's do it again soon!

[Vatsal Bajaj @vatsalbajaj]

Jake (@jakekang ) and I interviewed Ryan Tsang (@rvtsang), who is building Harmonize Health - an AI-native operating system for medical clinics.

Ryan spent four years at Microsoft working on Azure OpenAI's fine-tuning product, then trained models for open-source AI safety benchmarks at MLCommons.

We get deep into how Harmonize Health builds and grades evals for voice agents (including a user-simulator agent that role-plays the patient calling in), why you should get to production before you build the perfect eval set, how Harmonize scrubs patient data out of transcripts, and the tension between forward-deployed engineering and scaling across multiple clinics.

Hope you enjoy the third episode of The Interlude Show!

Chapters:

00:00 - Move fast and break things doesn't work in healthcare
00:41 - Meet Ryan Tsang and what Harmonize is building
02:40 - Why healthcare, and the path from pre-med to CS
05:45 - Inside a primary care clinic: the work nobody sees
10:50 - Safety is the hardest technical problem
13:55 - Data plumbing vs. AI: EHRs and tribal knowledge
20:36 - Encoding a clinic's operating procedures into agents
23:35 - Inside their eval harness for voice agents
27:07 - What can we really not screw up?
30:16 - Healthcare moves at the speed of trust
38:57 - TempShot: pitching his high school app to us as VCs
49:53 - The YC all-nighter, quitting our jobs, and why compounding wins

[Ryan Tsang @rvtsang]

@Radiologysheep The ordering doc is burdened with that responsibility right?

I’m assuming they can’t steer the pt well because they don’t they have an up to date menu of the options + what’s covered.

Aka they’re shooting blind.

Have you seen a setup where this isn’t the case?

[Ryan Tsang @rvtsang]

@InvestingDoc @bilalfarooqui What do you think went wrong.

The one medical on my street is also always dead. Carbon health too I think.

[Ryan Tsang @rvtsang]

legendary olympic coach accidentally gives advice for RL loops

[Bob Bowman @coach_bowman]

Set the goal. Build the environment. Do the work. Fix the details. Let them fail while it's cheap. Keep going.

It isn't complicated. It's just hard to do for long period of time

[Ryan Tsang @rvtsang]

@akiffpremjee @YounisJoseph who is benchmarking the benchmarks benchmarking the benchmarks

[Ryan Tsang @rvtsang]

@YounisJoseph @akiffpremjee 8. cure cancer
9. world peace

etc

[Ryan Tsang @rvtsang]

@faruk_parhat @spc wait where do i get one lol

[Ryan Tsang @rvtsang]

Ok that's consistent. There's a big psychological leap of faith. This also implies that you need to be in a place financially to stomach those first few months right?

Have you seen others "dip their toe in" so to speak? Where they start seeing patients on the side or via telehealth to build up a small panel before fully leaving?

[Ryan Tsang @rvtsang]

@ossy147 @MichaelAlbertMD what's stopping you with the hipaa legal situation haha. do you need BAA's with the model providers?

[Ryan Tsang @rvtsang]

@DrDadBuilder @DrJgps Agreed. Hmm so based on your experience starting, why don’t you think there are more independents?

What were the biggest leaps of faith you took? What were the scariest and most difficult parts to getting started?

[Ryan Tsang @rvtsang]

I’m an ML engineer who built AI infra at Microsoft Azure OpenAI (and building in healthcare now)

1. Security (especially for cloud services) is even more relevant now. Understanding cloud environments, sandboxes, and some basics on “containerization”.

2. How to build always on monitors, alerting, and advanced guardrails for any AI product.

3. Keeping up with latest eval techniques that frontier labs are using. Maintaining evals over time and managing drift.

There are more. A lot of online material is dated, but depending on what you’re interested I can share some content that is still relevant. A lot of slop out there. I’d recommend talking regularly with people that have shipped real software product at scale.

[Ryan Tsang @rvtsang]

@DrDadBuilder @DrJgps how can you re-establish continuity as a core principle?

[Ryan Tsang @rvtsang]

@RojaGarimella @BeckersHR They need to get on cloud agents lol. They’re so good now haven’t needed to worry about this in months

[Brendan Keeler @healthapiguy]

We need to build the agent-native healthcare network.

[Ryan Tsang @rvtsang]

@sahirjaggi Congrats. You should swing by SPC to share more about this with the community. Would love to have you, this is important work. DM me

[Ryan Tsang @rvtsang]

@pgipe How many of these require your full attention? Would you trust AI to prefill or triage/deflect some of these?

[Ryan Tsang @rvtsang]

@pau67211382 @YounisJoseph What were some of the other best things they taught?

[Ryan Tsang @rvtsang]

@YounisJoseph @Hamp_The_Doc Why is that you think?

[Ryan Tsang @rvtsang]

@InvestingDoc It’s also reasonable to follow custom per-physician guidelines with clear escalation policies.

If you wrote these down, would you trust say an MA to follow these instructions? Importantly, would you trust them to escalate appropriately?

[Ryan Tsang @rvtsang]

@RojaGarimella I’ve been looking for how to describe what it feels like: second-principles is perfect.