How can we log, audit, and improve deployed AI models and agents for medicine? 🏥
AI is rapidly entering the clinic. Although models are already deployed across millions of patients, there is no consistent approach to monitor model use, assess performance, detect failure modes, or measure clinical impact. Model cards and data sheets describe a model before deployment. After deployment, there is no standard to record how, when, by whom, and for whom health AI models are used.
We need a “black box” flight recorder for medical AI. ✈️
Co-supervised by @zakkohane and @marinkazitnik, I led a team across 53 institutions and 11 countries to develop MedLog, a protocol for event-level logging of medical AI. Each time an AI model makes a prediction, MedLog creates a record with nine core fields: header, model, user, target, inputs, artifacts, outputs, outcomes, and feedback.
In our updated pre-print, we now report four real-world pilot implementations of MedLog across the US, Switzerland, and Vietnam. Predictive and generative AI, with 3M+ logged model invocations across up to 791K patients.
🇨🇭ICU deterioration monitoring in Switzerland: 223,840 records, 114 days, 212 patients
🇻🇳Wearable tetanus progression monitoring in Vietnam: 3,406 records, 289 days, 15 patients
🇺🇸LLM-based sepsis quality reporting in San Diego: 3,766 records, 89 days, 60 patients
🇺🇸Patient non-attendance prediction in New York: 2,914,264 records, 244 days, 791,319 patients
A thread on what MedLog revealed and how to get involved 🧵👇 1/n
The multitude of ways that jack up American healthcare costs (without improving outcomes)
@armollica@WSJGraphics@annawmathews
gift link https://t.co/FjseZsYQiT
I don’t think people realize how much healthcare costs are driving big companies to fire and not hire.
It costs them $30k per family, per year for premiums and care. Most of that goes to the massive, vertically integrated insurance companies that send weekly bills that no one reviews in details. And it doesn’t include the company overhead to deal with it all. It’s usually the 2nd largest expense after payroll. Which is insane
It’s far easier to blame AI than it is to blame Healthcare costs.
Want to increase jobs, wages and improve affordability for every American ?
Break up the biggest insurance companies. Make divest non insurance companies. They don’t need thousands of subsidiaries. That’s how they game and abuse the system and increase costs for all of us.
Call your senator and tell them to support the BreakUp Big Medicine Bill by @HawleyMO and @SenWarren.
American identity isn’t a scalar quality that varies based on your ancestry. It’s binary: either you’re an American, or you’re not. You are an American if you believe in the rule of law, in freedom of conscience & freedom of expression, in colorblind meritocracy, in the U.S. Constitution, in the American dream, and if you are a citizen who swears exclusive allegiance to our nation. That’s the truth & it’s why I wrote this piece.
For the record. This was not unanimous. I was muted on the call and not allowed to speak or voice my opposition to this move. Also for the record, this was not on the agenda. This was not consensus. This is censorship.
3 years ago we could showcase AI's frontier w. a unicorn drawing. Today we do so w. AI outputs touching the scientific frontier: https://t.co/ALJvCFsaie
Use the doc to judge for yourself the status of AI-aided science acceleration, and hopefully be inspired by a couple examples!
I promised to come back to @X after I investigated the facts concerning @EPotterMD's video post about @UHC and its health insurance subsidiary, UnitedHealthcare.
To review, I made an @X post in response to Dr. Potter's videos and X posts about an overzealous representative of United Healthcare ("UNH") that had apparently interrupted her while in the operating room, and denied coverage for her patient's treatment. In response to her January 7th video about the experience, Clare Locke, defamation counsel to UNH, sent a six-page demand letter to Dr. Potter, which begins:
"We are writing to demand you correct your knowingly false, misleading, and defamatory social media posts regarding UnitedHealthcare."
In the second paragraph of the letter, UNH demands that:
"You must promptly correct the record by removing your videos, posting a public apology to UnitedHealthcare, and condemning the threats of violence aimed at our client result from your posts."
The six-page demand letter can be found here:
https://t.co/YUxHlBj4MO
Before I get into the details, I want to emphasize that regardless of the facts of this situation that there is no justification whatsoever for violence and/or threats of violence against company officers or their legal or other representatives. This is particularly poignant in this case as we all know that the CEO of UnitedHealthcare was murdered in cold blood on the streets of New York, a horrendous tragedy for all involved, and for society at large. I understand the emotions of those who have felt harmed or been harmed by a failure of their insurer to pay for healthcare that was needed. I get it, but violence is not the solution to solving this problem.
Getting back to my post about United Healthcare, I said that if I still shorted stocks, I would short UNH because based on Dr. Potter's experience I believed that UNH's "profitability is massively overstated due to its denial of medically necessary procedures." I also encouraged the @SECGov to do a thorough investigation of the company.
UNH responded to my post by releasing a public statement that said:
"Health insurance has long been subject to significant regulatory oversight and earnings caps. Any claims that health insurers, which typically have low- to mid-single digit margins, can somehow over-earn are grossly uninformed about the structure and strong regulatory oversight of the sector."
UNH also stated that it had contacted the SEC because of its concerns with my post.
Contemporaneously, a partner at Clare Locke contacted our firm, and said that Dr. Potter's claims were false, and that I should therefore take down my post. I took down the post, not wanting to have an inaccurate post on X. We have used the Clare Locke firm and respect their work, so I took their request seriously.
My CLO was also contacted by the general counsel of UNH who told her that the underlying facts in Dr. Potter's posts and videos were false, and that UNH employees were under considerable stress due to the murder of their CEO -- which is understandable to say the least, and for which I greatly empathize. The UNH GC also asked to speak with me directly.
When my CLO reported the call to me, I said that before I would agree to speak with the UNH GC, I would like him to provide a detailed explanation of what Dr. Potter had said that was wrong in her videos. Our CLO then contacted the UNH general counsel who said that he would send this information to her, and he took her email address.
After days went by and we did not receive anything from UNH, our CLO again reached out to the UNH GC. He explained that he understood that we now had a copy of the Clare Locke demand letter, and that the letter provided all of the information we needed in order to understand what Dr. Potter had gotten wrong.
Since my post, I have had the opportunity to speak with Dr. Potter and her counsel numerous times. Dr. Potter and her lawyer have sent me supporting documentation of the statements she made in her video, which I have reviewed carefully and about which I have had the opportunity to ask any questions that I have had. I have also reviewed the defamation claims that UNH made in the letter from Clare Locke.
Based on all of the above, I believe that Dr. Potter told the truth in her initial video and in her statements and advocacy since that date. I also believe that UNH's threatening defamation letter to Dr. Potter and its public statements about my post and SEC complaint are simply brazen attempts to silence UNH's critics.
Bear in mind that I have extensive experience with companies that attempt to silence and bully their critics. Herbalife and MBIA, in particular, were expert in shutting down criticism and regulatory interest through their aggressive approach to public relations and the media, by threatening and bringing litigation, by asking regulators to investigate market participants who questioned their accounting and business methods, by using their political influence, and by other more unseemly methods.
I believe that you can learn a lot about a company by how it responds to its critics. UNH's response here parallels how Herbalife attacked its critics through its public statements, threatened litigation, SEC complaint, and other activities.
Let's first examine all of Dr. Potter's statements in her January 7th video that triggered UNH's response here:
"It's 2025 and insurance keeps getting worse."
This is a statement of opinion by Dr. Potter and free speech permits it.
She continued:
"I just did two bilateral DIEPs and two bilateral tissue expanders for patients and I've never had this happen before."
I believe Dr. Potter is telling the truth, which explains why she was inspired to do a video in the first place, and which I explain further below.
She continued:
"But during the second DIEP I got a phone call um into the operating room, saying that United Healthcare wanted me to call them about one of the patients who was having surgery today, who's actually asleep having surgery. And um you know said I had to call right now."
Dr. Potter is referring to a representative from UNH who called the hospital operating room front desk and asked to speak to Dr. Potter. When the nurse on duty explained that Dr. Potter was unavailable because she was in the OR, the UNH representative explained that he had to speak Dr. Potter right away. This caused the nurse on duty to escalate the message to the head nurse on duty who delivered a sticky note message into the operating room to Dr. Potter. The note said the first name of the UNH representative, included a phone number and the words 'United Healthcare Pt. JL for Dr. Potter.'
While UNH denies that its representative insisted on speaking to Dr. Potter right away, the facts on the ground suggest otherwise:
First, the UNH representative called the operating room front desk at the hospital, rather than Dr. Potter's office and/or staff or billing department.
Second, the nurse on duty believed it was sufficiently urgent that she gave the message to the head nurse on duty. The head nurse in turn also thought it sufficiently urgent that she delivered the message into the operating room.
All of the above actions are consistent with Dr. Potter's statements in her video. According to Dr. Potter, the head nurse said that in her 15 years of experience she never had an insurance company seek to speak with a surgeon in the operating room so she assumed it had to be urgent.
Dr. Potter continues in the video:
“…so I scrubbed out of my case and I called UnitedHealthcare, and the gentleman said he needed some information about her, wanted to know her diagnosis, and whether um whether uh her inpatient stay should be justified. And I was like do you understand that she’s asleep right now and she has breast cancer?” [Dr. Potter of course did not leave the patient alone during the two-minute call. There was another surgeon, nurse, etc. in the operating room.]
I believe what Dr. Potter is saying is true. But before we go further, why did Dr. Potter 'scrub out of her case' and call UNH?
The answer is that Dr. Potter is an advocate for her breast cancer patients, not just for their health, but also for their financial well being. As we all know, many families have been financially wiped out by their healthcare bills that are not covered by insurance. It's bad enough to have breast cancer and have a double mastectomy, but imagine then being wiped out financially after the surgery.
[For context, challenges to insurance coverage for modern breast reconstruction have been increasing. In 2021, CMS (Medicare) announced a coding change that threatened access to modern breast reconstruction techniques. United Healthcare was the first to adopt the change in April 2022. Recognizing the danger to patients and the practice of breast reconstruction through insurance, Dr. Potter started a national effort to reverse the change. She used her own savings to fund this effort. The change was reversed by CMS in August of 2023. I have a lot of respect for activists generally and for Dr. Potter's work on behalf of patients.]
Receiving a note to call an insurer mid surgery was a first for Dr. Potter, and she stepped out to call UNH because she was afraid for her patient that UNH was going to deny coverage. She had to believe the call was urgent, otherwise there is no credible reason for her to have scrubbed out and called back the UNH representative on her cell phone.
When you read the transcript of Dr. Potter's video remarks or even better when you watch the video, you can hear the emotion and exasperation in her voice, which is of someone frustrated with big insurers and very concerned about her patients. I have also found Dr. Potter to be extremely credible in all of my communications with her.
Dr. Potter continues:
“And um the gentleman said actually I don’t that’s a different department that would know that information. And I was like well um she does need to stay overnight tonight and um you have all the information with you because I got approval for this surgery, and I need to go back and be with my patient now.”
Again, here I believe Dr. Potter when I examine all of the facts and documents that were made available by both parties. UNH was apparently calling to create a record that it had discussed the case with Dr. Potter and to make the case that her patient should not have an inpatient overnight stay in the hospital. I am not an expert in the insurance law here, but this is my understanding.
Dr. Potter required the patient to stay overnight because her patient had a lung infection on the morning of the surgery, i.e., histoplasmosis that required a strong anti-viral medication. [Dr. Potter's patient has permitted Dr. Potter to share her medical information.] When the infection was considered along with the surgery, Dr. Potter believed an inpatient overnight stay was required because of concerns she had with potential interactions between the antiviral and post-surgical medications, as well as the stress to the patient from the surgery. That was her judgment as the patient's surgeon, and that is why she placed an order for the overnight stay with UNH before she did the surgery. [If I get any of these details wrong, I am sure Dr. Potter will correct the record.]
In the demand letter, UNH accuses Dr. Potter of making an error in ordering an inpatient stay. Dr. Potter disputes this vociferously and as simply gaslighting by UNH.
Why was UNH trying to speak to Dr. Potter so urgently? The difference between a one-day, in-patient stay, and the patient being released the same day from surgery was a bill to the insurer of more than $100,000, in this patient's case $110,356, coverage that was denied by UNH.
[As a side note, the amount of this inpatient overnight stay is absurd and speaks to the fundamental problems with the system. The $100k plus charge is typically if not always dramatically negotiated down by the insurer, but when the insurer does not pay, the individual can get stuck with the face amount of the bill and without the negotiating leverage of a large insurer. These absurd large invoice amounts remind me of what it is like buying a prescription when you don't have your insurance card and CVS tells you the $25 drug will cost $3,000. This system is broken and fundamentally corrupt, and hopefully @RobertKennedyJr and the @realDonaldTrump and @DOGE will do something about it.]
Under Texas law (the surgery took place in Austin, Texas) according to the Clare Locke letter, an insurer apparently has one day to discuss the plan of treatment with the physician before issuing a denial. Therefore, apparently, if UNH didn't reach the doctor before the end of the day, it would not have had as credible an argument to deny coverage. UNH did deny coverage in writing later that same day of the surgery, before the patient even left the hospital. The above explains why I believe UNH's representative was urgently trying to reach Dr. Potter.
Dr. Potter finishes the video by saying:
“But um yeah, it’s out of control. Insurance is out of control. Uh I have no other words.”
The above is a statement of opinion, and based on Dr. Potter's experience here it is entirely accurate.
Now, let's examine UNH's statement in response to my initial post, which among other things, said that:
“I would not be surprised to find that the company’s profitability is massively overstated due to its denial of medically necessary procedures and patient care.”
UNH's statement:
"Health insurance has long been subject to significant regulatory oversight and earnings caps. Any claims that health insurers, which typically have low- to mid-single digit margins, can somehow over-earn are grossly uninformed about the structure and strong regulatory oversight of the sector."
The statement begins by saying that health insurance is subject to 'strong regulatory oversight' and 'earnings caps.' This statement is meant to give the reader the impression that I must be wrong because regulators are watching the insurers closely, and that earnings are somehow 'capped.'
UNH states that I must be 'grossly uninformed' for how can UNH's earnings be overstated if health insurers have low- to mid-single digit margins?
While the above statements from UNH are true, they are highly misleading.
First, the fact that UNH is subject to strong regulatory oversight does not mean that the company is properly adjudicating claims. As we all know, regulators often fail to do their jobs. In fact, I have personal experience with regulators failing to do their jobs (See MBIA and Herbalife) because regulators can be intimidated by powerful companies and the big law firms that represent them. That is why regulators often shy from going after big targets, and it is only after the problem companies collapse that the regulators step in and punish the people responsible. I can't think of an example where a regulator found fraud at a large company before it collapsed. It is usually the short sellers who find fraud, and the regulators who come in afterwards to clean up the mess.
MBIA collapsed six years after we brought our concerns to the company's insurance regulator and the SEC. Herbalife stock collapsed years after the FTC failed to shut the company down. The facts about MBIA and Herbalife were manifestly true when we shared them with the regulators, but still the regulators did not do their jobs.
Second, the fact that insurers have low- to mid-single margins is not evidence that they are properly adjudicating claims. Rather, the fact that UNH has low profit margins gives it a huge incentive to minimize the claims that it pays.
When a company has low margins, it by definition has high operating leverage. This means that small changes in revenues up or down have a huge impact on bottom line profits. Public company management teams are compensated based on meeting and exceeding profit targets which drive earnings-per-share growth and long-term stock price increases. If management can drive revenues up slightly in a low margin business, profits can explode upwards because of operating leverage. So the fact that an insurer has low margins does not in any way prove or support the fact that its earnings are not overstated, but it clearly creates an incentive to minimize claims paid by an insurer.
When you step back and look at this situation, it gives you better perspective on what likely transpired.
A surgeon posted a video about her frustration with a healthcare insurer. When she posted it, she did not know it would go viral. When it did go viral, the company responded by having its defamation counsel send a threatening letter accusing the doctor of making "knowingly false, misleading, and defamatory" social media posts, and demanded that she take down the posts, retract her claims, and post a public apology.
[UNH did so in my view for two principal reasons: (1) because it wants to minimize negative press and the risk of regulatory inquiries into its business, and (2) it wants to minimize negative press to reduce the risk to its executives in light of recent events, an important and legitimate concern.]
In response to a threatening letter from UNH's defamation counsel, the doctor, rather than taking down her posts, makes more posts, and then sits down for an upcoming interview on a major TV show. Why would she double down and expose herself to more legal and career risk unless what she said was true?
When a market observer, in this case me, reposts the doctor's video and criticizes the company, the company responds by issuing misleading statements to the public, and contacts the SEC, our principal regulator, in an attempt to intimidate me even though I have publicly stated that we have no investment in UNH long, short or otherwise.
When you look at the above facts and watch Dr. Potter's videos, I strongly believe that a jury of Dr. Potter's peers would conclude that she is telling the truth.
What is her incentive to make "knowingly false, misleading and defamatory" statements about UNH? She has none. In fact, she has the opposite.
She is a breast cancer surgeon with a small, not particularly profitable practice, going up against a publicly trading insurance holding company with a $482 billion market cap, the 16th most valuable U.S. company. She has no incentive to lie and double down and go on network television unless she is telling the truth.
Dr. Potter put herself at significant personal and financial risk by going public about her experience with UNH because of her passion for protecting her patients and her frustration with our healthcare system and its insurers. There is no other credible explanation for her video and other social media posts.
Now what about UNH?
I suspect that the employees and other representatives of UNH that help manage its insurer's claim expenses are given large financial incentives to keep claims payments as low as possible. That would explain the tenacity with which the UNH representative operated when he called the operating room front desk, and the urgency with which he expressed a desire to speak to Dr. Potter. That, in my view, is the only credible explanation for why the front desk nurse gave the message to the head nurse who brought the message into the operating room, and explains what has transpired here. Occam's razor. And according to Dr. Potter, all of the nurses and other witnesses involved have offered to testify on her behalf.
With respect to my thoughts on shorting UNH from my first post, I don't recommend shorting stocks, but I wouldn't recommend anyone invest in UNH, certainly at this valuation. Since my post, I have heard many other bad stories about the company's approach to paying claims so I don't think Dr. Potter's experience here is a one off. Based on all of the above, in my opinion, there is likely something systematically wrong with this company.
Compare UNH with the other top 20 U.S. companies by market cap. When you do so and you consider each of these companies contributions to humanity, does it cause you to question a bit why UNH is so valuable compared to the others? Yet, another reason, I would argue, why one might question the company's reported profitability and valuation. And UNH's earnings don't appear to in any way be 'capped.' Certainly, the company's shareholders and analysts are not valuing the stock assuming 'capped earnings' for otherwise you could not justify a half of a trillion dollar market cap.
With respect to Dr. Potter, I think she is a hero. I have offered to pay her legal expenses, but her lawyer was already handling her case pro bono, such was his confidence in her case and her character. If she needs funding to bring her own defamation case, she knows where to find me.
UNH owes Dr. Potter a public apology for defaming her and accusing of her lying.
And if I were on the UNH board, I would launch an immediate investigation of the company's approach to paying claims, the incentives it gives the employees and agents who work on its behalf, and the approach it takes in attacking the critics who challenge it. I am sure that Dr. Potter is not the first person to receive a threatening letter from UNH. I look forward to hearing from others on X about their experiences with the company good and bad.
In summary, the whole thing smells very bad to me. And yes, the SEC should take a very close look at UNH.
I think we can all agree on a few things:
We have $36 trillion of national debt growing at a rapid rate, that must be refinanced and issued at a materially higher interest rate as our bonds come due and as new funds are needed.
We had a $1.8 trillion deficit in fiscal 2024.
There is an enormous amount of fraud, waste, and inappropriate spending in our government budget.
It is illegal and/or otherwise should not be permitted for our government to fund activities that it cannot do directly through NGOs and other non-profits.
@elonmusk has a proven ability to takeover dysfunctional organizations and turn them into highly functioning ones while massively reducing wasteful overhead and other spending. See Twitter.
@realDonaldTrump won the election with a mandate for eliminating wasteful government spending and ending regulations that interfere with our country’s progress.
President Trump announced during his campaign that Elon would be appointed to oversee @DOGE. In other words, the American people voted for and signed up for DOGE and Elon as its leader.
We are extremely fortunate to have one of our greatest business leaders and entrepreneurs devote a considerable amount of his limited time for free to help our country.
We are extremely fortunate that some of our most talented young people have signed on to help DOGE succeed.
DOGE is just getting started and has already identified a massive amount of waste and likely fraud, and has begun cutting a material amount of costs.
It is axiomatic that those who are beneficiaries of waste, fraud and unnecessary government spending will be the most threatened by the cuts that DOGE is making in these programs. These beneficiaries of waste and fraud are also extremely worried about the reputational, legal and potential criminal risk they will suffer by being exposed by DOGE.
The beneficiaries of government waste, fraud and inappropriate spending will protest loudly and state false reasons for why DOGE must be stopped because their careers and livelihoods are threatened and put at risk by DOGE’s success. I have seen this movie many times before, but in the corporate world.
The president has stated clearly that he is overseeing DOGE’s activities and that no important decisions will be made without his consent.
Our legal system will ensure that no laws are broken as DOGE executes on its important mission.
Other efforts over the last many decades to eliminate government waste and fraud have totally failed.
DOGE is our best hope for fixing our fundamentally flawed and highly ineffective bureaucratic system for running our country.
Elon and Trump will face enormous pressure and pushback from those who benefit from the current corrupt system.
All Americans must therefore make their voices heard loud and clear about how much they support DOGE, Elon and our president’s efforts to help our country. We cannot let DOGE fail as our country is rapidly on the path to insolvency.
Let’s not let our country down. Let’s do our part in making sure our Congressmen and Senators know that we are 100% behind the president, Elon and DOGE.
It’s the least we can do.
One of the healthcare industry’s biggest problems is our ability to “call our baby ugly.”
Make no mistake—it’s an ugly baby.
Maybe the ugliest.
But when we fail to see things clearly, when we normalize the abnormal, we rob ourselves of opportunities to improve.
We hold ourselves to lower standards.
We start enabling and supporting industry practices that we ourselves would never want for ourselves.
So in 2025, I urge you to start seeing things clearly—and help others see them clearly yourself.
To quit gaslighting yourself and others.
To stop celebrating things that are fundamentally anti-patient.
To quit representing incremental as innovative.
And from whatever corner of the industry you work—start working towards making things *truly* better.
But it starts with:
1) Calling things what they are
2) Being more authentic; and
3) Leading from first principles.
Wishing you an impactful, honest, and authentic 2025.
@elonmusk Doesn’t this enable groupthink? Wouldn’t Jay Bhattacharya have been substantially muzzled by such a policy? Do you backtest this objective function to assess the trade offs ?