ICU Physician @uclh, ICU Family Liaison Team Lead, co-inventor of SEND and the EMAP real-time data platform, digital health researcher, human factors enthusiast
I started doing office hours on LLM evals and met with 8+ founders in the last 3 weeks. Common questions:
- Which components of our app do we start evaluating (RAG,tool calls, etc)?
- What metrics should I use?
- Where should I spend my time?
All have the same solution. LOOK AT THE DATA. What does this mean though?
It means look at your logs/traces - start with 30 or so. Start categorizing the errors and issues you see. Keep looking at logs and traces until you feel like you aren't learning anything new.
In the end, you will know where your biggest issues are. You prioritize those! You will also get a sense of what is most important to measure (and how).
That's it. Look at data, build evals and tests prioritized by patterns in the data. If you don't have data, generate synthetic inputs/interactions into your LLM application so you can generate data.
I didn't make this technique up fwiw. These are fundamentals of building machine learning systems and is often referred to as "Error Analysis". It is a fancy word for looking at data, categorizing errors, and then doing data analysis on those errors to understand what to prioritize and work on.
I've documented some of the office hours, and you can see in all cases the solution was performing error analysis. Here are links to those:
https://t.co/Z5CSGznzgP
One does not "become" agile. (We never should have chosen that word!) Working in an "agile" way is nothing more, and nothing less, than adopting a set of disciplines that reinforce a set of simple values.
The values come first. If you don't value people, working software, collaboration, and change (Communication, Feedback, Respect, and Courage) then you won't adopt the disciplines that reinforce those values.
An agile transition is all about installing those values into an organization. Organizations, and people, do not like to have their values changed. They will fight it. Some overtly. Some by subterfuge.
My experience is that most truly "agile" organizations are those who are composed of like-minded people -- people who were hired for their values.
Couldn't agree more! The impact that John has had on patient safety is not known by many - I'm so proud that he is recognised for his contribution 👏🏻👏🏻👏🏻👏🏻👏🏻👏🏻
This illusion – one of the best of the last few years – will really mess with your head. How many black dots are there? The answer is 12, but your visual system won’t let you see them all at once. https://t.co/XA6K6XLFlr
Important work from @CarolineJC5 and the team at Inclusion Barnet to change the fundamentals around how we think and talk about disability. Language matters so much in shaping our outlook and expectations.
Families of ICU patients encounter various challenges, revealing the complex interplay of emotions, needs and challenges within the ICU. This study highlights the intricate dynamics in this critical health care environment.
@BACCNUK
https://t.co/hkjiLbmfW0
Fourteen years later, my #ICUDiary continues to serve as a grounding mechanism when PTSD and depression tighten their grip.
Transitioning to reality: the diary of an ARDS survivor: https://t.co/NL00jmvZMh
#PICS#Walk4PICS#ARDS#ECMO
Intensivists - interested in improving your skills in donor optimisation? Please join us in London on 2/3 Oct for the 1st Donor Optimisation and Management in Intensive Care (DOMIC) course.
Happy World Organ Donation Day 👍🏾
(please do share…)
In my experience, people don't resist change because they have "resistant personalities" or because of "human nature". They mostly resist because of the way we go about introducing or implementing the change or imposing it on them. @Digitaltonto suggests five major categories of change resistance, most of which can be less of an issue if we think about them beforehand:
1) Lack Of Trust: by focussing on what's new & different, there's a danger that leaders communicate changing values. Need to focus on shared values & creating a sense of safety around the change conversation.
2) Change Fatigue: the change needs to be seen in the context of all the other initiatives that are happening.
3) Competing Incentives and Commitments: we shouldn't have to "incentivise" people to take part in change. If the change has real value, we should be able to find people who are enthusiastic about it & want to make it work.
4) Switching Costs: what we experience in the here & now always seems more tangible/concrete than the more distant benefits of change, which many will suspect will never come.
5) Identity, Dignity & Sense of Self: people's sense of identity can act as one of the strongest barriers against change. We resist things that challenge our perceptions of ourselves, including loss of status. Helping people navigate the fear of losing their identity & see new possibilities emerge is crucial for successful change outcomes.
https://t.co/9MdXSJwqkR. Graphic adapted from @FLIMPStudios.
Our thoughts and very best hopes are with our friend and former colleague Tom Doherty, his wife Anne and all family since his fall and disappearance in the Pyrenees. We desperately hope he is found soon. Our great thanks to all supporting the search.