Checking the source is not enough: I traced a number to a scientific paper and found a Teams meeting
Jamie Watters
Operational resilience and AI delivery practitioner.
Who this is for. Anyone who checks sources for a living or a habit, and anyone buying a tool on the strength of a published number. By the end you'll have three questions that catch a figure which passes the usual check and is still wrong.
Skip it if you already trace every number to the document that first measured it, and check what that document counted. That is the whole finding.
On the length. Eleven minutes. The three questions are the deliverable and come after the case that produces them.
You have been told, correctly, that the answer to being misled is to check the source.
Do not trust the headline, trust the study. Do not trust the summary an AI gives you, click through to what it is summarising. It is good advice. I have given it myself, in this blog, more than once.
Here is a case where a careful person does exactly that, opens the peer-reviewed paper, reads it properly, and still walks away believing a figure whose only source is a private meeting.
I spent a long time being the person who checks. At a bank, my job for years was deciding whether a company's claims about its own systems were good enough to trust, and then defending that decision when someone senior asked me why. It is the thing I was paid to get right, and I would have been caught by this.
The number is a better traveller than the thing that qualifies it. That has nothing to do with anyone's honesty.
The thing in the hospital
Epic is the company whose software holds the medical records of a very large number of American patients. Inside it sat an alarm that was supposed to predict sepsis, which is what happens when an infection tips into organ failure, and which kills people quickly when it is missed.
Epic said the alarm scored between 0.76 and 0.83. That score runs from 0.5, which is a coin toss, to 1.0, which is perfect. So 0.76 to 0.83 sounds like a tool that mostly works.
In 2021 a team at Michigan Medicine ran it against their own patients, nearly 28,000 of them, and measured 0.63 (Wong et al., 2021). Closer to the coin toss than to the claim.
Where did Epic's higher number come from? The researchers say so plainly in the paper. It came from internal documentation, shared with permission, plus a conference presentation Epic had co-written. Nowhere you could look it up. Nowhere anyone outside could check it.
Now the part a careful reader should stop at, because a specialist will raise it first. That 0.63 is the score across a whole hospital stay. The same team also measured the alarm over shorter windows, and four hours before sepsis set in it reached 0.76, which is the bottom of Epic's range (Wong et al., 2021).
So is the gap real or not?
You cannot tell, and that is the whole thing. To line those two numbers up you would need to know what Epic had measured its own score against: which patients, what counted as a case, over what window. The paper says that information is not public, because the model is proprietary. So the honest position is not that the company's number was wrong. It is that nobody outside the company was ever in a position to know what it was a number about.
Only one of those two numbers had travelled. By the time the Michigan team measured it, the alarm was running in what the same paper calls hundreds of hospitals.
What it was actually doing, at the setting hospitals were told to use: it caught a third of sepsis cases and missed two thirds. It raised an alarm on nearly a fifth of everyone admitted. And of the roughly 1,700 people with sepsis it missed, about 1,030 got their antibiotics in time anyway, because the doctors found them without it (Wong et al., 2021).
None of that was visible from 0.76 to 0.83. And 0.76 to 0.83 is what the people deciding had in front of them.
It kept running for two and a half more years
The correction was published in June 2021, in a major medical journal, and widely covered.
A separate study looked at two safety-net hospitals in Harris County, Texas, across nearly 146,000 patient encounters covering the whole of 2023. The original version was still running there, catching around 15% of cases at a six-hour window (Ostermayer et al., 2024).
Two and a half years after the number was corrected. In the hospitals serving the people with the fewest other options.
The gap between a number being corrected in public and the correction reaching the ward.
Then the part that made me sit up
Epic rebuilt the thing. In February 2026 the same lead researcher published a proper check of the new version: four hospital systems, 227,091 patient stays, no Epic authors, no Epic money. It scored much better, and the researchers attached their own warnings to that: the results varied a lot between hospitals, most of the alarms it raised were still false, and there were a lot of them (Wong et al., 2026).
This is the paper you would send someone as the happy ending. Independent, large, peer-reviewed, in a serious journal. If you were being careful, this is exactly where checking the source would take you.
So read how it describes the company's own figures for the rebuilt version. It says Epic has reported scores between 0.83 and 0.86 across three internal sites. And the citation attached to that sentence is not a paper. It is a Microsoft Teams meeting with two named Epic employees, held on 25 February 2025 (Wong et al., 2026).
A meeting. In the paper that exists because the last set of company figures could not be looked up.
Nothing about that is improper. The authors disclosed it exactly as the rules require, which is the only reason I know about it. That is what makes it worth your time. The disclosure worked perfectly and the problem walked straight through it.
So the careful reader, the one who did what we all tell each other to do, reads a peer-reviewed paper and receives the company's claim at full strength in the body text, with where it came from sitting in a bracket. And the bracket looks like every other bracket on the page. You would have to already know that a citation naming a meeting is unusual to feel anything when you passed it.
What is actually happening
A number travels well. Whatever qualified it travels badly.
So at every step between the measurement and the decision they drift further apart, and the gap never closes on its own.
This is not about science, or healthcare, or even AI, although AI has made it much worse. When you ask a model a question, you get the number. You almost never get the conditions the number was measured under, because the conditions did not make it into the training data intact either, and a confident sentence is what the thing is built to produce.
Three questions, and one habit
The questions take under a minute on any number that matters to you.
- Where can I look it up? A paper, a filing, a report with a page number. If the honest answer is "their own testing", "a briefing", "they told me", then it is a claim, not a measurement. Both are allowed. They are not the same thing and should not feel the same.
- Who measured it, and who paid? Independent in who did the work and who funded it. And check for the softer version: people taking no money from a company can still have taken that company's advice on how to run the test.
- Measured on what, and when? A number with no population, no time period and no conditions attached cannot be compared to anything, including an earlier version of itself.
Then the habit, which is the part that actually changes anything. When you repeat a number, repeat where it came from in the same breath. Not in a footnote. In the sentence, out loud, in the meeting.
It sounds like this, and none of these takes longer to say than the bare number did:
- Not "it scores 0.86", but "it scores 0.86 on the company's own testing, which isn't published".
- Not "the study says nine in ten", but "one study says nine in ten, and it was paid for by the people selling the thing".
- Not "this market will be worth billions", but "an analyst firm forecast billions, three years ago".
That is the whole technique. It does not read as pedantry. It reads as someone telling you how much weight to put on what they have just said, which is a thing people are grateful for rather than annoyed by.
And when you cannot complete the sentence, you have learned something more useful than the number.
Is this just telling me to be sceptical?
Fair challenge, and no. General scepticism is useless, because you cannot doubt everything and you will end up doubting whatever you already disliked.
This is narrower and it is mechanical. You are not asking whether a number is true. You are asking one checkable thing about it, which is whether anyone outside the organisation that benefits from it has ever been in a position to test it. That question has an answer, you can usually get it in under a minute, and it does not require you to know anything about sepsis, or software, or statistics.
The uncomfortable part of this story is not that a company's number was optimistic. It is that five years after that was pointed out in public, the correction still arrives carrying a figure from a meeting nobody can read.
Sources
- Wong, A., Otles, E., Donnelly, J.P. et al. (2021) 'External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients', JAMA Internal Medicine, 181(8), pp. 1065-1070. The 27,697 patients and 38,455 hospitalisations, the measured 0.63 against Epic's claimed 0.76 to 0.83 "in internal documentation (shared with permission)" and a conference proceeding co-authored with Epic, the 33% caught and 67% missed, the alerts on 18% of hospitalisations, and "hundreds of hospitals". The missed-but-treated figure is the paper's own sentence: "The ESM did not identify 1709 patients with sepsis (67%), of whom 1030 (60%) still received timely antibiotics." The shorter-window scores are from its Table 2: 0.72 at 24 hours, 0.73 at 12, 0.74 at 8 and 0.76 at 4. On why the two numbers cannot be lined up, the paper says that "owing to the proprietary nature of the ESM, only limited information is publicly available about the model's performance". https://doi.org/10.1001/jamainternmed.2021.2626
- Wong, A., Currey, D., Schwinne, M. et al. (2026) JAMA Network Open, 9(2), e260181, 27 February 2026. Four health systems, 227,091 inpatient encounters, no Epic affiliation or funding declared, and the sentence citing Epic's internal figures of 0.83 to 0.86 to "Tyler Sundberg and Elliot First, Epic Systems, Microsoft Teams meeting, February 25, 2025". https://doi.org/10.1001/jamanetworkopen.2026.0181
- Ostermayer, D. et al. (2024) JAMIA Open, 7(4), ooae133. 145,885 encounters across two Harris County, Texas safety-net hospitals during calendar year 2023, sensitivity 14.7% at a six-hour window.
- Ross, C., STAT, 3 October 2022, reporting from Epic corporate documents on the rebuilt version.
- The three questions and the habit are mine. They are not a protocol I have been running for years under those names. They are how I would put, in four lines, what I was always trying to establish about a number when it was my job to ask.
- Companion piece: four questions to ask any AI statistic before you repeat it, which sorted nineteen widely quoted AI failure figures by what each one counts. That one is about numbers that measure the wrong thing. This one is about a number that measures the right thing and arrives without its conditions.