Testing vs Bayes Theorem

I saw an interview with this correspondent and looked up the print version...to present her story as food for thought. She had many classic symptoms for 5 weeks yet tested negative 4 times with various tests.....


"Why do these doctors feel they can override my negative COVID tests? The answer lies in something called Bayes’ Theorem and its idea of pretest probability."


www.gq.com/story/julia-ioffe-false-negative-covid-testing

Doctors Tell Me I Have COVID. Why Won't the Tests?

She's been sick for over a month with what doctors long-ago diagnosed as COVID. So why has Julia Ioffe repeatedly tested negative—and what does her search for answers tell us about the faith we invest in a testing system that's trickier than we think?

By Julia IoffeSeptember 28, 2020
Prayer hands with cotton swab and covid 19 tag
Photo Illustration by C.J. Robinson
I first started feeling sick on Sunday night, August 16. I had spent the previous week dealing with excruciating neck pain whose origins I couldn’t quite explain. My mother, a doctor who takes great pride in her formidable diagnostic skills, was sure that this was the opening salvo of a COVID infection. I waved it off, insisting that I had probably just gone a little too hard on the ab workout. But by Sunday night, it was clear something was wrong. I barely made it through grocery shopping and had to lie down several times while making dinner. By morning, I had chills, a runny nose, and a scratchy throat. Something told me my mother was right. I called my primary-care physician’s office in Washington, D.C., and went downtown to their testing site. A quick swab of the nose yielded a negative result two days later. I was elated. I wasn’t feeling awful, just a little under the weather, and now the test confirmed what I wanted to hear: I didn’t have COVID-19. But my mother and sister, a doctor who took care of COVID patients at the peak of the pandemic, were adamant: I was to stay home and quarantine for two weeks.

Gradually, the symptoms came back—though they never went away, really. They came in waves. I’d spend most of the day feeling fine, only to have my temperature spike as I swaddled myself in layers of clothing and blankets to fight the chill. Within an hour or two, I’d feel completely fine again. My senses of taste and smell started fading in and out. I could no longer smell my cat’s litter box and ice cream just tasted cold. Then my sense of taste would come back, but some things, like sweets, tasted intolerably intense. A brain fog pulled in, and I found myself constantly searching for simple words. It was the first time I had felt this way while speaking English, rather than a foreign language.

I was becoming increasingly convinced that I had COVID, so I tried to remember everyone I’d seen in the two weeks before I’d gotten sick in order to warn them that, by coming in contact with me, they might have been exposed. I told them by text that, even though I tested negative for COVID, I suspected I was infected and encouraged them to get tested. My friends’ responses surprised me: The vast majority said they felt fine and that they did not feel the need to get tested because my own result had been negative. In their minds, they could not have been exposed if I had tested negative—never mind the symptoms I was showing. There wasn’t much I could do to convince them, so I decided to get a repeat test. This time, I went to one of the walk-up testing sites the government of Washington, D.C., had set up all over the city. I filled out the questionnaire: “Have you experienced any of the following symptoms?” It listed the symptoms of COVID—cough, sore throat, body aches, chills, loss of taste, loss of smell, difficulty breathing. I checked nearly every box. A quick nose swab and I was home, where, 48 hours later, I received another negative result.

In the meantime, the ebb and flow intensified. Every time I thought I was feeling better, a new wave of illness covered me. I’d have a good day, only to sleep 16 hours the next and feel utterly unable to move unless I also took a three-hour nap. Shit, I thought, convinced I was getting sicker, only to feel fine the day after that.

About halfway through my third week of being sick, a tightness crept into my chest. It felt like someone had grabbed a hold of my trachea and was steadily strengthening their grip. I went from chills to sweats and back. My mother and sister were convinced it was COVID, but how could that be if I had twice tested negative? Friends checking in on me were growing increasingly certain: Two negative tests meant I did not have COVID. “Girl,” one friend texted, “stop telling people you have COVID when you just have a cold!”

But it didn’t feel like a cold. In fact, it didn’t feel like anything I’d ever had before. No cold or even flu had ever been this wobbly. Not even mono or whooping cough had been this much of a roller coaster. All of them had been straightforward, linear illnesses, and none of them made me question my sense of reality. I felt like I was bobbing around in a kind of viral purgatory, with no end in sight. Still, two negative COVID tests felt pretty definitive.

It made me wonder: Was it possible to have COVID-19 and still test negative for SARS-CoV-2, the virus that causes it—twice? And if false negatives were possible, how frequent were they? I had read stories about false negatives in the spring, and I’d heard anecdotes from friends working in New York hospitals who didn’t fully trust the tests after having to place critically ill patients on ventilators, even though the patients had repeatedly tested negative for the novel coronavirus. A friend’s elderly grandmother had tested negative four times, but was still treated on the COVID floor of her Florida hospital. And I remembered my mother, who had been sick for a month and tested negative for COVID twice, despite sharing a home with my father, who had been sick and tested positive for the coronavirus. But my impression was that the tests had gotten much better since then.

Moreover, compared to the early days of the pandemic, the tests had come to mean something different to many people. They’d become the key to getting life back to something approaching normal in the absence of a vaccine. A new kind of faith had been placed in them. All summer, people I knew were taking tests in order to join friends in beach houses or to visit family if the result came back negative. But now I wondered, if it were possible for someone with the virus to test negative for it—and given the politicization of the CDC’s testing guidelines—had we grown too confident in the tests and in our ability to find and isolate infected people?

I started calling the directors of prominent labs, interviewing doctors who had cared for coronavirus patients and asking them: How likely was a false negative? And did I have COVID? In the meantime, I was feeling worse and worse. The fatigue and brain fog were overwhelming, the chest tightness wasn’t letting up, and a dry, barking cough was making it impossible to sleep. Against all medical advice, I went to get one more test—my third. I drove to a testing site in D.C. and, because I suspected the nasal swabs might have been missing something, asked the man administering the test if he could go back further than the nostril, to do the brain tickle I had read about. Oh, he told me, we don’t do those here. We only do nasal swabs. He showed me the short medical Q-tip before gently caressing the inside of each nostril.

“You obviously have COVID,” my doctor said, brushing aside my now three negative tests. She told me to go to the hospital immediately.

On the Saturday morning of Labor Day weekend, after three weeks of being sick, I got an email informing me that I had again tested negative. But by evening, it was becoming harder and harder to breathe. I had been measuring my blood oxygen levels with an oximeter, a gadget that clamps onto your index finger. Given how quickly some patients can deteriorate without even realizing that their oxygen levels are falling, doctors have said oximeters are helpful for COVID patients to have at home. For the three weeks I had been sick, my oxygen levels remained safe and steady. That weekend, as it became harder to breathe, the levels began dropping. By Sunday morning, my oxygen level was at 89—alarmingly low. I called my PCP. “You obviously have COVID,” she said, brushing aside my now three negative tests. She told me to go to the hospital immediately.

How did this happen? How did I get so sick, and how did the tests miss it? And was it even possible that I had COVID after three negative tests?

The answer, I realized, was extremely complicated.

“Just because you’ve tested negative twice doesn’t mean that you don’t have it,” Geoffrey Baird, M.D., Ph.D., told me after my second negative test. Baird runs the lab at the University of Washington, which dealt with the first COVID hotspot in the U.S., and I had written about his team’s herculean efforts to develop their own test for the novel coronavirus. He was the first person I called in my quest to understand what was going on with me and if I was an outlier. “Your story is unusual,” Baird said, “but it is not unique.”

During the peak of the pandemic in the spring, Baird told me it was not uncommon for doctors treating patients in the University of Washington hospital to call down to his lab, confused by the COVID test results they had just received. In front of them would be a patient with a fever and severe respiratory symptoms, but the lab results would say the patient was negative for the coronavirus. The two pieces of evidence—what the doctor saw and what the test said—seemed to contradict each other. Resolving this contradiction—and doing so quickly—was not only a matter of life and death, but of containing the pandemic. If a patient presenting in the emergency room with COVID-like symptoms tested negative for SARS-CoV-2 and was not, in fact, infected with the virus—what’s known as a true negative—then a doctor’s course of action would be very different than for someone who really did have COVID-19. They might even send the patient home to recover without telling them to quarantine. But what if it was a false negative? That is, what if the patient actually did have SARS-CoV-2 in their system, but, for whatever reason, it was not showing up on the test? If the doctors relied only on the test and that patient were sent home or put in a regular wing of the hospital, they might end up infecting others with this new, highly contagious, and deadly virus.

Part of the issue concerns the tests themselves. At the beginning of the pandemic, the United States government was using a testing kit—put out by the CDC—that turned out to be faulty. Infections in the U.S. were skyrocketing, and the FDA eventually allowed hospitals, university labs, and private manufacturers to create their own tests. The ad hoc approach helped fill America’s testing gap, but as a result, there was no gold standard for the coronavirus test. There still isn’t. Each of the test manufacturers who applied to the FDA for emergency permission to create a test—and there are 165 of them—had to submit their own data for how well those tests worked.

Some tests were very accurate, but others weren’t. One review of available tests found that the likelihood of their yielding a false negative could be anywhere from 2% to 29%. Another found that up to 30% of tests could spit out a false negative. Other physicians gave me a range from 10 to 20%. One faster test, the Abbott ID NOW, which President Trump promoted in a Rose Garden press conference, was found to be only 51% sensitive. That is, this one test was just as likely to give you a false negative as it was to give you an accurate result: not much better than a coin toss. (Public debate has now shifted to the more rapid antigen tests, which use a different technique than the PCR tests that are in use now. But they, too, have a vast range of accuracy, and are far less sensitive than PCR tests. The chances of getting a false negative is, on average, around 30%, though on some of the tests, it can be as high as 50 or even 70%.)

And though PCR tests have become far more sensitive—that is, less likely to yield a false negative—they are still not 100% accurate. They are also far more likely to produce a false negative than a false positive. “There’s no perfect test,” Baird told me. “The ones we have now are 80 to 90% sensitive. That’s really good, but that means there is still the one-in-five chance of a false negative.”

In fact, when I went back to look at my test results, the disclaimers were there, in black and white. I had just missed them. Right under my first negative test result, for example, was the following warning: “When diagnostic testing is negative, the possibility of a false negative result should be considered in the context of a patient's recent exposures and the presence of clinical signs and symptoms consistent with COVID-19.” That is, if you tested negative for the coronavirus but still have symptoms of COVID, you might have gotten a false negative. You might still have COVID.

And that’s just the probability for error built into the test itself. There is a whole other range of factors that all have the potential to chip away at the test’s accuracy.

The standard COVID test is done through the nasopharyngeal swab, which samples the back of your nasal passage and throat. But this sampling technique, though highly sensitive, is not perfect either. “The virus enters through the nose, but it does a lot of its business in the lungs,” Baird explains. “The further away you get from your lungs, the less virus there is.” Baird, as well as clinicians I spoke to from New York, San Francisco, and Atlanta, told me that they had seen several instances of patients who had tested negative on the swab, but whose lung fluid was found to be teeming with the coronavirus. Of course, one would have to be pretty sick—hospitalized and intubated—to have their lung fluid tested. If you only have mild symptoms, as I did at the beginning, no one is going to go burrowing in your lungs.

Conversely, the test might also miss a COVID infection, according to studies, if you’re swabbed too early after being exposed.

One missed infection—and the false sense of confidence created by a false negative—can create a chain of transmission that snares dozens of people.

Moreover, if there’s something specific in the way you are built—about the physical structure of your nose, for example—or if, for whatever reason, when the virus hit you, it went straight to your lungs, repeating a COVID test wouldn’t give you a more accurate result. Here, the laws of probability run into your specific physiology. “If there’s something about your nose or your disease that makes it harder for the test to catch,” explains Baird, “it’s more likely to repeat the false negative because whatever it is about your nose or your disease is also true the second time you get tested.”

Then there is the question of how good of a sample the swab gets. Nasopharyngeal swabs—those long Q-tips that go so far back into your nose that they feel like they’re going to come out the back of your head—are more sensitive than simple nasal swabs that only make contact with the insides of your nostrils. But the city of Washington, D.C., for example, conducts COVID tests using nasal-only swabs. When I called the city’s public health department to ask why the city had opted for nasal swabs instead of the more sensitive brain ticklers, a spokesperson there told me that it was “mostly based on supply availability.” In the early days of the pandemic, when there were shortages of testing supplies, the city went with what was available and it stuck.

Even when it’s done right, “there is no perfect test,” said David Hirschwerk, M.D., the vice chair of infectious diseases at Northwell, one of the largest private health care networks in the New York area. He and his team conducted a study in which they swabbed each patient with two different swabs. “We did see a discordance,” he told me. That is, the same patient yielded a negative and a positive result at the same time.

The sheer scale of the pandemic—and the volume of testing—means that there are far more false negatives occurring than we think. “Even if a test were 98% sensitive and 99% specific,” warned a recent article in the New England Journal of Medicine, “it would still produce a false negative result in 2 of every 100 people infected. If we test 5 million Americans daily and only 1% of them have COVID-19, a total of 1000 positive cases will be missed, which increases the risk of spread.” And as we know, one missed infection—and the false sense of confidence created by a false negative—can create a chain of transmission that snares dozens of people.

So what is a physician to do? Do you trust the test or the patient? And what was I to do?

“I would probably tell you you still have COVID and you should quarantine and act as if you have COVID,” Neda Frayha, M.D., an internal medicine doctor who teaches medicine at the University of Maryland, told me. “If someone has symptoms that look and feel and sound like COVID but the result is negative, we need to treat it as if it’s COVID.”

One young doctor, who treated the crush of COVID patients when her hospital in New York City was overwhelmed in the spring, told me that back then, “we relied heavily on clinical diagnoses.” At the time, she said, “if the picture fit COVID, then it’s COVID unless you can convince us otherwise. At the end of the day, the COVID test was very helpful to have when it was positive, but there were patients that clearly had COVID, based on things like their X-rays, their labs, and their symptoms.” (This doctor asked for anonymity because she was not authorized to speak to the press.)

Though a false negative is far less likely when the virus isn’t raging in your community, “if you have symptoms that are concordant with COVID, and it’s the time of year when there isn’t much flu going around,” Hirschwerk said, referring to August, when I got sick, “there’s probably a very good chance that, irrespective of the test, you have COVID.”

“If you come in with COVID symptoms, you’re severely ill and have the classic presentation, you’re still pretty likely to have it,” said Baird. “Using just the test is a rookie mistake,” he explained, likening such a move to using merely glucose levels to diagnose something as complex as diabetes. “It’s why we have doctors and not robots.”

“I must sound totally insane to you,” I told Neda Frayha, M.D. “I must sound like I’m making it up.”

Why do these doctors feel they can override my negative COVID tests? The answer lies in something called Bayes’ Theorem and its idea of pretest probability. Since the beginning of the pandemic, medical journals have been publishing articles reminding doctors to use Bayes’ Theorem, which encourages a physician to integrate all the pieces of evidence in front of them, both the test and what their clinical acumen is telling them. According to Bayes’ Theorem, if a patient’s pretest probability is high—they have all the COVID symptoms, say, or they had close contact with a confirmed COVID patient—getting a positive test result slides their probability to 100% probability of having the coronavirus, but getting a negative test does not take them to zero. “There’s a moving dot on a continuum between zero and 100,” Baird explained. “With a negative test, we’ve lowered the probability, but it would still not be zero.”

James Crawford, M.D., who runs the diagnostic labs at Northwell, put it even more bluntly. “When the pretest probability is really high, you’re going to ignore the test,” he said. “Yes, there are some crystal-clear diagnoses in medicine, but you spend a lot of time as a doctor doing probabilistic medicine and trying to do right by your patient.”

In my case, I had all the cardinal symptoms of COVID (including the loss of taste and smell), I live in an area of Washington with some of the highest COVID rates in the city, and I got sick during the height of summer, making the flu far less likely. My pretest probability, according to Frayha, was very high. “It’s the fact that you have symptoms and that they feel and sound a whole lot like COVID,” she said. If she is right, according to Bayes’ Theorem and a research study recently published in the BMJ (originally the British Medical Journal), the likelihood of my having COVID is not negligible—in fact, it’s quite high. “For a pretest probability of 90%, someone with a negative test has a 74% chance of having COVID-19,” the authors of the paper wrote. “With two negative tests this risk is still around 47%.”

When Frayha and I spoke, I asked her if it was significant that whatever I had was so in and out, and felt so different from anything I’d had before. By that point, I had been sick for nearly three weeks and had had two negative tests, and though I was tired of non-doctor friends questioning whether it was really COVID, there was a growing part of me that was afraid they were right, even as I was spending more and more energy trying to prove to them—and to myself—that I wasn’t crazy.

“I must sound totally insane to you,” I told Frayha. “I must sound like I’m making it up.”

“Actually, it makes me think more of COVID than anything else,” she replied. “Other upper respiratory viruses are very linear. This virus is very unpredictable, and there’s a lot of mystery to it still. But people know when what they’re going through is something different than what they’ve been through before. When my patients tell me, ‘This is nothing like anything I’ve ever experienced,’ I pay attention to that. Those personal memory points are very important in interpreting what’s going on. People know when something different is going on, and that’s very important to pay attention to. That should be empowering, even more than what a laboratory says. Listen to your body.”

On Monday, September 14, I got another COVID test. This time it was a brain tickler, and this time it was in the emergency room. After my Labor Day scare, I was prescribed steroids, which recent studies have shown to be effective in dealing with the fact that the coronavirus sends the immune system into overdrive, sometimes to lethal effect. The steroids worked like magic. No more coughing, no more fatigue, and I could breathe without having to think about it or struggle for air. But then the course of treatment was over, the steroids left my system, and by Monday afternoon, I was sitting on my couch, pale and sweating and panting for air. My oxygen levels started going wonky again.

In the ER, I told the staff that I had tested negative for COVID three times, but the nurse taking care of me repeated what I had heard from all the doctors I had interviewed when I was still well enough to work. “We’re going to treat you like it’s COVID unless you can convince us otherwise,” he said.

“The CDC has guidelines on this,” the ER doctor told me ruefully, “but I don’t trust the CDC anymore.”

A few hours later, the doctor treating me came in to discuss the results of the tests they’d run. I didn’t have blood clots, nor did I have pneumonia, but the X-ray did show that I had bibasilar atelectasis—partially deflated lungs. A few days later, at the pulmonologist’s office, I would learn that I also had what is called reactive airway disease, a vicious cycle of post-viral inflammation that results in coughing fits and difficulty breathing. (“Sure sounds like COVID,” the pulmonologist said.) I also found out that my last COVID test—my fourth—had come back negative. The ER doctor had warned me this might happen. Most people this far along in their illness—five weeks in my case—were usually no longer positive for the coronavirus. Plus, he added, “the tests aren’t perfect. They’re only about 80% accurate.”

This made me wonder. “Do you think I’m still contagious?” I asked the ER doctor.

“I don’t know,” he said, explaining that the science had not reached a verdict on this yet. But that was his educated guess. “The CDC has guidelines on this,” he told me ruefully, “but I don’t trust the CDC anymore.”

“You’re not crazy, you’re not alone,” one doctor I spoke to assured me, as did the other physicians treating me. Still, my case is an outlier. The PCR tests are mostly very good, said Caitlin Rivers, Ph.D., an epidemiologist at Johns Hopkins. “It’s very possible to receive a false positive or a false negative, but I suspect that number is very small,” she told me. Even if people like me occasionally fall through the cracks, she added, “on a public health level, the tests are accurate enough to, on average, serve our needs to contain the epidemic. On a population level, the tests work.”

Testing, Doctor Baird reminded me, “is not curative and it is not protective." He went on: “The clear answer is, if you don’t follow all the rules, it will keep spreading."

Baird and Crawford, who run big diagnostic labs that were once at the center of COVID hotspots, are also pretty confident in the tests. Crawford estimates that the test he’s using at Northwell is 98% sensitive. It’s not 100%, but it’s close. They also stress that this is why it is important to get a handle on the spread of the virus: The less virus circulating in your community, the lower your pretest probability and the more you can trust a negative COVID test to mean you are really free from the virus.

The other message is a more somber one: Tests are important, but they’re not the solution. Behavior is. Getting tested before you join friends at a beach house won’t protect you; social distancing and mask-wearing will. Getting tested before going on a trip doesn’t keep you from catching the virus en route; not going on the trip will. Testing, Baird reminded me, “is not curative and it is not protective, it’s an adjunctive to all the other things we’re doing.” He went on: “The clear answer is, if you don’t follow all the rules, it will keep spreading. There’s a reason the COVID pandemic maps onto the electoral map. It’s not testing or a lack of therapy, it’s choice and behavior. It’s a hugely bummer message to send out there, but COVID is real and it’s totally unforgiving. The virus will punish us right away for doing what we shouldn’t be doing.”

Julia Ioffe is a GQ correspondent.