Getting “Testy” with Medical Testing!

Back in the 1700’s an English mathematician and Presbyterian minister named Thomas Bayes published two works in his life, one on math and the other on theology. Mr. Bayes never published the work he is most remembered for yet Bayes Theorem has been widely applied from mathematics to bioethics.

On the face of it Bayes Theorem isn’t that complicated. Essentially what the math symbols say is that predicting the future likelihood of any event is based on the present likelihood of that event and whatever evidence (test results) you have that might alter prediction of that future event.

Breast cancer offers a concrete example. To keep things simple, ignoring familiar cancer risk, the prevalence of breast cancer increases with age. A 75-year-old woman has a greater chance of breast cancer than does a 40-year-old woman all other things being equal. A clinician may do an examination in women these ages and find a lump yet the post-examination odds of breast cancer are going to be less in the 40-year-old woman because the pre-examination odds were lower to begin with.

Small numbers multiplied by typically small factors (the technical term is Likelihood ratios) render small results. If my odds for a disease are 1:1000 and a physical examination finding or test only contributes a multiplier (a.k.a. Likelihood ratio) of 2 then the odds of the disease remain low. Thought of differently, if my Likelihood ratio was 10 (and that’s an uncommonly good test) the change in probability of my subject having the outcome of interest increases 45%. That sounds like a lot but a 45% increase of a very small number is still a very small number.

Healthy people are well…healthy. The prevalence of many diseases just aren’t so common to make meager test characteristics add up to much diagnostic clarity. Herein lies the problem of screening healthy people. Among the sick some tests don’t improve upon what could be fairly well estimated based on cheaper studies. Bladder testing among low risk women is a good example. In this setting, basic office examinations have been shown to be as effective in predicting suitability for surgical treatment as expensive testing. Mr. Bayes was a genius.

Recently the American Board of Internal Medicine conveniently culled together the Five Things Physicians and Patients Should Question based on the evidence-based recommendations of specialty societies representing 500,000 physicians (check it out at http://www.abimfoundation.org/Initiatives/Choosing-Wisely.aspx). Among the recommendations submitted by the American College of Obstetricians and Gynecologists was to not treat mild cervical dysplasia that is less than 2 years duration and not to screen for ovarian cancer in asymptomatic women at average risk. The reasoning behind these and the many other excellent society recommendations can often be traced back to Bayes Theorem. Either the prevalence of the disease is rare or the test/treatment performance is poor or both rendering Bayes equation to calculate it’s just not worth the effort. Indeed, beyond the cost of testing, false positive tests cause a lot of harm to a lot people every year.

Dancing gorillas say a lot about another problem with testing. In a recent study researchers snuck the image of a dancing gorilla on a radiograph that was later reviewed by radiologists. The radiologists did their job and reported what clinically they saw in the radiograph but when asked about the dancing gorilla they were stumped. Few saw the dancing gorilla. Is this really that surprising? Not really. There is collusion between our eyes and brain such that what we see is what our brain says is to be seen. Dan Gilbert’s book, Stumbling on Happiness nicely describes research supporting this eye-brain conspiracy. The problem of dancing primates is that a test might not always clarify one diagnosis or another but merely confirm the plan already hatched in the mind of the person ordering the test.

Such seems to be the case for a young woman I saw who had undergone three procedures to treat her bladder problem. She had undergone the usual testing to justify the procedures done. Surprisingly none of the tests confirmed the diagnosis pursued in doing the procedures. In other words, the tests didn’t matter. This scenario could be helped with a better understanding of Mr. Bayes theorem. For this young woman, is a troublesome bladder problem common such that three procedures would prove ineffective? Not so much. Herein was a red flag that the pre-test odds might make any test result suspect inspiring needed diagnostic and therapeutic caution. Understanding how common a disease is in your population can help you see the dancing gorilla.

I have surely abused some treasured statistical concept in my simplification of what can quickly become very complex. For example I have mixed odds and probabilities as if they are synonyms. They are related but importantly different. I also haven’t mentioned that tests results can be combined. Pre-test disease prevalence multiplied by test 1 and test 2 and test 3 may offer a remarkably good prediction for the presence of a given disease. Nonetheless Bayes theorem says a lot about the limits of evidence-based medicine even if it is at the heart of many of its recommendations. The “art” of medicine may be a clinician’s capacity to understand the prevalence of a given condition among her population. Good medicine will always rely on the merger of common sense and science and in a way this is what Bayes Theorem is saying.

Mr. Bayes’ ideas on conditional probability were not immediately embraced by his mathematics peers. Many clinicians would not embrace statistical methods in their daily practice. I am reminded of George Santayana who said, “Those who do not remember the past are condemned to repeat it.” In health care, for both patient and practitioner, Mr. Santayana’s sentiment echo Mr. Bayes theorem and could as much be reconfigured as, those who do no appreciate mathematics are doomed to spend lots of money.

Standard

Compared to what?

A flyer appeared in my mailbox the other day advertising a surgical technique for a particular hospital system. I’ll refrain from specifics. The advertisement, however, wasn’t unlike a lot of health-related advertisements. In this case the advertisement claimed, “less pain, “shorter hospital stay,” and “faster return to your regular activities” with this new surgical technique.  When confronted with this sort of claim, the savvy health consumer should ask, “compared to what?”

 

Health care is expensive and fee for service still keeps the lights on for most hospitals. Getting folks in the door is the first step in getting those fees rolling. I hate to sound like medicine is just another business but in some ways it is. It would be naïve to think advertisements by health systems are motivated exclusively by some sense that coming to one health system over another is in your best interests. That would be altruism and there are some who legitimately feel this way about advertising health services. In some cases they are right (e.g. going to one hospital is better than going to another) but in many/most ways the consumer of health services needs to embrace the adage, “let the buyer beware.”

 

Most of medicine is practiced without substantial proof that it works. Indeed according to the now defunct Office of Technology Assessment (OTA), fewer than 30% of procedures currently used in conventional medicine have been rigorously tested. Since the OTA was retired in 1995, and given the introduction of new procedures and technologies have likely exceeded credible investigations as to their clinical merit, that 30% number is likely vastly underestimated. This is certainly true for the surgical tool marketed in my advertisement mailer. Indeed credible investigations comparing this surgical approach with other convention approaches do not fully support the claims made. Even more poignant is the innovation was being promoted for a procedure one study estimated was done without good cause in 70% of cases. Ouch!

 

To be sure physicians and by extension the health systems they represent, want to do well for their patients. At least part of the problem is that much of medicine doesn’t understand what constitute “proof” of effectiveness. For example, one study found sixty-percent of gynecologists asked to estimate a woman’s chance of having breast cancer following a positive screening mammogram overstated the risk. Statistics, that branch of mathematics that is often central to judging the comparisons made in clinical studies, can be really confusing and acquiring and maintaining a working knowledge of this is difficult for most clinicians.  To make matters worse, industry influence and just plain bad study methods, has made identifying credible investigations almost like looking for a lost needle on the hospital operating room floor…or haystack.

 

So what are you suppose to do? Here are some helpful rules of thumb about consuming medicine:

1)   Carefully weigh any recommended therapy with what you find important seeking to understand what evidence exists regarding how the therapy works for addressing what you find important.

2)   Take advantage of a medical librarian. Many hospitals have medical librarians who can help you find studies relevant to your problem. See what they can find. You can “Google” your problem but sometimes that just brings up junk. A medical librarian can help sift some of the junk out.

3)   Don’t be afraid to ask your physician hard questions about what proof justifies the treatment recommendation. If they appear offended check out the next rule of thumb.

4)   Get a second opinion…maybe a third?

5)   Understand the bias. Bias occurs anytime there are hidden influences that push opinions in a specific direction. Bias can be good or bad but you’d like to minimize bias in medical studies or in treatment recommendations. One problem folks have with industry buying physician lunches is the problem of bias. On September 30, 2014, as part of the Sunshine Act, the federal government will release most of the data showing how much money industry gave to physicians. In pursuit of understanding the bias, that data might be helpful to look over when it becomes available.

6)   Seek a consultation with a physician who publishes research. If a physician conducts and publishes research then there is a better chance they are better at interpreting the current literature on a topic. Note I just said “better.”

7)   If waiting is clinically appropriate, resist the temptation to decide on any aggressive therapy the day you are offered it.

 

My list isn’t perfect. There are always going to be gaps between what you and your clinician know about a given medical problem. That gap is safeguarded historically by a privileged ethical relationship wherein the patient’s interests are to be given priority over those of the clinician or the health system. There are lots of good and bad reasons why understanding what’s best is not such an easy task. Facing that challenge, when presented medical facts regarding a treatment being offered to you don’t forget to ask, “compared to what?”

Standard