In Search of the Holy Grail

By Bill Benda, MD, FACEP, FAAEM

Photo by speckyfish2000 via Flickr, used under the Creative Commons License.

I sat on an NIH panel a couple of weeks ago in Washington, D.C. Our mandate: to determine which of a few dozen grant applications deserved, through purity of methodology and virtue of intent, to be awarded funding by a commercial enterprise whose name I cannot reveal.

In fact, I wish I could tell about you the subject of the grants, or the people who sat on the panel with me, or the amount of monies given away, or how we determined who got the cash. But the NIH is very, very adamant that nothing ever, ever leaks to the outside world on what transpired those two days. Non-disclosure forms are signed, warnings are given, and papers and discs are shredded after the proceedings are completed. I imagine there is a sophisticated surveillance system set up in Bethesda, MD, to watch our every step, and should I inadvertently write the wrong words in this blog, an alarm sounds on the east coast: NIH Central Alert! Reviewer violation! Activate liquidation team! And within the hour an ominous black Prius would pull up in front of my house and bespectacled men in trench coats would emerge.

But I digress from the topic of this article. My time on the panel did serve to reinforce one particular belief I have harbored over the past couple of decades as I have explored the world of scientific investigation – that research, for all of its academic and cultural value, is not health care’s Holy Grail. It is not representative of any irrefutable fact, it is not the courier of absolute truth, and it certainly is nothing close to purity of thought and action.

Now before those of you who spend most of your waking hours in academia start sharpening your ninja swords, let me state firmly that research itself is indeed the best tool we have at our disposal to differentiate reality from fiction in the clinical arena. But it is a fragile tool, and prone to breakage and misuse. And I do hold a touch of credibility on this particular topic - I have conducted and published two randomized control trials (one funded by NCCAM), and have been providing peer review to six medical journals over the past decade or so. So I can state with some authority one thing that I have never seen echoed in any journal, except perhaps on the editorial page or in a letter to the editor, and that is: All research is flawed, and likely quite badly flawed.

Lets take a look at the process of publishing a study, from the first inkling of an idea to the final inking on the journal page. Someone, somewhere, comes up with a hypothesis. After much thinking and mulling, the hypothesis transmogrifies into a research plan–usually in conjunction with other co-investigators–then into a methodology, complete with inclusion and exclusion criteria, plans for statistical analysis, recruitment, carrying out the study, collation of data, actual statistical analysis, final conclusions, writing of the paper, peer review by a journal, revisions as requested by said journal (and perhaps repetition of this step several times), publication of the article, reading and attempted understanding by the subscriber and, finally perhaps, incorporation into clinical use.

There are countless steps where bias and error can creep into the final findings, and this doesn’t take into account influence by industry if the piece happens to be on a pharmaceutical or other medical appliance. And speaking of pharmaceuticals (and supplements), the study usually recruits a narrow demographic of males between this age and that age with this clinical condition but not complicated by that clinical condition (exclusion criteria!) and then treats at a specified dose for a set period of time. And if the positive findings are somewhat higher than the placebo effect, that pharmaceutical (or supplement) is marketed to men and women from 18 to 88 with all sorts of confounding conditions and with each very diverse patient getting pretty much the same dosage. This is part of the reason why, say, hormone replacement therapy protected against heart disease last month, but increases heart attacks this month, while protecting against breast cancer last year, but thought to stimulate breast cancer next year, and on, and on.

OK, I’m going to back up once more and restate a very important point: Research is the best tool we have at our disposal to differentiate reality from fiction in the clinical arena! But it’s simply not as perfect as we, and our media, purport. As I tell my students, no research study, no matter how well done, is the truth; it is simply another finger pointing in the general direction of the truth.

So the next time that your organization wants to put research at the top of its priority list, consider… Wait a minute...

I need to go. A black Prius just pulled up in front of my house.