Clinical trials rarely get a second look—and when they do, their findings are not always what the authors originally reported. That’s the conclusion of a new study, which compared how 37 studies that had been reanalyzed measured up to the original. In 13 cases, the reanalysis came to a different outcome—a finding that suggests many clinical trials may not be accurately reporting the effect of a new drug or intervention.
To conduct the study, published today in The Journal of the American Medical Association (JAMA), Shanil Ebrahim, a clinical epidemiologist at the Stanford University School of Medicine in Palo Alto, California, and colleagues combed through the MEDLINE database. The online archive contains almost all life sciences and biomedical research studies conducted over the past 64 years. The team found 37 clinical trials that been reanalyzed. Of these, 13 came to conclusions at odds with the original authors’ findings. Moreover, only five of the reanalyses were by an entirely different set of authors, which means they did not get a neutral relook.
In one of the trials, which examined the efficacy of the drug methotrexate in treating systemic sclerosis—an autoimmune disease that causes scarring of the skin and internal organs—the original researchers found the drug to be not much more effective than the placebo, as they reported in a 2001 paper. However, in a 2009 reanalysis of the same trial, another group of researchers including one of the original authors used Bayesian analysis, a statistical technique to overcome the shortcomings of small data sets that plague clinical trials of rare diseases such as sclerosis. The reanalysis found that the drug was, as it turned out, more effective than the placebo and had a good chance of benefiting sclerosis patients.
In another study, researchers reanalyzed a 1984 clinical trial of sclerotherapy, in which a drug is injected to treat dilated veins in the esophagus. The original researchers found that the therapy reduced mortality but did not reduce the risk of bleeding recurring at a later time. Based on this finding, they recommended sclerotherapy to patients. In 2001, however, another group of researchers analyzed the same data with a different statistical method that looked at the interplay between recurring bleeding and mortality, and concluded that sclerotherapy did not reduce mortality. Their findings imply that doctors ought to avoid prescribing sclerotherapy in patients with high risk of mortality, in contrast with the original conclusion. Sclerotherapy is still commonly used for the treatment of hemorrhoids and varicose veins.
Other reanalyses found faults with the original methodology. For example, in a study of the efficacy of mechanical cardiopulmonary resuscitation (CPR) versus manual CPR in patients suffering cardiac arrest outside hospitals, one of the hospitals participating in the trial changed its protocols midway. This altered the trial outcome in favor of manual CPR, a flaw that the reanalysis corrected. The authors of this reanalysis belonged to the same research group as the original study.
“It was alarming that a substantial portion of trials arrived at different conclusions,” Ebrahim says. Although the reanalyses didn’t always mean the original research was flawed, the altered conclusions indicate a greater need for the sharing of clinical trial data, he says.
Tom Jefferson, a Rome-based researcher who reviews studies for the nonprofit Cochrane Collaboration, says the results of the JAMA study do not surprise him. “The process [of analyzing clinical trials] is so subjective, you can twist it any way you want.” He notes that drug companies often do not release all-important raw data from clinical trials, instead choosing to publish highly compressed summaries of their findings.
Last year, the Cochrane Collaboration succeeded in a 4-year battle with Swiss pharmaceutical giant Roche to access raw data from unpublished clinical trials on its influenza drug Tamiflu. Governments across the world had stockpiled the drug to guard against a flu epidemic, after clinical trials found the drug to be effective in the disease. But during a reanalysis of these trials, the Cochrane Collaboration discovered that the raw data underlying several of the published studies had not been released. When it was eventually able to access these data several years later, a reanalysis showed that Tamiflu wasn’t as effective as thought earlier.