LYMEPOLICYWONK: New Study Reveals Fatal Flaws in NIH Klempner Trial Statistical Analysis. Is this Error Human, Incompetence or Worse?

Byldoadmin

So what was the fatal error in the Klempner study?

Researchers who design clinical trials define when a treatment is considered a success. Does the treatment work or not?  How much does a patient have to improve before you can say the treatment helped? Do they have to return to perfect health?  Clinical trials usually require that patients improve, but not that they return to perfect health.  DeLong found that the Klempner trial set the level for determining treatment success excessively high. For instance, in the seronegative arm of the trial, treatment success required that patients not simply return to the norm for the general population, but instead surpass the norm by essentially one full standard deviation.  That type of success is unheard of in clinical trials.

More specifically, the Klempner study used too high a level to determine success on something called the SF-36 scale. It used a measure that was not clinically meaningful.  Let’s look at this in context.  While other chronic diseases set the bar for improvement on the SF-36 between 2 to 5, the Klempner trial set the bar at roughly 7 and 9 in the seropositive arm and between 9 and 13 in the seronegative arm. This amount of improvement is far higher than the amount of improvement that could reasonably be expected among the chronically ill.

If the Klempner trial had used a clinically meaningful measure of success, say between 2 and 5, it would have had to have used much, much larger sample sizes, which it did not have.  For instance setting the clinically meaningful measure of success at 2 would have required a sample of 400 or more per treatment arm. The sample sizes in Klempner were only 78 (seropositive) and 51 (seronegative)—a sample size which would not yield statistically significant results. When researchers select an excessively large treatment effect required for success, as was done here, the size of the sample required is reduced.  The trade-off, however, is that the results are not clinically meaningful.

Many researchers say that small sample size trials are unethical because they can literally steer scientists down the wrong path.  That’s what the Klempner study did.  It was a waste of money, a waste of time, and it has led research down the wrong path for the last 10 years.

Was it a matter of researcher incompetence?  DeLong and colleagues point out that studies on the level of improvement that is clinically meaningful for other chronic diseases had not been done at the time the Klempner trial was conducted. So the Klempner researchers did not know that they were using an excessively large treatment effect at the time they conducted the study.  Does that take the researchers off the hook?  Not really.  First of all, once the studies on the way to measure success in chronic disease had been conducted, the Klempner team could have acknowledged a possible error.

Second, the Klempner team could have avoided “overstating” its findings and highlighted the potential for error.  The ethical implications of exaggeration in science are understood by other scientific societies.  For example, the code of conduct for the American Chemical Society states: “Public comments on scientific matters should be made with care and accuracy, without unsubstantiated, exaggerated, or premature statements.”

Instead, the Klempner study concluded flatly that retreatment was ineffective.  The NIH press release headline read: “Chronic Lyme Disease Symptoms Not Helped by Intensive Antibiotic Treatment.”  It further concluded that “ it is unlikely that a longer course of treatment or different antibiotic combination would result in greater improvement than what we found in these studies.” This conclusion was clearly beyond the scope of the study since it did not test other antibiotics or other durations of antibiotic therapy.

In short, rather than acknowledging the potential weaknesses of its findings and encouraging further research, the Klempner study overstated its conclusions and firmly shut the door on future science.  Dr. Klempner himself then sat on the 2006 IDSA Lyme disease guidelines panel, which used his study to deny seriously ill patients any access to care. Research of fellow panel members on a guideline panel may be less likely to be critically challenged, and the IDSA guidelines are riddled with references from the panel members.

This guideline recommendation has caused untold suffering by patients, who are essentially told to simply live with the symptoms of chronic Lyme, which can cause disability equivalent to that of congestive heart failure. For example, 25% of over 4,000 patients who responded to a survey by LymeDisease.org have been on disability at some point in their illness.  If a guidelines panel is going to preclude further care of patients this ill, they should be absolutely certain that further treatment will not be beneficial.  And, they should err on the side of treatment. Denying patients access to care when treatment options are available results in unnecessary suffering.

All of this brings us back to what Congressman Smith referred to as the “lost decade” of research in Lyme disease.  He was referring to the delay in publishing the Embers research trial which found persistent infection in monkeys after treatment using the Klempner protocol.  The lack of scientific rigor in the Klempner trial coupled with the delay in publication of the Embers trial has caused the Lyme community enormous harm.  Both studies were NIH funded, and, again we are left to wonder why the NIH and Dr. Phil Baker, who headed up Lyme research for the NIH, failed to properly oversee and follow-up on this research.  Why aren’t they setting the record straight now that they have the information?

It brings to mind a comment that Dr. Stephen Barthold made at the congressional hearing chaired by Congressman Smith:

There is overwhelming evidence in a variety of animal species as well as humans that B. burgdorferi persists without treatment, but the crucial question is does it survive following treatment, and if so, do surviving spirochetes cause “chronic” Lyme Disease or PLDS? These questions cannot be answered by speculative and expensive human clinical trials motivated by firmly held dogmatism.

Hear! Hear!

References:

Delong, A. K., B. Blossom, et al. (2012). “Antibiotic retreatment of Lyme disease in patients with persistent symptoms: A biostatistical review of randomized, placebo-controlled, clinical trials.” Contemp Clin Trials.

Klempner, M., L. Hu, et al. (2001). “Two controlled trials of antibiotic treatment in patients with persistent symptoms and a history of Lyme disease.” N Engl J Med 345(2): 85-92.

Embers, M. E., S. W. Barthold, et al. (2012). “Persistence of Borrelia burgdorferi in Rhesus Macaques following Antibiotic Treatment of Disseminated Infection.” PLoS ONE 7(1): e29914.

My previous blog postings regarding the Embers monkey trials and their importance for Lyme patients can be found here.

The LYME POLICY WONK blog is written by Lorraine Johnson, JD, MBA, who is the Chief Executive Officer of LymeDisease.org, formerly CALDA. Contact her at lbjohnson@lymedisease.org.

 

Similar Posts

  • LYMEPOLICYWONK: Expired Drugs—Are they Effective, Safe?

    This is a little off topic for my blog, but something that is of high interest to patients. Patients ask all the time about the effectiveness and safety of “expired” drugs. A recent article that appeared in the Medical Letter examines the issue in detail. The article notes that generally manufacture expiration dates merely reflect the fact that the drug is “stable” at that point, they do not reflect when the drug becomes unstable. The conclusion of the article indicates that outdated drugs may be effective and safe for at least 5 years after the expiration. Notable exceptions include liquid suspensions and epinephrine in Epipen, which are not stable over time. The article also notes the following: " There are no published reports of human toxicity due to ingestion, injection or topical application of current drug formulations after their expiration date. Renal tubular damage has been reported after use of degraded tetracycline in a formulation that is no longer available (REF 2)." Other sources contain warnings about expired doxycycline, which may (or may not) be based on this outdated tetracycline warning, but patients may want to check with their doctors when using expired doxycycline to be on the safe side.

  • IDSA Lyme Hearing: Wormser– Talk About Exaggeration!

    Talk about exaggeration. No really–at the hearing. Dr. Carol Baker asked Dr. Wormser the $200 question: Why exaggerate if the truth will do? She was talking about the use in the guidelines of words like "vast majority" when the real percentage was 65%. And her question was why not let percentages speak for themselves? If the truth is 65%–why not simply say 65%. Is it just me, or does Wormser sound irritated at the question? His response after a drill down is that he would not use the expression "vast majority" to refer to anything less than say–90%. Excuse me? Did I hear that right? 90%. Really? Ok, let's break it down for him.

  • Talking the talk

    Terrence McNally at KPFK, a local public radio station in Los Angeles interviewed me, Andy Wilson, Mandy, and Dr. Horowitz on June 23rd about the film, Under Our Skin, and about the antitrust settlement. It is a good listen if you have not heard it. The link to the podcast is below. A close friend snapped the picture of me in the booth.

  • LYMEPOLICYWONK: IOM LIVE BLOG PART III Public Comment

    I have summarized the public comments made at the IOM hearing today. I intend to post the statements made by those speaking to the panel that are provided to me. Those who spoke included Pat Smith of the national Lyme Disease Assn., Diane Blanchard and Ann Lyons of Time for Lyme, Gregg Skall of National Capital, Helene Jargensen, author of Sick and Tired, Candy Brassard of the EPA, Rick Smith, Bob Smith, and Julia Rice, a former nurse. Others listed to speak who did not attend were Arthur Weinstein, Phil Baker, and Lynn Shepler.

  • LYMEPOLICYWONK: And, the Number Is? 5,200 Responses! IDSA Guidelines and Patient Preference Survey

    Let me thank everyone who responded to the Patient Preferences Survey! We have drawn over 5,200 responses, which is just simply fabulous. We will now start the process of reviewing the responses with an eye toward publication of the results. Our last survey was published in Health Policy, a widely respected peer reviewed journal. Those results now help inform the base of knowledge of the access to care problems that Lyme patients face. Our goal with these surveys is to open the space for conversations in the public health arena about what is really going on with Lyme patients from their perspective. We are trying to give voice to the patient community, which is all to often completely ignored in conversations about Lyme disease. Your willingness to participate in these surveys allows us to do that. So hats off to you for being so vocal and letting us know your views.