Quit While You're Behind

When should a medical study of a drug/therapy be abandoned? Merck has announced that a trial of a candidate HIV vaccine was stopped early. Their original plan was to continue through 2008 and enroll 3,000 HIV negative volunteers. Instead, the study has stopped after enrolling just over 1,500 participants. Why?

Because of discouraging results. The study found that about 3% of subjects became infected with HIV with slightly higher rate among those who got the vaccine compared to those who got placebo injections.

The decision to cut a study short has to weigh complex, competing considerations. If the vaccine is truly a dud, it is unethical to continue to expose new volunteers to the burdens and risks of participating in the study -- the money and resources could be better deployed to test other promising HIV prevention stratgies (see disclosure). But are we sure it doesn't work? We'd want to be sure.. After all, formidable resources have been invested to answer the question and scientists owe the public a conclusive answer.

Statisticians look at the data in hand and ask "If the study was completed, what the chance the study would favor the vaccine?" This calculation depends on how far into the study we are and projections about how the vaccine will look in the (yet uncollected) future data.

The study was planned to continue until there were around 90 HIV infections (a grim reality). Based on this, the vaccine would be proved effective if 35 (or fewer) of the 90 infections occured among the vaccinated. Once there are more than 35 HIV+ among the vaccinated, the vaccine will be condsider a failure. At the half-way point, there were 24 HIV+ subjects among the vaccined.

How could the vaccine to come back from that bad start? Only if among the next 45 infections, fewer than 11 of those infections (35 - 24) occur in the vaccinated group - a split of 11 v. 34. How likely is that?

Let's assume the vaccine is very effective (even though the current data contradicts that) and it cuts the risk of HIV infection by 50%.

Adopting this optimism, we can calculate the change of a "positive" trial using the binomial distribution and get an approximate probability of 0.13. Thus, even if the vaccine worked, the odds are more than 6:1 against a positive result. Unlikely. Even someone who believes the vaccine works would concede, given the available data, the vaccine is going to "fail". I can post my calculations later for the technically inclined.

Decisions about whether a trial should stop or continue are made by an independent board of experts who consider numerous aspects of the data. For instance, the count of infections was 19 vaccine v. 11 placebo among those who'd received more than 1 injection. This raises a theoretical, if not conclusive, possibility of increased risk of HIV from the vaccine. This is discussed an article in the San Francisco Chronicle.

Disclosures I am the biostatistician for a trial of a non-vaccine HIV prevention strategy -- pre-exposure prophylaxis.