Five weeks ago, when I raised questions about the results of Pfizer’s and Moderna’s covid-19 vaccine trials, all that was in the public domain were the study protocols and a few press releases. Today, two journal publications and around 400 pages of summary data are available in the form of multiple reports presented by and to the FDA prior to the agency’s emergency authorization of each company’s mRNA vaccine. While some of the additional details are reassuring, some are not. Here I outline new concerns about the trustworthiness and meaningfulness of the reported efficacy results.

“Suspected covid-19”

All attention has focused on the dramatic efficacy results: Pfizer reported 170 PCR confirmed covid-19 cases, split 8 to 162 between vaccine and placebo groups. But these numbers were dwarfed by a category of disease called “suspected covid-19”—those with symptomatic covid-19 that were not PCR confirmed. According to FDA’s report on Pfizer’s vaccine, there were “3410 total cases of suspected, but unconfirmed covid-19 in the overall study population, 1594 occurred in the vaccine group vs. 1816 in the placebo group.”

With 20 times more suspected than confirmed cases, this category of disease cannot be ignored simply because there was no positive PCR test result. Indeed this makes it all the more urgent to understand. A rough estimate of vaccine efficacy against developing covid-19 symptoms, with or without a positive PCR test result, would be a relative risk reduction of 19% (see footnote)—far below the 50% effectiveness threshold for authorization set by regulators. Even after removing cases occurring within 7 days of vaccination (409 on Pfizer’s vaccine vs. 287 on placebo), which should include the majority of symptoms due to short-term vaccine reactogenicity, vaccine efficacy remains low: 29% (see footnote).

If many or most of these suspected cases were in people who had a false negative PCR test result, this would dramatically decrease vaccine efficacy. But considering that influenza-like illnesses have always had myriad causes—rhinoviruses, influenza viruses, other coronaviruses, adenoviruses, respiratory syncytial virus, etc.—some or many of the suspected covid-19 cases may be due to a different causative agent.

But why should etiology matter? If those experiencing “suspected covid-19” had essentially the same clinical course as confirmed covid-19, then “suspected plus confirmed covid-19” may be a more clinically meaningful endpoint than just confirmed covid-19.

However, if confirmed covid-19 is on average more severe than suspected covid-19, we must still keep in mind that at the end of the day, it is not average clinical severity that matters, it’s the incidence of severe disease that affects hospital admissions. With 20 times more suspected covid-19 than confirmed covid-19, and trials not designed to assess whether the vaccines can interrupt viral transmission, an analysis of severe disease irrespective of etiologic agent—namely, rates of hospitalizations, ICU cases, and deaths amongst trial participants—seems warranted, and is the only way to assess the vaccines’ real ability to take the edge off the pandemic.

There is a clear need for data to answer these questions, but Pfizer’s 92-page report didn’t mention the 3410 “suspected covid-19” cases. Nor did its publication in the New England Journal of Medicine. Nor did any of the reports on Moderna’s vaccine. The only source that appears to have reported it is FDA’s review of Pfizer’s vaccine.

The 371 individuals excluded from Pfizer vaccine efficacy analysis

Another reason we need more data is to analyse an unexplained detail found in a table of FDA’s review of Pfizer’s vaccine: 371 individuals excluded from the efficacy analysis for “important protocol deviations on or prior to 7 days after Dose 2.”  What is concerning is the imbalance between randomized groups in the number of these excluded individuals: 311 from the vaccine group vs 60 on placebo. (In contrast, in Moderna’s trial, there were just 36 participants excluded from the efficacy analysis for “major protocol deviation”—12 vaccine group vs 24 placebo group.)

What were these protocol deviations in Pfizer’s study, and why were there five times more participants excluded in the vaccine group?  The FDA report doesn’t say, and these exclusions are difficult to even spot in Pfizer’s report and journal publication.

Fever and pain medications, unblinding, and primary event adjudication committees

Last month I expressed concern about the potential confounding role of pain and fever medications to treat symptoms. I posited that such drugs could mask symptoms, leading to underdetection of covid-19 cases, possibly in greater numbers in people who received the vaccine in an effort to prevent or treat adverse events. However, it seems their potential to confound results was fairly limited: although the results indicate that these medicines were taken around 3–4 times more often in vaccine versus placebo recipients (at least for Pfizer’s vaccine—Moderna did not report as clearly), their use was presumably concentrated in the first week after vaccine use, taken to relieve post-injection local and systemic adverse events. But the cumulative incidence curves suggest a fairly constant rate of confirmed covid-19 cases over time, with symptom onset dates extending well beyond a week after dosing.

That said, the higher rate of medication use in the vaccine arm provides further reason to worry about unofficial unblinding. Given the vaccines’ reactogenicity, it’s hard to imagine participants and investigators could not make educated guesses about which group they were in.  The primary endpoint in the trials is relatively subjective making unblinding an important concern. Yet neither FDA nor the companies seem to have formally probed the reliability of the blinding procedure, and its effects on the reported outcomes.

Nor do we know enough about the processes of the primary event adjudication committees that counted covid-19 cases. Were they blinded to antibody data and information on patients’ symptoms in the first week after vaccination?  What criteria did they employ, and why, with a primary event consisting of a patient-reported outcome (covid-19 symptoms) and PCR test result, was such a committee even necessary? It’s also important to understand who was on these committees. While Moderna has named its four-member adjudication committee—all university-affiliated physicians—Pfizer’s protocol says three Pfizer employees did the work. Yes, Pfizer staff members.

Vaccine efficacy in people who already had covid?

Individuals with a known history of SARS-CoV-2 infection or previous diagnosis of Covid-19 were excluded from Moderna’s and Pfizer’s trials. But still 1125 (3.0%) and 675 (2.2%) of participants in Pfizer’s and Moderna’s trials, respectively, were deemed to be positive for SARS-CoV-2 at baseline.

Vaccine safety and efficacy in these recipients has not received much attention, but as increasingly large portions of many countries’ populations may be “post-Covid,” these data seem important—and all the more so as the US CDC recommends offering vaccine “regardless of history of prior symptomatic or asymptomatic SARS-CoV-2 infection.” This follows on from the agency’s conclusions, regarding Pfizer’s vaccine, that it had ≥92% efficacy and “no specific safety concerns” in people with previous SARS-CoV-2 infection.

By my count, Pfizer apparently reported 8 cases of confirmed, symptomatic Covid-19 in people positive for SARS-CoV-2 at baseline (1 in the vaccine group, 7 in the placebo group, using the differences between Tables 9 and 10) and Moderna, 1 case (placebo group; Table 12).

But with only around four to 31 reinfections documented globally, how, in trials of tens of thousands, with median follow-up of two months, could there be nine confirmed covid-19 cases among those with SARS-CoV-2 infection at baseline? Is this representative of meaningful vaccine efficacy, as CDC seems to have endorsed? Or could it be something else, like prevention of covid-19 symptoms, possibly by the vaccine or by the use of medicines which suppress symptoms, and nothing to do with reinfection?

We need the raw data

Addressing the many open questions about these trials requires access to the raw trial data. But no company seems to have shared data with any third party at this point.

Pfizer says it is making data available “upon request, and subject to review.” This stops far short of making data publicly available, but at least leaves the door open. How open is unclear, since the study protocol says Pfizer will only start making data available 24 months after study completion.

Moderna’s data sharing statement states data “may be available upon request once the trial is complete.” This translates to sometime in mid-to-late 2022, as follow-up is planned for 2 years.

Things may be no different for the Oxford/AstraZeneca vaccine which has pledged patient-level data “when the trial is complete.” And the ClinicalTrials.gov entry for the Russian Sputnik V vaccine says there are no plans to share individual participant data.

The European Medicines Agency and Health Canada, however, may share data for any authorized vaccines much earlier.  EMA has already pledged to publish the data submitted by Pfizer on its website “in due course,” as has Health Canada.

Peter Doshi, associate editor, The BMJ

Competing interests: I have been pursuing the public release of vaccine trial protocols, and have co-signed open letters calling for independence and transparency in covid-19 vaccine related decision making.

Footnote

Calculations in this article are as follows:  19% = 1 – (8+1594)/(162+1816); 29% = 1 – (8 + 1594 – 409)/(162 + 1816 – 287). I ignored denominators as they are similar between groups.