Share

Writing the protocol was heavy going; it took six months to finalise. It was the first time (to our knowledge) that someone had attempted to update a merged Cochrane review by only looking at unpublished evidence from different sources. That is, mostly pharma internal reports and regulatory documents from Europe, Japan and the US which were written for a different purpose than publication - regulation.

We knew there was a mass of evidence running to many thousands of pages written in English and Japanese and that we would have to essentially create new methods to review such data. 

Some things became clearer as we progressed. First, there was no complete list of trials of Neuraminidase inhibitor (NIs) inhibitors undertaken or underway. Even in the many Health Technology Appraisal publications on the topic, and because of the age of certain trials, we could not look for them in a clinical trial registry. It was only in 2005 that the International Committee of Medical Journal Editors required clinical trials to be published in a clinical trial registry to qualify for journal publication. We needed to create the list.  

It became clear that we would have to ask the three manufacturers of NIs for help. But instead of asking them: “please give us a list of all your trials”, we had to come up with a credible list to ensure that the manufacturers took our request seriously. 

Creating an exhaustive list to send to Roche, GSK, and BioCryst (the latter makes the intravenous NI peramivir)  took months because we had to put together all that we knew from different sources, published and unpublished. The difficulty increased because each trial is primarily identified by a pharma code. For example, some of the Roche trials have a WV prefix followed by 5 digits, e.g. WV15670), but some trials were also published, and often the relationship wasn’t clear, so we had to try and match the two. 

Share

Other trials were cited by one regulator or one pharma document but were unpublished and never will be for unknown reasons. We referred to these as “traces” because they were mentioned in one or two documents that no one had ever seen outside the pharmaceutical companies and perhaps the regulators. 

In trying to reconstruct a complex list of trials using codes, it is very easy to make mistakes. The first draft of the list was double-checked by Peter Doshi, who edited and expanded it again. This was also done on the basis of the responses from pharma. Chris del Mar had written to Pharma on behalf of us all and was still keeping the correspondence going. 

At this point, Pharma was on board: correcting our mistakes and adding several trials that we were unaware of. The reconstruction was complicated by the sheer number of trials of the three NIs. Eventually, we ended up with a tally of 146 (mostly pharma-funded but not all eligible for our review). However, we found a trial carried out in China that Roche knew nothing about. To say we were confused is an understatement. The list was more than five times the number of studies we had included in our 2009 reviews (20 adult trials and seven in children).   

We stored the list in a monstrously complex spreadsheet nicknamed Marge (after the Simpson’s character who still maintains order in the Simpson household despite all their antics). Marge also served as a sort of scrapbook where we jotted down impressions, comments, quotes and logged all correspondence. Of course, all this is a far cry from the nice, neat systematic review based mainly or wholly on published evidence that we had been producing for decades.

The sheer complexity of the trial sheets in Marge and the timings involved (the trials were run over a decade and are still being published in dribs and drabs either as full publications, abstracts (or in what we call extended abstracts) brought us to our next step. 

Share

We realized that we needed to change our thinking from single trials or groups of trials (e.g. influenza treatment with one of the NIs in a defined population) to the whole trial programme. This entailed merging the two existing Cochrane reviews (on adults and children) into one. 

To understand the trial programme, we needed to understand why a certain trial was conducted at a certain time. We needed to construct a timeline of development, marketing and use of the drugs to set the trials into the particular context they were designed for. We thought this might also help us understand why the majority have never been published. 

Publication, of course, is a sort of extreme synthesis of the internal trial report. We did not know how many pages a full CSR would run to because we had never seen one. Roche was quick to point out that what we had in our archives was far more than anyone else had ever seen outside pharma, especially since we hadn’t signed any secrecy agreement, and that probably meant we had also seen more than the “authors” of the published versions of the trials. 

We know from the BMJ/Channel 4  investigation they only saw summary tables, and the ghostwriters inserted key messages into the text. The problem with this state of affairs, though, is that we were promised full CSRs, but at present, Roche had only given us access to only 1 of the 4 or 5 modules that each CSR contains.

But how do you appraise and review a trial programme? Single trials can be assessed using tried and tested critical appraisal tools to assess bias, but we had to come up with a “macro tool” to look at the direction and reliability of the whole trial programme. And that also meant its visibility: the way that it was reported, how much was reported - did what we have in our possession constitute the full trial?  Ultimately this meant we were answering two questions. A narrower question: what are the effects of NIs? And a broader question:  what decisions can you make on the basis of the visible evidence? 

In our quest for a tool to assess bias, we were helped by a well-written narrative review by McGauran and colleagues describing the various types of reporting biases in the medical literature, including examples taken from our NI review published in December 2009. 

The narrative was good, if somewhat shocking, for those who read medical journals avidly or regularly. Largely because the examples were brought to light as a consequence of litigation or detailed outside investigations and not by regulators (although some of them might have had a covert hand in one or two cases) or by peer reviewers. 

Aided by other reviews and our own experience, we constructed a table of macro-biases to be tested for. These are in the form of null hypotheses (e.g. “there is no difference between analysis plan in the trial protocol and final report - or the differences are listed and annotated”). The list can be found in Table 1 of our review protocol. Some of the bias names are pretty exotic (such as “swamping bias” when reviewers are swamped by unpublished data and may miss something important as a consequence).

The radical change of approach in conducting the review was somewhat watered down by the insertion of a traditional literature search “to placate Cochrane referees”. Some of us thought this was utterly illogical, but we were outvoted by Cochrane’s Editor in Chief. The reality is we didn’t trust publications anymore. Some of us simply could not take them seriously for reasons which we shall try and explain. 

So far, none of the handful of reviewers who had seen our protocol made any substantive comments on our approach as if we had pulled the methods out of some well-worn manual. However, the outline must have appeared sound enough to the UK’s NIHR, who decided to fund us – even in the midst of a financial crisis. 

The decision to throw out publications was a difficult one - we didn’t initially agree, but with the advent of our meeting in Oxford and Tom’s explosive presentation, we coalesced around a uniform position.