We trundled along to 2009, when the world confronted the first influenza pandemic of the century. 

But everything was under control -  the supplies of Tamiflu were protected by an armed guard. The drug had taken on a near-mythical status as the pandemic’s magic bullet.  Specially guarded sites would hand out the drug, and to further heighten its value, the word on the street was the ‘thieves’ were out to ‘target official distribution points.’  Apparently, it would sell on the black market for a fortune.  The upcoming UK Civil Contingencies Committee (COBR) would iron out any problems. In April, it would decide when and to whom to hand out the wonder drug. 

In reality,  it wasn't much of a pandemic, but it was enough to trigger the pre-arranged plans and, on a tiny scale, the updating of our Cochrane review. One thing puzzled us, though: how exactly did the miracle status of Tamiflu emerge? 

Expert opinion certainly played a role in this status. Professor John Oxford was one of the essential experts in ramping up the hype. An English virologist and, at the time, professor at Queen Mary, University of London, set out in the Journal Expert Review of Anti-infective Therapy that, in his view, the first two designer antiviral drugs were a ‘ground breaking advance.’  Oxford expected the neuraminidase inhibitors would be effective against the influenza pandemic strain - quoting the most well-known studies of the time also included in our reviews. 

Our Cochrane review (A047 for short) had been updated in 2006; as we mentioned in Antivirals 2, it had been co-published in the Lancet.  On 23 Dec 2008, we further updated the review. This latest update didn’t add much more to the 2006 review. It cleaned up the text but added no new studies.

The onset of the Pandemic necessitated another update - was there anything new? By December 2009, we’d published an update in the BMJ and in Cochrane in 2010. At the time, the Cochrane Library only updated four times a year: it wasn't in print and came by snail mail on a CD, so we took a pragmatic decision to turn to the BMJ to publish the update by the year's end.  

The conclusions had become more supportive. In 2006, we reported that ‘neuraminidase inhibitors should not be used in seasonal influenza control and should only be used in a serious epidemic or pandemic alongside other public-health measures.’

However, by 2009 we concluded they had ‘modest effectiveness,’ and "the drugs were effective post-exposure against laboratory-confirmed influenza, but this is a small component of influenza-like illness, so for this outcome neuraminidase inhibitors are not effective.’

We were perplexed by the seemingly neutral or negative acceptance of the results of our review. The chief complaint appeared to be that we had found no evidence of effect against “bird flu”.  This was correct. None of the trials meeting our inclusion criteria had been carried out on people exposed to influenza H5N1. 

The comment belied a fundamental ignorance of how systematic reviews work: authors are bound by pre-specified inclusion criteria, which cannot be changed (as we shall see) without introducing unfathomable amounts of bias. No H5N1 influenza virus was in the trials, so none was reported.  We were also puzzled as to how to look at the harms, which had been underinvestigated and underreported. This was important as reports coming out of Japan (the world’s largest consumer of Tamiflu) seemed to indicate bizarre neuropsychiatric effects in minors.

As we were gathering our wits to update the review (funded by the UK’s NIHR and the Australian NHMRC) on 14 July 2009, we received a comment through the Cochrane post-peer review system from a Japanese paediatrician. Dr Hayashi turned everything we knew about the publication of trials results on its head.

Share

What became known as Haysashi’s comment and then Hayashi’s problem questioned the veracity of the evidence we had included in our updates.

‘We have some questions on the conclusion in your  Oseltamivir review, especially about the prevention of complications. You  described that “Oseltamivir 150 mg daily  prevented lower  respiratory tract complications (OR 0.32,  95% CI 0.18 to 0.57).” (in the abstract). However, we have found that this conclusion is based on the other  review (Kaiser 2003)  and not on your own data analysis.’

Hayashi’s criticism centred on one paper, a meta-analysis of the effects of oseltamivir on complications of influenza, published in the Archives of Internal Medicine. Only two of the ten randomised datasets had been published. Hayashi was rightly concerned the information was insufficient to assess the methods, reliability, and applicability of the eight unpublished datasets.

Hayashi also pointed out the concerning nature of the author list of this review: four were employees of F. Hoffman-La Roche Ltd, one paid consultant to F. Hoffman-La Roche Ltd, and the review was lacking raw data. 

‘Only two RCTs  (Nicholson 2000 and Treanor  2000) were published , but other 8 RCTs were proceedings of  congress (5  RCTs), abstracts of the congress (one RCT) and meeting  (one RCT) and data on file, Hoffmann-La Roche, Inc, Nutley,  NJ (one RCT). The lower respiratory tract complication rates  of these  articles were summarized on table: there was no significant  difference between Oseltamivir and placebo, and  their Odds Ratios  (ORs) were 1.81. But ORs of the other 8 RCTs were 4.37.’

We strongly suppose that the reviewer’s conclusion about the  complications was mainly determined by these 8 RCTs, we should  appraise the 8 trials rigidly. Without this process it’s difficult to  conclude that oseltamivir can prevent lower respiratory tract  complications.’

Then a review similar to ours but on children was published in BMJ in August. Its conclusions were more conservative than ours: ‘Neuraminidase inhibitors provide a small benefit by shortening the duration of illness in children with seasonal influenza and reducing household transmission. They have little effect on asthma exacerbations or the use of antibiotics. Their effects on the incidence of serious complications, and on the current A/H1N1 influenza strain remain to be determined.’

In the course of the updates, we wrote to Roche to see if any further trials had been published since 2005. This was a routine enquiry when conducting an update before the widespread use of trial registers; no one knew what trials had been done by whom. Politely, Roche sent us the reprints of what they considered were the pivotal trials of their drug. Biggish trials published in prestigious journals: New England Journal of Medicine, JAMA, The Lancet and Annals of Internal Medicine. The authors were well-known scientists with a long track record in influenza research. 

At this time, Roche had the opportunity to send us the unpublished data from the eight trials, but they made no mention of them and disclosed none of the data.

We were still working out how to respond to Hayashi’s comment. One answer was to ask the authors of the review and meta-analysis: Professor Laurent Kaiser and his co-author Professor Frederick Hayden.

On the 10th of August, Tom wrote to Professor Hayden: we needed to understand the raw data to allow us to either expand and reinforce the current conclusions or change them altogether. 

Four days later came the answer:

‘Dear Tom and colleagues, apologies for this late reply.  I have searched but cannot find the original files related to this 2003 publication. Before and again after my 2+ years at WHO in Geneva, I was obliged to move offices at the University several times and downsize.  The files appear to have been discarded.  My co-author Laurent Kaiser, now professor at the University of Geneva, is copied on this reply, as he may have his own sources.  The questions posed by the inquirer are not clear to me, but if original data or unpublished study reports are required, they will likely need to come from Roche, the sponsor of these studies.  At present my own interest is in finding good quality data about antiviral treatment responses in pandemic H1N1 infections, especially in those with more serious illnesses, and the effects of treatment in those hospitalized with seasonal or pandemic H1N1 infections. Best regards, FGH

Share

Professor Kaiser responded  three days later: ' 

- The original idea, the main question, the selection of the published and unpublished studies and the design of the analysis have been elaborated by Fred and I. Then we had to convince the Roche's people. Thus this has been done thanks to a close collaboration with the Roche's group that pooled the published and some unpublished data and then conducted the analysis according to our needs. Each step was validated independently of the Roche's team by ourselves and we then drafted the MS. For this reason and since the analysis was performed by Roche's statisticians the database everything was centralized there. This study was a long quest that we tried to initiate in 1999 following a similar analysis for zanamivir (when nobody was interested in this type of analysis). Data on AB use and complications were collected prospectively but to the best of my knowledge this was not a predefined primary end-point in most studies.Given the organization I do not have the files anymore unfortunately (that was mainly paper files). 

Of note these studies focused mainly on previously healthy adults and adolescents and unfortunately this type of endpoints has not been included as a primary end-point in any subsequent large study or in hospitalized patients.  

- I suggest contacting Roche directly to get access to the files.

So it appeared that the only people who could answer Hayashi’s question were the Roche personnel who had carried out the crucial analysis, as neither the first nor the second author had seen the data. The Kaiser meta-analysis reports it was supported by F. Hoffmann-La Roche Ltd, Basel, Switzerland. The authors acknowledged help with statistical analysis provided by Vivien Mitchell, MSc (Roche Global Development). But there is no indication in the Kaiser paper that the authors had no access to or never saw the data. 

The Kaiser meta-analysis reports it was supported by F. Hoffmann-La Roche Ltd, Basel, Switzerland. The authors acknowledged help with statistical analysis provided by Vivien Mitchell, MSc (Roche Global Development). But there is no indication in the Kaiser paper that the authors had no access to or never saw the data. 

But what about the first authors of the two published trials in the Kaiser meta-analysis? Professor Treanor did not respond; Professor Nicholson did:

Dear Tom,

Raw data for multicentre studies like this would be collated and stored

by the Industry. I suspect that you will get a similar response from

both XX and XX.

A047 was withdrawn in 2010 and overtaken by our A159 strategy. We met in September in New College, Oxford and agreed we couldn't sustain the review based on what had been published to date.  In the BMJ, we noted, ‘we were left feeling that conclusions drawn from only a proportion of all existing trials (that is, just the published ones) are wholly inadequate.’ 

In written evidence to the UK’s Science and Technology Committee, we reported that the result of our efforts was ‘a refusal to consider published trials (either on their own or as part of reviews) for inclusion in our Cochrane reviews.’ Everything we had learnt about published evidence was now out the window. In 2011, A047 was withdrawn from The Cochrane Library as we commenced our journey to find the hidden data.