I am writing to let you know that we are starting work on a further update of our Cochrane neuraminidase inhibitor review. We are going to conduct the update alongside our fellow reviewers of the paediatric dataset. Because of the presence of publication bias, reporting bias of harms, lack of access of authors to "their own datasets", inconsistencies of the data presented to different regulators and these data with the published reports, we have decided to ignore the published evidence initially and concentrate on the regulator and internal Glaxo and Roche reports. 

We are planning on submitting and publishing our protocol together with an analysis piece explaining our rationale in the BMJ as well as in the Cochrane Library.

Any thoughts would be very welcome and would be treated in the strictest confidence (freddyflu is a dead email box created to facilitate keeping a log of traffic on the review).

With best wishes, Yours, Tom.

Share

Given our lack of experience working with Clinical Study reports (CSR), we were fishing for input.  We had changed our position: journal publications, given all their problems, were to be ignored. From now onwards, we had to change our language.  Terms like CSR, with which we were unfamiliar, became our new lingo; as the narrative unfolds, we will explain, but this interactive glossary and taxonomy will help. We had to do a language conversion “course” the hard way, hence the glossary.

We also made the momentous decision to re-write the protocol to include the same types of studies (placebo-controlled randomised trials) exclusively from regulatory sources. I.e. to use only those documents that we now knew were CSRs.

A CSR is an unabridged report of a clinical study written for regulators following E3 reporting guidelines developed by the regulatory industry.  They represent the most complete reporting of the planning, execution, and results of a clinical trial. While they contain some of the same information as a journal article, they are substantially more detailed, with numerous large tables, figures, and datasets not constrained by page limits. 

Above the water line is the research you can see - the journal publications, maxing out at about 3,000 words with tables and appendices and only representing a small amount of the details of a trial. This issue leads to substantial problems with reporting bias: ‘a systematic distortion that arises from the selective disclosure or withholding of information by parties involved in the design, conduct, analysis, or dissemination of a study or research findings.’

Reporting bias is a massive problem in medical research, which includes withholding study data, underreporting of harms and the active attempt by manufacturers to suppress the publication of findings.  Consequently, journal articles overestimate the effects of an intervention and underestimate the harms. 

We no longer trusted any of the publications of the Tamiflu trials (nor the people named as “authors”) and had no intention of contaminating our review with spun studies. It’s, therefore, not that much of a leap to ditch the journals and focus solely on the CSRs that could run into thousands of pages. 

Original drawing Yuko Hara: BMJ 2013;346:f4223

Share

While we waited and worked our new methods out, the Seventh Cavalry was about to arrive in the shape of the Danish Cochrane Centre. There had been a landmark recommendation by (of all institutions) the EU, which made accessing CSRs possible. But we only realised this when Deborah Cohen told us.

Why was the Nordic Cochrane Centre so crucial to our story? On 29 June 2007, the Nordic Cochrane Centre asked the EMA to provide access to the clinical study reports and their corresponding protocols for two anti-obesity drugs. Over the following three years, the EMA put forward several arguments to avoid disclosing the documents: protection of commercial interests. EMA also did not respond to the Ombudsman’s letters by the deadline. The EU Ombudsman is like a court of last resort for matters regarding administration.

On 7 June 2010, the Ombudsman publicly accused the EMA of maladministration because it refused to grant access to the CSRs. Finally, on 31 August, EMA caved in and informed the Ombudsman that it would provide access. By November, EMA formally announced its change of policy which was fully articulated four years later. 

As quickly as possible, we emailed EMA and asked for their trial holdings for Tamiflu and Relenza. Tamiflu had been centrally approved so they could send us what they had, but Relenza had been approved by a National Competent Authority (NCA), so they couldn’t provide us with data they did not hold. NCA is another piece of slang - do go and consult the glossary. Translated into English, it means your national regulator. MHRA in the UK, AIFA in Italy, MEB in Holland, etc. Because of EU law, if MEB grants a licence to X, then the licence is recognised in all member states. So EMA did not have Relenza documents, only Tamiflu.

We were starting to get somewhere, but we weren’t ready for - or aware of - what was about to be dumped into our inboxes.