I have touched on this topic briefly in some of our other posts, [e.g. 1, 2 & 3] but following a recent discussion I decided to explore the topic of irreproducible results further and encourage conversation. During a recent discussion I was told that researchers with a well-established quality system in place were considering dropping it because funders of research did not take it into consideration when awarding grants. Laboratory quality systems help to ensure accuracy, reliability, and timeliness of reported test results. At NFMRI we ask researchers to identify any quality systems including ISO, GMP and GLP that they have in place. There are many reasons why quality systems can be helpful to the translation of research including; confidence in the robustness of methodologies and results, regulatory requirements, attracting potential collaborators, and the training of researchers who may wish to be more competitive with their career opportunities and move into industry. Whilst there are many reasons why research results may be irreproducible the introduction of quality systems can assist to increase the ‘research quality’ of the studies being undertaken. ‘Research quality’ in this case refers not to the importance of the discovery, but the robustness and usability of the results. This focus can be lost when funders of research focus only on “sexy” publishable research and not on other aspects of quality. There are examples of important published discoveries that have been retracted at a later point, as the research could not be reproduced. As it is not a requirement to publish whether or not quality systems are in place and complied with, few academic laboratories have such systems in place. It is unlikely that fraud is a significant issue but that other factors are responsible. One contributing factor may be the focus on publications as discussed “Looking back through the microscope”. In November 2104, Nature Journal published a special issue on the “challenges of irreproducible research” including how they are addressing the issue. Funders of research should also consider how they too influence policy and administration decisions that support research. The global need to address the issue of irreproducible results is clear. In June 2015, the Boston Globe reported on a study that claimed $28 billion is spent in the United States each year on preclinical research that can’t be reproduced by other researchers. This study examined around 24 other studies and concluded that around 53% of all preclinical studies have errors that mean they are not reproducible. 
Earlier studies found that only 20–25% of published data associated with new drug targets were consistent with their in-house findings and the landmark Amgen study found that only 6 of 53 preclinical cancer papers could be reproduced in the company’s labs. The impact of non-reproducible papers is further exacerbated when the hundreds of future articles and citations are considered. Funders of medical research should remember that researchers and their institutions will put extra effort on what is measured and rewarded. If the primary focus is increasingly on headline grabbing new discoveries, then the direction is clear. Worst-case scenario is that translation becomes obsolete if funders of research only reward the opening chapters on a story of discovery and the confidence of potential collaborators is eroded. “Do you know if you’re asking the right questions, including those beyond research, to support the delivery of community benefits?” Join the conversation online and at our conference ‘putting rubber on the road’ on September 9/10th 2015 at the Australian Maritime Museum.
I also agree that the impact of irreproducible research should not be underestimated, especially in the context of its impacts on the health and wellbeing of communities.
Whilst peer review of papers is integral to the evaluation of research, it does not attest to the reliability and reproducibility of the findings and data on which the conclusions of the research are based.
Some years ago in response to stakeholder demand, NATA developed a program for the accreditation of research that is based on various international (ISO) standards. Facilities accredited in the R&D Program value accreditation as a means for: ensuring that technically robust and traceable practices are in place; providing opportunity to evaluate and manage the risks inherent in research; and for providing a level of certainty in the process, even though results/outcomes are unknown, without hindering the freedom or direction of the research. It is also seen that accreditation assists with collaborations nationally and globally and with the translation of research to industry.
A barrier to facilities pursuing R&D accreditation seems to be the perception of the cost attributable to introducing a quality management system. The initial and ongoing cost of the accreditation process needs to be separated from the costs of good/best practice which we all presumably, are aiming for. Furthermore, linking the tangible benefits of R&D accreditation to the funding of research continues to be challenging.
We definitely need hearty discussion with the funders of research about “how they too influence policy and administration decisions that support research”.
I agree, the impact of unreproducible research should not be underestimated. Not only does it waste valuable research funding but it can lead to additional wasted funding as other groups attempt to build on flawed data. That said, introducing quality systems in basic research incurs cost and this cost may place research groups using them at a disadvantage as they use up their limited funds on ensuring to data is fit for purpose. That being the case, in my opinion we need to look to funders and publishers of research for leadership. I would agree that there is merit in making improvements by including an assessment of the quality system in a grant application review. Obviously imposing a quality standard might lead to the need for auditing and if there is a standard there also needs to be consequences for not adhering to them. Such measures will add cost into research system but you need to weigh this cost against the impact of wasted funding highlighted above. You only need to look as far as industry who clearly see the value of investing some of its research funds to quality systems. They know that money spent on ensuring any data they produce is reliable improves their return on investment. I would even go further, and argue that if you accept that the problem of unreproducible research is truly global, a nation-wide approach to data reliability might provide a national competitive advantage as overseas funders such as industry might be encouraged to invest in an innovation system that takes a systematic approach to the issue.
On a practical point, in many cases it may be difficult for researchers to employ GxP or total quality system in an academic research setting given the costs involved. A pragmatic alterative might be a Quality by Design (QbD) approach. QdD is very much in favour with the global therapeutic regulatory agencies and, in the research environment, might be employed to systematically identify potential sources of irreproducibility and take steps in the experimental design to minimise them. Such an approach might be required by funders at the application stage and then verified upon publication. Its use might be expected to lead to greater standardisation of experimental approaches leading to increased reproducibility between labs. To be clear, I am not advocating for the removal of a research labs ability to design innovative experiments, but to a more deliberate consideration of factors that leads to irreproducibility and, ultimately, increasing the return on investment on research funding.
I could not agree more. ISO has had available for some time, a research standard and it is unacceptable that Universities and similar research entities have not been obliged to adhere to this as a prerequisite for research funding. Peer review of papers is not a substitute for adhering to an objective quality system. We insist on GLP/GMP for data supporting product registration, why should this not apply to all research? Perhaps if the journals started to insist on a quality system being in place in laboratories from those seeking to publish, we might get somewhere. Funders of research have a right to demand evidence of quality management for work they are funding!