How we assess the quality of evidence

Evidence syntheses, like scoping and systematic reviews, bring all the studies on a particular issue or intervention together to evaluate what they mean. It’s a process with specific steps designed to minimize bias and to ensure rigor and transparency, so that someone else could replicate the process and reach the same conclusion.

Each of the eight intervention research teams is supported by research synthesis experts. The first critical step is for the authors to create a protocol for each review. This is the roadmap setting out how the review is going to be done, how the reviewers will decide what studies or data to include or exclude in the review, and how those studies and data will be reviewed. 

One particular issue facing agricultural research is that it has fewer randomized control trials than, say, medicine, and needs to be inclusive of many different kinds of evidence and data. This makes scientific appraisal more difficult. For this reason, we are taking a mixed-methods approach, which combines quantitative and qualitative evidence on complex and pressing questions, and which has been successful in previous agricultural systematic reviews. Groups with deep experience in mixed-methods reviews, such as the Campbell Collaboration and the Center for Evidence-Based Agriculture have worked closely with us to explore appropriate methods and offer expert advice.

Once each research team reached a consensus on the protocol, it was published—and it cannot be changed. Publishing ahead of doing the review protects its replicability and transparency. It also gives us a chance to share our work, what we are doing, and allow for scientific dissent as part of the process.

All the protocols have been uploaded to the OSF open-science platform. You can find links to these on each question page.

Evidence synthesis

1. Formulate a research question

2. Search for similar systematic reviews

3. Identify all relevant evidence bases

4. Develop and test search strategies

5. Write inclusion and exclusion criteria

6. Publish protocol

7. Execute searching and screen results

8. Conduct quality of evidence assessment

9. Review and synthesize results