Applied Evidence Based Medicine

Understand where to look for systematic reviews and protocols

Understand what's involved in locating studies for a systematic review that you are working on

"A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making." (from the Cochrane Library)

Let's break this definition down:

Identify: The literature search is the data collection method for a systematic review and is, therefore, key to the quality of the review. Authors attempt to find every study (whether completed or ongoing, published or unpublished) that has been done on a particular intervention or question. They search literature databases, such as Ovid MEDLINE or PubMed; clinical trial registries; and sources of "grey" literature, such as government publications. The search strategies used for systematic reviews are more complex and comprehensive than those taught in this course. Higher quality systematic reviews usually include a librarian on the search team to ensure a thorough and rigorous search.

Appraise: After the relevant studies have been identified, they are appraised using standard tools specific to the type of study (e.g., RCTs, cohort studies). Generally, only studies considered by the authors to be high quality should be included in the final review. In the absence of high-quality studies, lower-quality ones may still be reviewed, as long as the review's authors discuss the limitations of those studies.

Synthesize: The purpose of a systematic review is to draw conclusions based on the studies reviewed and to explain inconsistent or contradictory findings. Where the data are quantitative and homogenous, the review may include a meta-analysis. This involves extracting and combining the data from different studies to create a single large data set from which larger conclusions can be made. Meta-analyses are complicated and should only be undertaken by individuals with appropriate training and expertise in statistical analysis.

Systematic Review Protocols and Health Technology Assessments

A protocol is a document outlining a plan to carry out a systematic review. It usually includes a preliminary literature review, and some indication of the methods used to locate, select, and appraise the research that the review team finds. Protocols are registered in order to place "dibs" on a particular question and let others know that you are working on this issue, so efforts are not duplicated.

A health technology assessment (HTA) is similar to a systematic review, but is usually carried out by a health care organization to inform its own policy around the use of a drug, device, or other healthcare technology. Because there is an immediate policy need, an HTA may include evidence of lower quality, as well as expert opinion, in its analysis. It also tends to place more weight on context. While most systematic reviews concern themselves strictly with medical evidence, HTAs also study "the social, economic, and ethical implications of investments in health care" (from the International Network of Agencies for Health Technology Assessment).

A systematic review is a lengthy, rigorous process that involves a team of experts in areas like literature searching and statistical data analysis. It should not be undertaken lightly. However, some AEBM students choose to do a systematic review for their research elective. If you have a very focused question with a finite body of literature to analyze, it is possible to do a systematic literature search and a basic review of the selected studies. The library can help you with developing your search strategy. Contact hslibr@ucalgary.ca to book a consultation.

The research guides below also offer guidance for researchers undertaking systematic reviews: