Systematic reviews and Meta-analysis in Business/Management

Developing and refining your research question is one of the most important steps in a synthesis project. The question should be well-defined, with an appropriate and reasonable scope.  

Question frameworks can be used to help focus your research question. Two examples of frameworks relevant to management questions are presented below.

PICO

P - Population (this can be groups or categories of individuals or organizations, such as knowledge workers, post-secondary students, or SMEs)

I/E - Intervention or Exposure (this is your independent variable)

C - Comparison (if relevant)

O - Outcome of interest (this is your dependent variable)

Example question: Does gamification of training programs (I) increase motivation for compliance training (O) than non-gamified approaches (C) in banking and insurance industry employees (P)? 

CIMO

C - Context (individuals, groups of individuals, institutions, etc. Similar to population in the PICO) 

I - Intervention (event or action being studies; independent variable)

M - Mechanism (that explain, bridge, or enable the intervention to lead to the outcome)

O - Outcomes (measurable effects; dependent variable)

Eligibility criteria determine which studies will be included in your systematic review, and which studies will not. They may sometimes be referred to as inclusion/exclusion criteria.

1) Eligibility criteria related to PICO elements. These are additional details about each element that show where the boundaries lie and what variations are acceptable. Criteria include:

• Population characteristics (adult, child, employee sub-group, etc.)
• Geographic region 
• Setting (organizational, public services, educational setting, etc)
• Reported outcomes (self-report instruments versus objective measures)

Example

A systematic review is being conducted that looks at "what factors increase the likelihood of X in leaders". Leaders are the population (P) in the research question.

You might decide to specify further that: 1) leaders in public service organizations will be excluded (since they do not operate in a similar environment), and 2) leaders in private corporations have to be at the level of middle management or higher in order to be included. 

2) Additional criteria. These are some commonly used criteria that may be used to determine study eligibility:

• Study design (interventional, observational, cross-sectional, case study, etc)
• Sample size (minimum sample size for a study to be included)
• Date of publication 
• Language of publication
• Data type (scholarly only, or scholarly and trade, but not dissertations, etc)
• Type of publication (empirical studies, theoretical, editorials, etc)

Eligibility criteria should be pre-specified at the protocol development stage, before your data collection (search for studies) has been completed. Your comprehensive database search strategies may incorporate some of the specified eligibility criteria (such as publication date, language, publication type, etc)

Changing criteria after the results are known may introduce bias, and should be avoided. If you change your criteria at a later stage, you may need to revisit your database search strategies, as they may be affected.

Additionally, you must report (in your final manuscript) any changes that were made in a effort to be transparent, and provide justification for the changes.

The review protocol

A primary study would never be conducted without a research plan; similarly, evidence synthesis should not be undertaken without a research protocol. A protocol describes the planned methodology and intended process for the various steps of your review, including:

• The research question
• Data collection
• Eligibility criteria and study selection process (how you will select studies for inclusion)
• Data extraction
• Critical appraisal 
• Synthesis and/or meta-analysis

PRISMA - P - This checklist describes the items and level of detail that needs to be reported in a protocol for a systematic review. 

Registering/publishing your protocol

It is a good idea to have your protocol done before you begin collecting data for your review. This shows that you did not hypothesize after you knew the direction of the effect or what the results would show, and in this way reduces the opportunity for reporting bias. In some disciplines, journals may require proof that your protocol was pre-registered. 

Organizations such as Cochrane, Campbell Collaboration, JBI and others that conduct evidence synthesis publish their own protocols. However you can also upload a copy of your protocol to an open access registry or even your institutional repository.

Open Science Framework - You can pre-register your protocol on the Open Science Framework registry. It can be submitted for blinded peer-review, or can be embargoed for a set period of time (maximum 4 years).

PROSPERO - This is an international database of prospectively registered systematic reviews. Their disciplinary coverage includes: health, social care, welfare, public health, education, crime, justice, and international development, where there is a health related outcome.

Similar to other research projects, you should make a plan for how you will manage the vast amounts of data acquired during the course of your review project. Some of the types of data you will acquire/create are:

• Exported files from each database
• Citation libraries (in your citation management software)
• Article PDFs
• Data extraction files (Excel, Word docs)
• Notes (documenting each step of the process)

Data management tips

1. It is good to have a central backup of all data from your project, regardless of which member of the review team is responsible for acquiring or creating that particular type of data. This ensures that the team can always access a copy of all data, in the case of a technology accident (laptop crash, lost USB) or if someone needs to be away for an extended period of time or a personnel change.
2. Name your files appropriately. It is good to think about a naming convention for the files themselves, as well as the folders in which you will organize your backups. Remember that 12 months from now, you may not be able to remember what 1.ris means.
   • You may want to use a multi part name which includes aspects such as: the project name, step of the review process, database name, individuals name, date, etc.
   • For naming files exported from a database during the search stage, you might name your file ProjectName_NameofDatabase_DateofSearch_#ofRecords.
   • For naming the files that were used to pilot the screening criteria, you might name your file ProjectName_PilotScreening_#ofRecords_Date_InitialsofScreener
   • For naming each of the full-text article PDFs, you might consider naming your files  FirstAuthor_Year_FirstFourWordsOfTitle