Systematic review & meta-analysis

What is a meta-analysis?

This guide assumes you have never run a systematic review or a meta-analysis before. It starts with the most basic question, namely what these things even are, and builds up one layer at a time, until you can run a complete review in Synthesis with a team and submit the result to a journal.

It is written in two interleaved tracks. The first track teaches the underlying research method, so you understand what you are doing and why. The second track shows you how Synthesis performs each step, so you are never just clicking buttons you do not understand. Look for the two callout styles throughout.

Start with a single study and its problem

Imagine one clinical study tests whether a new drug lowers blood pressure. It enrolls 120 patients, half on the drug and half on placebo, and finds the drug group did slightly better. Is that enough to change how doctors treat patients?

Usually, no. A single study of 120 people can be wrong for ordinary reasons: the patients happened to be unusually healthy, the effect was a fluke of chance, the clinic was unusually good, or the result was simply too imprecise to trust. Any one study, however well run, is a single data point in a noisy world.

The idea behind a meta-analysis

Now imagine fifteen separate teams around the world each ran a similar study of that same drug. Individually each is limited. But if you could carefully gather all fifteen and combine their results into one pooled answer, you would effectively have a study of thousands of patients, far more reliable than any single one.

A meta-analysis is exactly that: a statistical method for combining the numerical results of many separate studies into one pooled estimate. The whole point is to turn many small, uncertain answers into one larger, more trustworthy answer.

Researchers do this because better-pooled evidence leads to better decisions for clinical guidelines, for policy, and for patients. Meta-analyses sit at the very top of the evidence hierarchy for precisely this reason.

Systematic review vs meta-analysis: not the same thing

These two terms are constantly confused, so fix the distinction now:

• A systematic review is the whole rigorous process of finding, screening, appraising, and summarizing all the evidence on a question, following a pre-declared method so it can be reproduced and audited.
• A meta-analysis is the optional statistical step inside a systematic review where you actually pool the numbers. Not every systematic review contains one. If the studies are too different to combine, you report them narratively instead.

Put simply: the systematic review is the entire factory, from raw literature to finished paper. The meta-analysis is one machine inside that factory, the statistics engine. This guide teaches the whole factory, because that is what you will actually run.

The seven-stage pipeline

Every systematic review with meta-analysis follows the same sequence of stages. Learn this map first; the rest of the guide is just each stage in detail.

Stage 1: Frame the question and write the protocol

PICO: turning a vague idea into an answerable question

A research question like “is this drug any good?” cannot be answered systematically because it is not specific. You make it answerable with the PICO framework, which forces you to name four things:

• P, Population: who exactly? (e.g. adults with type 2 diabetes)
• I, Intervention: what treatment or exposure? (e.g. drug X, 10 mg daily)
• C, Comparison: compared against what? (e.g. placebo)
• O, Outcome: measuring what, and when? (e.g. HbA1c reduction at 12 weeks)

Strung together: “In adults with type 2 diabetes (P), does drug X (I) compared with placebo (C) reduce HbA1c at 12 weeks (O)?” That is a question a review can actually answer.

The protocol and why you write it first

A protocol is a written plan that states your question, your search strategy, your inclusion and exclusion rules, and exactly how you will analyze the data, all decided and written down before you look at any results. You then register it publicly, most commonly on a registry called PROSPERO.

This pre-commitment is the single most important safeguard for honesty in a review. If you decide your methods after seeing the data, you can, even unconsciously, pick the rules that produce the answer you were hoping for. Locking the plan first makes that impossible and is what reviewers and journals look for.

Stage 2: Search the literature

Your goal is to find every relevant study, not just the convenient ones. A biased search produces a biased review no matter how good the later steps are.

How a real search is built

You translate your PICO terms into a structured search query using Boolean logic (AND, OR, NOT) and controlled vocabulary. In medicine, these standardized tags are called MeSH terms. You then run that query across multiple databases, because no single database contains everything. The usual set includes PubMed, Embase, the Cochrane Library, and Web of Science, plus clinical-trial registries to catch studies that were run but never published.

You also hand-search the reference lists of the studies you find, and sometimes email authors for unpublished data. A typical search returns several hundred to several thousand records.

Stage 3: Screen and select studies

Screening is a funnel. You start with everything the search returned and progressively remove what does not qualify, in two rounds.

The two screening rounds

1. Title and abstract screening: you read only the title and abstract of each record and decide whether it could plausibly meet your criteria. This is fast and removes the large majority of records.
2. Full-text screening: you read the complete paper of every survivor and apply your inclusion and exclusion criteria strictly. What remains is your set of included studies.

A worked example of the funnel:

Why two people screen independently

Each record is screened by two reviewers working separately, neither able to see the other’s decisions. Where they disagree, a third reviewer breaks the tie. This is called dual or double screening, and the blinding matters: if reviewer two can see that reviewer one already said “include,” they tend to go along with it, and the second check becomes worthless.

Stage 4: Extract the data (the heart of the review)

Extraction is where you pull the raw numbers out of each included study so they can be pooled. The single most important principle to internalize:

Two passes: study characteristics, then outcomes

Extraction has two parts. The first describes the study once; the second captures its results, of which there may be several.

Pass 1: study characteristics (one set per study). This is context: who was studied and how. Typical fields are study design, country or region, setting, enrollment period, total participants, mean age, percentage male, the diagnostic criteria or definition used, and free-text notes. It is also good practice to record the study’s funding source and any conflicts of interest, because these feed into your bias assessment later.

Pass 2: outcomes (one or more per study). A single study can report several outcomes, for example mortality, response rate, and an adverse event, and each can be stratified by its own criteria. For each outcome you record what was measured and the numbers behind it. What numbers you need depends entirely on the type of outcome.

The data you extract, by outcome type

This is the part new reviewers most often get wrong, so go slowly. The kind of outcome decides which numbers you must capture.

Binary (dichotomous) outcomes: did the event happen or not? Examples: died vs survived, cured vs not, relapsed vs not. For a two-group comparison you need four numbers, the events and the group total, for each of the two arms:

Continuous outcomes: a measured value. Examples: blood pressure, weight, a pain score. For a two-group comparison you need six numbers, the mean, the standard deviation (SD), and the sample size, for each arm.

Other outcome types:

• Proportion: events divided by total in a single group (e.g. prevalence).
• Incidence rate: events over person-time observed.
• Single mean: one mean and SD with its sample size.
• Generic effect: used when a paper only reports a pre-computed effect (an odds ratio, risk ratio, or hazard ratio) with its 95% confidence interval, rather than the raw counts.

The details that quietly decide whether your pooling is valid

Beyond the headline numbers, capture these or your later analysis can be silently wrong:

• Timepoint: when was it measured? Pooling a 12-week result with a 1-year result as if they were the same is an error.
• Endpoint vs change-from-baseline: a value at the end is not the same as the change from the start. Never mix them in one analysis.
• Dispersion type: a reported spread might be an SD, a standard error (SE), or a confidence interval, and they are not interchangeable. Record which one it is, and let the engine convert, rather than converting by hand.
• Effect measure: if you record a pre-computed effect, note whether it is an OR, RR, or HR.
• Unit: mmHg is not mmol/L. A unit mismatch ruins a pooled continuous result.
• Derived flag and source quote: if you calculated a number rather than reading it directly, mark it as derived; and always paste the verbatim sentence you took the number from.

Stage 5: Assess risk of bias

Before combining studies, you judge how trustworthy each one is. A flawless 2,000-patient randomized trial deserves more confidence than a small, poorly controlled one. This judgement is separate from how big the study is; it is about how well it was conducted.

You use a standardized tool so the judgement is consistent and reproducible. For randomized trials the common one is the Cochrane Risk of Bias tool (RoB 2); for observational studies, the Newcastle-Ottawa Scale. These rate each study across domains, such as whether the randomization was proper, whether participants and assessors were blinded, whether outcome data was complete, and whether there was selective reporting, and each domain is rated as low risk, some concerns, or high risk.

Note the distinction from statistical weighting: risk-of-bias is your structured judgement of study quality, whereas the weighting in Stage 6 is an automatic mathematical function of each study’s precision. Both matter, and they are not the same thing.

Stage 6: Pool the studies statistically

This is the meta-analysis proper. It happens in three conceptual moves.

Move one: convert each study to a common effect size

Studies report results in different ways, so you first translate each into a single comparable number called an effect size:

• For binary outcomes: an odds ratio, risk ratio, or risk difference.
• For continuous outcomes: a mean difference, or a standardized mean difference when studies used different measurement scales.

An effect size of “no difference” is 1.0 for ratio measures (odds and risk ratios) and 0 for difference measures. That reference point is the line everything is judged against.

Move two: combine them as a weighted average

You do not take a simple average. Larger, more precise studies are given more weight. Formally, each study’s weight is proportional to the inverse of its variance, which means precise studies with tight confidence intervals count for more. A 2,000-patient study moves the pooled result far more than a 50-patient one.

There are two weighting models, and choosing between them matters:

• Fixed-effect model: assumes every study is estimating the exact same single true effect. Appropriate only when studies are very similar.
• Random-effects model: assumes the true effect varies a little between studies (different populations, doses, settings) and accounts for that extra variation. Most real-world medical meta-analyses use this, because identical studies are rare.

Move three: read the forest plot

The result is displayed as a forest plot, the signature image of meta-analysis. Here is how to read one:

• Each horizontal row is one study.
• The square sits at that study’s effect size; the size of the square shows its weight, so a bigger square means more influence.
• The horizontal line through the square is its 95% confidence interval; a wide line means an imprecise study.
• The vertical line down the middle is “no effect.”
• The diamond at the bottom is the pooled result. Its center is the combined estimate and its width is the combined confidence interval. If the whole diamond sits to one side of the no-effect line and does not touch it, the pooled effect is statistically significant.

The two checks: heterogeneity and publication bias

Heterogeneity: should these studies be combined at all?

If one study shows the drug helps and another shows it harms, averaging them produces a meaningless middle number. Heterogeneity is the degree to which study results disagree beyond what chance alone would explain.

You measure it most commonly with the I-squared statistic, a percentage where 0% means the studies are perfectly consistent and 75% or more signals high inconsistency. High heterogeneity is not necessarily fatal, but it is a signal to stop and investigate why the studies disagree, often by running subgroup analyses (for example, splitting by dose or by population) to find the source.

Publication bias: are the missing studies skewing you?

Studies with exciting positive results get published more readily than studies that found nothing. If half the “no effect” studies are sitting unpublished in researchers’ file drawers, the studies you can find will overstate the true effect.

You probe for this with a funnel plot, which is a scatter of studies by size and effect that should look like a symmetric inverted funnel if nothing is missing (asymmetry suggests absent studies), and with formal tests such as Egger’s test.

Stage 7: Rate the certainty and write the manuscript

GRADE: how confident should a reader be?

Producing a pooled number is not the end. You then rate the overall certainty of the whole body of evidence using a system called GRADE, which assigns one of four levels: High, Moderate, Low, or Very Low. You start high for randomized trials and downgrade for problems you already measured: serious risk of bias, high heterogeneity, imprecision, or evidence of publication bias. The GRADE rating is what tells a guideline committee how much to trust your conclusion.

PRISMA: reporting it so others can reproduce it

Finally you write up the review following PRISMA, the reporting standard for systematic reviews. PRISMA is essentially a checklist plus a flow diagram (the screening funnel from Stage 3) that ensures you have transparently reported what you searched, what you included and excluded, and how you analyzed it. Journals expect PRISMA compliance; a review that cannot be reproduced from its own methods section will struggle to pass review.

Working as a team in Synthesis

Real reviews are team efforts, and the integrity of the method depends on people doing distinct jobs. Synthesis supports multiple collaboration roles so that responsibilities, and the audit trail, stay clean.

Why roles matter

The dual-blinded screening above only works if the two screeners are genuinely different people with their own accounts, and if a third person resolves conflicts. The same logic applies to extraction, where two independent extractions catch transcription errors. Roles enforce this separation rather than relying on good intentions.

A practical team workflow

1. The lead defines the protocol and PICO, and invites the team.
2. Two reviewers are assigned to screen each record independently and blind.
3. A senior reviewer resolves any screening disagreements.
4. Two extractors independently extract data from included studies; disagreements are reconciled into an agreed consensus record.
5. A statistician or the lead runs the pooled analysis and the two checks.
6. The team appraises risk of bias and agrees the GRADE rating.
7. The lead generates and edits the manuscript, with the team reviewing before submission.

Your end-to-end checklist

Use this as a single reminder of the whole journey from question to submission.

Glossary