Module 03Literature Search & Synthesis

Learning Outcomes

# Module 3: Literature Search and Synthesis

Estimated time: 3–4 hours

Type: Technical (HTML lab + workbook)

Prerequisites: Module 1 (How LLMs Work) recommended

What you'll produce: A screened bibliography with evidence table and bounded synthesis paragraph

1. The Literature Problem

AI makes literature work faster. That is precisely the problem.

A model can summarise a paper, find related work, and draft a literature review in seconds. What it cannot do — because no model can — is guarantee that:

• Every paper it cites actually exists
• Every claim it attributes to a paper is supported by that paper
• The search it ran found all relevant work (or even most of it)
• The synthesis it wrote accurately represents the field rather than the most-cited or most-accessible papers

These are not implementation problems that will be fixed in the next model version. They are consequences of the architecture: the model predicts statistically plausible text. A plausible-sounding literature review is not the same thing as an accurate one, and the model cannot tell the difference.

This module teaches you to use AI for literature work while keeping the researcher — you — in control of scope, screening, evidence, and claims.

2. Bounded Search

A literature search that "finds everything" is impossible. A search with no stopping rule runs until you get tired. A search with no scope drifts. A search with no provenance record cannot be described in a methods section.

Before you search, define:

Scope

What is the question the search should answer? Be specific. "AI and political science" is not a scope. "How have researchers used LLM-based classification to measure political polarisation in social media text since 2020?" is a scope.

Stopping rule

When do you stop searching? Possibilities: after N results screened, after no new relevant results in the last K records, after a fixed time budget. State the rule before you start; do not adjust it after seeing results.

Query provenance

Record every search you run: the database, the query string, the date, the filters, and the number of results. If someone cannot reproduce your search from your record, your search is not reproducible. Save query snapshots, not just final results.

Inclusion and exclusion criteria

Write your screening criteria before you see any results. A good criterion answers: "What would make me include (or exclude) this paper, regardless of whether I find its conclusions congenial?" Test your criteria on a few papers you already know. Revise the criteria, not the screening decisions, if you discover ambiguity.

3. Screening with Explicit Criteria

Screening is the most consequential step in a literature review — and the one where AI assistance is most dangerous if the criteria are vague or the researcher defers to the model's judgment.

The screening workflow

1. Write your criteria first. Before you see any search results, write down what makes a paper relevant or irrelevant. Be specific. "Interesting" is not a criterion. "Reports original empirical results using LLM-based classification on political text" is a criterion.

1. Screen titles and abstracts only. At this stage, you are deciding which papers to read, not which to include in the final review. When in doubt, retain. A retained paper gets a full-text assessment; an excluded paper does not.

1. Record a reason for every exclusion. "Not relevant" is not a reason. "Does not use LLM-based methods" is a reason. The reason should reference your criteria.

1. Treat missing information as a reason to retain, not exclude. If the abstract does not say whether the paper uses LLM-based methods, retain it for full-text assessment. Do not guess.

1. Verify before citing. A paper that passes screening is a candidate for inclusion. It becomes a citation only after you have read enough of it to confirm it exists, it is what the abstract claimed, and it supports the specific point you are citing it for.

The AI role in screening

You can ask an AI assistant to apply your criteria to a set of abstracts. This can save time — but it introduces a measurement problem (see Module 4 for the full treatment). The AI is making classification decisions, and those decisions can be wrong. Always:

• Screen a sample yourself first, so you know what the task feels like
• Compare the AI's decisions to yours on at least a random subset
• Resolve disagreements yourself — do not default to the AI's label
• Treat the AI's screening as a first pass, not a final decision

4. The Evidence Table

Before you write a synthesis paragraph, build an evidence table. This is the bridge between screening and writing.

An evidence table has one row per source you are citing and these columns:

| Column | What goes in it |

|---|---|

| Source | Full citation (author, year, title, journal/venue) |

| Verified? | Yes (you personally checked the source exists and says what you claim) |

| Claim supported | The specific claim this source supports in your synthesis |

| Evidence type | Empirical? Theoretical? Review? Commentary? |

| Locator | Page, section, or figure where the evidence appears |

| Notes | Any caveats: the evidence is partial, the claim is contested, the methods have known limitations |

Build the table before you write. If a row has "No" in the Verified column, do not cite that source. If you cannot fill the "Claim supported" column for a source, it does not belong in your synthesis.

5. Source-Bounded Synthesis

A source-bounded synthesis is a paragraph (or several) that:

1. States the question the synthesis addresses
2. Reports what the retrieved and screened sources say about that question
3. Stays within the bounds of the search that was actually run
4. Discloses what the search did not cover
5. Distinguishes the sources' claims from your own interpretation

A source-bounded synthesis is not a comprehensive literature review. It is a transparent report on what a specific search found, what those sources claim, and what limitations apply.

Template

What the synthesis should NOT do

• Claim to be "a systematic review" unless it follows systematic review methodology (dual screening, risk-of-bias assessment, registered protocol, etc.)
• Cite sources you have not personally verified
• Draw conclusions that go beyond what the retrieved sources support
• Use the model's summary of a paper as a substitute for reading it

6. Demonstration: A Bounded Literature Search

Let's walk through a concrete example. The research question:

Step 1: Define scope and criteria

Scope: Empirical studies (quantitative or qualitative) examining the effects of deplatforming on content spread, published 2015–2026, in English.

Stopping rule: Screen up to 100 results; stop after 100 or when 20 consecutive results are irrelevant, whichever comes first.

Inclusion criteria:

• Reports original empirical results on deplatforming effects
• Examines content spread, user migration, or community-level outcomes
• Published in a peer-reviewed venue or as a working paper with methods described

Exclusion criteria:

• Opinion, commentary, or legal analysis without empirical component
• Studies of content moderation generally (not specifically deplatforming)
• Studies of platform design or policy without outcome measurement

Step 2: Run the search

Database: OpenAlex (open academic graph)

Query: (deplatforming OR "account suspension" OR "community ban") AND (political OR participation OR expression)

Date: 15 October 2026

Filters: publication_year:2015–2026, type:article

Results: 47 matching works (first page of 20 retrieved)

Step 3: Screen

Of the 20 retrieved records, applying the criteria above:

• Retain (6): Papers with empirical evidence on deplatforming effects
• Exclude (14): Opinion pieces (5), general moderation studies without deplatforming focus (6), platform policy descriptions (3)

Step 4: Build evidence table (partial)

| Source | Verified? | Claim supported | Evidence type | Locator |

|---|---|---|---|---|

| Ali et al. (2025). "Deplatforming and content diffusion on Reddit." ICWSM. | Yes | Deplatforming reduced toxic content in target communities but increased it in neighbouring communities | Empirical (quantitative) | §§3–4 |

| Thomas et al. (2024). "The migration patterns of deplatformed influencers." JCMC. | Yes | Deplatformed users who migrated to alternative platforms maintained approximately 40% of their original audience | Empirical (mixed) | §2, Table 3 |

Step 5: Draft synthesis

7. The Lab

The HTML lab for this module provides a self-contained literature screening environment:

1. Pre-loaded search results: A set of real academic records from OpenAlex (pre-fetched — no API key required)
2. Screening interface: Apply your criteria, record decisions with reasons
3. Evidence table builder: Connect claims to verified sources
4. Synthesis workspace: Draft a source-bounded paragraph

Open lab/literature.html in any browser. All computation is local; no data is sent anywhere. Read the instructor notes at lab/instructor.md for the full walkthrough.

8. The Four Questions for Literature Work

Apply the course lens before, during, and after your literature task:

1. What is being delegated? The search execution? The screening decisions? The synthesis writing? Each has different risks.
2. What would make the result believable? Every cited source personally verified. A search record someone else could reproduce. Screening decisions checked against human judgment.
3. What could be lost or flattened? Non-English literature. Grey literature and working papers. Work from adjacent fields that uses different terminology. Minority or critical perspectives that are less cited.
4. What must remain visible or human? The search provenance. The screening criteria and decisions. The distinction between what the sources say and what you conclude. Every citation.

9. Summary

• Define scope, stopping rules, and screening criteria before you search
• Screen with explicit, pre-registered criteria; record a reason for every exclusion
• Treat AI-assisted screening as a first pass, not a final decision
• Build an evidence table before you write; verify every source
• Write source-bounded synthesis: transparent about what the search covered and what it did not
• The lab provides a complete, self-contained environment — no API keys, no accounts

What's Next

Module 4 applies the measurement principles from this module to text annotation: building codebooks, designing prompts, evaluating reliability, and understanding the difference between agreement and validity.