← course index

Module 04Text Annotation & Measurement

⏱ 3–4h 📋 technical HTML lab 📌 Module 1 recommended 🎯 Miniature annotation study with reliability evidence

§ workbook

Learning Outcomes

# Module 4: Text Annotation and Measurement

Estimated time: 3–4 hours

Type: Technical (HTML lab + workbook)

Prerequisites: Module 1 (How LLMs Work) recommended; Module 7 (Validation) helpful

What you'll produce: A miniature annotation study with codebook, model labels, reliability evidence, and validity limits

1. Annotation as Measurement

Social scientists classify text constantly. Is this news article about economic policy? Does this tweet express support or opposition? Does this legislative speech invoke a particular frame?

When you ask an LLM to do this classification, you are measuring a construct — the same way a survey question measures an attitude or a census category measures an identity. The fact that the measurement is performed by a model rather than a human coder or a dictionary method does not exempt it from measurement standards.

The key standards are:

  • Agreement: Do different coders (or runs) produce the same label for the same text?
  • Reliability: Is the agreement higher than chance, and is it consistent across the full range of texts?
  • Construct validity: Does the label actually capture the concept you intend to measure?
  • Consequential validity: Do the errors that remain change your substantive conclusions?

A model that labels 92% of texts "correctly" may still produce biased estimates if the 8% of errors are concentrated in one category, or if "correct" means "agrees with a single human coder whose own reliability is unknown."

2. Human Judgment Before Model Judgment

This is the most important rule in annotation work — and the one most often violated:

Make your own judgments before you see the model's labels.

Once you have seen a label, you are reacting to it — defending it, questioning it, accepting it. Before you see it, you are producing one. The difference matters because:

  • You discover ambiguity in your own codebook when you try to apply it
  • You notice edge cases that the codebook does not address
  • You form a view about which cases are difficult, which informs your reliability assessment

The lab enforces this order: you define your codebook, code a set of items yourself, then compare your labels to reference labels and (optionally) model-generated labels.

3. The Codebook

A codebook is a document that operationalises your construct for classification. A good codebook contains:

Categories

Each category needs a name, a definition, and decision rules. "Positive," "Negative," and "Neutral" are not definitions. A definition states what makes a text belong to that category:

Support: The text explicitly advocates for the policy, expresses approval of it, or argues that it should be adopted. The support must be directed at the policy itself, not at a related goal or value.

Decision rules

Rules for resolving ambiguity. What do you do when:

  • A text mentions the policy but expresses no position? → Neutral
  • A text supports a related policy but not the target policy? → Not applicable
  • A text is ironic or satirical? → Exclude (or code the literal meaning with a flag)

Boundary cases

Examples of texts that are difficult to classify, with the correct label and the reasoning. These are your training material for human coders and your test material for model evaluation.

The codebook is YOUR instrument

Do not ask a model to write your codebook. The codebook records your theoretical decisions about what the construct means. You can ask a model to suggest edge cases or to check whether a definition is ambiguous, but the categories, definitions, and decision rules must be yours.

4. From Codebook to Prompt

Translating a codebook into a prompt is itself a methodological decision. Different prompt structures can produce different labels from the same codebook — this is the "prompt variation" check from the validation menu (Module 7).

A good classification prompt includes:

  1. The task description: What you are asking the model to do
  2. The categories with definitions: Copied from your codebook
  3. Decision rules: Especially for ambiguous cases
  4. Output format: Specify exactly what the model should return (e.g., "Return only the label, one per line, in the same order")
  5. What NOT to do: Explicitly forbid explanations, confidence scores, or extra columns unless you want them

Example prompt structure

```

Classify each text below according to the following codebook.

CODEBOOK

[Category definitions and decision rules]

RULES

  • Return exactly one label per text, in the same order
  • Use only the labels defined above
  • If the text does not contain enough information to classify, use "INSUFFICIENT"
  • Do not explain your decisions
  • Do not add confidence scores

TEXTS

[Text 1]

[Text 2]

...

```

5. Evaluating Annotation Quality

Agreement and reliability

How often do two coders (or two model runs) agree? Simple percentage agreement is a start, but it does not account for agreement that would occur by chance. Cohen's kappa and Krippendorff's alpha adjust for chance agreement. The lab calculator provides these metrics.

A kappa of 0.80 is a common threshold for "acceptable" — but this is a convention, not a law of nature. The threshold that matters depends on your research use. If you are using the labels to filter a dataset for qualitative analysis, 0.70 may be fine. If you are using them as an independent variable in a regression, 0.70 may introduce enough measurement error to bias your estimates.

Construct validity

High agreement does not mean you are measuring the right thing. Two coders can agree perfectly that a text is "positive" while both misunderstanding what "positive" means in your theoretical framework. Construct validity requires evidence that the labels correspond to the concept you intend to measure — typically through comparison with an external criterion or through theoretical argument about what the labels should correlate with.

Consequential error

The question that matters most: do the errors change your conclusions?

Egami et al. (forthcoming) show that even high-accuracy LLM labels can produce biased regression estimates if the errors are correlated with the outcome or the treatment. A model that is 92% accurate overall but systematically misclassifies theoretical papers (which tend to make different substantive claims) will bias a literature review toward empirical findings.

Always test: run your analysis with the LLM labels and with a human-coded subset. Does the substantive conclusion change? If yes, the labels are not good enough for your use case, regardless of the accuracy number.

6. The Lab

The HTML lab (lab/classification.html) provides a complete annotation environment:

  1. Codebook editor: Review and revise the supplied codebook
  2. Human coding interface: Code items yourself before seeing any labels
  3. Model request builder: The lab constructs a prompt from your codebook and the text items — paste it into any AI chat interface
  4. Scoring calculator: Compare your labels (or model labels) to reference labels; compute accuracy, precision, recall, and F1 per category
  5. Baseline comparison: See what "retain all" and "exclude all" baselines would achieve
  6. Screening handoff: Export your decisions for the next stage of review

All computation is local. No data is sent anywhere. The dataset is constructed teaching data (not real publications), so you can experiment freely without misrepresenting real research.

7. The Four Questions for Annotation

  1. What is being delegated? The classification decisions for each text. You retain the codebook design, the validation, and the interpretation.
  2. What would make the result believable? Human-model agreement on a representative sample. Reliability above chance. No systematic error pattern that biases the downstream analysis. Evidence that the labels capture the intended construct.
  3. What could be lost or flattened? Ambiguous texts forced into categories. Minority or unexpected uses of language. The tacit knowledge you used when coding by hand.
  4. What must remain visible or human? The codebook (your instrument). The comparison between human and model labels. The error analysis. The decision about whether the labels are good enough for your use.

8. Summary

  • Annotation is measurement. Apply measurement standards: agreement, reliability, construct validity, consequential error.
  • Always code items yourself before seeing model labels. The order matters.
  • Your codebook is your instrument. Do not delegate its design to a model.
  • The prompt is part of the method. Vary it and check whether conclusions change.
  • High accuracy can mask systematic errors that bias downstream estimates. Test the full pipeline.

What's Next

Module 5 extends these principles to code and data analysis: using AI to write, debug, and verify analysis scripts. The validation framework from this module applies directly — every AI-generated statistic needs the same scrutiny as an AI-generated label.

If you want the full treatment of validation and reproducibility, Module 7 covers the evidence framework in depth, including the seven validation checks, data boundaries, and disclosure standards.

Module 4 of 8 · AI as a Research Tool: Workflow Design for the Social Sciences · University of Zurich

§ exercises

# Module 4: Exercises

Exercise 4.1: Code Before You See

Goal: Experience the difference between forming a judgment and reacting to one.

Open the lab at lab/classification.html. Navigate to the "Code it yourself" section. You will see 6 short texts (titles and abstracts of constructed teaching records). Before looking at any reference labels or model output, classify each text using the supplied codebook.

For each text, record:

| ID | Your label | Your reason | Confidence (1-5) |

|---|---|---|---|

| | | | |

| | | | |

| | | | |

| | | | |

| | | | |

Now compare your labels to the reference labels (provided in the lab). For each disagreement:

  1. Did you misapply the codebook, or is the codebook ambiguous?
  2. Does the reference decision make sense to you, or would you defend yours?
  3. What would you change in the codebook to resolve the disagreement?

Answer Key (click to expand)

The reference labels are in the lab (under "Reference Labels"). Specific answers depend on the dataset version, but general patterns:

  • Disagreements that reveal codebook ambiguity are findings, not errors. If you and the reference disagree and you can both justify your decisions from the codebook, the codebook needs clarification.
  • Confidence matters. A low-confidence correct label and a high-confidence wrong label are different problems. The first suggests the task is inherently difficult; the second suggests you misunderstood the codebook.
  • The exercise is not about being "right." It is about discovering where your understanding of the codebook diverges from the reference — and what that reveals about the measurement instrument.

Exercise 4.2: Revise the Codebook

Goal: Practice improving a measurement instrument based on discovered ambiguity.

Based on the disagreements you found in Exercise 4.1, revise the codebook. Make at least one of these changes:

  1. Clarify a definition: Add detail to a category definition that was ambiguous
  2. Add a decision rule: Specify what to do in a case that was not covered
  3. Add a boundary example: Include an example of a text that is genuinely ambiguous and how to handle it

Write your revised codebook here or in a separate document.

Answer Key (click to expand)

Good codebook revisions are:

  • Specific: "If the abstract mentions a method but does not describe results, classify as METHOD_ONLY, not EMPIRICAL" is better than "be more careful about abstracts without results"
  • Testable: The revision should resolve the specific disagreement you encountered, not introduce new ambiguity
  • Minimal: Don't rewrite the whole codebook. Fix the part that broke.

A common pitfall: adding so many rules that the codebook becomes harder to apply, not easier. The goal is clarity, not comprehensiveness.

Exercise 4.3: Compare Human and Model Labels

Goal: Evaluate the quality of AI-generated labels against a human benchmark.

Using the lab's "Request builder," copy the constructed prompt and paste it into any AI chat interface. The prompt includes the codebook, instructions, and the text items. Paste the model's output into the lab's scoring calculator.

Record the results:

Accuracy: ____%

Precision (RETAIN): ____%

Recall (RETAIN): ____%

F1 (RETAIN): ____%

Now examine the errors:

  1. Which category has the most errors?
  2. Are errors concentrated in a particular type of text (look at the titles)?
  3. Does the model tend to over-classify or under-classify any category?
  4. Would these errors change your conclusions if this were a real study?

Answer Key (click to expand)

Results vary by model, prompt wording, and dataset version. What to look for:

  • Systematic over-classification: The model labels more texts as RETAIN than the human reference. This is common — models often default to inclusion when uncertain. Consequence: your review may include irrelevant papers.
  • Systematic under-classification: The opposite. The model misses relevant papers. Consequence: your review may omit important evidence.
  • Category-specific errors: Errors concentrated in one category (e.g., the model cannot distinguish EMPIRICAL from THEORETICAL). This may bias your review toward or away from certain types of evidence.
  • The accuracy number is not enough. 85% accuracy with errors concentrated in one category is worse than 80% accuracy with errors spread evenly across categories.

Exercise 4.4: The Downstream Consequence Test

Goal: Understand why high accuracy can still produce biased estimates.

Imagine you have classified 1,000 abstracts as RETAIN or EXCLUDE using an LLM. Your accuracy is 92%. You now use the retained papers to estimate the proportion of studies in your field that report positive results.

Here is the confusion matrix:

| | Reference RETAIN | Reference EXCLUDE |

|---|---|---|

| Model RETAIN | 270 | 40 |

| Model EXCLUDE | 30 | 660 |

  1. What is the true proportion of RETAIN papers? (____/1000 = ____%)
  2. What proportion does the model estimate? (____/1000 = ____%)
  3. The model's accuracy is 93%. Does the proportion estimate match the truth?
  4. If the 30 missed papers (false EXCLUDEs) are disproportionately theoretical papers that report null results, how would this bias your estimate of "proportion reporting positive results"?

Answer Key (click to expand)

  1. True RETAIN: 270 + 30 = 300. 300/1000 = 30%.
  2. Model estimate: 270 + 40 = 310. 310/1000 = 31%.
  3. The proportion is close (30% vs. 31%), but the composition is different — the model retained 40 papers it should have excluded and excluded 30 it should have retained. If those groups differ systematically, conclusions drawn from the retained set may be biased.

The deeper point (from Egami et al.): even when accuracy is high and the proportion looks right, which specific papers are retained matters. If the errors are correlated with the outcome you are studying, your estimate will be biased. This is why "92% accuracy" is not sufficient evidence that the labels are good enough for your use case.

Exercise 4.5: Apply the Four Questions

Goal: Integrate the course lens with annotation methodology.

For the annotation task you just completed (or a real annotation task from your research):

  1. What is being delegated? _______________
  2. What would make the result believable? _______________
  3. What could be lost or flattened? _______________
  4. What must remain visible or human? _______________

Now choose 2–3 checks from the validation menu (see modules/shared/validation_menu.md) that would address your answer to question 2.

Answer Key (click to expand)

For a typical annotation task:

1. Delegated: The classification of each text according to the codebook. You retain: codebook design, validation design, error analysis, and the decision about whether labels are fit for use.

2. Believable: Human-model agreement on a representative sample (check 2). Reliability above chance. No systematic error pattern that biases the downstream analysis (check 7).

3. Lost or flattened: Ambiguous texts forced into categories. Context that the model misses because it only sees the text, not the broader document or discourse.

4. Visible/human: The codebook. The agreement analysis. The decision about fitness for use. A record of model, date, prompt, and settings.

Checks: Human comparison (check 2), boundary cases (check 5), downstream consequence (check 7).

Module 4 Exercises · AI as a Research Tool

§ readings

---

Core Readings

Grimmer, J., Roberts, M. E., & Stewart, B. M. (2022). Text as Data: A New Framework for Machine Learning and the Social Sciences. Princeton University Press. Chapters 5–7 (on classification and measurement).

Why read it: The standard social-science treatment of text classification. Chapters 5–7 cover the measurement framework that applies whether the classifier is a dictionary method, a supervised learning model, or an LLM. The principles do not change because the technology does. What to take: The distinction between discovery (finding patterns), measurement (classifying according to a known schema), and inference (using classifications in statistical models). LLM-based annotation is measurement, and it inherits all the validity concerns of measurement. ---

Egami, N., Hinck, M., Stewart, B. M., & Wei, H. "Using Large Language Model Annotations for the Social Sciences: A General Framework of Using Predicted Variables in Downstream Analyses." American Journal of Political Science, forthcoming.

Why read it: The paper that formalises the problem: when you use LLM-predicted labels in a regression as if they were observed, your estimates can be biased even when accuracy is high. The bias depends on the pattern of errors, not just their frequency. What to take: Three key insights: (1) measurement error in predicted labels does not simply attenuate coefficients — it can bias them in any direction; (2) the bias depends on whether errors are correlated with the outcome or the treatment; (3) the standard fix — running the analysis on a human-coded subset — works, but you need to know how large the subset must be. ---

Halterman, A., & Keith, K. A. (2025). "Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts." Political Analysis, 34(2).

Why read it: A direct evaluation of LLMs for codebook-based classification in political science. Provides a framework for what to report: the codebook, the prompt, the model and settings, the agreement metrics, the error analysis. What to take: Reporting standards. A paper (or portfolio) that claims LLM-based classification should report: the codebook, the exact prompt, the model and date, the agreement with human coders, and the pattern of errors — not just the accuracy number. ---

Abdurahman, S., et al. (2025). "A Primer for Evaluating Large Language Models in Social-Science Research." Advances in Methods and Practices in Psychological Science, 8(2).

Why read it: A practical guide to evaluation, covering classification, generation, and reasoning tasks. The checklist approach helps ensure you have not missed a critical evaluation step. What to take: The evaluation checklist. Before claiming your LLM-based annotation is "good enough," work through the checklist items: construct definition, prompt documentation, human benchmark, reliability, validity, error analysis, replicability. ---

Methodological Readings

Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology, 4th ed. Sage. Chapters on reliability.

Why read it: The definitive treatment of inter-rater reliability. Krippendorff's alpha is the most widely accepted reliability coefficient for content analysis, and understanding what it does (and does not) measure is essential. What to take: Reliability coefficients adjust for chance agreement. Percentage agreement does not. Reporting only percentage agreement without a chance-corrected coefficient is insufficient. ---

Artstein, R., & Poesio, M. (2008). "Inter-Coder Agreement for Computational Linguistics." Computational Linguistics, 34(4), 555–596.

Why read it: A comprehensive survey of agreement measures, with guidance on which to use when. Covers Cohen's kappa, Krippendorff's alpha, Scott's pi, and weighted variants. What to take: Different coefficients make different assumptions about the data. For multi-category classification with more than two coders, Krippendorff's alpha is the standard. For two coders and binary classification, Cohen's kappa is common. ---

Optional: Going Deeper

Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). "Synthetic Replacements for Human Survey Data? The Perils of Large Language Models." Political Analysis, 32(4), 401–416.

Why read it: The survey-data parallel to the annotation problem. LLM-generated survey responses look plausible — the means are often close to human survey means — but the patterns of association between variables are wrong. The same can happen with LLM-generated labels: the marginal distribution looks right, but the joint distribution is wrong. What to take: Always check whether the relationships between variables are preserved, not just the marginal distributions. ---

Read Later (Across Modules)

- Barrie, Palmer & Spirling (forthcoming) on replication for language models — what to record so that your annotation can be reproduced. See Module 7.

- Kapoor & Narayanan (2023) on leakage and reproducibility — especially relevant if the model might have been trained on the texts you are classifying. --- Module 4 Reading List · AI as a Research Tool