Module 07Validation & Research Integrity

Learning Outcomes

# Module 7: Validation, Reproducibility, and Research Integrity

Estimated time: ~3 hours

Type: Conceptual workbook

Prerequisites: Module 1 (How LLMs Work) — the four questions and the capability map

What you'll produce: A validation plan for your portfolio experiment and a journal-ready disclosure paragraph

1. The Evidence Problem

Fluency is not validation

You have now spent several modules working with AI on research tasks: writing, literature searching, annotation, coding, data exploration. In each case, you encountered a fundamental asymmetry. The model produces output that is fluent, confident, and structured — and it does so instantly. Checking that output — verifying sources, comparing against human judgments, testing on known cases, running it twice — is slow, tedious, and unglamorous.

This asymmetry is not a design flaw. It is a direct consequence of how language models work. A model trained to predict the next token from a vast corpus of human writing will tend to produce output that looks like competent human work. It will not tend to produce output that is correct, because correctness was never the training objective. The model optimises for plausibility, not truth (see Module 1, Section 1).

The result is a permanent temptation: accept the output, move on, and treat fluency as sufficient. The response this course offers is structural rather than motivational. You do not need to be more sceptical. You need a procedure.

What evidence is — and is not

For any AI-assisted research claim, evidence comes from outside the model output. The following are not evidence:

• The model's stated confidence ("I am highly certain that...")
• The model's internal reasoning ("Let me think through this step by step...")
• The fluency, coherence, or professional appearance of the output
• The fact that the model got something similar right before

The following are evidence:

• A source you personally verified exists and says what is claimed
• A human coder who independently reached the same label
• A second run that produced the same result
• A test on a case where you already knew the answer
• A record complete enough that someone else could reproduce the process

This distinction — between what the model says about itself and what you can demonstrate from outside it — is the organising principle of this module.

The course lens, applied to validation

The four questions from Module 1 (see modules/shared/four_questions.md) frame every validation decision. Question 2 — "what would make the result believable?" — asks you to name evidence outside the model output. This module is an extended answer to that question: the validation menu is a catalogue of possible answers, and the reproducibility record is how you make your answer inspectable. Question 4 — "what must remain visible or human?" — is answered by the disclosure standard and the accountability principle.

2. The Validation Menu: Seven Checks in Depth

The validation menu (modules/shared/validation_menu.md) lists seven checks. That document provides a summary table and examples; this section expands each check with the full rationale, costs, when to use it, and common pitfalls. You do not need to run all seven. You need to choose the ones that could actually change your interpretation.

Check 1: Source Verification

Question it answers: Does every cited source exist and say what is claimed?

What it involves. For every reference, fact, or claim the model produces, you locate the original source and confirm that (a) it exists and (b) it supports the claim the model attributed to it. This includes checking page numbers, quoted passages, author names, journal volumes, and DOIs.

Why it matters. Language models fabricate references with high fluency. A fabricated reference — correct journal format, plausible author name, plausible title, plausible volume and page numbers — is indistinguishable from a real one by reading it. The scale of the problem is substantial: fabricated references in the published literature rose approximately twelvefold between 2023 and early 2026 (see Syllabus, "Use of AI in this course"). The people involved did not think this rule applied to them.

The demonstration. If you completed Exercise 1.4 (Hallucination Hunt), you have already seen this. If not: ask any LLM for a reference to "Smith, J. (2023). 'Artificial Intelligence and Survey Methodology.' Journal of Political Methodology, 45(2), 112-138." The model will likely return a confident, formatted reference. Then try to find the paper. The Journal of Political Methodology does not exist. Neither does the paper. The model produced a statistically plausible reference — nothing more.

Cost. Time to look up each reference. For a literature search returning 50 papers, verifying all of them might take an hour. For a single writing-assistance session producing five citations, it takes five minutes. The cost is proportional to the number of claims you accept.

When to use it. Whenever the model produces a claim that depends on the existence or content of an external source. This covers literature searches, summarisation, fact-checking, and any output that names authors, papers, or data sources.

Pitfall. Verification by asking the same model "does this reference exist?" — a model cannot check its own output against reality. Verification by asking a different model — a second model has the same fundamental limitation. Verification means you, or a colleague, locating the source.

Check 2: Human Comparison

Question it answers: Would a competent human produce the same result?

What it involves. Take a sample of the AI's output — labels, summaries, classifications, search results — and have a human perform the same task independently on the same material. Compare the results. Report agreement (e.g., Cohen's kappa, Krippendorff's alpha, percentage agreement) and, critically, disagreement patterns: where do human and model diverge, and why?

Why it matters. Agreement with a human is the most intuitive validation standard in social science — it is what we do when we train and check human coders. But it is also the most easily misinterpreted. Three cautions:

1. Agreement is not validity. High inter-rater reliability between human and model does not mean the construct is being measured correctly. Both could be making the same systematic error. (See the glossary entry on construct validity in modules/shared/glossary.md.)

1. Human coders are not gold standards. Humans make errors, have biases, and disagree with each other. A model that agrees with one human coder may disagree with another. The comparison you want is with a competently performed human task — which means you need a codebook, training, and some evidence of human reliability first.

1. The comparison is most informative where there is disagreement. If human and model agree on 90% of cases, the interesting question is what distinguishes the 10% where they disagree. Is there a systematic pattern? A category the model consistently misses? A type of case the codebook does not address?

Cost. Recruiting, training, and (if necessary) paying coders — or your own time for a smaller validation sample. For a 2,000-item classification task, double-coding a random sample of 200 items by a second human is usually sufficient to characterise agreement patterns.

When to use it. Whenever the AI is performing a task that, in the non-AI version of your workflow, a human would perform: classification, coding, screening, summarisation, or judgment.

Pitfall. Using a single agreement number (e.g., "89% agreement") without examining disagreement patterns. An 89% agreement rate where all disagreements are concentrated in one category is a different result from 89% agreement with evenly distributed disagreements. The first pattern may produce biased downstream estimates; the second may not. (See Check 7.)

Check 3: Repeated Runs

Question it answers: How much does the output vary across runs with the same input?

What it involves. Run the same prompt, on the same model, with the same settings, on the same input, multiple times. Compare the outputs. If the task produces categorical labels, compute agreement across runs. If it produces continuous scores, compute the distribution. If it produces text, compare the substantive claims.

Why it matters. As Module 1 explained, language models use sampling: at each token, they choose from a distribution of plausible next tokens rather than always picking the most probable one. The same prompt can produce different outputs on different runs. For a classification task, this means the label assigned to a specific case might change between runs. For a summarisation task, different runs might emphasise different points. If your research conclusion depends on which run you happened to get, your measurement is not reliable.

What varies. Across runs, you may observe variation in:

• Classification labels (for the same item)
• The ordering or ranking of items
• The presence or absence of specific claims in a summary
• The wording, length, or structure of generated text
• The specific examples or citations included

What to report. At minimum, report the number of runs, whether outputs were identical, and if not, the nature and magnitude of the variation. For classification: report agreement metrics across runs. For generation: describe what stayed constant and what varied.

Temperature and systematic variation. The sampling settings control how much variation to expect. At low temperature (0.0–0.2), outputs are near-deterministic: repeated runs will be almost identical, and the variation you observe mostly disappears. At default or higher temperatures, variation increases. This cuts both ways. A low temperature gives you a more stable measurement — but stability at a single setting is not the same as correctness; a confidently consistent wrong answer is still wrong. A higher temperature reveals the model's underlying uncertainty — the set of plausible outputs — at the cost of noisier results. The defensible approach is to decide, in advance, which setting your task requires, then report what that setting was and run the repetition check under it.

Cost. Compute time or API cost for multiple runs. For a 200-item classification task with an API model, running it three times costs three times the API credits. For a local model, it costs three times the compute time.

When to use it. Always run at least twice for any task where the output is used in a downstream analysis. A single run gives you one sample from a distribution; you need at least two to know whether the distribution is narrow (consistent) or wide (unreliable).

Pitfall. Running twice, seeing different outputs, and picking the one you prefer. The point of repeated runs is not to find the "best" output — it is to characterise the variation so that a reader knows how much uncertainty attaches to any single run.

Check 4: Prompt Variation (Sensitivity)

Question it answers: Does the result hold when you reword the instructions?

What it involves. Write your prompt. Then write at least one alternative prompt that asks for the same thing in different words — different structure, different examples, different ordering of instructions. Run both on the same input. Compare the results.

Why it matters. Prompt sensitivity is a form of measurement sensitivity: if small changes in how you operationalise a construct produce large changes in the result, the measurement is not robust. This is the LLM equivalent of asking whether your survey question would produce the same answer if you reworded it — a standard concern in survey methodology.

Prompt variation can surface several problems:

• The model is latching onto a specific word or phrase rather than the underlying concept
• The examples you provided are driving the result more than the instructions
• The instruction ordering (what comes first vs. last in the prompt) is affecting the output
• The construct itself is underspecified — different reasonable phrasings of "what counts as economic policy" produce different classifications

Cost. Time to design and test alternative prompts. Usually an hour or two of careful rewriting, followed by running both prompts on the same material.

When to use it. Whenever the prompt contains a construct definition, classification rule, or instruction that could be reasonably phrased in multiple ways — which is to say, nearly always. If you cannot think of an alternative phrasing that a reasonable colleague might use, the construct may be unusually well-specified.

Pitfall. Testing a trivial rewording ("Classify these texts" → "Please classify these texts") and concluding the prompt is robust. The alternative should be a genuine rephrasing — change the structure, not just the politeness.

Check 5: Boundary Cases

Question it answers: Does the system fail sensibly on edge cases, or does it produce confident wrong answers?

What it involves. Identify or construct cases that sit at the boundary of your categories: texts that are ambiguous between labels, search queries that should return nothing, claims that are half-true, data points that are outliers. Run the model on these cases and examine what it does.

Why it matters. Language models tend to produce confident output regardless of whether they are correct. On a boundary case — a text that could reasonably be classified either way — the model will tend to pick one category and justify it confidently, rather than flagging the ambiguity. This is the "confident wrong answer" problem.

Boundary cases also reveal what the model is actually attending to. If a text that mentions "economic" once in passing is classified as "economic policy" while a text that discusses fiscal policy without using the word "economic" is not, the model is matching surface features rather than the construct.

Designing boundary cases. Good boundary cases include:

• Cases where the correct label depends on context outside the text itself
• Cases that match the surface features of one category but the substance of another
• Cases that are genuinely ambiguous and should be flagged as uncertain
• Cases where a human coder would need to consult the codebook decision rules

Cost. Designing or collecting difficult cases — your time, and ideally a colleague's time to independently suggest boundary cases.

When to use it. Whenever your task involves classification, screening, or any form of "does this belong in category X?" Boundary cases are particularly important when the categories are fuzzy, contested, or context-dependent.

Pitfall. Only testing cases you already know how to classify. Ask a colleague: "what is a case you would find ambiguous under my definition?" Their answer is a better boundary case than yours.

Check 6: Sensitivity to Model or Settings

Question it answers: Would a different model or different sampling settings produce a different conclusion?

What it involves. Run your task on at least one alternative model (e.g., if you used GPT-5, try Claude 4; if you used a proprietary model, try an open-weight model like Llama). Alternatively, vary the sampling settings: different temperature, different top-p. Compare the substantive conclusions.

Why it matters. Different models are trained on different data, with different architectures, and different alignment procedures. They have different strengths, weaknesses, and failure patterns. If your conclusion holds across two models with different training data and architectures, it is more robust than if it depends on a specific model.

This check also addresses a practical problem: model deprecation. The model you use today may not be available in six months. When the provider retires the model, can you reproduce your result? If your conclusion depends on a specific model version that is no longer accessible, your work cannot be replicated.

Cost. Access to multiple models or API credits for an alternative provider. Free-tier access to a chat interface for a different model is often sufficient for a sensitivity check.

When to use it. Whenever your claim is substantive enough that a reader would care whether it depends on the specific tool you chose. For exploratory work, this may be optional. For a portfolio experiment or publication, it is expected.

Pitfall. Running on two models, getting different results, and concluding "the models disagree" without examining why. The disagreement itself is data: does one model systematically misclassify a category? Does one model produce more variation across repeated runs? The pattern of disagreement tells you something about your construct or your prompt.

Check 7: Downstream Consequence

Question it answers: Do the errors that exist actually change the estimate, decision, or interpretation?

What it involves. Run the full analysis with the AI-generated data. Then run the same analysis with the best available alternative — human labels, a different model's output, a subset of verified cases. Does the substantive conclusion change? Does the coefficient change sign? Does the significance level cross a threshold? Does the ranking of items shift?

Why it matters. This is the Egami et al. problem (Egami, Hinck, Stewart & Wei, forthcoming; see readings.md): a classifier can be 95% accurate and still produce biased regression estimates if the 5% of errors are concentrated in a way that correlates with the outcome variable. High overall accuracy does not guarantee that the downstream analysis is unaffected.

Consequential error is a specific concept: an error is consequential if it changes the substantive conclusion a reader would draw. Not all errors are consequential. The errors that matter depend on your research question. For a classification task feeding into a regression, the consequential errors are those that change the coefficient estimate. For a literature screening task, the consequential errors are the missed relevant papers and the included irrelevant ones. For a summarisation task, the consequential errors are the omitted or distorted claims that would change a reader's understanding.

Cost. Running the full analysis — which you would need to do anyway. The additional cost is running it with the alternative data to compare.

When to use it. Whenever the AI-generated output enters a downstream analysis. This is the check that connects measurement quality to research conclusions. If you are using AI to produce labels, classifications, or extracted data that then enter a regression, a table, or an interpretive claim, you need this check.

Pitfall. Running the downstream analysis only on the cases where human and model agree. This will artificially inflate performance because it excludes the cases where errors are most likely to be consequential. Run the analysis on the full dataset, then on the verified subset, and compare.

How to Choose Checks

The validation menu is not a checklist to be completed. It is a set of tools to be selected. The selection rule is:

1. Start with the claim. What exactly are you asserting?
2. Ask what would falsify it. What kind of evidence would show you were wrong?
3. Pick the checks that would detect that falsification.

| If your claim is... | The falsification would be... | Run these checks... |

|---|---|---|

| "This search found all relevant papers" | A known paper is missing; an included paper is about the wrong topic | 1 (source verification), 2 (human comparison), 5 (boundary cases) |

| "These labels are accurate enough for regression" | The regression coefficient changes with human labels | 2 (human comparison), 7 (downstream consequence), 5 (boundary cases) |

| "This summary is faithful to the source" | A claim in the summary is not in the source | 1 (source verification), 3 (repeated runs), 4 (prompt variation) |

| "This analysis is reproducible" | Another researcher cannot produce the same result | 3 (repeated runs), 6 (model sensitivity), documentation |

One check is better than none; two well-chosen checks are better than five run superficially. The validation section of your portfolio should name which checks you ran and what they found — not list all seven and claim coverage.

3. Reproducibility with LLMs

What to record

Reproducibility means that another researcher, given your materials and description, could repeat your process and evaluate whether they reach the same conclusion. For LLM-based research, this requires recording information that is typically invisible in a chat interface.

The Barrie, Palmer, and Spirling framework (Barrie, Palmer & Spirling, forthcoming; see readings.md) identifies the minimum reproducible record:

1. Model. Which model, which version, accessed how (API, chat interface, local). Include the date(s) of access. Example: "Claude 4 (Anthropic), via claude.ai, 10–12 September 2026."

1. Prompt. The exact prompt text, including any system prompt, custom instructions, or project-level configuration. If you used few-shot examples, include all of them. The prompt is your instrument; it must be reported as completely as a survey questionnaire.

1. Settings. Temperature, top-p, top-k, maximum output length, and any other sampling parameters. If you used default settings, say so — and name what the defaults were if the interface documents them.

1. Input data. What text, data, or files were provided to the model. If the input is a dataset, provide the dataset or describe exactly how it can be obtained. If the input is search results, describe the search and the date.

1. Output. The complete model output. For classification: the label for every item, not just aggregate statistics. For generation: the full text, not a summary. If output is too large to include in an appendix, provide it as supplementary material.

1. Post-processing. Any transformations applied to the model output before it entered your analysis. Did you filter, reorder, edit, or select among outputs? Every step must be documented.

Model deprecation

Proprietary models are retired, updated, and replaced. The model you used in September 2026 may not be available in September 2027. This creates a reproducibility problem that does not arise with traditional statistical software: you cannot reinstall the exact version of GPT-5 that was available in September 2026.

The response, per Barrie, Palmer, and Spirling, has several components:

• Record the model and date precisely so that a reader knows what was available when.
• Preserve the complete output so that the result can be inspected even if the model cannot be re-run.
• Run sensitivity checks with alternative models (Check 6) so that the conclusion is not tied to a single model version.
• Consider open-weight models for tasks where long-term reproducibility is essential. An open-weight model (like Llama or Qwen) can be archived and re-run indefinitely, though it may not match proprietary model performance.

The Spirling/Smith/Palmer standard for proprietary models

Spirling, Smith, and Palmer (2023; see readings.md) argue that using a proprietary model in academic research requires explicit justification. The justification should address: why a proprietary model was chosen over an available open alternative, what reproducibility limitations this choice creates, and what steps were taken to mitigate those limitations. This is not a prohibition on proprietary models — it is a requirement to acknowledge and address the trade-off.

4. Data and Institutional Boundaries

The four piles

Not all data can be handed to an AI system. Before using any AI tool on research data, sort your material into one of four piles:

| Pile | Description | Examples | Rule |

|---|---|---|---|

| Safe | Public, non-sensitive, no access restrictions | Published papers, public government data, your own unpublished solo-authored drafts | Can be used with any tool. |

| Safe after transformation | Contains identifiers or confidential elements that can be removed | Survey data with respondent IDs stripped, interview transcripts with names replaced by pseudonyms | Transform first, verify the transformation removed all identifying information, then use. |

| Not safe | Cannot be shared with a third party under any circumstances | Identifiable personal data under GDPR, IRB-restricted data, data with contractual prohibitions on third-party access, unpublished co-authored material without co-author consent | Do not upload to any external service. Use local, open-weight models only if your institutional security policy permits. |

| Uncertain | You are not sure which pile it belongs in | Data with unclear consent terms, institutional data where the data owner has not been consulted, material you received under informal sharing arrangements | Do not proceed until you have clarified the boundary. Ask your supervisor, your institution's data protection officer, or the data provider. |

The rule from the syllabus applies throughout: do not upload material you lack the authority to share.

What rules apply

Your use of AI in research is governed by overlapping regimes. The specific rules depend on your institution, your funder, your data, and your publication venue. The following are the most relevant for UZH social scientists:

UZH Recommendations (September 2023). The University of Zurich's "Recommendations on the Use of Generative Artificial Intelligence" implements seven guiding principles: (1) responsibility remains with the researcher; (2) transparency about AI use; (3) data protection and information security; (4) scientific integrity; (5) no AI authorship; (6) AI literacy; (7) subject-specific standards. The principle that matters most for this module: you are responsible for everything you submit, regardless of what tools contributed.

SNSF on AI in funding applications (February 2024). The Swiss National Science Foundation states that AI tools may be used in preparing funding applications, but the applicant remains responsible for content. The critical paragraph addresses confidentiality: information submitted to an AI tool is considered disclosed to a third party. If your research proposal contains confidential material, do not submit it to an external AI service. The same logic applies to research data: uploading data to a commercial API is disclosure to a third party.

ERA Living Guidelines (updated May 2026). The European Research Area's Living Guidelines on the Responsible Use of Generative AI in Research establish cross-European expectations: researchers must be transparent about AI use, must verify AI-generated output, must respect data protection and confidentiality, and must maintain accountability. The factsheet version is sufficient for most purposes; the full document is referenced in readings.md.

Journal policies (AJPS, 2025; APSR). The American Journal of Political Science and American Political Science Review require disclosure of AI use. The standard: name the tool, the purpose, the location in the manuscript, and the verification performed. AI tools cannot be listed as authors or cited as sources of fact. These policies are representative of the broader trend in political science and social science journals.

A note on jurisdiction

GDPR (European General Data Protection Regulation) and the Swiss Federal Act on Data Protection (FADP) impose legal obligations on the handling of personal data. Uploading personal data to a commercial AI service — particularly one whose servers are outside Switzerland or the EU — may constitute a data transfer that requires specific legal justification. This module does not provide legal advice. If your research involves personal data, consult your institution's data protection officer before using any external AI service.

5. The Disclosure Standard

What a journal editor needs to see

The disclosure is not an institutional formality. It is the answer to a reviewer's first question: "what exactly did the AI do, and how do I know the result is reliable?"

The disclosure template (modules/shared/disclosure_template.md) specifies five required elements:

1. Tool(s). Model name, version, interface, and date(s) of use.
2. Purpose. The specific task the AI performed.
3. Location. Where in the work the AI contribution appears.
4. Verification. What checks were performed, with evidence.
5. Human decisions. What judgments were retained by the author.

A complete disclosure also includes the accountability statement: "The author takes full responsibility for all content in this document. No AI tool was used as a source of fact, and no AI-generated reference was included without personal verification."

Examples

The disclosure template contains annotated examples for writing assistance, annotation, and no-AI use. These are reproduced and contextualised in Exercise 7.3. The key principle: the disclosure must be specific enough that a reviewer can evaluate whether the verification was adequate for the claim.

Rules that cannot be disclosed around

Disclosure does not excuse certain failures:

• Fabricated references. Disclosing AI use does not make a fabricated reference acceptable. If you include a reference you have not personally confirmed exists and says what you claim, the disclosure does not protect you.
• Confidentiality breaches. Disclosing that you uploaded confidential data to an AI service does not remedy the breach. The breach occurred when you uploaded the data, not when you failed to disclose it.
• Delegated accountability. A disclosure that says "the AI generated this section" without specifying verification or retained human judgment is an admission that you did not do the work, not a disclosure that makes it acceptable.

6. Research Integrity

Fabricated references: the scale of the problem

The course syllabus states: "Fabricated references in the published literature rose roughly twelvefold between 2023 and early 2026." This is not hyperbole. AI-generated text, submitted without verification, has produced retractions across disciplines. The pattern is consistent: an author uses an AI tool to assist with writing, the tool produces plausible-looking citations, the author does not verify them, and the fabricated references enter the published record.

The mechanism is the same one you studied in Module 1: next-token prediction produces statistically plausible text, and statistically plausible citations look exactly like real ones. The only defence is verification — and verification cannot be delegated to the same tool that produced the output.

Confidentiality: don't upload what you can't share

The confidentiality rule is simple: do not submit to an external AI service any material you lack the authority to share. This includes:

• Unpublished co-authored work without co-author consent
• Interview transcripts, field notes, or other data collected under confidentiality agreements
• Administrative or institutional data with access restrictions
• Personal data subject to GDPR, FADP, or IRB restrictions
• Peer-review materials (manuscripts you are reviewing, grant proposals you are evaluating)

When in doubt: keep the material on your own machine. Use a local, open-weight model if your institutional security policy permits it. Ask before uploading.

Accountability: you are responsible, not the tool

The accountability principle is the position of every relevant body — ICMJE, COPE, the SNSF, the ERC, UZH, and your eventual publishers:

If your submitted work contains an error — a fabricated reference, a misclassification, a misleading summary, a confidentiality breach — the fact that an AI tool contributed the error does not reduce your responsibility. You submitted it. It carries your name. You are accountable.

This principle has a positive counterpart: it also means you get credit for the work. The judgments you made, the checks you performed, the decisions you defended — these are yours, and they are what the portfolio rubric rewards.

7. Building a Validation Plan

From claim to checks

A validation plan is a short document that states, for a specific AI-assisted claim:

1. The claim. What exactly are you asserting?
2. The falsification condition. What would show the claim is wrong?
3. The checks. Which checks from the validation menu would detect that falsification? For each check: what you will do, what material you need, and what counts as passing.
4. The record. What you will preserve so that someone else can inspect what happened.

Example validation plan

Claim: "The LLM-labelled dataset correctly classifies 500 news articles as covering economic policy or not, according to the codebook in Appendix A, with accuracy sufficient for use in the regression in Table 2."

Falsification condition: The regression coefficient in Table 2 changes sign or significance when human labels replace LLM labels on a random validation sample.

Checks:

• Check 2 (human comparison): Double-code a random sample of 100 articles. Compute agreement (expected: >85% raw agreement; kappa >0.70). Examine disagreement patterns: are errors concentrated in one category?
• Check 3 (repeated runs): Run the classification twice. Compute agreement across runs. If agreement is <95%, the measurement is unreliable regardless of human-model agreement.
• Check 7 (downstream consequence): Run the regression in Table 2 with LLM labels and with human labels on the validation sample. Does the coefficient of interest change sign? Does it cross a significance threshold?
• Check 5 (boundary cases): Select the 20 articles where LLM and human disagree. Is there a systematic pattern? Are articles from a particular source, topic, or style consistently misclassified?

Record: The complete prompt, model version and date, the labelled dataset with item-level labels and run IDs, the human validation labels, the disagreement analysis, and both regression outputs.

Your turn

Exercise 7.6 asks you to build a validation plan for your own capability map task. If you have not yet defined that task, use the supplied claim in Exercise 7.1.

8. Exercise Instructions

The exercises for this module are in exercises.md. Complete them in order:

• Exercise 7.1: Select validation checks for a given claim
• Exercise 7.2: Classify data types by risk level
• Exercise 7.3: Write a disclosure paragraph
• Exercise 7.4: Design a reproducibility record
• Exercise 7.5: Spot the missing validation
• Exercise 7.6: Apply the four questions + validation menu to your capability map

Each exercise has a collapsible answer key. Work through the exercise before opening the key.

9. Summary

This module is the evidence spine of the course. It answers the question that every other module raises: how do you know the AI-assisted result is reliable?

The answer has several layers:

• The evidence problem. Fluency, confidence, and internal reasoning are not evidence. Evidence comes from outside the model output.
• The validation menu. Seven checks — source verification, human comparison, repeated runs, prompt variation, boundary cases, model sensitivity, downstream consequence — each appropriate to different claims and different risks.
• Reproducibility. Record the model, prompt, settings, input, output, and post-processing. Model deprecation is a real threat; open-weight models and sensitivity checks are partial mitigations.
• Data and institutional boundaries. Sort your data into safe, safe-after-transformation, not-safe, and uncertain. Do not upload what you lack the authority to share. Know which rules apply: UZH, SNSF, ERA, and journal policies.
• Disclosure. A journal-ready disclosure names the tool, purpose, location, verification, and retained human judgments. Disclosure does not excuse fabricated references or confidentiality breaches.
• Accountability. You are responsible for everything you submit. A tool is not a co-author, not a source of fact, and not an excuse.

The four-question course lens now has its full evidential apparatus. Question 2 can be answered with specific checks from the validation menu, not "the model sounded confident." Question 3's threats — variation, boundary cases, prompt sensitivity — are surfaced by the checks that a single run hides. Question 4 — what must remain visible — is specified by the reproducibility record and the disclosure standard.

In Module 8, you will bring everything together: your capability map, your experiment, your validation plan, your disclosure, and your decision. For now, the validation plan is your deliverable. Build it carefully. It is the evidence that supports everything else.

Cross-References to Other Modules

• Module 1 (How LLMs Work): The four questions, hallucination, and sampling variation — revisited with the full validation apparatus.
• Module 2 (Writing): Check 1 (source verification) is the primary check for AI-assisted writing. The disclosure template was introduced there.
• Module 3 (Literature Search): Checks 1, 2, and 5 are the core checks for literature screening.
• Module 4 (Text Annotation): All seven checks apply; checks 2, 3, and 7 are the priority for annotation claims.
• Module 5 (Code): Checks 1, 3, and 5 for code correctness and reproducibility.
• Module 6 (Agents): Check 7 (downstream consequence) is critical for multi-step agent workflows; the reproducibility record becomes more complex.
• Module 8 (Portfolio): Your validation plan becomes Section 4 (Evidence and Failures) of the portfolio.

Module 7 of 8 · AI as a Research Tool: Workflow Design for the Social Sciences · University of Zurich