Module 05Code & Data Analysis

Learning Outcomes

# Module 5: Code and Data Analysis with AI Assistants

Estimated time: ~3.5 hours

Type: Technical — Python notebook + markdown workbook

Prerequisites: Module 1 (How LLMs Work); Modules 4 (Text Annotation and Measurement) or 7 (Validation) recommended

What you'll produce: A documented data analysis with at least one independently verified statistic and interpretation, plus an AI use disclosure

1. Why AI for Code and Data?

Social scientists increasingly encounter code in their research even when they are not programmers. A collaborator shares a replication script. A journal requires data deposit. A reviewer asks about a transformation. An RA hands you a .py file and moves on.

Until recently, engaging with code required either learning to program or finding someone who could. AI assistants change the terms of engagement: they can read the code you cannot write, explain what it does, spot problems, and generate new analysis. But they also generate code that looks correct and is not — and you cannot always tell the difference by looking.

This module is not a programming course. It is a course about supervised delegation: how to ask an AI assistant to work with code and data while keeping the research decisions visible and the results verifiable.

The scaffold approach

The core principle of this module is that you never need to write code from a blank editor. Instead, you work from a scaffold — a pre-written script or notebook that provides the structure — and adapt it using AI assistance. The instructor provides the scaffold; you provide the research questions and the verification.

This is the pattern you will use:

1. Start with a working script or notebook (provided, or one you already have)
2. Ask the AI to explain what each part does, in plain language
3. Ask the AI to extend or modify the analysis to answer your question
4. Verify at least one statistic and one interpretation independently
5. Document the AI's contribution and your checks

2. AI for Code: What It Can Do

AI assistants can perform several distinct tasks with code. Each has a different risk profile and requires different verification.

Explanation

What it is: You paste a script into a chat window and ask "What does this do?" The AI produces a plain-language explanation of the logic, the variables, and the output.

When to use it: Whenever you inherit code you did not write — a replication script, an RA's analysis, a methods appendix snippet. This is the lowest-risk use: the AI is describing code that already exists. The code is the ground truth; you can check the explanation against it.

What can go wrong: The AI may invent a plausible interpretation of an ambiguous variable name. It may claim the code does something it does not, or miss something it does. Always cross-check: read the explanation alongside the code and confirm at least one claim by tracing a value through the logic yourself.

Debugging

What it is: You give the AI a script that produces an error or the wrong result, describe what you expected, and ask for a fix. The AI identifies the problem and proposes a corrected version.

When to use it: When a script fails with an error message you do not understand, or when the output does not match what the codebook says it should produce.

What can go wrong: The AI may fix the surface error but introduce a new error. It may change the analysis logic in a way you do not notice. It may drop observations, change the unit of analysis, or silently reinterpret a missing-data code. Always run the fixed script and verify the output against a value you computed independently.

Documentation

What it is: You give the AI an uncommented script and ask it to add comments explaining what each section does, or to generate a docstring describing the function's purpose, inputs, and outputs.

When to use it: Before sharing code with a collaborator, depositing replication materials, or submitting to a journal.

What can go wrong: Comments may be factually wrong about what the code does. They may describe what the AI expected the code to do rather than what it actually does. Always read the comments alongside the code and verify at least one claim by tracing it.

Translation

What it is: You give the AI a script in one language (R, Stata, Python) and ask it to produce an equivalent script in another language.

When to use it: When you are more comfortable reading one language than another, or when a collaborator works in a different language.

What can go wrong: Statistical functions with the same name may behave differently across languages (e.g., default standard error calculations, missing-value handling, factor variable treatment). The translation may look correct but produce different numbers. Always run both versions on the same data and compare outputs.

Generation (with caution)

What it is: You describe an analysis you want ("load this CSV, compute means by group, make a bar chart") and the AI writes the script.

When to use it: For exploratory work on non-sensitive data, when you can verify every output. In this course, we prefer the scaffold approach — modifying existing working code — over generating from scratch. Generation is riskier because there is no baseline to compare against.

What can go wrong: Everything listed above, plus: the AI may use packages that do not exist, call functions with wrong arguments, or write code that runs without error but produces the wrong result. The fluency of the code is not a guarantee of its correctness.

3. Data Exploration with AI

The most common research use of AI-assisted coding is data exploration: loading a dataset, producing summaries, checking for problems, creating plots, and running simple models. Here is a workflow you can use, structured around verification:

Step 1: Load and inspect

Ask the AI to load your dataset and display the first few rows, the column names, the data types, and the dimensions (rows × columns). Verify one of these values yourself: count the columns by hand, or confirm that the number of rows matches what you expect from your data source.

Step 2: Summary statistics

Ask the AI to produce summary statistics: means, medians, standard deviations, minimums, maximums, and counts of missing values for each variable. Pick one statistic for one variable and compute it yourself — with a calculator, in a spreadsheet, or by counting cases — before trusting any other number the AI reports.

Step 3: Data quality checks

Ask the AI to check for:

• Missing values (how many, in which variables)
• Impossible values (e.g., age = 250, a Likert scale with a value of 99)
• Duplicate rows
• Inconsistent categories (e.g., "male", "Male", "m", "M" all appearing in the same column)

For each issue the AI flags, confirm at least one example manually: find the row it identified and check that the problem is real.

Step 4: Transformations

Ask the AI to create new variables: recode a categorical variable, compute a scale mean, create a binary indicator. Before running the transformation, predict what the result should be for one specific case. After the AI produces the transformation, check whether your prediction matches.

Step 5: Visualisation

Ask the AI to produce a plot: a histogram, a bar chart, a scatterplot. Read the plot: does the pattern make sense given what you know about the data? Are the axis labels correct? Are the units right? Pick one data point on the plot and confirm it against the raw data.

Step 6: Simple models

Ask the AI to run a regression, a t-test, or a correlation. Before looking at the result, write down what direction you expect the relationship to have and why. After seeing the result: did the AI use the right variables? The right unit of analysis? The right standard errors? Verify one coefficient by running a simpler calculation: if the model says the mean of group A is 4.3, compute that mean yourself.

4. What to Verify Independently

You cannot verify everything — that would defeat the purpose of using AI. But you must verify something, or you are accepting output on faith. The validation menu from Module 4 (modules/shared/validation_menu.md) provides seven checks you can choose from — source verification, human comparison, repeated runs, prompt variation, boundary cases, sensitivity to model or settings, and downstream consequence. For code and data analysis, checks 2 (human comparison — compute one value yourself), 5 (boundary cases — test on a known-answer case), and 7 (downstream consequence — does the error change your interpretation?) are the most directly applicable.

Here is a minimum verification checklist for any AI-assisted data analysis:

| What to verify | How |

|---|---|

| One descriptive statistic | Compute it by hand or in a spreadsheet |

| One count | Count rows, cases, or groups yourself |

| One plot point | Find the corresponding value in the raw data |

| One transformation | Predict the result for one case before running, then check |

| The unit of analysis | Confirm that each row is what you think it is (respondent? country-year? article?) |

| Missing-data handling | Check what happened to cases with missing values — were they dropped, imputed, or silently included? |

| One interpretation | Ask: "Does the number actually support this claim?" Read the output yourself |

| Edge cases | Test the code on a case you constructed with a known answer |

If you verify nothing, you know nothing. If you verify one thing, you have a toehold. If you verify three things across different aspects of the analysis (a count, a statistic, and an interpretation), you have a basis for evaluating the rest.

5. Common Failure Modes

AI-assisted data analysis fails in predictable ways. Knowing what to watch for is half the defence.

Hallucinated functions

The AI calls a function that does not exist — pandas.read_csv_smart() or ggplot.geom_smooth_robust(). This is usually easy to spot because the code produces an error. Less visible: the AI calls a real function with wrong arguments that have plausible defaults, and the code runs silently with unexpected behaviour.

Plausible wrong analyses

The AI runs the analysis you asked for, but on the wrong subset of data, with the wrong denominator, or with the wrong variable coding. The output looks correct — coefficients, p-values, R-squared — but every number is wrong. You cannot detect this by looking at the output; you must trace one value back to the raw data.

Off-by-one errors

The AI selects rows 1–10 when you wanted the first ten rows but the data has a header row, and row 1 is the header. Or it splits data at index 100 but the index is zero-based. These are classic programming errors that AI inherits from its training data.

Dropped observations

The AI silently drops rows with missing values and reports results on a smaller N than you expect. Or it fills missing values with zeros or means without telling you. Always compare the N in the model output to the N you expect from your data description.

Silent recoding

The AI treats "99" as a number rather than a missing-value code, or converts a categorical variable to numbers in a different order than you expected. The analysis runs; the results are wrong. This is why the codebook matters: the AI cannot infer data semantics from numbers alone.

The self-assessment gap

Becker et al. (2025) found that developers using AI felt more productive, but objective measures showed smaller and less consistent gains. The same gap appears in data analysis: the AI makes you feel faster and more confident, but that feeling is not evidence that the results are correct. Confidence and correctness are separate dimensions.

6. The Four Questions Applied to Code and Data Tasks

Before every AI-assisted code or data task, answer the four questions. (Full reference: modules/shared/four_questions.md — keep it open alongside this workbook.) Here they are applied to a typical scenario: asking an AI to explore a survey dataset.

1. What is being delegated?

Task: The AI loads the data, computes descriptive statistics, produces plots, and runs a regression.

Decisions being delegated: Which statistics to report (the AI selects from available variables). How to handle missing values (the AI chooses a default strategy). Which regression specification to run (the AI infers from the prompt). How to interpret the coefficient (the AI produces a narrative interpretation).

Notice that "task" and "decision" blur together in data analysis. Every computational step embeds a methodological choice. Separating them is the first step toward deciding what you need to verify.

2. What would make the result believable?

• I compute one mean by hand and match it to the AI's output.
• I trace one case through a transformation and confirm the result.
• I read the regression output myself and confirm the coefficient sign matches my expectation.
• I check that the N in the model matches the N of complete cases I counted.
• I run the same analysis on a different AI system (or the same system twice) and compare.

3. What could be lost or flattened?

• Cases with missing values might be silently dropped, changing my effective sample.
• Outliers might be included in means without my knowledge, skewing the results.
• The AI might choose a default reference category for a categorical variable that obscures the comparison I care about.
• My own judgment about which variables are substantively important is replaced by the AI's statistically-driven selection.
• The data provenance — where it came from, how it was collected, what the codes mean — is invisible to the AI.

4. What must remain visible or human?

• The codebook must be written by me, not the AI, because the AI cannot know what the data mean.
• The decision about which cases to include or exclude must be documented and justified by me.
• The interpretation of what a coefficient means for my research question is mine.
• The disclosure must record what the AI did, what I checked, and what I decided.
• I must be able to say: "Here is the exact code the AI ran, here is the output, here is what I verified, and here is what I would need to see to change my interpretation."

7. Demonstration: Debug a Broken Analysis

Let's walk through a concrete example. Below is a short Python script that computes the mean of a survey variable by region. The script runs without error — but it contains at least two problems.

```python

SURVEY_ROWS = [

{"respondent_id": 1, "region": "urban", "trust_eu": 6},

{"respondent_id": 2, "region": "urban", "trust_eu": 7},

{"respondent_id": 3, "region": "urban", "trust_eu": 99},

{"respondent_id": 4, "region": "rural", "trust_eu": 4},

{"respondent_id": 5, "region": "rural", "trust_eu": 5},

{"respondent_id": 6, "region": "rural", "trust_eu": 3},

{"respondent_id": 7, "region": "rural", "trust_eu": None},

{"respondent_id": 8, "region": "urban", "trust_eu": 5},

]

def mean_trust_by_region(rows):

totals = {}

counts = {}

for row in rows:

if row["trust_eu"] is None:

continue

region = row["region"]

totals[region] = totals.get(region, 0) + row["trust_eu"]

counts[region] = counts.get(region, 0) + 1

return {region: totals[region] / counts[region] for region in totals}

# What does this actually compute?

```

Without reading the codebook, can you identify what is wrong? The script runs. It produces numbers. But:

1. Problem 1: Respondent 3 has trust_eu = 99. In many survey datasets, 99 is a missing-value code, not a genuine trust score. Including it in the mean inflates the urban average. The AI cannot know this from the code alone — it needs the codebook.

1. Problem 2: Respondent 7 has trust_eu = None and is excluded. But the script does not report how many cases were excluded, so the user does not know the denominator changed.

1. Problem 3 (less visible): The output reports only the mean, not the count. A mean of 5.0 from 100 cases means something different from a mean of 5.0 from 3 cases. Without the N, the statistic is uninterpretable.

Now, with the codebook in hand (see materials/day1_codebooks.md), we learn:

• trust_eu: 0 = no trust, 10 = complete trust
• 99 = "no answer" (missing)
• None = system missing

The fix is to exclude 99 as well as None, and to report the count alongside the mean. An AI can suggest these changes — but it can only do so after you provide the codebook. The AI cannot infer data semantics from numbers.

Verify one statistic

Urban respondents (excluding 99 and None): 6, 7, 5 → mean = (6 + 7 + 5) / 3 = 6.0. Compute this yourself. Do not trust the printed output until you have confirmed one value independently.

Document the finding

8. Exercise Instructions

The exercises for this module are in exercises.md. Work through them in order:

• Exercise 5.1: Ask AI to explain a short script; verify the explanation
• Exercise 5.2: Ask AI to add comments; check for errors in the comments
• Exercise 5.3: Give AI a dataset and ask for summary statistics; independently verify three
• Exercise 5.4: Ask AI to debug a script with an intentional error; identify what was fixed and what was missed
• Exercise 5.5: Apply the four questions to a code/data workflow

Each exercise includes a collapsible answer key. Work through the exercise before reading the key. The point is to form a judgment before seeing a prepared answer.

The demo notebook

The notebooks/ directory contains analysis_demo.ipynb, a Python notebook you can open in Google Colab (free, no installation). It demonstrates the full scaffold workflow: load a small CSV, request summary statistics, create a plot, run a regression, and verify each step. Open it, run the cells, and use it as a template for your own data.

9. Summary

• AI can explain, debug, document, translate, and generate code — but every use requires independent verification.
• The scaffold approach lets you work with code without writing from scratch: start with working code, ask the AI to modify it, and verify the output.
• For data exploration, verify one statistic, one count, one plot point, and one transformation before trusting the rest.
• Common failure modes — hallucinated functions, plausible wrong analyses, off-by-one errors, dropped observations, silent recoding — are predictable. Check for them explicitly.
• The AI cannot infer what your data mean. The codebook, the research question, and the interpretive judgments are yours.
• Apply the four questions before and after every AI-assisted code or data task.
• Every output you use in your research must carry a disclosure naming the tool, the task, the verification performed, and the decisions the researcher retained.

10. Connection to Module 6

Module 6 (Agents) extends the code-and-data workflow in one direction: automation. Where Module 5 uses AI as an assistant — you ask, AI responds, you verify — Module 6 introduces the agent loop: the AI decides what to do next, executes it, and reports back, potentially without asking at every step. The verification discipline you build in this module is the foundation for that delegation. An agent running unattended is only as trustworthy as the checks you built into its workflow. If you have not learned to verify a single statistic independently, you cannot supervise an agent that produces a hundred.

Module 7 (Validation, Reproducibility, and Research Integrity) extends the workflow in the other direction: evidence standards. It asks how much verification is enough for a publication claim, what a reviewer can inspect, and what happens to your results when the model is updated or deprecated.

Module 5 of 8 · AI as a Research Tool: Workflow Design for the Social Sciences · University of Zurich