← course index

Module 06Agents & Workflow Automation

⏱ 3–4h 📋 conceptual + demo 📌 Module 1; Modules 2–5 helpful 🎯 Documented agent workflow with audit trail

§ workbook

Learning Outcomes

# Module 6: Agents and Workflow Automation

Estimated time: 3–4 hours

Type: Conceptual + practical (workbook + interactive demo)

Prerequisites: Module 1 (How LLMs Work); Modules 2–5 helpful

What you'll produce: A documented agent workflow with an audit trail

1. What Makes Something an Agent?

In Module 1, we distinguished models, interfaces, tools, and agents. Here we go deeper into the agent end of the spectrum.

An agent is a system that combines:

  1. An LLM — the text-in/text-out core
  2. Tools — things the system can do: search the web, read files, write files, execute code, call APIs
  3. A loop — the model decides what to do, the system executes it, the result enters the context window, and the model decides the next action
  4. Memory — the context window plus any persistent storage the system provides

The key property of an agent is that it acts in a loop without needing a human to approve every step. This is what makes agents powerful — and dangerous.

The autonomy spectrum

| Level | Name | What it does | Human role |

|---|---|---|---|

| 0 | Chat | Responds to each message; no tools, no memory across conversations | Human drives every exchange |

| 1 | Chat + tools | Can search, calculate, or run code when asked | Human asks for each tool use |

| 2 | File-working assistant | Reads and edits files; asks before acting | Human approves each action |

| 3 | Assisted agent | Proposes a plan; executes with confirmation at checkpoints | Human approves the plan and key decisions |

| 4 | Agent | Acts on its own within defined boundaries; reports what it did | Human defines the boundaries and reviews the report |

| 5 | Unattended agent | Runs without a human in the loop | Human checks the output later — or never |

For research, levels 2–4 are where most useful work happens. Level 5 is rarely appropriate: if you are not checking the output, you are not doing research — you are publishing model output.

2. Bounded Delegation

The central challenge of agent design is bounded delegation: giving the agent enough authority to be useful without giving it so much that you lose control of the research process.

Before every agent run, answer five questions:

The Five-Box Safety Check

  1. What can it access? Files, databases, APIs, the internet. Be specific. "The data/ folder" is better than "my computer." "Read-only access to the Zotero library" is better than "access to Zotero."
  1. What can it do without asking? Read files? Write files? Delete files? Send emails? Publish to the web? Each capability you grant is a decision you have delegated.
  1. Where is the human decision? Name the specific step where you must approve, reject, or modify. "Before writing the final CSV" or "before including a paper in the review." The decision point must be visible in the workflow, not buried in the agent's internal reasoning.
  1. What record remains? What can a reviewer inspect after the run to understand what happened? The prompt, the model, the date, the actions taken, the outputs produced, and any errors or overrides. If the record is "the agent's final output," that is not enough — you cannot audit a decision from its result.
  1. How does it fail safely? What happens if the agent misunderstands the task, encounters an error, or produces output that does not meet your criteria? Does it stop and ask? Does it continue with wrong assumptions? Does it overwrite good data with bad?

Answer all five questions in writing before the run. After the run, answer them again — what was different from what you predicted?

3. Persistent Instructions

A prompt is text you write once. A persistent instruction is text the system loads every time — before you type anything.

Persistent instructions go by many names:

  • System prompt (the hidden instructions set by the provider)
  • Custom instructions (the part you can edit in some chat interfaces)
  • Project instructions (Claude, ChatGPT)
  • CLAUDE.md / AGENTS.md / SOUL.md (files that coding agents read on startup)
  • Skills (reusable instruction bundles in some agent frameworks)

Despite the different names and loading mechanisms, these are all the same thing: text that enters the context window before the conversation begins, shaping every response.

What persistent instructions can do

  • Define the agent's role and expertise ("You are a research assistant specialising in quantitative text analysis")
  • Set constraints ("Never include a citation you have not personally verified")
  • Specify output formats ("Return results as CSV with these columns")
  • Document methodological decisions ("Use Krippendorff's alpha for reliability, not percentage agreement")
  • Define stopping conditions ("If you cannot find the answer after three searches, report what you found and stop")

What persistent instructions cannot do

  • Make the model verify anything. An instruction that says "verify all citations" does not give the model the ability to verify citations. It may make the model say it verified them — which is worse.
  • Serve as a gate on their own. An instruction is text, not a check. If the model violates an instruction, the instruction does not catch it. A separate verification step — outside the model — is the gate.
  • Guarantee consistent behaviour. The same instruction, on the same model, with the same input, can produce different behaviour across runs. Instructions shape probabilities; they do not determine outcomes.

Designing an instruction artifact

An instruction artifact is a persistent instruction that documents a complete methodological workflow. It should answer six questions:

  1. Task authorised: What is the agent allowed to do?
  2. Inputs allowed: What data, files, or sources can it access?
  3. Decisions already specified: What methodological choices have you made that the agent should not revisit?
  4. When to stop or ask: Under what conditions should the agent pause and request human input?
  5. Output and record: What should the agent produce, and what record of its actions should it keep?
  6. How checked: How will you verify the agent's output?

4. The Agent Demonstration

This module includes an interactive demonstration: a pre-computed agent run with inspection tools. Open lab/agent_demo.html in any browser.

The demonstration shows:

  1. The task: Screen 20 abstracts using a supplied codebook, write a screening CSV, and draft a paragraph summarising the retained papers.
  2. The instruction artifact: The exact persistent instructions given to the agent.
  3. The run transcript: Every action the agent took, in order, with timestamps.
  4. The output: The CSV and the paragraph the agent produced.
  5. The audit: What the agent got right, what it got wrong, and what the human had to override.

Your job is to inspect the run and answer: would you accept this output for your own research? If not, what would you change in the instruction artifact?

5. Peer Stress Testing

The best way to find weaknesses in an instruction artifact is to give it to someone who did not write it.

The stress test protocol

  1. Exchange only the artifact and the input. Do not explain your intent. Do not provide oral instructions. The artifact must stand alone.
  2. First attempt: silent. The reviewer attempts to follow the artifact exactly, without asking questions. They record every point where they had to guess.
  3. Second attempt: with questions. The reviewer can now ask: "What did you mean by X?" "What should I do when Y happens?" The author answers, and both revise the artifact.
  4. Record the changes. What did you add, clarify, or remove because the reviewer could not follow it?

A good instruction artifact survives the silent first attempt: someone who has never seen it before can produce substantially the same output you would produce, given the same input.

6. The Four Questions for Agents

  1. What is being delegated? Not just the task — the sequence of decisions. An agent that screens, classifies, and writes is delegating a chain of judgments, not a single action.
  2. What would make the result believable? The audit trail. Can you reconstruct what happened from the record? Can you identify where a judgment was made and whether it was correct?
  3. What could be lost or flattened? The moment-by-moment decisions that a human researcher makes implicitly. The agent makes them explicitly or not at all — and what is not in the instruction artifact does not happen.
  4. What must remain visible or human? The five-box answers. The decision points. The record. The verification step that is outside the agent's control.

7. Summary

  • Agents combine models, tools, loops, and memory. Their defining feature is autonomy — acting without per-step human approval.
  • Bounded delegation: grant enough authority to be useful, not enough to lose control. Use the five-box check.
  • Persistent instructions document methodological decisions. They are text, not gates — they shape behaviour but do not guarantee it.
  • Stress-test instruction artifacts with a peer. If someone else cannot follow it silently, it is not ready.
  • The audit trail is the evidence. If you cannot reconstruct what the agent did and why, you cannot publish the result.

What's Next

Module 7 provides the evidence framework for evaluating everything you have built: the seven validation checks, reproducibility standards, data boundaries, and disclosure requirements. Every module so far has pointed toward this — now you get the full treatment.

Module 8 integrates everything into a coherent research workflow and helps you build the final portfolio: task, baseline, experiment, evidence, decision, disclosure.

Module 6 of 8 · AI as a Research Tool: Workflow Design for the Social Sciences · University of Zurich

§ exercises

# Module 6: Exercises

Exercise 6.1: Classify by Autonomy Level

Goal: Practice identifying where a system falls on the autonomy spectrum.

For each scenario, identify the autonomy level (0–5) from the workbook:

  1. You paste a prompt into a chat window and receive a text response. Level: ___
  2. You ask the system to read a CSV and produce summary statistics; it does so and displays them. Level: ___
  3. You give the system access to a folder of papers and ask it to screen them, write a CSV, and save it. It reads, classifies, writes, and reports what it did — without asking between steps. Level: ___
  4. You schedule a script to run every morning: search for new papers, screen them, and email you the ones that pass. Level: ___
  5. You ask the system to draft a literature review. It searches for papers, reads them, writes the review, and saves it — all while you are at lunch. Level: ___

Answer Key (click to expand)

  1. Level 0 (Chat) — Pure text-in/text-out, no tools.
  2. Level 1 (Chat + tools) — The system used a code interpreter tool once, at your request.
  3. Level 4 (Agent) — The system acted in a loop (read → classify → write → report) without per-step approval, but within defined boundaries.
  4. Level 5 (Unattended agent) — The system runs without a human in the loop at all. You check the output later.
  5. Level 5 (Unattended agent) — Multiple tool uses, no human in the loop, output delivered after the fact.

Note: scenario 3 and scenario 5 are both agent runs, but scenario 5 delegates more (search strategy, paper selection, synthesis writing) and removes the human further. The difference between level 4 and level 5 is whether a human reviews before publication or use.

Exercise 6.2: The Five-Box Check

Goal: Apply the safety check to a real or hypothetical agent task.

Choose a research task you might delegate to an agent — ideally from your capability map (Module 1). Fill in all five boxes:

Task: _______________

  1. What can it access? _______________
  2. What can it do without asking? _______________
  3. Where is the human decision? _______________
  4. What record remains? _______________
  5. How does it fail safely? _______________

Now ask: if you handed these five boxes to a colleague, could they predict what the agent would do and what you would review? If any box is vague, make it specific.

Answer Key (click to expand)

A good five-box check is:

  • Specific about access: Names folders, databases, or APIs — not "my computer"
  • Honest about autonomy: Distinguishes actions the agent can take without asking from actions that require approval
  • Explicit about decisions: Names the exact point where the human reviews and can override
  • Auditable: The record would let a reviewer reconstruct what happened
  • Safe by default: The failure mode is "stop and report," not "continue with assumptions"

Common gaps:

  • Access that is too broad ("my whole drive")
  • No named human decision point (the agent "just does it")
  • Record is "the final output" (not the actions that produced it)
  • Failure mode is "try again" (which may compound errors)

Exercise 6.3: Write an Instruction Artifact

Goal: Practice writing persistent instructions that someone else could follow.

Write an instruction artifact (CLAUDE.md, AGENTS.md, or a skill) for a bounded research task. Use the six-question template from the workbook (Section 3):

  1. Task authorised: _______________
  2. Inputs allowed: _______________
  3. Decisions already specified: _______________
  4. When to stop or ask: _______________
  5. Output and record: _______________
  6. How checked: _______________

Keep it short — a good instruction artifact is precise, not long.

Answer Key (click to expand)

Here is an example for a literature screening task:

```markdown

# Literature Screening Agent

Task authorised

Screen the supplied abstracts (data/screening_input.csv) using the

codebook in codebook.md. For each abstract, return RETAIN or EXCLUDE

with the specific rule applied and a one-sentence reason.

Inputs allowed

  • data/screening_input.csv (read-only)
  • codebook.md (read-only)

Decisions already specified

  • The codebook is fixed. Do not revise categories or decision rules.
  • If an abstract is ambiguous, RETAIN for full-text assessment.
  • Missing information is a reason to RETAIN, not EXCLUDE.

When to stop or ask

  • If you cannot determine the document type from the available metadata,

flag the record and continue.

  • If the input file is malformed or empty, stop and report the error.
  • Stop after processing all records. Do not paginate or search for more.

Output and record

  • Write results to data/screening_output.csv with columns:

item_id, decision, rule, reason, timestamp

  • Write a run log to data/screening_log.md with:

model, date, total records, counts per decision, any flags

How checked

  • The human will review all EXCLUDE decisions and a random 20% of RETAIN

decisions against the codebook.

  • Any record flagged for ambiguity will be resolved by the human.
  • The screening_output.csv will be compared to a human-coded sample.

```

Exercise 6.4: Stress-Test an Artifact

Goal: Find the gaps in an instruction artifact through peer testing.

Exchange your instruction artifact from Exercise 6.3 with a partner (or use the example in the answer key above). Without asking the author any questions:

  1. Follow the artifact exactly. Record every point where you had to guess.
  2. List what the artifact did NOT specify that you needed to know.

Now discuss with the author. What did they assume you would know? What did you interpret differently than they intended?

Answer Key (click to expand)

Common gaps that stress testing reveals:

  • Assumed knowledge: "Screen using the codebook" — but how? One abstract at a time? All at once? In what format?
  • Undefined terms: "Flag the record" — what does flagging mean? A column? A separate file? A note in the log?
  • Missing edge cases: What if an abstract is in a language the agent cannot read? What if the codebook references a category that does not appear in any abstract?
  • Implicit workflow: The author knows the steps; the artifact does not state them. "Screen the abstracts" assumes the reader knows whether to do it in one prompt or many, in what order, with what checks between steps.

The stress test is not a test of the reader. It is a test of the artifact. Every guess the reader had to make is a gap in the artifact.

Exercise 6.5: Audit the Agent Run

Goal: Inspect a real agent run and decide whether to trust it.

Open the agent demonstration at lab/agent_demo.html. Review the run transcript and output. Then answer:

  1. What did the agent get right? _______________
  2. What did it get wrong? _______________
  3. Would you have caught the errors from the output alone (without the transcript)? _______________
  4. What would you change in the instruction artifact to prevent the errors? _______________
  5. Would you accept this output for a publication? Why or why not?

Answer Key (click to expand)

The specific errors depend on the demonstration version, but common patterns:

  • Errors invisible from output alone: The agent classified a paper incorrectly, but the output CSV looks correct — only the transcript reveals the misclassification. This is why the transcript matters.
  • Plausible wrong answers: The agent wrote a confident-sounding synthesis that misrepresents some papers. The fluency of the output does not correlate with its accuracy.
  • Instruction gaps: The agent made a reasonable decision that violated the researcher's intent because the instruction artifact did not specify the constraint. The fix is a clearer artifact, not a smarter model.

The core lesson: an agent run without an auditable transcript is a black box. If you cannot inspect what the agent did, you cannot verify its output — and you cannot publish it.

Module 6 Exercises · AI as a Research Tool

§ readings

---

Core Readings

Spirling, A., Smith, N. A., & Palmer, A. (2023). "Using Proprietary Language Models in Academic Research Requires Explicit Justification." Nature Computational Science.

Why read it: The case for justifying which model you use, not just that you used one. Proprietary models can be deprecated, modified, or withdrawn without notice. Open-weight models can be preserved and re-run. The choice affects reproducibility. What to take: Your disclosure should name the model and justify the choice. "Convenience" is not a methodological justification. If you use a proprietary model, explain why an open-weight alternative was not suitable. ---

Kapoor, S., & Narayanan, A. (2023). "Leakage and the Reproducibility Crisis in Machine-Learning-Based Science." Patterns, 4(9).

Why read it: A clear explanation of data leakage and its consequences. Relevant to agents because an agent that searches the web or accesses a database may retrieve information that contaminates what should be a held-out test. What to take: Know what your agent can access. If it can search the web, it can find the papers you are asking it to classify — and its "classification" may be retrieval, not judgment. ---

Christin, A. (2020). "The Ethnographer and the Algorithm: Beyond the Black Box." Global Perspectives, 1(1).

Why read it: An ethnographic perspective on algorithmic systems. Christin argues that the "black box" metaphor is misleading — algorithms are not opaque by nature; they are made opaque by the practices around them. The same applies to agents. What to take: Opacity is a choice. An agent whose actions are recorded is less of a black box than a chat interface whose internal reasoning is hidden. Design for auditability. ---

Methodological Readings

Wu, T., Terry, M., & Cai, C. J. (2022). "AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts." Proceedings of the 2022 CHI Conference.

Why read it: A framework for breaking complex AI tasks into chains of smaller, inspectable steps — each with a human check between them. This is the alternative to giving an agent a large task and hoping. What to take: Prefer chains to monoliths. A chain of small, verifiable steps (search → screen → extract → synthesise, with human review between each) is more auditable than one large agent run. ---

Zhou, S., et al. (2024). "Swe-agent: Agent-Computer Interfaces Enable Automated Software Engineering." arXiv.

Why read it: A technical paper on agent design, but the principles transfer: define the interface (what the agent can see and do), provide feedback (what happened after each action), and maintain a state (what the agent knows so far). What to take: An agent's design — what it can access, what it can do, what feedback it receives — shapes its behaviour more than the underlying model. A well-designed agent with a modest model can outperform a poorly-designed agent with a powerful model. ---

Optional: Going Deeper

Park, J. S., et al. (2023). "Generative Agents: Interactive Simulacra of Human Behavior." Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology.

Why read it: The paper that introduced generative agents — LLM-powered agents with memory, planning, and reflection. It is a research paper, not a how-to guide, but it illustrates what agents can do when given persistent memory and a planning loop. What to take: Memory changes agent behaviour qualitatively. An agent that remembers previous runs is a different system from one that starts fresh each time. Your instruction artifact should specify whether the agent persists state between runs. ---

Read Later (Across Modules)

- ERA Living Guidelines (2026) and UZH Recommendations (2023) on responsible AI use — particularly relevant for unattended agent runs. See Module 7.

- Barrie, Palmer & Spirling (forthcoming) on replication — what to record so that an agent run can be reproduced. See Module 7. --- Module 6 Reading List · AI as a Research Tool