Generative AI: Evidence, Failure, and Judgment
New for Spring 2027. Open to all majors. No programming and no math background needed. A 4-credit course for about 30 students. Registration on SPIRE opens November 9, 2026.
Description
How do generative AI systems turn words into answers, and when should we trust them? Students learn the core mechanics of language models (tokens, sampling, embeddings, attention, retrieval and agents), then test real systems through structured experiments, document where and why they fail, and contribute findings to a shared record of failures. Written position papers take up understanding, authorship, labor and trust. Open to all majors; no programming or math background required. Substantial direct interaction with generative AI tools is a required part of nearly every assignment.
In plain words
You use AI tools every day. This course is about knowing when to trust them. You will learn how systems like ChatGPT turn your words into answers, run careful experiments to see where they break, and keep a record of what you find. Along the way we take on the big questions: can a machine that produces the right answer understand anything, who owns what it writes, and what should we believe when it sounds sure of itself.
Who it is for
Every major. You work all term in a permanent team of 5 or 6, mixed by major.
No programming, ever. You never write or install code. Everything happens in a web browser: the UMass AI portal and a course workbench built for this class (a prompt bench, a tokenizer, a sampling control, an embedding plot, an attention heatmap and a pipeline builder).
No math background beyond everyday arithmetic. Fractions, percentages and reading a simple chart are enough. The course teaches the rest of what it uses, and you do the key calculations by hand before checking them against a working system.
All you need is a laptop. There is no textbook and nothing to buy.
At a glance
| Course | CICS 190G, Generative AI: Evidence, Failure, and Judgment |
| Term | Spring 2027 (first offering, as an experimental course) |
| Credits | 4 |
| Prerequisites | None |
| Who can enroll | All undergraduates, any major |
| Class size | About 30 |
| Format | Two 75-minute class meetings and one 60-minute lab each week, in person |
| Days and times | To be set; check SPIRE when registration opens |
| Grading | Letter grade only |
| General education | An R2 (Analytic Reasoning) designation has been proposed; check SPIRE for whether it applies in Spring 2027 |
| Instructor | Tim Richards, tdr@umass.edu |
What you will learn to do
- Write prompts carefully and test whether they work, instead of trusting the first answer.
- Explain how a language model works (tokens, embeddings, attention, sampling) well enough to predict where it will fail.
- Build and test a small working AI system, such as one that answers from a set of sources, and say what each step adds and where it breaks.
- Audit a real AI product: design a test, run it enough times to report a failure rate, and say what the result does and does not show.
- Take a position on a question AI raises, then change your mind in writing when the evidence says you should.
- Tell a marketing claim apart from what a system can actually do.
- Add a documented finding to a shared record of AI failures that outlives the semester.
How the course runs
- Class meetings. Short talks, never longer than 15 minutes, followed by team tasks on real AI systems.
- Labs. Hands-on work on that week’s workbench lab, with the instructor in the room.
- Experiments and an audit. You run structured tests on deployed AI products. The midterm is a system audit you design yourself.
- Dialogues. Four short position papers on questions like “Does producing correct output require understanding anything?”, each followed by a seminar.
- Capstone. Teams finish with a project and a demo day.
Schedule
Thirteen units in three parts: How it works (Weeks 1 to 5), Building and breaking (Weeks 6 to 9), and Judgment (Weeks 10 to 14).
| Week | Unit | Topics |
|---|---|---|
| 1 | Seeing past the marketing | Hype and reality; what does “understanding” mean? |
| 2 | Prompting as a discipline | Roles and framing; few-shot, decomposition, self-critique |
| 3 | Tokens and chance | Tokenization; sampling and temperature |
| 4 | Meaning as position | Embeddings; similarity search |
| 5 | Attention | Dialogue 1 seminar; attention, and a transformer block by hand |
| 6 | Grounding answers in sources | Why retrieval exists; building a retrieval pipeline |
| 7 | Red-teaming | Red-teaming method; midterm audit studio |
| 8 | Agents and tools | Tools and structured output; how agents fail |
| 9 | Choosing the approach | Prompting, retrieval and fine-tuning compared |
| 10 | Whose data, whose defaults | Bias and data provenance; Dialogue 2 seminar |
| 11 | Confidence and what we can see inside | Hallucination and grounding; the limits of looking inside |
| 12 | Authorship and labor | Dialogue 3 seminar; labor and economics |
| 13 | Capstone | Capstone studio |
| 14 | Capstone | Demo day; Dialogue 4 seminar |
The week-by-week schedule with every assignment and due date is in the syllabus.
Questions
Write to Tim Richards at tdr@umass.edu. Enrolled students get the course Canvas site, with the dated calendar and appointment hours, at the start of the term.