Teaching Large Language Models: A Pipeline-Centered Course Design and Mixed-Methods Evidence From a Cross-Listed Undergraduate/Graduate Offering
Journal of Information Technology Education: Innovations in Practice · 2026 · 25 · 45
DOI: 10.28945/5899 ↗Kurz gefasst
Mixed-Methods-Studie in der Zeitschrift Journal of Information Technology Education: Innovations in Practice, 2026. Verfasst von Adu A Baffour. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Abstract (Originalsprache)
Aim/Purpose: This study presents and evaluates an innovative course design for teaching generative artificial intelligence (AI) and large language models (LLMs) as technical subjects at a university. It addresses two problems: the lack of reusable course architectures for teaching how LLM systems function and the shortage of evidence on how students experience and perform in such courses, particularly in cross-listed undergraduate/graduate offerings.
Background
Most educational responses to generative AI treat it as a tool that changes how students write, code, and are assessed for their work. This study documents a course in which the LLM is the subject of instruction. The innovation is a pipeline-centered design that sequences instruction through five stages: representation, model internals, training and evaluation, adaptation, and system integration. It aligns cumulative assessments to that pipeline and differentiates undergraduate and graduate rigor through epistemic demand. The paper also describes a proposed misconception-repair activity, the Pipeline Debugger, which targets relational understanding and awaits formal evaluation. Methodology: A mixed-methods evaluation of a single offering of the course (Spring, 2026) combined two intentionally unlinked data sources: an anonymous end-of-semester survey (n = 45) and a de-identified coursework dataset (n = 49). Analyses included ordinal descriptives, a retrospective pre/post design with non-parametric effect sizes and bootstrap confidence intervals, undergraduate/graduate comparisons, reflexive thematic analysis of open-ended responses, and group-level triangulation of perceived and assessment-based difficulties. Contribution: This paper contributes a transferable, low-compute course architecture for teaching the LLM pipeline, a reusable evaluation toolkit (survey instrument, qualitative codebook, topic-to-assessment mapping, and difficulty-matrix protocol), and early empirical evidence on how the design operates in practice.
Findings
Students most often named attention mechanisms as difficult and rated them among their least-mastered topics, yet the cohort’s performance on related assessments was near the ceiling. Students also reported positive perceptions and self-reported knowledge gains across all five pipeline stages. These gains were measured using a retrospective pre/post self-rating and reflect perceived, rather than directly measured, learning; near-universal gains are an expected feature of this design. Live coding labs were the most frequently cited contributor to learning. Graduate students, who had more machine learning experience, modestly outperformed undergraduates on several assessments, although most scores clustered near the ceiling. Perceptions were similar across levels. Recommendations for Practitioners: Organize LLM courses around the pipeline rather than isolated tools; give implementation-focused, live coding activities a central role; differentiate cross-listed rigor by the evidence students must produce rather than by separate content; and combine perception and performance data to identify instructional priorities. Recommendation for Researchers: The design and instruments support replication and extension across different institutions. Priority areas include the direct evaluation of misconception-repair activities, such as the proposed Pipeline Debugger and a focused study of the confidence-performance divergence observed for model internals. Impact on Society: As LLM systems spread through work and daily life, courses that teach how these systems operate, not only how to use them, can help prepare a workforce and citizenry able to build, evaluate, and critically appraise AI systems in the future. Future Research: Future research should include multi-institution replication, designs that link perception and performance data at the individual level under appropriate privacy protections, and measures that discriminate above the performance ceiling observed in the course assessments.
Zitieren
APA
Baffour, A. A. (2026). Teaching Large Language Models: A Pipeline-Centered Course Design and Mixed-Methods Evidence From a Cross-Listed Undergraduate/Graduate Offering. Journal of Information Technology Education: Innovations in Practice, 25, 45. https://doi.org/10.28945/5899
Vancouver
Baffour AA. Teaching Large Language Models: A Pipeline-Centered Course Design and Mixed-Methods Evidence From a Cross-Listed Undergraduate/Graduate Offering. Journal of Information Technology Education: Innovations in Practice. 2026;25:45. doi:10.28945/5899
BibTeX
@article{Baffour2026,
author = {Baffour, Adu A},
title = {Teaching Large Language Models: A Pipeline-Centered Course Design and Mixed-Methods Evidence From a Cross-Listed Undergraduate/Graduate Offering},
journal = {Journal of Information Technology Education: Innovations in Practice},
year = {2026},
volume = {25},
pages = {45},
doi = {10.28945/5899}
}
EUVIMED