LLM-Supported Generation of ML Prototype Cards

Machine learning (ML) systems evolve through repository-centered development in which pull requests (PR) continuously change code, configuration, data interfaces, and operational assumptions. Documentation often lags behind, causing documentation drift in which system descriptions become outdated or lose traceability to evidence needed for review and accountability. This is especially pronounced in ML because relevant information is scattered across repository artifacts and must serve multiple stakeholders. This thesis introduces the ML System Card (MSC), a system-level documentation artifact maintained through a PR-first, evidence-based, review-governed workflow under a strict repository-only evidence boundary. The MSC is a single canonical file that consolidates stakeholder-relevant information in a version-controlled representation. A large language model-based generator produces structured change proposals grounded in repository evidence through file-and-line anchors and an evidence index. Reviewers assess proposals as diffs with explicit rationales, and the canonical card is updated only after review decisions. A multi-pass prompting pipeline extracts and consolidates information while enforcing schema and evidence constraints to prioritize traceability over autonomous edits. A case study on unfamiliar open-source repositories and a PR-bounded update scenario suggests that repository-grounded proposals surface documentation gaps and support consistent, reviewable updates while preserving human control when evidence is incomplete.

Project information

Status:

Finished

Thesis for degree:

Master

Student:

Baran Tanriverdi

Supervisor:
Part of research project:

SE4ML - Processes, People and Tools

Id:

2026-009