Evaluating Usefulness and Usability of JupyterRecSys

Machine Learning (ML) solution prototyping in Jupyter Notebooks often involves repetitive coding tasks, yet structured code reuse mechanisms remain limited. While Jupyter RecSys was developed to address this gap through context-aware recommendations, prior evaluations focused solely on technical accuracy, leaving user-centric quality attributes unassessed. This thesis presents a mixed-methods user study to evaluate the perceived usefulness and usability of JupyterRecSys during ML prototyping. Fourteen participants with varying levels of expertise completed a Jupyter Notebook task while optionally employing the system. Data were collected via interaction logs, Think-Aloud Protocols, the System Usability Scale, and semi-structured interviews. Quantitative results revealed active engagement and excellent perceived usability. Qualitative findings indicated significant reductions in cognitive load and manual search effort, particularly benefiting less experienced developers. All five predefined propositions regarding usefulness, effort reduction, workflow integration, usability, and adoption were supported by the empirical evidence. The study concludes that JupyterRecSys effectively bridges the gap between technical accuracy and practical developer value, confirming its viability as a supportive tool for ML solution prototyping development. These findings contribute a replicable evaluation methodology and offer actionable insights for enhancing code cell recommendation systems.

Project information

Status:

Finished

Thesis for degree:

Bachelor

Student:

Bo-Jun Hwang

Supervisor:
Part of research project:

SE4ML - Processes, People and Tools

Id:

2026-007