Dagger: A Data (not code) Debugger

With the democratization of data science libraries and frame- works, most data scientists manage and generate their data analytics pipelines using a collection of scripts (e.g., Python, R). This marks a shift from traditional applications that communicate back and forth with a DBMS that stores and manages the application data. While code debuggers have reached impressive maturity over the past decades, they fall short in assisting users to explore data-driven what-if sce- narios (e.g., split the training set into two and build two ML models). Those scenarios, while doable programmati- cally, are a substantial burden for users to manage them- selves. Dagger (Data Debugger) is an end-to-end data de- bugger that abstracts key data-centric primitives to enable users to quickly identify and mitigate data-related problems in a given pipeline. Dagger was motivated by a series of interviews we conducted with data scientists across several organizations. A preliminary version of Dagger has been in- corporated into Data Civilizer 2.0 to help physicians at the Massachusetts General Hospital process complex pipelines.