Development and Validation of the Metric-Based Assessment of a Robotic Dissection Task on an Avian Model

Introduction: The introduction of robot-assisted surgical devices requires the application of objective performance metrics to verify performance levels. Objective: To develop and validate (face, content, response process, and construct) the performance metrics for a robotic dissection task using a chicken model. Methods: In a procedure characterization, we developed the performance metrics (i.e., procedure steps, errors, and critical errors) for a robotic dissection task, using a chicken model. In a modified Delphi panel, 14 experts from four European Union countries agreed on the steps, errors, and critical errors (CEs) of the task. Six experienced surgeons and eight novice urology surgeons performed the robotic dissection task twice on the chicken model. In the Delphi meeting, 100% consensus was reached on five procedure steps, 15 errors and two CEs. Novice surgeons took 20 min to complete the task on trial 1 and 14 min during trial two, whereas experts took 8.2 min and 6.5 min. On average, the Expert Group completed the task 56% faster than the Novice Group and made 46% fewer performance errors. Results: Sensitivity and specificity for procedure errors and time were excellent to good (i.e., 1.0-0.91) but poor (i.e., 0.5) for step metrics. The mean interrater reliability for the assessments by two robotic surgeons was 0.91 (Expert Group inter-rater reliability = 0.92 and Novice Group = 0.9). Conclusions: We report evidence which supports the demonstration of face, content, and construct validity for a standard and replicable basic robotic dissection task on the chicken model.