Development and Validation of Objective Performance Metrics for Robot-Assisted Radical Prostatectomy: A Pilot Study
Abstract
Purpose:
We explore and validate objective surgeon performance metrics using a novel recorder (“dVLogger”) to directly capture surgeon manipulations on the da Vinci® Surgical System. We present the initial construct and concurrent validation study of objective metrics during preselected steps of robot-assisted radical prostatectomy.
Materials and Methods:
Kinematic and events data were recorded for expert (100 or more cases) and novice (less than 100 cases) surgeons performing bladder mobilization, seminal vesicle dissection, anterior vesicourethral anastomosis and right pelvic lymphadenectomy. Expert/novice metrics were compared using mixed effect statistical modeling (construct validation). Expert reviewers blindly rated seminal vesicle dissection and anterior vesicourethral anastomosis using GEARS (Global Evaluative Assessment of Robotic Skills). Intraclass correlation measured inter-rater variability. Objective metrics were correlated to corresponding GEARS metrics using Spearman’s test (concurrent validation).
Results:
The performance of 10 experts (mean 810 cases, range 100 to 2,000) and 10 novices (mean 35 cases, range 5 to 80) was evaluated in 100 robot-assisted radical prostatectomy cases. For construct validation the experts completed operative steps faster (p <0.001) with less instrument travel distance (p <0.01), less aggregate instrument idle time (p <0.001), shorter camera path length (p <0.001) and more frequent camera movements (p <0.03). Experts had a greater ratio of dominant-to-nondominant instrument path distance for all steps (p <0.04) except anterior vesicourethral anastomosis. For concurrent validation the median experience of 3 expert reviewers was 300 cases (range 200 to 500). Intraclass correlation among reviewers was 0.6-0.7. For anterior vesicourethral anastomosis and seminal vesicle dissection, kinematic metrics had low associations with GEARS metrics.
Conclusions:
Objective metrics revealed experts to be more efficient and directed during preselected steps of robot-assisted radical prostatectomy. Objective metrics had limited associations to GEARS. These findings lay the foundation for developing standardized metrics for surgeon training and assessment.
References
- 1 : Teaching surgical skills–changes in the wind. N Engl J Med2006; 355: 2664. Google Scholar
- 2 : Surgical skill and complication rates after bariatric surgery. N Engl J Med2013; 369: 1434. Google Scholar
- 3 : Outcomes for radical prostatectomy: is it the singer, the song, or both?. J Clin Oncol2012; 30: 476. Google Scholar
- 4 : Cancer control and functional outcomes after radical prostatectomy as markers of surgical quality: analysis of heterogeneity between surgeons at a single cancer center. Eur Urol2011; 59: 317. Google Scholar
- 5 : Diffusion of surgical innovations, patient safety, and minimally invasive radical prostatectomy. JAMA Surg2014; 149: 845. Google Scholar
- 6 : Does training on a virtual reality robotic simulator improve performance on the da Vinci surgical system?. J Endourol2010; 24: 467. Google Scholar
- 7 : Robotic surgical education: a systematic approach to training urology residents to perform robotic-assisted laparoscopic radical prostatectomy. Urology2006; 68: 75. Google Scholar
- 8 : Learning curves for urological procedures: a systematic review. BJU Int2014; 114: 617. Google Scholar
- 9 : From proficiency to expert, when does the learning curve for robotic-assisted prostatectomies plateau? The Columbia University experience. World J Urol2007; 25: 105. Google Scholar
- 10 : Validation of the AUA BLUS tasks. J Urol2016; 195: 998. Link, Google Scholar
- 11 : Objective Structured Assessment of Technical Skill (OSATS) for surgical residents. Br J Surg1997; 84: 273. Google Scholar
- 12 : Objective assessment of technical surgical skills. Br J Surg2010; 97: 972. Google Scholar
- 13 : Global Evaluative Assessment of Robotic Skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol2011; 187: 247. Google Scholar
- 14 : Development and validation of an objective scoring tool for robot-assisted radical prostatectomy: Prostatectomy Assessment and Competency Evaluation. J Urol2017; 197: 1237. Link, Google Scholar
- 15 : Structured and modular training pathway for robot-assisted radical prostatectomy (RARP): validation of the RARP assessment score and learning curve assessment. Eur Urol2016; 69: 526. Google Scholar
- 16 : Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills. Am J Surg2016; 211: 398. Google Scholar
- 17 : Measuring to improve: peer and crowd-sourced assessments of technical skill with robot-assisted radical prostatectomy. Eur Urol2016; 69: 547. Google Scholar
- 18 Intuitive Surgical, Inc.: 2007 Annual Report. Google Scholar
- 19 Intuitive Surgical, Inc.: 2015 Annual Report. Google Scholar
- 20 : Content and construct validation of a robotic surgery curriculum using an electromagnetic instrument tracker. J Urol2012; 188: 919. Link, Google Scholar
- 21 : Beyond task time: automated measurement augments fundamentals of laparoscopic skills methodology. J Surg Res2014; 192: 329. Google Scholar
- 22 : Objective measures for longitudinal assessment of robotic surgery training. J Thorac Cardiovasc Surg2012; 143: 528. Google Scholar
- 23 : Quantitative objective assessment of preoperative warm-up for robotic surgery. UW BioEngineering General Exam. Seattle: University of Washington2012: 50. Google Scholar
- 24 : Comparative assessment of three standardized robotic surgery training methods. BJU Int2013; 112: 864. Google Scholar
- 25 : Development and validation of a novel robotic procedure specific simulation platform: partial nephrectomy. J Urol2015; 194: 520. Link, Google Scholar
- 26 : Structured learning for robotic surgery utilizing a proficiency score: a pilot study. World J Urol2017; 35: 27. Google Scholar
- 27 : External validation of Global Evaluative Assessment of Robotic Skills (GEARS). Surg Endosc2015; 29: 3261. Google Scholar
