Multi-Stream Deep Architecture for Surgical Phase Recognition on Multi-View RGBD Videos

Overview & Synopsis

With the increasing volume of surgeries and the advancement in medicine and technology, having a context-aware system (CAS) is becoming a necessity in the operating room (OR). By understanding the surrounding physical environment, a CAS will allow the possibility to improve OR scheduling, to design context-sensitive user interfaces, and to develop the automatic transcription of medical procedures. In this paper, we address one of the essential components of a CAS, namely the ability to recognize surgical phases during a surgical procedure. Here, we focus on performing this recognition task on vertebroplasty procedures recorded by a ceiling-mounted multi-view RGBD camera system. Instead of using hand-crafted visual features, we propose to learn multi-modal visual features using deep learning techniques. We design a neural network architecture which takes RGB, depth, and motion images as input and computes a visual feature representation shared among the modalities. Using this network, visual features are then extracted and passed to a recognition pipeline, which consists of SVM and HHMM. This pipeline is used to enforce the temporal constraints from surgical workflow into the recognition process. To investigate the performance and generalizability of the network, we perform the task on two new multi-view RGBD datasets, capturing in total 37 surgeries performed in two different hybrid ORs. Through an extensive comparison with other visual features, we show that the features extracted from the proposed network yield state-of-the art results for this recognition task.