Augmented Reality Based on Image Features in Minimally Invasive Ear Surgery

Overview & Synopsis

The field of otolaryngological surgery has seen significant advancements with the integration of augmented reality (AR) technologies. AR has the potentials of providing enhanced precision and safety during delicate procedures. This research focuses on developing a specialized registration system dedicated for endoscopic ear surgery.The initial approach involved aligning 2D images derived from endoscopic frames and virtual endoscopy generated from CT scans. Using a deep learning-based segmentation method, anatomical features of the malleus and tympanic membrane were identified. Then, by applying the Iterative Closest Point (ICP) algorithm we align the segmented features. This produced a transparent overlay of preoperative data onto live surgical views.Further developments in the research included transitioning from 2D-to-2D registration to a more sophisticated 3D-to-2D alignment process. The algorithm included an optimization process that enabled the rendering of an image with a clear view of middle ear structures. Evaluation in a controlled surgical setting using temporal bone phantoms demonstrated the system's high accuracy in registering images and its potential to minimize human errors during operations. Moreover, in a novel study, we evaluated the approach of target registration reprojection error (TRRE) as a metric for measuring registration error. TRRE evaluates the error when one of the inputs is of 2D dimensionality. It was found that the metric is bias to several factors such as the distance of the target from the camera lens.Additionally, a systematic review of image-to-patient registration techniques was conducted. It highlighted the strengths and limitations of various methods, including anatomy-based, marker-based, surface-based, and computer vision-based approaches. The review underscored the necessity for precise and quick registration methods in computer-assisted surgeries (CAS). The findings suggested that the invasive marker-based methods remain the gold standard in terms of accuracy, while surface-based methods offer speed but with lower precision. The emerging computer vision-based methods promise real-time performance but require further refinement to handle surgical challenges like tissue shifts and rapid anatomical changes.The research also explored the employment of Persistent Homology, a mathematical theory, in the segmentation task. PH was incorporated in the loss function of the U-Net, aiming at teaching the model to capture the topological features of the target anatomies. As execution times were long, current work is focusing on developing a PH library compatible with a GPU rather than a CPU.In practical terms, the developed AR system offers several advantages for endoscopic ear surgeries. From a medical standpoint, it enables highly precise interventions, reducing possible postoperative complications. From a technical perspective, the research tackled challenges related to image processing, latency reduction, and the creation of a user-friendly system.Future work may focus on integrating additional imaging modalities like intraoperative MRI, and refining deep learning models for even more robust feature extraction.