Contributions to the Automation of Preoperative 3D Model to 2D Image Registration in Mini-Invasive Liver Surgery : Primitive Detection, Pose Estimation, Registration and Anatomical Modelling

Overview & Synopsis

Minimally invasive liver resection consists in removing liver parts enclosing tumours using surgical tools, while visualising the abdominal cavity through an endoscope, both inserted through small incisions in the abdominal wall. It offers significant advantages over open liver resection, including fewer postoperative complications and shorter hospital stays. However, the localisation of liver inner structures, such as tumours and blood vessels, remains challenging.This information can be extracted from preoperative imaging and used for building a 3D model of the liver with its inner structures. However, this model must be moved and deformed for its projection to be aligned with the 2D image of the intra-abdominal surgical scene; this is the 3D/2D registration problem. Augmented reality enhances mini-invasive images with information from the registered preoperative model. Accurate augmented information could alleviate the limitations of mini-invasive surgery.To this end, previous computer-based approaches have established a patient-specific registration pipeline, mainly relying on 3D/2D liver surface landmark correspondences to estimate pose (rigid movement) and then deformation. However, these methods still contain manual steps. In clinical practice, this is not handy for the surgeon, whose focus should not be disturbed and gloves should be kept sterile. The main objective of this thesis is to improve upon this baseline by automating the 3D-2D intraoperative registration process using 3D/2D correspondence information. We propose two approaches: automating the manual intraoperative steps of the registration pipeline, or using a learning-based framework.We first review the baseline pipeline and redefine the landmarks. This facilitates the identification of relevant 3D/2D correspondences. Additionally, we compare different deformation models and select one based on biomechanical simulations followed by dimension reduction.Next, we automate the manual intraoperative steps from the baseline pipeline, comprising landmark annotation on mini-invasive images and pose estimation. We formulate the former as an image segmentation task and compare segmentation neural networks based on encoder-decoder architectures. The best results for image independent inputs are achieved with a fully attention-based network, but these are further improved when incorporating additional information from other images and masks. Pose estimation is tackled using an iterative visibility-aware algorithm, refining 3D/2D landmark point correspondences to estimate pose according to the visible 3D surface landmark parts from the previous iteration. This method obtains competitive results compared to manual pose estimation, while executing in a few seconds.Regarding the learning-based framework, we draw connections to human body shape reconstruction to adapt an encoder-regressor architecture network to the 3D/2D liver registration problem. Distance maps of automatically or manually annotated landmarks are input to the encoder, while pose and deformation parameters are iteratively regressed. Preoperative training involves simulating corresponding inputs and outputs. This patient-specific approach obtains registration results on par with previous state-of-the-art methods, while ensuring real-time network inference.Instead of a patient-specific deformation model, the second learning-based approach uses a generic liver shape model, which is built using anatomical priors. This leads to very low surface registration and reconstruction errors. This patient-generic approach also includes a preoperative block for processing patient-specific data. Although the 3D-2D registration accuracy is slightly lower than that of patient-specific methods, it does not require per-patient retraining and can be applied without patient-specific data, facilitating both patient-generic and patient-specific image augmentation.