Leakage of Sensitive Data from Deep Neural Networks

Overview & Synopsis

It has been shown that Machine Learning (ML) models can leak information about their training sets. This is a critical issue in the case where the training data is of a sensitive nature, e.g., medical applications where the data belongs to patients.A popular approach for measuring the leakage of information from ML models is to perform inference attacks against the models. The goal of this approach is to measure the privacy of the system as the robustness to inference attacks. These attacks are mainly categorized into Membership Inference Attacks (MIAs) and Attribute Inference Attacks (AIAs). The goal of a MIA is to determine if a sample or group of samples are part of the training set of the model, while an AIA tries to infer or reconstruct a sample from the trained model.Although there exist other methods for measuring privacy in ML, such as differential privacy, the main focus of this thesis is on inference attacks.First, we derive theoretical bounds on the success rate of an attacker. This result provides an upper bound on the success probability of an inference attack in the specific case where the attacker has access to the model parameters of the trained model, and therefore in any other scenario where the attacker possesses less information. Second, we derive bounds that link the generalization gap of a ML model to the success rate of an attacker against this model. Third, we derive a list of results that relate the mutual information between the trained model and its training set to the generalization gap and the success rate of an attacker.These theoretical results are illustrated in a toy scenario. The lower bound relating the generalization gap to the success rate is tested and compared to state of the art MIAs in a more realistic scenario.Fourth, we use our framework to describe a set of AIAs and test their effectiveness against a model trained to classify handwritten digits. The data set contains the identity of the writers, and we use this as the sensitive information to be determined by the AIAs.Fifth, we benchmark the performance of different MIAs strategies against state of the art image classification models. We describe and categorize the existing state of the art strategies. We explore the use of out of distribution detection techniques and diversity measures for MIAs and compare their effectiveness to the state of the art. We also propose a technique based on the norm of the minimum perturbation necessary to make a model change its prediction using an adversarial attack. We empirically show that having access to additional samples that can be used as training data for the attacker does not provide an advantage over strategies that do not require additional data. We rank different strategies based on their performance against the state of the art image classification models.