Bias in Face Recognition : Assessment and Post-Processing Mitigation

Overview & Synopsis

Face Recognition (FR) systems have reached unprecedented accuracy due to deep learning advances, enabling widespread deployment. Yet, concerns about fairness—especially across demographic subgroups defined by sensitive attributes like gender and ethnicity— persist. This thesis addresses algorithmic bias in FR through two complementary lenses: evaluation and mitigation.The first part introduces a statistically rigorous framework for assessing FR performance and fairness. Focusing on similarity-based ROC analysis, we apply U-statistics theory to derive asymptotically valid confidence intervals for key metrics. We also propose a scalar uncertainty score to quantify the statistical reliability of fairness assessments. This enables practitioners to identify disparities that are not only large but statistically significant.The second part tackles post-processing bias mitigation for frozen, pre-trained FR models. We begin with gender bias, proposing a novel mitigation approach using two operationally relevant fairness metrics: BFAR and BFRR, which capture disparities in false acceptance/rejection rates under fixed-security constraints. We introduce the Ethical Module, a shallow neural network that transforms embeddings using a Fair von Mises-Fisher loss. This loss models group-specific intra-class variance, enabling fairness control without access to sensitive attributes at deployment. The method improves fairness with minimal performance loss and low computational cost.Next, we address racial bias via a post-processing strategy aligned with FR objectives. Using centroid-based pseudo-scores as proxies for similarity, we develop the Centroid Fairness loss, implemented through a lightweight Fairness Module. This module, trained on top of a frozen model, facilitates subgroup alignment and effectively mitigates racial disparities while preserving overall accuracy—challenging the conventional fairness-performance trade-off.Together, these contributions offer a statistically grounded and practically effective toolbox for evaluating and mitigating bias in face recognition. By integrating uncertainty quantification, fairness-aware representation learning, and scalable post-hoc debiasing, this thesis is an additional step towards the development of equitable and trustworthy biometric technologies.