Automated Data Error Cleaning Impact on Federated Learning Utility and Fairness

Overview & Synopsis

Data cleaning is a critical process for reliable and effective Federated Learning (FL). While current FL data cleaning methods mainly focus on their impact on model accuracy, their effects on FL model fairness and overall utility remain largely unexplored. This paper presents the first comprehensive empirical study that investigates the impact of automated data cleaning for tabular data on both FL model utility and fairness. Our study encompasses 2,365 different FL workloads, involving 21 data cleaning methods, applied to five real-world datasets, all implemented within our extensible TITANIA framework, resulting in FL workload traces consisting of 1,764,750 records. The key finding of our analysis is twofold: (i) data cleaning actually influences both FL model utility and fairness, with disparities in impacts on different datasets; (ii) FL data cleaning is not robust to high data error rates or high non-IID data distributions, as is usually the case in FL. This opens interesting research directions for novel robust and multi-criteria FL data cleaning, where model fairness and quality are provided by design.