Deep learning methods for Action Unit detection

Overview & Synopsis

Action Unit (AU) detection aims at automatically caracterizing facial expressions with the muscular activations they involve. Its main interest is to provide a low-level face representation that can be used to assist higher level affective computing tasks learning. Yet, it is a challenging task. Indeed, the available annotated databases display limited face variability and are imbalanced toward neutral expressions. Furthermore, as AU involve subtle face movements they are difficult to annotate so that some of the few provided datapoints may be mislabeled. Therefore, a deep neural network trained on such datasets is likely to overfit and to consequently display poor generalisation performance. In this thesis, we explore three different ways to improve the generalisation performance of an AU detector.First, we aim at exploiting known inter-dependencies between Action Units to better structure the prediction of the network. To do so, we leverage multi-task recurrent neural networks that process tasks sequentially and help each task prediction with the results of the previous ones. However, such method requires imposing an arbitrary order into an unordered task set. Crucially, it has been shown that this ordering choice impacts predictive performance. For that purpose, we propose Multi-Order Networks (MONET) that jointly learns multiple tasks along with the order in which they should be predicted. Experimentally, we first show that MONET is able to learn the optimal prediction order in a controlled environment. Second, we validate MONET architecture by showing that MONET outperform existing multi-task baselines on multiple attribute detection problems chosen for their wide range of dependency settings. More importantly, we demonstrate that MONET significantly extends state-of-the-art AU detection performance on both DISFA and BP4D. Second, we build upon the observation that each Action Unit is locally constrained to a known region of the face. Consequently we attempt to guide our deep Action Unit detector towards paying attention to AU-relevant zones. For that purpose, we draw inspiration from the recent success of vision transformer architectures and their multi-head attention mechanism. More precisely we benchmark several strategies to guide the multi-head attention of vision transformers with either landmarks-based and face parsing-based priors. The main outcome of our study is that, whatever the nature of the prior, forcing different attention heads to pay attention to different zones is an useful strategy as it improves AU detection performance on both DISFA and BP4D. Finally, we tackle the noisy annotation problem of AU detection. To this end, we first attempt to use label smoothing to reduce the network confidence and consequently mitigate the influence of noisy examples on the training. However, we experimentally observe that label smoothing degrades AU detection performance. To explain this drop in performance, we hypothetize that applying label smoothing in imbalanced scenarios worsens the pre-existing under-confidence in the minority class. To circumvent that issue, we propose a Vanilla Asymetric Label Smoothing (VALS) that use separate smoothing coefficients for positive and negative examples. Consequently VALS provides the flexibility to reduce over-confidence in the majority class while leaving the minority class untouched. We further refine this strategy by proposing Robin Hood Label Smoothing (RHLS) that smoothes only the majority class with a smoothing coefficient that is scaled based on AU empirical frequencies. Experimentally, we show that both VALS and RHLS display promising performance on BP4D and outperforms state-of-the-art results on DISFA.