Controllability and Efficiency in Generative models

Overview & Synopsis

Scene understanding and synthesis are naturally related since on is the inverse process of the other. In the case of comprehension, the goal is to produce a semantic (what type) and parametric (what properties) description of an image or a video, whereas in scene synthesis, the goal is to produce a realistic image or video depicting a semantic and parametric description. In this PhD, we want to solve both tasks jointly by using image editing as a pretext to train deep neural networks for analysis and synthesis. First, we focus on human editing in videos, where we aim at replacing an actor in a video by anthoer one. In order to generate a realistic output, a first neural network has to produce a relevant analysis of the scene, and a second neural network has to use this analysis correctly so as to generate the modified video. Then we focus on 3D scene editing, in which not only humans but also objects or context can be edited.