In-Network Learning : Distributed Training and Inference in Communication Networks

Overview & Synopsis

We study the problem of distributed statistical learning. More specifically, we consider the distributed training and inference setting. In the considered scenario, multiple nodes need to work together to predict some quantity of interest/event at a central node. Some nodes observe data related to the target quantity, while others do not; however, the nodes that do not have access to data form a link between the central node and the nodes that observe data. We consider the case in which a directed acyclic graph can model such a network. We study how one can train such a complex network in a distributed manner, i.e., without the exchange of raw data, for the case in which each node is equipped with a neural network to process the data. Inspired by the famous information bottleneck problem, we first use rate-distortion theory to derive an achievable trade-off between the network's performance under logarithmic loss and the rate of the messages passed between the nodes. Then, using the obtained trade-off, we showcase how one can derive a loss function that can jointly train the neural networks at the nodes in a distributed manner. We also study the problem of scheduling and power allocation for the proposed model for the star network topologies, i.e., in which the data observing nodes are connected directly to the central node making the prediction. Due to the distributed nature of the data and the lack of a known prior, no node can provide a definitive picture of the system and correctly decide the allocation of the resources. To overcome this issue, Prof. Abdellatif Zaidi patented a solution in which 1. Each device measures its own local assessment of how relevant is the data it holds for the inference task at hand and transmits it to the decision node. 2. That device also simultaneously sends the output activation values to the decision node. 3. The decision node combines all received information (activation values from the various devices as well as their individual local assessments) and then forms its own (global) assessment of how relevant each device's observation is. 4. The central nodes used the obtained assessments for scheduling for power allocation. In this thesis, we build upon the patent by presenting a method for computing the local and global assessments and proposing new scheduling and power allocation algorithms