Echo State Transformer

Overview & Synopsis

We present a new hybrid architecture that combines the advantages of Transformer and Reservoir Computing to overcome the inherent quadratic complexity of traditional Transformer. Our approach integrates working memory through multiple reservoirs per layer, each acting as an independent memory with its own internal dynamics, and using attention mechanisms to select relevant information from the current input and the state of other memories. Although innovative, this work is heavily inspired by Echo State Networks (ESN), which have demonstrated their ability to encode and retain information – including chaotic information – over long periods. Thus, in line with ESNs, we propose a memory functioning by "Echo," based on non-learned random matrices, allowing for rich, diverse, and robust encoding of information and constant complexity at each step. We conducted preliminary evaluations of Echo State Transformer (EST) on the STREAM benchmark – a set of sequential tasks modifiable in difficulty – thus allowing us to have an initial comparison with models such as LSTM and Transformer. The preliminary results show promising performance, indicating that this hybrid approach could offer an effective alternative to traditional Transformer models.