SSMSFuse: A Spectral and Spatial Multiscale Coupling Fusion Model for Hyperspectral and Multispectral Image
1939-1404
Abstract
Hyperspectral image (HSI) has more spectral information than conventional images, which helps to distinguish targets in a complex scene more accurately. However, HSI typically has a low spatial resolution, which limits their application scenarios. To achieve high-resolution HSI, we propose a spectral and spatial multiscale coupling fusion model (SSMSFuse) for hyperspectral and multispectral image (MSI). SSMSFuse couples the spatial information of MSI and the spectral information of HSI at multiscales by means of a two-branch network structure, thus obtaining the fused images with high spatial and spectral resolution. SSMSFuse consists of two branches, namely the spatial embedding network (Spa-Net) and the spectral embedding network (Spe-Net). Spa-Net is constructed using a multiscale convolutional neural network to better mine multilevel spatial features from MSI. Spe-Net is constructed using self-attention, which can model the long-distance spectral dependencies of HSI to better extract spectral information from HSI. Finally, to achieve interactive coupling of dual-branch information, we designed a spatial–spectral guidance fusion block to fuse features at different scales to avoid loss of spatial and spectral details. Experiments are carried out on four public datasets, and the results show that the proposed method can effectively improve the objective indicators of the fusion results, such as the peak signal to noise ratio, which is increased by 1.36%, and the root mean square error, which is increased by 9.72% on the CAVE dataset, and satisfactory subjective results are also obtained.