PDI, an approach to decouple I/O concerns from high-performance simulation codes

Overview & Synopsis

High-performance computing (HPC) applications manipulate and store large datasets for scientific analysis, visualization purposes and/or resiliency. Multiple software libraries have been designed for interacting with the parallel file system and in some cases with intermediate storage levels. These libraries provide different level of abstraction and have been optimized for different purposes. The best I/O library for a given usage depends on multiple criteria including the purpose of the I/O, the computer architecture or the problem size. Therefore, to optimize their I/O strategies, scientists have to use multiple API's depending on the targeted execution. As a result, simulation codes contain intrusive and library dependent I/O instructions interwoven with domain instructions. In this paper, we propose a novel interface that transparently manage the I/O aspects and support multiple I/O libraries within the same execution. We demonstrate the low overhead of our interface and perform experiments at scale with the Gysela production code.