Invisible Glue: Scalable Self-Tuning Multi-Stores

Overview & Synopsis

Next-generation data centric applications often involve di-verse datasets, some very large while others may be of mod-erate size, some highly structured (e.g., relations) while others may have more complex structure (e.g., graphs) or little structure (e.g., text or log data). Facing them is a variety of storage systems, each of which can host some of the datasets (possibly after some data migration), but none of which is likely to be best for all, at all times. Deploying and efficiently running data-centric applications in such a complex setting is very challenging. We propose Estocada, an architecture for efficiently han-dling highly heterogeneous datasets based on a dynamic set of potentially very different data stores. Estocada pro-vides to the application/programming layer access to each data set in its native format, while hosting them internally in a set of potentially overlapping fragments, possibly dis-tributing (fragments of) each dataset across heterogeneous stores. Given workload information, Estocada self-tunes for performance, i.e., it automatically choses the fragments of each data set to be deployed in each store so as to op-timize performance. At the core of Estocada lie powerful view-based rewriting and view selection algorithms, required in order to correctly handle the features (nesting, keys, con-straints etc.) of the diverse data models involved, and thus to marry correctness with high performance.