Compression, Deduplication, and Execution Order in Datacenter Storage Pipelines

Overview & Synopsis

Compression and deduplication are routinely combined in datacenter storage and streaming-service backends, yet the benefits and trade-offs of such pipelines remain unclear. This paper presents a reproducible single-node benchmark of four compression tools and three deduplication tools, evaluated individually and in combination on two contrasting workloads: a large structured CSV dataset and the Linux Kernel source tree. Using containerized experiments, we measure compression ratio, execution time, CPU and RAM usage, and I/O activity to study storage savings, execution order, and workload sensitivity. Results show that combining tools rarely improves space savings over the best standalone method, while execution order matters: deduplication before compression systematically reduces time and resource usage. Deduplication is most effective on redundancy-rich, multi-file data, whereas compressors dominate on flat structured datasets, and high-ratio compressors incur higher CPU/RAM costs. Overall, there is no universal "compress+dedupe" recipe; effective optimization is workload-aware and order-conscious.