On agnostic post hoc approaches to false positive control

Overview & Synopsis

Classical approaches to multiple testing grant control over the amount of false positives for a specific method prescribing the set of rejected hypotheses. On the other hand, in practice, many users tend to deviate from a strictly prescribed multiple testing method and follow ad-hoc rejection rules, tune some parameters by hand, compare several methods, or pick from their results the one that suits them best. This will invalidate standard statistical guarantees because of the selection effect. To compensate for any form of such """data snooping""", an approach that has garnered interest recently is to derive """user-agnostic""", or post hoc, bounds on the false positives valid uniformly over all possible rejection sets; this allows arbitrary data snooping from the user. In this chapter, we present a general approach to this problem using a family of candidate rejection subsets (call this a reference family) together with associated bounds on the number of false positives they contain, the latter holding uniformly over the family. We illustrate the interest of our approach in different contexts, for different choices of the reference family, and apply it to a genomic example (differential gene expression) and a neuroimaging example (functional Magnetic Resonance Imaging).