Auditing Semantic and Relational Reliability in Open-Vocabulary Human-Object Interaction Evaluation

Overview & Synopsis

Open-vocabulary human-object interaction detectors describe interactions in free text, and this creates a problem for evaluation. Category scoring rejects valid paraphrases, while semantic similarity scoring can reward predictions that name the wrong action or reverse the roles of the participants. We present a diagnostic protocol that holds every prediction fixed and scores it under five views, namely official category evaluation, an identity control, lexical normalization, graded semantic credit, and strict propositional equivalence. Comparing these views separates the effect of the matching engine from the effect of semantic credit. The protocol pairs controlled transformations that preserve or break meaning with a blinded audit of natural predictions, and we derive consistency checks that link mean credit, invalid acceptance, and transformation counts. Applied to six systems on HICO-DET, the views disagree on which system leads. GEN-VLKT leads under official evaluation, SL-HOI leads under SHOE, and GEN-VLKT leads again under strict equivalence. Semantic gains range from 1.6 to 20.3 mAP, and AgentHOI keeps less than half of its gain once equivalence is enforced. We recommend that semantic scores be reported together with relational validity and paired uncertainty, so that recovered credit is not read as better interaction understanding.