When Agentic Retrieval Fails Silently: Framework and Production Evidence from a Creative AI Assistant

Overview & Synopsis

CTR, MRR, and behavioral NDCG assume the user sees ranked results and acts on them. In an agentic assistant, a generation layer consumes retrieval output first, and that assumption breaks in three ways. The agent reads the results before the user does (who). The user's action can come several turns or sessions after retrieval (when). Outcomes blend retrieved content with the model's own knowledge (what caused it). We call this the attribution gap. We propose Layered Agentic Retrieval Assessment (LARA), which measures retrieval quality, engagement and outcome, and product success as three independent layers joined by a four-field lineage schema that requires no new modeling. We examine the motivation on Adobe Express, where one retrieval infrastructure serves a gallery search surface and an agentic assistant at roughly one-eighth its session volume for the same user population. Chat users reformulate more than gallery users (44.4% vs. 33.1%) because chat reformulation refines generated output. LARA restores attribution for the retrieval.