Evaluation of Large Language Models for Blockchain Transaction Explanation

Overview & Synopsis

Blockchain data is public but difficult for nonspecialists to interpret because transactions expose low-level identifiers, encoded calls, and protocol-specific concepts. We evaluate whether Retrieval-Augmented Generation (RAG) improves the clarity and factual reliability of LLM-generated explanations of Ethereum transactions. Four configurations -GPT-5 and Mistral 7B, with and without RAG -are tested on four Ethereum cases through a 99-participant user study, an LLM-based questionnaire assessment, and an independent expert factual review. Results reveal a fluency-fidelity divergence: participants most often preferred Mistral 7B non-RAG, although expert scores identified it as factually weak. RAG slightly improves GPT-5's expert-rated reliability through stronger hallucination control, but does not improve perceived clarity and degrades Mistral 7B overall. These results suggest that on-chain explanation systems should combine deterministic numerical extraction, retrieval-based grounding, and controlled simplification.