Officially corrected errors persist in AI training data: preregistered measurements on EU law, Internet standards and retracted science

Overview & Synopsis

Official texts are corrected after publication: EU corrigenda, verified RFC errata, retractions. We measured whether these corrections are present in four public datasets used to train language models, and what models answer about the corrected passages. In EurlexResources, a legal training dataset, 1,989 of 2,082 testable single-language corrections to EU acts (95.5%, 94.6-96.3) remain in their erroneous form; in a random sample of 100, 46 change the meaning, a figure, a cross-reference, the competent authority or the scope (20 without scope). At least 124 corrected EU-law errors and at least 60 verified RFC errata (38 classified "Technical") are reproduced in their erroneous form in FineWeb-2 and in a sample of FineWeb. About 12,200 retracted papers (11,700-12,500) are present in the peS2o corpus, 79% with no mention of the retraction; separately, 53% of a random sample of retracted papers are still cited at least a year after retraction (OpenAlex). Asked 59 questions on serious corrected errors without documents, five models gave the erroneous version in 45% of the answers in which they committed to one version (31-60% allowing for clustering by question): 53% for scope corrections, 32% for the others. One answer in 295 said it did not know. Shown both versions side by side, the models chose the corrected one in 53% of cases (49-57), and naming the act made no difference; when each version carried its date and the corrigendum label, the four larger models chose it in 99-100% of cases. A comparison between a legal masked language model and its base model is null, against our bet: we do not show that model answers come from the datasets measured. All steps were preregistered and their fingerprints timestamped in the Bitcoin blockchain; six controls failed, several bets were lost, and every deviation is reported. Data and proofs: doi:10.5281/zenodo.23642239.