Do LLM-derived text features improve electricity price forecasts beyond document activity? Evidence from Australia's National Electricity Market

Overview & Synopsis

Large language models (LLMs) can turn market notices and news into features for electricity price forecasting. Such features also record when documents arrive and whether their extraction failed, so a gain measured against a model without text can reflect document activity as well as content. We add the counts of the same documents and of their failed extractions to a no-text LightGBM model and measure each LLM representation against this activity reference, which needs no additional LLM requests. With gpt-6-luna as the extractor, we forecast Australian electricity prices from operator notices (5-30 minutes ahead) and WattClarity news (day-ahead) after the extractor's documented knowledge cutoff, averaging forecasts over three training seeds. For news, the counts alone lower mean absolute error (MAE) by 3.0%, most of the 3.6% reduction that the better LLM representation achieves relative to the no-text model; its increment beyond the counts is not significant. For notices, two of the three LLM representations lower MAE beyond the counts, by 0.14-0.21% and 0.45-0.52%, whether the counts are cumulative or cover recent windows, whereas the third shows no significant reduction. Time-shifted placebos give mixed evidence on whether these gains require text aligned with the target. Benchmarks using the operator's pre-dispatch forecasts (notices) or price history alone (news) are more accurate than all these text models. Gains against the no-text model and the activity reference differ in sign at individual seeds and, for notices with cumulative counts, in seed-averaged forecasts. Text gains should be reported against both references.