Interpretable Intersection Control by Reinforcement Learning Agent With Linear Function Approximator

ABSTRACT Reinforcement learning (RL) is a promising machine‐learning solution to traffic signal control problems, which have been extensively studied. However, variants of non‐linear, deep artificial neural network (ANN) function approximators (FAs) have been predominantly employed in previous studies proposing RL‐based controllers, leaving a significant interpretability issue due to their black‐box nature. In this work, the use of the linear FA for a value‐based RL agent in traffic signal control problems is investigated along with the least‐squares Q‐learning method, abbreviated as LSTDQ. The interpretable linear FA was found to be adequate for the RL agent to learn an optimal policy. This leads to the proposal to replace a non‐linear ANN FA with the linear FA counterpart, resolving the interpretability issue. Moreover, the LSTDQ learning method shows superior behaviour convergence compared to a gradient descent method. In a low‐intensity arrival pattern scenario, the control by the RL agent cuts about half of the average delay resulting from the pretimed control. Owing to the conciseness of the linear FA, a direct interpretation analysis of the converged linear‐FA parameters is presented. Lastly, two online relearning tests of the agents under non‐stationary arrivals are conducted to demonstrate the online performance of LSTDQ. In conclusion, the linear‐FA specification and the LSTDQ method are together proposed to be used for its control algorithm interpretability property, superior convergence quality, and lack of hyperparameters.

# Transportation engineering # Electronic computers. Computer science # computer simulation # learning systems # traffic control

Somporn Sahachaiseree, Takashi Oguchi (2025). "Interpretable Intersection Control by Reinforcement Learning Agent With Linear Function Approximator". IET Intelligent Transport Systems. https://doi.org/10.1049/itr2.70034