Asynchronous multi-agent reinforcement learning for coordinated control of natural ventilation and radiant cooling

Abstract: Optimal control for building systems requires coordinating different types of systems and balancing multiple objectives, yet most Building Management Systems still rely on isolated, rule-based controls that fail to optimize across interdependent systems. This paper addresses two critical challenges in advanced building control: (1) coordinating systems with fast (automated window) and slow (radiant floor cooling) dynamics, and (2) eliminating the need for complex physical models through a fully model-free approach. 

We propose a novel Multi-Agent Deep Reinforcement Learning Control (MA-DRLC) framework that introduces sQ-learning, a slow-response Q-learning variant that incorporates multiple past valve actions to account for thermal lag in radiant systems, and a hierarchical action protocol enabling the valve agent to guide the window agent, with seamless bidirectional state sharing. A unified reward function balances thermal comfort, indoor air quality, and cooling energy. 

In a real-world office deployment, MA-DRLC maintained thermal comfort and good indoor air quality for more than 90% of occupied hours while reducing mechanical cooling hours by 21% compared to conventional rule-based control. These results demonstrate the scalability of the method, its adaptability to changing weather forecasts, and its potential to transform building automation through autonomous, model-free coordination of multi-timescale subsystems.