MARS: An Adaptive Multi-Agent DRL-based Scheduler for Multipath QUIC in Dynamic Networks

Abstract

The multipath extension of the Quick UDP Internet Connection (QUIC) protocol, also called MPQUIC, is currently attracting increasing attention from both industry and academia. The multipath scheduler of MPQUIC determines how to distribute the packets onto different paths. However, our experimental results show that they fail to adapt to various receive buffer sizes and Quality of Service (QoS) requirements while applying current multipath schedulers into MPQUIC due to the diversity of devices and applications. These problems are especially severe under heterogeneous and dynamic network environments. To tackle these problems, we propose MARS, a Multi-Agent deep Reinforcement learning (MADRL) based Multipath QUIC Scheduler, which is able to promptly adapt to dynamic network environments. It exploits the MADRL method to learn a neural network for each path and generate scheduling policy. Besides, it introduces a novel multi-objective reward function that takes out-of-order (OFO) queue size and different QoS metrics into consideration to realize adaptive scheduling optimization. We implement MARS in an MPQUIC prototype and compare it with the state-of-the-art multipath schedulers in both emulated and real-world networks. Experimental results show that MARS outperforms the other schedulers with better adaptive capability regarding the receive buffer sizes and QoS.