多代理Nash Q-Learning模型行动选择策略研究

doi:10.16381/j.cnki.issn1003-207x.2023.0142

摘要/Abstract

摘要：

多代理Q-Learning模型的行动选择策略优化是复杂经济学博弈模拟过程中亟待解决的问题之一。本文将强制 $ε - g r e e d y$ 行动选择策略引入多代理Nash Q-Learning模型中，通过博弈实验对比该行动选择策略与经典 $ε - g r e e d y$ 策略的效果，探究该行动选择策略对算法计算速度和收敛情况的影响；同时，根据实验结果进行了算法真实性理论验证，并基于多代理模型的性质给出强制 $ε - g r e e d y$ 的普适性推论。模拟结果表明，强制 $ε - g r e e d y$ 适用于更复杂、涉及状态行动更多、回合更多的博弈，此时能有效提升多代理Q-Learning算法运行性能，但由于其本质是初期增加对行动的探索，这会消耗一些回合，导致均衡收敛率下降。因此，强制 $ε - g r e e d y$ 带来的性能提升与损失的均衡收敛率是使用者在应用该策略时需要权衡的问题。

关键词: Nash Q-Learning, $ε - g r e e d y$ ')">强制 $ε - g r e e d y$ , 行动选择

Abstract:

Optimization of action selection strategy for multi-agent Q-Learning model is one of the urgent problems to be solved in the simulation process of complex economics games. The forced ε-greedy action selection policy is introduced into the multi-agent Nash Q-Learning model， the effect of this action selection policy is compared with the classical ε-greedy policy through game experiments， and the effect of this action selection policy on the computational speed and convergence of the algorithm is explored； at the same time， theoretical verification of the veracity of the algorithm is carried out according to the experimental results， and a theoretical verification of the algorithm based on the nature of the multi-agent model is given. The simulation results show that forced ε-greedy is suitable for more complex games involving more state actions and more rounds， at which time it can effectively improve the running performance of the multi-agent Q-Learning algorithm， but due to its nature of initially increasing the exploration of actions， which will consume some rounds and lead to a decrease in the equilibrium convergence rate. Therefore， the performance improvement brought by enforcing ε-greedy versus the lost equilibrium convergence rate is an issue that users need to weigh when applying this policy.The main contributions of this paper are：1） comparing the model performance under the classical action selection strategy and the extended action selection policy， and conducting theoretical validation of the algorithm's veracity based on the simulation results， and summarizing the characteristics of the application of this extended policy in the multi-agent Nash Q model； 2） based on the analysis of the theoretical properties of multiple multi-agent models and the process of proof of convergence， the conclusion of the generalization of this action selection policy is given for the multi-agent reinforcement learning algorithms with action selection link provide a strategy to enhance the model computing speed and improve the convergence results. This promotes the operation efficiency of complex reinforcement learning models in economics， and to a certain extent， it can reduce the cost of experiments， and increase the number of experiments within the allowable range of computational volume to improve the reliability of experimental results.

Key words: Nash Q-Learning, $ε - g r e e d y$ ')">forced $ε - g r e e d y$ , action selection

中图分类号:

F224-3

韩松,李璨. 多代理Nash Q-Learning模型行动选择策略研究[J]. 中国管理科学, 2025, 33(12): 110-120.

Song Han,Can Li. A Study of Action Selection Policies for Multi-Agent Nash Q-Learning Model[J]. Chinese Journal of Management Science, 2025, 33(12): 110-120.

图/表 18

表 1

单代理和多代理Nash Q函数和最优Q值形式对比"

函数与最优Q值	单代理	多代理
Q函数	$Q * k s, a$	$Q * k s, a 1, …, a n$
最优Q值	本期回报+从当前状态起，之后的每个阶段博弈都选择最大化自身收益的策略所获的报酬的贴现	本期回报+从当前状态起，之后的每个阶段博弈都选择纳什均衡策略所获的报酬的贴现

表 1

图1

图2

图3

图 4

表2

表3

图5

表4

短时间经典和强制 ε-greedy各指标对比"

模型	平均运行时间	每回合平均步数	平均均衡收敛率（%）
经典 $ε - g r e e d y$	63.46	6.07	17.98
强制 $ε - g r e e d y$	70.02	6.29	10.56

表4

图6

图7

表5

中等时间经典和强制ε-greedy各指标对比"

模型

平均运行时间

每回合平均步数

平均均衡收敛率(%)

经典

$ε - g r e e d y$

257.42

6.11

17.91

强制

$ε - g r e e d y$

259.39

5.57

12.24

表5

表6

长时间经典和强制 ε-greedy各指标对比"

平均运行时间

每回合平均步数

平均均衡收敛率(%)

经典

$ε - g r e e d y$

1346.02

6.06

19.11

强制

$ε - g r e e d y$

1277.01

5.37

12.77

表6

表7

表8

表9

表10

短时间过程ε变化对模型性能的影响"

$ε$ 取值	平均运行时间	每回合平均步数	平均均衡收敛率（%）
$ε = 0.1$	69.74	6.35	17.27
$ε = 0.2$	70.19	6.29	11
$ε = 0.3$	71.06	6.45	8

表10

表11

参考文献 32

[1]	Charpentier A， Élie R， Remlinger C. Reinforcement learning in economics and finance［J］. Computational Economics， 2023， 62（1）： 425-462.
[2]	Dogan I， Güner A R. A reinforcement learning approach to competitive ordering and pricing problem［J］. Expert Systems， 2015， 32（1）： 39-48.
[3]	Calvano E， Calzolari G， Denicolò V， et al. Artificial intelligence， algorithmic pricing， and collusion［J］. The American Economic Review， 2020， 110（10）： 3267-3297.
[4]	Littman M L. Markov games as a framework for multi-agent reinforcement learning［C］//Proceedings of the Eleventh International Conference， Rutgers University， New MORGA Brunswick， NJ， July 10-13， 1994： 157-163.
[5]	Hu J， Wellman M P. Nash q-learning for general-sum stochastic games［J］. Journal of Machine Learning Research， 2003， 4： 1039-1069.
[6]	Sridharan M， Tesauro G. Multi-agent Q-learning and regression trees for automated pricing decisions［M］// Parsons S， Gmytrasiewicz P， Wooldridge M. Game Theory and Decision Theory in Agent-Based Systems. Boston，MA：Springer US，2002：217-234.
[7]	Ye M A， Tianqing C， Wenhui F. A single-task and multi-decision evolutionary game model based on multi-agent reinforcement learning［J］. Journal of Systems Engineering and Electronics，2021，32（3）： 642-657.
[8]	dos Santos Mignon A， de Azevedo da Rocha R L. An adaptive implementation of ε-greedy in reinforcement learning［J］. Procedia Computer Science， 2017， 109： 1146-1151.
[9]	Tokic M. Adaptive ε-greedy exploration in reinforcement learning based on value differences［C］//Proceedings of Annual Conference on Artificial Intelligence， Berlin， Heidelberg， September 21-24 ， Springer Berlin Heidelberg， 2010： 203-210.
[10]	Sutton R S， Barto A G. Reinforcement Learning： An Introduction［M］. 2nd ed. Cambridge， MA： MIT Press， 2018.
[11]	Angelopoulos G， Metafas D. Forced ε-greedy， an expansion to the ε-greedy action selection method［C］//Proceedings of the 1st International Conference on Novelties in Intelligent Digital Systems， Athens， Greece， September 30-October 1 ， IOS Press， 2021： 12-20.
[12]	Dabney W， Ostrovski G， Barreto A. Temporallyextended ε-greedy exploration ［EB/OL］.（2020-06-02）［2023-01-30］. .
[13]	Gimelfarb M， Sanner S， Lee C G. ε-BMC：A Bayesian ensemble approach to epsilon-greedy exploration in model-free reinforcement learning［EB/OL］.（2020-07-02）［2023-01-30］..
[14]	Vidhate D A， Kulkarni P. A framework for dynamic decision making by multi-agent cooperative fault pair algorithm （MCFPA） in retail shop application［C］//Proceedings of Information and Communication Technology for Intelligent Systems， Singapore， April 6-7 ， Springer Singapore， 2018： 693-703.
[15]	Li X， Qin X， Zang C， et al. Pricing scheme based Nash Q-learning load control in smart grids［C］// Proceedings of 2013 25th Chinese Control and Decision Conference （CCDC）， Guiyang， China， May 25-27， IEEE， 2013： 5198-5202.
[16]	Lopez V G， Lewis F L， Liu M， et al. Game-theoretic lane-changing decision making and payoff learning for autonomous vehicles［J］. IEEE Transactions on Vehicular Technology， 2022， 71（4）： 3609-3620.
[17]	Zhou X， Kuang D， Zhao W， et al. Lane-changing decision method based Nash Q-learning with considering the interaction of surrounding vehicles［J］. IET Intelligent Transport Systems， 2020， 14（14）： 2064-2072.
[18]	Ma M， Zhu A， Guo S， et al. Intelligent network selection algorithm for multiservice users in 5G heterogeneous network system： Nash Q-learning method［J］. IEEE Internet of Things Journal， 2021， 8（15）： 11877-11890.
[19]	丁雨，李晨凯，韩会梅，等. 基于5G无人机通信的多智能体异构网络选择方法［J］. 电信科学， 2022， 38（8）： 28-36.
	Ding Y， Li C K， Han H M， et al. Multi-agent heterogeneous network selection method based on 5G UAV communication［J］. Telecommunications Science， 2022， 38（8）： 28-36.
[20]	Guo J， Harmati I. Evaluating semi-cooperative Nash/Stackelberg Q-learning for traffic routes plan in a single intersection［J］. Control Engineering Practice， 2020， 102： 104525.
[21]	Yang L， Sun Q， Ma D， et al. Nash Q-learning based equilibrium transfer for integrated energy management game with We-Energy［J］. Neurocomputing， 2020， 396： 216-223.
[22]	闫雪飞，李新明，刘东，等. 基于Nash-Q的网络信息体系对抗仿真技术［J］. 系统工程与电子技术， 2018， 40（1）： 217-224.
	Yan X F， Li X M， Liu D， et al. Confrontation simulation for network information system-of-systems based on Nash-Q［J］. Systems Engineering and Electronics， 2018， 40（1）： 217-224.
[23]	Zhang Y， Gan R， Shao J， et al. Path selection with Nash Q-learning for remote state estimation over multihop relay network［J］. International Journal of Robust and Nonlinear Control， 2020， 30（11）： 4331-4344.
[24]	Wang J， Cao L， Chen X， et al. General proof of convergence of the Nash-q-learning algorithm［J］. Fractals， 2022， 30： 2250027.
[25]	Zhuang Y， Chen X， Gao Y， et al. Accelerating Nash Q-learning with graphical game representation and equilibrium solving［C］//Proceedings of 2019 IEEE 31st International Conference on Tools with Artificial Intelligence （ICTAI）， Portland， OR， USA， November 4-6， IEEE， 2020： 939-946.
[26]	Ghazanfari B， Mozayani N. Enhancing Nash Q-learning and Team Q-learning mechanisms by using bottlenecks［J］. Journal of Intelligent & Fuzzy Systems， 2014， 26（6）： 2771-2783.
[27]	Watkins C J C H， Dayan P. Q-learning［J］. Machine Learning， 1992， 8（3）： 279-292.
[28]	Mitchell T M. Does machine learning really work？［J］. AI Magazine， 1997， 18（3）： 11-20.
[29]	Chang Y H， Ho T， Kaelbling L. All learning is local： Multi-agent learning in global reward games［J］. Advances in Neural Information Processing Systems， 2003， 16.
[30]	Laurent F， Schneider M， Scheller C， et al. Flatland competition 2020： MAPF and MARL for efficient train coordination on a grid world［C］//Proceedings of NeurIPS 2020 Competition and Demonstration Track， Online， December 6-12 ， PMLR， 2021： 275-301.
[31]	Greenwald A， Hall K， Serrano R. Correlated Q-learning［C］//Proceedings of the 20th ICML， Washington， DC USA， August 21 ， AAAI Press， 2003， 3： 242-249.
[32]	Littman M L. Friend-or-foe Q-learning in general-sum games［C］//Proceedings of the 18th ICML， San Francisco， United States， June 28-July 1 ， Morgan Kaufmann， 2001： 322-328.

访问次数下限	平均运行时间	每回合平均步数	平均均衡收敛率(%)
times=2	69.95	6.26	10.52
times=3	73.95	6.81	8.08

访问次数下限	平均运行时间	每回合平均步数	平均均衡收敛率（%）
times=2	260.60	5.62	12.08
times=3	259.96	5.67	11.8

状态（2 5）	玩家2（Up）	玩家2（Left）
玩家1（Up）	81,0	81,0
玩家1（Left）	0,81	0,81
状态（3 5）	玩家2（Left）	玩家2（Up）
玩家1（Right）	64.7,64.7	72.9,72.9
玩家1（Up）	72.9,72.9	72.9,72.9
状态（6 8）	玩家2（Left）	玩家2（Down）
玩家1（Right）	71.9,71.9	81,0
玩家1（Down）	0,81	65.7,65.7