基于深度强化学习算法的跨区域多微电网系统扩展规划研究

doi:10.16381/j.cnki.issn1003-207x.2023.1947

Abstract

Abstract:

Microgrids have shown great potential in improving the resilience of power supply and reducing greenhouse gas （GHG） emissions. The interconnection of island microgrids into a multi-area multi-microgrids （MMGs） system will help improve the economy and power supply resilience of microgrids. Aiming at the expansion planning problem for MMGs， a long-term expansion planning framework is proposed with the goal of minimizing the total cost of the MMGs while taking power supply resilience and environmental benefits as constraints. The energy sharing between adjacent microgrids is also considered. Based on the real data， a MMGs system containing three microgrids is constructed as a case study to demonstrate the effectiveness of the proposed model. The dynamic and stochastic optimization problem is solved by deep reinforcement learning algorithm. The results show that the planning framework for MMGs can improve the resilience of the microgrid power supply and reduce GHG emissions. The proposed framework also considers the impact of the interconnection structure of MMGs and appropriately adjusts strategies based on the frequency of outages and outage losses of individual microgrid. This research has important practical significance for the expansion planning of MMGs.

Key words: microgrid, resilience, expansion planning, multi-area, reinforcement learning

CLC Number:

TM73

Jian Zhou,Xiaoting Nie,Kexin Pang, et al. Multi-area Multi-Microgrid System Expansion Planning Based on Deep Reinforcement Learning Algorithm[J]. Chinese Journal of Management Science, 2025, 33(11): 41-53.

Figures/Tables 13

	算法: 微电网扩展规划的DDQN算法
1:	确定微电网的运行模式
2:	初始化: 经验回放缓冲区b和采样批量m的大小
3：	初始化:主网络的参数 $θ A$ 和目标网络的参数 $θ B$
4：	for episode = 1 : Num
5：	初始化微电网环境状态s
6：	for decision period k = 1 : K
7：	结合ε-greedy策略和表选择动作 $a$
8：	执行动作a，模拟主电网故障时的备用微电网运行
9：	得到奖励r和下一个状态s’
10	将经验存入经验回放缓冲区b
11	从经验回放缓冲区b中随机采样批量大小为m的经验
12：	目标网络A选择动作，网络B估计目标值： $Q * = r t + 1 + γ Q B (s t + 1, a r g m a x a Q A (s t + 1, a; θ t A); θ t B)$
13：	利用随机梯度下降 $Q * - Q A s, a, θ A 2$ 最小化训练主网络并更新参数 $θ A$
14：	每经过C个episodes更新 $θ B$ ，令 $θ B$ = $θ A$
15：	结束for循环
16：	结束for循环

用电设施	数目	VOLL(美元/千瓦时)	$C p$	$β$	$ζ$	$ψ$
医院	2	25	0.8	0.2	0.8	0.3
门诊	2	19	0.8	0.2	0.8	0.3
超市	3	10	0.6	0.5	0.5	0.3
旅舍	3	9	0.5	0.5	0.5	0.3
办公楼	5	8	0.5	0.5	0.5	0.3
学校	3	7	0.4	0.5	0.5	0.3
餐厅	7	6	0.9	0.5	0.5	0.3
住宅	300	5	0.3	0.5	0.5	0.3

References 44

[1]	衣博文，范英. 高比例可再生能源接入下的电力系统规划研究［J］. 管理科学学报， 2023， 26（10）： 21-35.
	Yi B W， Fan Y. Power system planning with a high share of renewable energy［J］. Journal of Management Sciences in China， 2023， 26（10）： 21-35.
[2]	殷辉，陆信辉，周开乐. 分布式与集中式储能并存的微电网负荷优化调度［J］. 中国管理科学， 2023， 31（2）： 118-128.
	Yin H， Lu X H， Zhou K L. Optimal load dispatch for microgrid with distributed and centralized energy storage systems［J］. Chinese Journal of Management Science， 2023， 31（2）： 118-128.
[3]	Tsianikas S， Yousefi N， Zhou J， et al. A storage expansion planning framework using reinforcement learning and simulation-based optimization［J］. Applied Energy， 2021， 290： 116778.
[4]	巨云涛，陈希，李嘉伟，等. 基于分布式深度强化学习的微网群有功无功协调优化调度［J］. 电力系统自动化， 2023， 47（1）： 115-125.
	Ju Y T， Chen X， Li J W， et al. Active and reactive power coordinated optimal dispatch of networked microgrids based on distributed deep reinforcement learning［J］. Automation of Electric Power Systems， 2023， 47（1）： 115-125.
[5]	杨家豪. 基于一致性算法的孤岛型微电网群实时协同功率分配［J］.电力系统自动化，2017，41（5）：8-15+38.
	Yang J H. Consensus algorithm based real-time collaborative power dispatch for island multi-microgrid［J］. Automation of Electric Power Systems， 2017， 41（5）： 8-15+38.
[6]	高蕾，龚晶. 基于线性分段恢复函数的基础设施韧性分析模型——以C县电力系统网络为例［J］. 中国管理科学， 2022， 30（12）： 86-95.
	Gao L， Gong J. Analysis model for infrastructure resilience based on linear piecewise recovery function： A case study of C county power network［J］. Chinese Journal of Management Science， 2022， 30（12）： 86-95.
[7]	刘天浩，朱元振，孙润稼，等. 极端自然灾害下电力信息物理系统韧性增强策略［J］. 电力系统自动化， 2021， 45（3）： 40-48.
	Liu T H， Zhu Y Z， Sun R J， et al. Resilience-enhanced strategy for cyber-physical power system under extreme natural disasters［J］. Automation of Electric Power Systems， 2021， 45（3）： 40-48.
[8]	黄文鑫，吴军，郭子辉，等. 台风灾害下电网韧性评估及差异化规划［J］.电力系统自动化，2023， 47（5）： 84-91.
	Huang W X， Wu J， Guo Z H， et al. Power grid resilience assessment and differentiated planning against typhoon disasters［J］. Automation of Electric Power Systems， 2023， 47（5）： 84-91.
[9]	Ma S， Chen B， Wang Z. Resilience enhancement strategy for distribution systems under extreme weather events［J］. IEEE Transactions on Smart Grid， 2018， 9（2）： 1442-1451.
[10]	Ashrafi R， Amirahmadi M， Tolou-Askari M， et al. Multi-objective resilience enhancement program in smart grids during extreme weather conditions［J］. International Journal of Electrical Power & Energy Systems， 2021， 129： 106824.
[11]	杜敏，刘绚，周元刚. 考虑极端事件下的高比例可再生能源电力系统韧性增强策略［J］. 电力系统自动化， 2023， 47（12）： 19-27.
	Du M， Liu X， Zhou Y G. Resilience enhancement strategy for power system with high proportion of renewable energy considering extreme events［J］. Automation of Electric Power Systems， 2023， 47（12）： 19-27.
[12]	Xu Y， Xing Y， Huang Q， et al. A review of resilience enhancement strategies in renewable power system under HILP events［J］. Energy Reports， 2023， 9： 200-209.
[13]	Abdelmalak M， Benidris M. Proactive generation redispatch to enhance power system resilience during hurricanes considering unavailability of renewable energy sources［J］. IEEE Transactions on Industry Applications， 2022， 58（3）： 3044-3053.
[14]	Gilani M A， Kazemi A， Ghasemi M. Distribution system resilience enhancement by microgrid formation considering distributed energy resources［J］. Energy， 2020， 191： 116442.
[15]	Yan M， He Y， Shahidehpour M， et al. Coordinated regional-district operation of integrated energy systems for resilience enhancement in natural disasters［J］. IEEE Transactions on Smart Grid， 2019， 10（5）： 4881-4892.
[16]	Hussain A， Bui V H， Kim H M. Microgrids as a resilience resource and strategies used by microgrids for enhancing resilience［J］. Applied Energy， 2019， 240： 56-72.
[17]	Hemmati R， Saboori H， Siano P. Coordinated short-term scheduling and long-term expansion planning in microgrids incorporating renewable energy resources and energy storage systems［J］. Energy， 2017， 134： 699-708.
[18]	Mehraban S A， Eslami R. Multi-microgrids energy management in power transmission mode considering different uncertainties［J］. Electric Power Systems Research， 2023， 216： 109071.
[19]	Zhong X， Zhong W， Liu Y， et al. Optimal energy management for multi-energy multi-microgrid networks considering carbon emission limitations［J］. Energy， 2022， 246： 123428.
[20]	Han D， Lee J H. Two-stage stochastic programming formulation for optimal design and operation of multi-microgrid system using data-based modeling of renewable energy sources［J］. Applied Energy， 2021， 291： 116830.
[21]	Pang K， Zhou J， Tsianikas S， et al. Deep reinforcement learning for resilient microgrid expansion planning with multiple energy resource［J］. Quality and Reliability Engineering International， 2024， 40（1）： 34-56.
[22]	Bukar A L， Tan C W， Yiew L K， et al. A rule-based energy management scheme for long-term optimal capacity planning of grid-independent microgrid optimized by multi-objective grasshopper optimization algorithm［J］. Energy Conversion and Management， 2020， 221： 113161.
[23]	Zhou B， Zou J， Chung C Y， et al. Multi-microgrid energy management systems： Architecture， communication， and scheduling strategies［J］. Journal of Modern Power Systems and Clean Energy， 2021， 9（3）： 463-476.
[24]	Zhang Z， Zhang D， Qiu R C. Deep reinforcement learning for power system applications： An overview［J］. CSEE Journal of Power and Energy Systems， 2020， 6（1）： 213-225.
[25]	Du Y， Li F. Intelligent multi-microgrid energy management based on deep neural network and model-free reinforcement learning［J］. IEEE Transactions on Smart Grid， 2020， 11（2）： 1066-1076.
[26]	Yu Y， Cai Z， Liu Y. Double deep Q-learning coordinated control of hybrid energy storage system in island micro-grid［J］. International Journal of Energy Research， 2021， 45（2）： 3315-3326.
[27]	Remani T， Jasmin E A， Imthias Ahamed T P. Residential load scheduling with renewable generation in the smart grid： A reinforcement learning approach［J］. IEEE Systems Journal， 2019， 13（3）： 3283-3294.
[28]	Stroe D I， Świerczyński M， Stan A I， et al. Accelerated lifetime testing methodology for lifetime estimation of lithium-ion batteries used in augmented wind power plants［J］. IEEE Transactions on Industry Applications， 2014， 50（6）： 4006-4017.
[29]	Tsianikas S. Microgrid expansion planning using Simulation-based optimization and reinforcement learning［D］.New Brunswick： Rutgers， the State University of New Jersey， School of Graduate Studies， 2020.
[30]	US Energy Information Association （EIA）. Levelized costs of new generation resources in the annual energy outlook 2022［R］.Working Paper， US Energy Information Association， 2022.
[31]	Smith K， Saxon A， Keyser M， et al. Life prediction model for grid-connected Li-ion battery energy storage system［C］//2017 American Control Conference （ACC），Seattle， WA， USA， May 24-26， IEEE， 2017： 4062-4068.
[32]	Sutton R S， Barto A G. Reinforcement learning： An introduction［J］. IEEE Transactions on Neural Networks， 1998， 9（5）： 1054.
[33]	Dayan P. Motivated reinforcement learning［M］//Dietterich T G， Becker S， Ghahramani Z. Advances in Neural Information Processing Systems 14. Cambridge： The MIT Press， 2002： 11-18.
[34]	Mnih V， Kavukcuoglu K， Silver D， et al. Human-level control through deep reinforcement learning［J］. Nature， 2015， 518（7540）： 529-533.
[35]	Bui V H， Hussain A， Kim H M. Double deep $Q$-learning-based distributed operation of battery energy storage system considering uncertainties［J］. IEEE Transactions on Smart Grid， 2020， 11（1）： 457-469.
[36]	Van Hasselt H， Guez A， Silver D. Deep reinforcement learning with double Q-learning［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2016， 30（1）： 10295.
[37]	US Energy Information Association （EIA）. Annual electric power industry report， form EIA-861 detailed data files［R］. Working Paper， US Energy Information Association， 2021.
[38]	National Renewable Energy Laboratory （NREL）. Commercial and residential hourly load profiles for all TMY3 locations in the United States［R］. Working Paper， National Renewable Energy Laboratory， 2014.
[39]	Pang K， Zhou J， Tsianikas S， et al. Deep reinforcement learning based microgrid expansion planning with battery degradation and resilience enhancement［C］//Proceedings of 2021 3rd International Conference on System Reliability and Safety Engineering （SRSE）. November 26-28， 2021， Harbin， China. IEEE， 2021： 251-257.
[40]	National Renewable Energy Laboratory （NREL）. National solar radiation database［R］. Working Paper， National Renewable Energy Laboratory， 2021.
[41]	US Energy Information Association （EIA）. State electricity profiles［R］. Working Paper，US Energy Information Association， 2021.
[42]	Energy Information Association US. Gasoline and diesel fuel update［R］. Working Paper，US Energy Information Association， 2021.
[43]	Karimi H， Jadid S. Optimal energy management for multi-microgrid considering demand response programs： A stochastic multi-objective framework［J］. Energy， 2020， 195： 116992.
[44]	International Renewable Energy Agency （IRENA）. Renewable power generation costs in 2021［R］. Working Paper，International Renewable Energy Agency， 2021.

年份	2015	2016	2017	2018	2019	2020
CAIDI	1.48	1.67	0.90	1.51	1.70	12.56
SAIFI	1.08	0.82	1.29	1.35	1.12	2.60

发电单元	效率	退役成本(美元)	运维率	年退化率	投资等级
光伏	0.17	17.23	0.0063	0.01	2000, … ,20000
陆上风电	0.48	5738	0.007	0.0076	1, 2, 3,4, 5
热电联产	0.3	100	0.044	0.02	100,…, 2800
柴油机组	0.39	31	0.044	0.02	100, …, 2800
水力发电	0.8	303	0.025	0.05	100,…, 2800

年份	扩展动作	年份	扩展动作
1	区域1，光伏，98千瓦	9	/
2	区域1，锂电池，5076千瓦时	10	/
3	区域3，水力发电，1120千瓦	11	区域1，光伏，1554千瓦
4	区域2，水力发电，1120千瓦	12	/
5	区域3，锂电池，3240千瓦时	13	/
6	区域1，铅酸电池，1337千瓦时	14	/
7	/	15	区域1，锂电池，6111千瓦时
8	/

年份	扩展动作	年份	扩展动作
1	区域2，水力发电，1120千瓦	9	/
2	区域3，水力发电，520千瓦	10	区域2，光伏，1496千瓦
3	区域1，光伏，1093千瓦	11	/
4	区域1，柴油发电机，42千瓦	12	区域3，光伏，1611千瓦
5	区域2，钒电池，1480千瓦时	13	/
6	区域1，铅酸电池，445千瓦时	14	/
7	区域1，锂电池，855千瓦时	15	区域2，钒电池，4740千瓦时
8	/

年份	扩展动作	年份	扩展动作
1	区域2，水力发电，880千瓦	9	/
2	区域1，光伏，104千瓦	10	区域2，光伏，1496千瓦
3	区域1，锂电池，5922千瓦时	11	区域1，光伏，1554千瓦
4	区域3，水力发电，1120千瓦	12	区域3，光伏，1611千瓦
5	/	13	/
6	/	14	/
7	/	15	/
8	/

Multi-area Multi-Microgrid System Expansion Planning Based on Deep Reinforcement Learning Algorithm

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 44

Related Articles 15

Metrics

Comments

Recommended 0

年份	扩展动作	年份	扩展动作
1	区域3，水力发电，1120千瓦	9	区域2，光伏，1437千瓦
2	区域3，锂电池，2538千瓦时	10	区域3，光伏，1494千瓦
3	区域1，光伏，329千瓦	11	/
4	/	12	/
5	区域2，水力发电，1120千瓦	13	区域1，光伏，1498千瓦
6	/	14	/
7	区域1，锂电池，855千瓦时	15	区域3，铅酸电池，1370千瓦时
8	/

[1]	Zilin Zeng, Bin Hu, Zhichao Wang, Xiaomeng Ma. The Resilience and Sudden Change Mechanism of Group Opinion in Online Community [J]. Chinese Journal of Management Science, 2025, 33(9): 161-176.
[2]	Wenhe Chen, Zhipeng Chang, Hanting Zhou, Longsheng Cheng, Buyu Wen. A Deep Transfer Learning Evaluation Method for Community Resilience Governance Capacity under Major Epidemics [J]. Chinese Journal of Management Science, 2025, 33(8): 100-111.
[3]	Wenqiang Shi, Zhaojun Kong, Mingyue Wang, Jie He. System Dynamics Analysis of Emergency Materials Mobilization Chain Resilience under the Coupling of Multiple Disruptions [J]. Chinese Journal of Management Science, 2025, 33(8): 177-188.
[4]	Hua Zhang, Xin Gu. The Interplay of Supply Chain Digitalization and Manufacturing Firms' Competitive Advantage: The Moderating Effect of Supply Chain Resilience [J]. Chinese Journal of Management Science, 2025, 33(4): 285-298.
[5]	Daqian Shi, Xueqin Li, Dandan Li. How Can Intelligent Supply Chain Improve Enterprise Performance: Study on Supply Chain Optimization Resilience Perspective [J]. Chinese Journal of Management Science, 2025, 33(4): 325-334.
[6]	Yuzhen Hu, Jianxia Liu. Container Liner Disruption Recovery Based on Redundancy Elimination Time-band Network [J]. Chinese Journal of Management Science, 2025, 33(2): 118-130.
[7]	Qinghua Zhu, Yuxiao Chen, Guoyu Chen. The Evolution and Prospects of Research on Circular Supply Chain Management [J]. Chinese Journal of Management Science, 2025, 33(1): 221-231.
[8]	Jie Song, Jingsi Huang, Guannan He, Jianxiao Wang. Risk Measurement and Resilience Improvement of Water-Energy-Food Coupling Systems [J]. Chinese Journal of Management Science, 2025, 33(1): 323-334.
[9]	Lin Wang,Shenghui Gu,Weilan Suo. Research on a Resilience-Oriented Resource Allocation for the Protection and Restoration of Critical Infrastructures Considering Multiple Interdependencies [J]. Chinese Journal of Management Science, 2024, 32(6): 301-311.
[10]	Xiaowen Qi, Changyong Liang, Junling Zhang. Dual Hesitant Unbalanced Linguistic Group Decision Approaches with Application to Evaluating Suppliers' Resilience Capability [J]. Chinese Journal of Management Science, 2024, 32(12): 108-117.
[11]	Bingsheng Liu, Qiaoyan Cui, Dan Wang, Jianfeng Zhao. A Dynamic Game Model for Post-disaster Resilience Governance of Transport Infrastructure Based on Public Participation [J]. Chinese Journal of Management Science, 2024, 32(11): 189-200.
[12]	Ying SUN,Hui-ping LIU,Rui YAN,Qun ZHANG. Equilibrium Optimization of the Supply Chain of Emergency Medical Supplies Based on Resilience and Social Welfare [J]. Chinese Journal of Management Science, 2023, 31(8): 132-141.
[13]	YIN Hui, LU Xin-hui, , ZHOU Kai-le,. Optimal Load Dispatch for Microgrid with Distributed and Centralized Energy Storage Systems [J]. Chinese Journal of Management Science, 2023, 31(2): 118-128.
[14]	Gen LI,Xin-yu LIU,Jia-guo LIU,Ying ZHOU. How does Industrial Collaborative Agglomeration Affect the Green Resilience of Manufacturing Industry? ——Empirical Analysis Based on Catastrophe Progression and Double Fixed Dynamic Spatial Durbin Model [J]. Chinese Journal of Management Science, 2023, 31(12): 249-260.
[15]	GAO Lei, GONG Jing. Analysis Model for Infrastructure Resilience Based on Linear Piecewise Recovery Function——A Case Study of C County Power Network [J]. Chinese Journal of Management Science, 2022, 30(12): 86-95.