A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning

GAO Yuan; FANG Hai; ZHAO Yang; YANG Xu

doi:10.11999/JEIT211384

Volume 45 Issue 2

Feb. 2023

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2023 > 45(2): 455-463

GAO Yuan, FANG Hai, ZHAO Yang, YANG Xu. A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 455-463. doi: 10.11999/JEIT211384

Citation:

GAO Yuan, FANG Hai, ZHAO Yang, YANG Xu. A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 455-463. doi: 10.11999/JEIT211384

Citation:

PDF( 2087 KB)

A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning

doi: 10.11999/JEIT211384

Xi’an Institute of Space Radio Technology, Xi’an 710100, China

Funds: The National Key Research and Development Program of China (2020YFB1808003)

Received Date: 2021-11-30
Accepted Date: 2022-06-22
Rev Recd Date: 2022-06-06

Available Online: 2022-06-28

Publish Date: 2023-02-07

Abstract

Abstract

In view of the high dynamics in low-orbit satellite networks and complexity of space environment, the online provisioning of Service Function Chain (SFC) has become the key problem in satellite edge networks. Considering constraints in node and link capacity and switching costs in service migration, an online SFC deployment method based on natural gradient actor-critic reinforcement learning is proposed for low-orbit satellites equipped with Multi-access Edge Computing (MEC) servers. Firstly, the real-time capacity constraints and migration costs are formulated following the high environmental dynamics in low-orbit satellite networks, respectively. Secondly, involving the migration costs and satellite coordinates, Markov Decision Process (MDP) is introduced to describe the state transition in low-orbit satellite networks. Finally, a natural gradient method-based online SFC deployment method is proposed, which facilitates the training of neural network escaping from the local optimum as compared to the standard gradient. Simulation results show that proposed method could asymptotically approach the global optimum, and exceeds existing ones based on the standard gradient in terms of end-to-end delay.
- Service Function Chain(SFC),
- Reinforcement learning,
- Low-orbit satellite networks,
- Service migration

FullText(HTML)

References(20)

References

[1]	ZHANG Jiaxin, ZHANG Xing, WANG Peng, et al. Double-edge intelligent integrated satellite terrestrial networks[J]. China Communications, 2020, 17(9): 128–146. doi: 10.23919/JCC.2020.09.011
[2]	王鹏, 张佳鑫, 张兴, 等. 低轨卫星智能多接入边缘计算网络: 需求、架构、机遇与挑战[J]. 移动通信, 2021, 45(5): 35–46. doi: 10.3969/j.issn.1006-1010.2021.05.007 WANG Peng, ZHANG Jiaxin, ZHANG Xing, et al. Low earth orbit satellite intelligent multi-access edge computing networks: Requirements, architecture, opportunities and challenges[J]. Mobile Communications, 2021, 45(5): 35–46. doi: 10.3969/j.issn.1006-1010.2021.05.007
[3]	唐琴琴, 谢人超, 刘旭, 等. 融合MEC的星地协同网络: 架构、关键技术与挑战[J]. 通信学报, 2020, 41(4): 162–181. doi: 10.11959/j.issn.1000-436x.2020082 TANG Qinqin, XIE Renchao, LIU Xu, et al. MEC enabled satellite-terrestrial network: Architecture, key technique and challenge[J]. Journal on Communications, 2020, 41(4): 162–181. doi: 10.11959/j.issn.1000-436x.2020082
[4]	LI Guanglei, ZHOU Huachun, FENG Bohan, et al. Horizontal-based orchestration for multi-domain SFC in SDN/NFV-enabled satellite/terrestrial networks[J]. China Communications, 2018, 15(5): 77–91. doi: 10.1109/cc.2018.8387988
[5]	WANG Guangchao, ZHOU Sheng, ZHANG Shan, et al. SFC-based service provisioning for reconfigurable space-air-ground integrated networks[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(7): 1478–1489. doi: 10.1109/JSAC.2020.2986851
[6]	王婷, 黄昊楠, 张兴, 等. 空天地一体化网络基于服务功能链的资源分配[J]. 无线电通信技术, 2021, 47(5): 611–617. doi: 10.3969/j.issn.1003-3114.2021.05.014 WANG Ting, HUANG Haonan, ZHANG Xing, et al. Resource allocation based on service function chain in space-air-ground integrated networks[J]. Radio Communications Technology, 2021, 47(5): 611–617. doi: 10.3969/j.issn.1003-3114.2021.05.014
[7]	SHI Keyi, ZHANG Xiushe, ZHANG Shun, et al. Time-expanded graph based energy-efficient delay-bounded multicast over satellite networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69(9): 10380–10384. doi: 10.1109/TVT.2020.2988023
[8]	邱航, 汤红波, 游伟. 基于深度Q网络的在线服务功能链部署方法[J]. 电子与信息学报, 2021, 43(11): 3122–3130. doi: 10.11999/JEIT201009 QIU Hang, TANG Hongbo, and YOU Wei. Online service function chain deployment method based on deep Q network[J]. Journal of Electronics &Information Technology, 2021, 43(11): 3122–3130. doi: 10.11999/JEIT201009
[9]	唐伦, 曹睿, 廖皓, 等. 基于深度强化学习的服务功能链可靠部署算法[J]. 电子与信息学报, 2020, 42(12): 2931–2938. doi: 10.11999/JEIT190969 TANG Lun, CAO Rui, LIAO Hao, et al. Reliable deployment algorithm of service function chain based on deep reinforcement learning[J]. Journal of Electronics &Information Technology, 2020, 42(12): 2931–2938. doi: 10.11999/JEIT190969
[10]	LI Taixin, ZHOU Huachun, LUO Hongbin, et al. Service function chain in small satellite-based software defined satellite networks[J]. China Communications, 2018, 15(3): 157–167. doi: 10.1109/CC.2018.8331999
[11]	YANG Huiting, LIU Wei, LI Hongyan, et al. Maximum flow routing strategy for space information network with service function constraints[J]. IEEE Transactions on Wireless Communications, 2022, 21(5): 2903–2923. doi: 10.1109/TWC.2021.3116983
[12]	GAO Xiangqiang, LIU Rongke, and KAUSHIK A. Service chaining placement based on satellite mission planning in ground station networks[J]. IEEE Transactions on Network and Service Management, 2021, 18(3): 3049–3063. doi: 10.1109/TNSM.2020.3045432
[13]	QU Kaige, ZHUANG Weihua, YE Qiang, et al. Dynamic flow migration for embedded services in SDN/NFV-enabled 5G core networks[J]. IEEE Transactions on Communications, 2020, 68(4): 2394–2408. doi: 10.1109/TCOMM.2020.2968907
[14]	ZHOU Zhi, WU Qiong, and CHEN Xu. Online orchestration of cross-edge service function chaining for cost-efficient edge computing[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(8): 1866–1880. doi: 10.1109/JSAC.2019.2927070
[15]	ZHENG Gao, TSIOPOULOS A, and FRIDERIKOS V. Optimal VNF chains management for proactive caching[J]. IEEE Transactions on Wireless Communications, 2018, 17(10): 6735–6748. doi: 10.1109/TWC.2018.2863685
[16]	KARIMZADEH-FARSHBAFAN M, SHAH-MANSOURI V, and NIYATO D. A dynamic reliability-aware service placement for Network Function Virtualization (NFV)[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2): 318–333. doi: 10.1109/JSAC.2019.2959196
[17]	WEI Yifei, YU F R, SONG Mei, et al. Joint optimization of caching, computing, and radio resources for fog-enabled IoT using natural actor–critic deep reinforcement learning[J]. IEEE Internet of Things Journal, 2019, 6(2): 2061–2073. doi: 10.1109/JIOT.2018.2878435
[18]	BHATNAGAR S, SUTTON R, GHAVAMZADEH M, et al. Natural actor-critic algorithms[J]. Automatica, 2009, 45(11): 2471–2482. doi: 10.1016/j.automatica.2009.07.008
[19]	WANG Lingxiao, CAI Qi, YANG Zhuoran, et al. . Neural policy gradient methods: Global optimality and rates of convergence[C]. The 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020: 1–46.
[20]	吴琦, 郭孟泽, 朱立东. 大规模低轨卫星网络移动性管理方案[J]. 中兴通讯技术, 2021, 27(5): 28–35. doi: 10.12142/ZTETJ.202105007 WU Qi, GUO Mengze, and ZHU Lidong. Large-scale low earth orbit satellite network mobility management scheme[J]. ZTE Technology Journal, 2021, 27(5): 28–35. doi: 10.12142/ZTETJ.202105007