A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning
-
摘要: 鉴于低轨卫星网络的高动态性和空间环境的复杂性，如何提供在线的快速服务功能链(SFC)部署方法，成为低轨卫星边缘网络中亟待解决的问题。综合考虑节点和链路容量等约束以及服务迁移等切换代价，针对部署多接入边缘计算(MEC)服务器的低轨卫星网络，该文提出一种基于自然梯度参与者-评价者(Actor-Critic)强化学习架构的SFC在线部署方法。首先，针对低轨卫星网络的环境高动态性， 对实时容量约束和迁移代价进行建模；其次，引入马尔可夫决策过程(MDP)，综合考虑服务迁移和卫星坐标等因素，描述低轨卫星网络的状态转移过程；最后，提出一种基于自然梯度的在线SFC部署强化学习方法，不同于标准梯度，自然梯度法进行模型层面的更新，以避免神经网络的训练陷入局部最优解。仿真结果表明，该文方法可逼近全局最优解，并在端到端时延性能上优于基于标准梯度的强化学习部署方法。Abstract: In view of the high dynamics in low-orbit satellite networks and complexity of space environment, the online provisioning of Service Function Chain (SFC) has become the key problem in satellite edge networks. Considering constraints in node and link capacity and switching costs in service migration, an online SFC deployment method based on natural gradient actor-critic reinforcement learning is proposed for low-orbit satellites equipped with Multi-access Edge Computing (MEC) servers. Firstly, the real-time capacity constraints and migration costs are formulated following the high environmental dynamics in low-orbit satellite networks, respectively. Secondly, involving the migration costs and satellite coordinates, Markov Decision Process (MDP) is introduced to describe the state transition in low-orbit satellite networks. Finally, a natural gradient method-based online SFC deployment method is proposed, which facilitates the training of neural network escaping from the local optimum as compared to the standard gradient. Simulation results show that proposed method could asymptotically approach the global optimum, and exceeds existing ones based on the standard gradient in terms of end-to-end delay.
-
算法1 基于自然梯度的Actor-Critic算法 输入：当前网络状态${{\boldsymbol{s}}_t}$ 输出：在线SFC部署结果${{\boldsymbol{a}}_t}$ (1) 初始化神经网络参数$ {\boldsymbol{w}} $,$ {\boldsymbol{w}}' $,$ {\boldsymbol{\theta}} $和$ {\boldsymbol{\theta}} ' $和经验回放池$\mathcal{D}$ (2) for episode=1, 2,···, E do (3) 重置环境，并重置${r_0} = 0$ (4) for t = 0, 1,···, T–1 do (5) 根据Actor的策略函数${\pi _\theta }( \cdot |{s_t})$，随机抽样获取${a_t}$ (6) 求解松弛式(15)的松弛问题，获取奖励值${r_t}$，并观察系
统转移到的下一状态${{\boldsymbol{s}}_{t + 1}}$(7) 将4元组$({{\boldsymbol{s}}_t},{{\boldsymbol{a}}_t},{r_t},{{\boldsymbol{s}}_{t + 1}})$存储到经验回放池$\mathcal{D}$ (8) 随机抽取出包含$D$个样本的小批量 (9) 根据式(20)和式(27)，依序更新参数$ {\boldsymbol{w}} $和$ {\boldsymbol{\theta}} $ (10) 按照设定的更新频率，更新$ {\boldsymbol{w}}' $和${\boldsymbol{ \theta}} ' $ (11) end (12) end 
下载: 导出CSV
-
[1] ZHANG Jiaxin, ZHANG Xing, WANG Peng, et al. Double-edge intelligent integrated satellite terrestrial networks[J]. China Communications, 2020, 17(9): 128–146. doi: 10.23919/JCC.2020.09.011 [2] 王鹏, 张佳鑫, 张兴, 等. 低轨卫星智能多接入边缘计算网络: 需求、架构、机遇与挑战[J]. 移动通信, 2021, 45(5): 35–46. doi: 10.3969/j.issn.1006-1010.2021.05.007WANG Peng, ZHANG Jiaxin, ZHANG Xing, et al. Low earth orbit satellite intelligent multi-access edge computing networks: Requirements, architecture, opportunities and challenges[J]. Mobile Communications, 2021, 45(5): 35–46. doi: 10.3969/j.issn.1006-1010.2021.05.007 [3] 唐琴琴, 谢人超, 刘旭, 等. 融合MEC的星地协同网络: 架构、 关键技术与挑战[J]. 通信学报, 2020, 41(4): 162–181. doi: 10.11959/j.issn.1000-436x.2020082TANG Qinqin, XIE Renchao, LIU Xu, et al. MEC enabled satellite-terrestrial network: Architecture, key technique and challenge[J]. Journal on Communications, 2020, 41(4): 162–181. doi: 10.11959/j.issn.1000-436x.2020082 [4] LI Guanglei, ZHOU Huachun, FENG Bohan, et al. Horizontal-based orchestration for multi-domain SFC in SDN/NFV-enabled satellite/terrestrial networks[J]. China Communications, 2018, 15(5): 77–91. doi: 10.1109/cc.2018.8387988 [5] WANG Guangchao, ZHOU Sheng, ZHANG Shan, et al. SFC-based service provisioning for reconfigurable space-air-ground integrated networks[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(7): 1478–1489. doi: 10.1109/JSAC.2020.2986851 [6] 王婷, 黄昊楠, 张兴, 等. 空天地一体化网络基于服务功能链的资源分配[J]. 无线电通信技术, 2021, 47(5): 611–617. doi: 10.3969/j.issn.1003-3114.2021.05.014WANG Ting, HUANG Haonan, ZHANG Xing, et al. Resource allocation based on service function chain in space-air-ground integrated networks[J]. Radio Communications Technology, 2021, 47(5): 611–617. doi: 10.3969/j.issn.1003-3114.2021.05.014 [7] SHI Keyi, ZHANG Xiushe, ZHANG Shun, et al. Time-expanded graph based energy-efficient delay-bounded multicast over satellite networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69(9): 10380–10384. doi: 10.1109/TVT.2020.2988023 [8] 邱航, 汤红波, 游伟. 基于深度Q网络的在线服务功能链部署方法[J]. 电子与信息学报, 2021, 43(11): 3122–3130. doi: 10.11999/JEIT201009QIU Hang, TANG Hongbo, and YOU Wei. Online service function chain deployment method based on deep Q network[J]. Journal of Electronics &Information Technology, 2021, 43(11): 3122–3130. doi: 10.11999/JEIT201009 [9] 唐伦, 曹睿, 廖皓, 等. 基于深度强化学习的服务功能链可靠部署算法[J]. 电子与信息学报, 2020, 42(12): 2931–2938. doi: 10.11999/JEIT190969TANG Lun, CAO Rui, LIAO Hao, et al. Reliable deployment algorithm of service function chain based on deep reinforcement learning[J]. Journal of Electronics &Information Technology, 2020, 42(12): 2931–2938. doi: 10.11999/JEIT190969 [10] LI Taixin, ZHOU Huachun, LUO Hongbin, et al. Service function chain in small satellite-based software defined satellite networks[J]. China Communications, 2018, 15(3): 157–167. doi: 10.1109/CC.2018.8331999 [11] YANG Huiting, LIU Wei, LI Hongyan, et al. Maximum flow routing strategy for space information network with service function constraints[J]. IEEE Transactions on Wireless Communications, 2022, 21(5): 2903–2923. doi: 10.1109/TWC.2021.3116983 [12] GAO Xiangqiang, LIU Rongke, and KAUSHIK A. Service chaining placement based on satellite mission planning in ground station networks[J]. IEEE Transactions on Network and Service Management, 2021, 18(3): 3049–3063. doi: 10.1109/TNSM.2020.3045432 [13] QU Kaige, ZHUANG Weihua, YE Qiang, et al. Dynamic flow migration for embedded services in SDN/NFV-enabled 5G core networks[J]. IEEE Transactions on Communications, 2020, 68(4): 2394–2408. doi: 10.1109/TCOMM.2020.2968907 [14] ZHOU Zhi, WU Qiong, and CHEN Xu. Online orchestration of cross-edge service function chaining for cost-efficient edge computing[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(8): 1866–1880. doi: 10.1109/JSAC.2019.2927070 [15] ZHENG Gao, TSIOPOULOS A, and FRIDERIKOS V. Optimal VNF chains management for proactive caching[J]. IEEE Transactions on Wireless Communications, 2018, 17(10): 6735–6748. doi: 10.1109/TWC.2018.2863685 [16] KARIMZADEH-FARSHBAFAN M, SHAH-MANSOURI V, and NIYATO D. A dynamic reliability-aware service placement for Network Function Virtualization (NFV)[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2): 318–333. doi: 10.1109/JSAC.2019.2959196 [17] WEI Yifei, YU F R, SONG Mei, et al. Joint optimization of caching, computing, and radio resources for fog-enabled IoT using natural actor–critic deep reinforcement learning[J]. IEEE Internet of Things Journal, 2019, 6(2): 2061–2073. doi: 10.1109/JIOT.2018.2878435 [18] BHATNAGAR S, SUTTON R, GHAVAMZADEH M, et al. Natural actor-critic algorithms[J]. Automatica, 2009, 45(11): 2471–2482. doi: 10.1016/j.automatica.2009.07.008 [19] WANG Lingxiao, CAI Qi, YANG Zhuoran, et al. . Neural policy gradient methods: Global optimality and rates of convergence[C]. The 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020: 1–46. [20] 吴琦, 郭孟泽, 朱立东. 大规模低轨卫星网络移动性管理方案[J]. 中兴通讯技术, 2021, 27(5): 28–35. doi: 10.12142/ZTETJ.202105007WU Qi, GUO Mengze, and ZHU Lidong. Large-scale low earth orbit satellite network mobility management scheme[J]. ZTE Technology Journal, 2021, 27(5): 28–35. doi: 10.12142/ZTETJ.202105007 -
下载:
点击查看大图
图(5) / 表(1)
计量
- 文章访问数: 718
- HTML全文浏览量: 378
- PDF下载量: 172
- 被引次数: 0
