Advanced Search
Volume 45 Issue 2
Feb.  2023
Turn off MathJax
Article Contents
GAO Yuan, FANG Hai, ZHAO Yang, YANG Xu. A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 455-463. doi: 10.11999/JEIT211384
Citation: GAO Yuan, FANG Hai, ZHAO Yang, YANG Xu. A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 455-463. doi: 10.11999/JEIT211384

A Satellite Edge Network Service Function Chain Deployment Method Based on Natural Gradient Actor-Critic Reinforcement Learning

doi: 10.11999/JEIT211384
Funds:  The National Key Research and Development Program of China (2020YFB1808003)
  • Received Date: 2021-11-30
  • Accepted Date: 2022-06-22
  • Rev Recd Date: 2022-06-06
  • Available Online: 2022-06-28
  • Publish Date: 2023-02-07
  • In view of the high dynamics in low-orbit satellite networks and complexity of space environment, the online provisioning of Service Function Chain (SFC) has become the key problem in satellite edge networks. Considering constraints in node and link capacity and switching costs in service migration, an online SFC deployment method based on natural gradient actor-critic reinforcement learning is proposed for low-orbit satellites equipped with Multi-access Edge Computing (MEC) servers. Firstly, the real-time capacity constraints and migration costs are formulated following the high environmental dynamics in low-orbit satellite networks, respectively. Secondly, involving the migration costs and satellite coordinates, Markov Decision Process (MDP) is introduced to describe the state transition in low-orbit satellite networks. Finally, a natural gradient method-based online SFC deployment method is proposed, which facilitates the training of neural network escaping from the local optimum as compared to the standard gradient. Simulation results show that proposed method could asymptotically approach the global optimum, and exceeds existing ones based on the standard gradient in terms of end-to-end delay.
  • loading
  • [1]
    ZHANG Jiaxin, ZHANG Xing, WANG Peng, et al. Double-edge intelligent integrated satellite terrestrial networks[J]. China Communications, 2020, 17(9): 128–146. doi: 10.23919/JCC.2020.09.011
    [2]
    王鹏, 张佳鑫, 张兴, 等. 低轨卫星智能多接入边缘计算网络: 需求、架构、机遇与挑战[J]. 移动通信, 2021, 45(5): 35–46. doi: 10.3969/j.issn.1006-1010.2021.05.007

    WANG Peng, ZHANG Jiaxin, ZHANG Xing, et al. Low earth orbit satellite intelligent multi-access edge computing networks: Requirements, architecture, opportunities and challenges[J]. Mobile Communications, 2021, 45(5): 35–46. doi: 10.3969/j.issn.1006-1010.2021.05.007
    [3]
    唐琴琴, 谢人超, 刘旭, 等. 融合MEC的星地协同网络: 架构、 关键技术与挑战[J]. 通信学报, 2020, 41(4): 162–181. doi: 10.11959/j.issn.1000-436x.2020082

    TANG Qinqin, XIE Renchao, LIU Xu, et al. MEC enabled satellite-terrestrial network: Architecture, key technique and challenge[J]. Journal on Communications, 2020, 41(4): 162–181. doi: 10.11959/j.issn.1000-436x.2020082
    [4]
    LI Guanglei, ZHOU Huachun, FENG Bohan, et al. Horizontal-based orchestration for multi-domain SFC in SDN/NFV-enabled satellite/terrestrial networks[J]. China Communications, 2018, 15(5): 77–91. doi: 10.1109/cc.2018.8387988
    [5]
    WANG Guangchao, ZHOU Sheng, ZHANG Shan, et al. SFC-based service provisioning for reconfigurable space-air-ground integrated networks[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(7): 1478–1489. doi: 10.1109/JSAC.2020.2986851
    [6]
    王婷, 黄昊楠, 张兴, 等. 空天地一体化网络基于服务功能链的资源分配[J]. 无线电通信技术, 2021, 47(5): 611–617. doi: 10.3969/j.issn.1003-3114.2021.05.014

    WANG Ting, HUANG Haonan, ZHANG Xing, et al. Resource allocation based on service function chain in space-air-ground integrated networks[J]. Radio Communications Technology, 2021, 47(5): 611–617. doi: 10.3969/j.issn.1003-3114.2021.05.014
    [7]
    SHI Keyi, ZHANG Xiushe, ZHANG Shun, et al. Time-expanded graph based energy-efficient delay-bounded multicast over satellite networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69(9): 10380–10384. doi: 10.1109/TVT.2020.2988023
    [8]
    邱航, 汤红波, 游伟. 基于深度Q网络的在线服务功能链部署方法[J]. 电子与信息学报, 2021, 43(11): 3122–3130. doi: 10.11999/JEIT201009

    QIU Hang, TANG Hongbo, and YOU Wei. Online service function chain deployment method based on deep Q network[J]. Journal of Electronics &Information Technology, 2021, 43(11): 3122–3130. doi: 10.11999/JEIT201009
    [9]
    唐伦, 曹睿, 廖皓, 等. 基于深度强化学习的服务功能链可靠部署算法[J]. 电子与信息学报, 2020, 42(12): 2931–2938. doi: 10.11999/JEIT190969

    TANG Lun, CAO Rui, LIAO Hao, et al. Reliable deployment algorithm of service function chain based on deep reinforcement learning[J]. Journal of Electronics &Information Technology, 2020, 42(12): 2931–2938. doi: 10.11999/JEIT190969
    [10]
    LI Taixin, ZHOU Huachun, LUO Hongbin, et al. Service function chain in small satellite-based software defined satellite networks[J]. China Communications, 2018, 15(3): 157–167. doi: 10.1109/CC.2018.8331999
    [11]
    YANG Huiting, LIU Wei, LI Hongyan, et al. Maximum flow routing strategy for space information network with service function constraints[J]. IEEE Transactions on Wireless Communications, 2022, 21(5): 2903–2923. doi: 10.1109/TWC.2021.3116983
    [12]
    GAO Xiangqiang, LIU Rongke, and KAUSHIK A. Service chaining placement based on satellite mission planning in ground station networks[J]. IEEE Transactions on Network and Service Management, 2021, 18(3): 3049–3063. doi: 10.1109/TNSM.2020.3045432
    [13]
    QU Kaige, ZHUANG Weihua, YE Qiang, et al. Dynamic flow migration for embedded services in SDN/NFV-enabled 5G core networks[J]. IEEE Transactions on Communications, 2020, 68(4): 2394–2408. doi: 10.1109/TCOMM.2020.2968907
    [14]
    ZHOU Zhi, WU Qiong, and CHEN Xu. Online orchestration of cross-edge service function chaining for cost-efficient edge computing[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(8): 1866–1880. doi: 10.1109/JSAC.2019.2927070
    [15]
    ZHENG Gao, TSIOPOULOS A, and FRIDERIKOS V. Optimal VNF chains management for proactive caching[J]. IEEE Transactions on Wireless Communications, 2018, 17(10): 6735–6748. doi: 10.1109/TWC.2018.2863685
    [16]
    KARIMZADEH-FARSHBAFAN M, SHAH-MANSOURI V, and NIYATO D. A dynamic reliability-aware service placement for Network Function Virtualization (NFV)[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(2): 318–333. doi: 10.1109/JSAC.2019.2959196
    [17]
    WEI Yifei, YU F R, SONG Mei, et al. Joint optimization of caching, computing, and radio resources for fog-enabled IoT using natural actor–critic deep reinforcement learning[J]. IEEE Internet of Things Journal, 2019, 6(2): 2061–2073. doi: 10.1109/JIOT.2018.2878435
    [18]
    BHATNAGAR S, SUTTON R, GHAVAMZADEH M, et al. Natural actor-critic algorithms[J]. Automatica, 2009, 45(11): 2471–2482. doi: 10.1016/j.automatica.2009.07.008
    [19]
    WANG Lingxiao, CAI Qi, YANG Zhuoran, et al. . Neural policy gradient methods: Global optimality and rates of convergence[C]. The 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020: 1–46.
    [20]
    吴琦, 郭孟泽, 朱立东. 大规模低轨卫星网络移动性管理方案[J]. 中兴通讯技术, 2021, 27(5): 28–35. doi: 10.12142/ZTETJ.202105007

    WU Qi, GUO Mengze, and ZHU Lidong. Large-scale low earth orbit satellite network mobility management scheme[J]. ZTE Technology Journal, 2021, 27(5): 28–35. doi: 10.12142/ZTETJ.202105007
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(5)  / Tables(1)

    Article Metrics

    Article views (700) PDF downloads(170) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return