A Novel Beam Hopping Resource Allocation Scheme of  Low Earth Orbit Satellite Based on Transfer Deep Reinforcement Learning

CHEN Qianbin; MA Shiqing; DUAN Ruiji; TANG Lun; LIANG Chengchao

doi:10.11999/JEIT211457

Volume 45 Issue 2

Feb. 2023

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2023 > 45(2): 407-417

CHEN Qianbin, MA Shiqing, DUAN Ruiji, TANG Lun, LIANG Chengchao. A Novel Beam Hopping Resource Allocation Scheme of Low Earth Orbit Satellite Based on Transfer Deep Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 407-417. doi: 10.11999/JEIT211457

Citation:

CHEN Qianbin, MA Shiqing, DUAN Ruiji, TANG Lun, LIANG Chengchao. A Novel Beam Hopping Resource Allocation Scheme of Low Earth Orbit Satellite Based on Transfer Deep Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 407-417. doi: 10.11999/JEIT211457

Citation:

CHEN Qianbin, MA Shiqing, DUAN Ruiji, TANG Lun, LIANG Chengchao. A Novel Beam Hopping Resource Allocation Scheme of Low Earth Orbit Satellite Based on Transfer Deep Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 407-417. doi: 10.11999/JEIT211457

PDF( 4863 KB)

A Novel Beam Hopping Resource Allocation Scheme of Low Earth Orbit Satellite Based on Transfer Deep Reinforcement Learning

doi: 10.11999/JEIT211457

School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Funds: The National Natural Science Foundation of China (62071078, 62001076), the Science and Technology Research Program of Chongqing Municipal Education Commission (KJZD-M201800601, KJQN-201900645)

Received Date: 2021-12-08
Rev Recd Date: 2022-03-23

Available Online: 2022-03-29

Publish Date: 2023-02-07

Abstract

Abstract

In the Low Earth Orbit (LEO) scenario, traditional resource allocation schemes can cause unbalanced resource allocation in specific cells. A beam hopping resource allocation scheme of LEO based on Transfer Deep Reinforcement Learning (TDRL) is proposed in this paper. Firstly, considering on-board buffer information, service arrival status and channel status, a LEO resource allocation optimization model that supports beam hopping technology is proposed with the goal of minimizing the average delay of data packets. Secondly, in view of the dynamic variability of the LEO network, the dynamic and random change of communication resources and requirements are considered, then the Deep Q Network (DQN) algorithm is adopted, and its neural network is used as a nonlinear approximation function. Further, to realize and accelerate the convergence process of the Deep Reinforcement Learning (DRL) algorithm in other target tasks, the concept of Transfer Learning (TL) is introduced in this paper, which uses the scheduling task learned by the source satellite to find quickly the beam scheduling and power allocation strategy of the target satellite. The simulation results demonstrate that the algorithm can optimize the time slot allocation in the satellite service process while decreasing the average delay of data packets and improving the throughput and resource utilization efficiency of the system.
- Low Earth Orbit (LEO),
- Beam hopping,
- Resource allocation,
- Deep Reinforcement Learning(DRL),
- Transfer Learning(TL)

FullText(HTML)

References(22)

References

[1]	RADTKE J, KEBSCHULL C, and STOLL E. Interactions of the space debris environment with mega constellations—using the example of the OneWeb constellation[J]. Acta Astronautica, 2017, 131: 55–68. doi: 10.1016/j.actaastro.2016.11.021
[2]	NI Shuang, LIU Junyu, SHENG Min, et al. Joint optimization of user association and resource allocation in cache-enabled terrestrial-satellite integrating network[J]. Science China Information Sciences, 2021, 64(8): 182306. doi: 10.1007/s11432-020-3083-5
[3]	XIE Renchao, TANG Qinqin, WANG Qiuning, et al. Satellite-terrestrial integrated edge computing networks: Architecture, challenges, and open issues[J]. IEEE Network, 2020, 34(3): 224–231. doi: 10.1109/MNET.011.1900369
[4]	PANTHI S, BREYNAERT D, MCLAIN C, et al. Beam hopping-a flexible satellite communication system for mobility[C]. The 35th AIAA International Communications Satellite Systems Conference, Trieste, Italy, 2017: 16–19.
[5]	WANG Libing, HU Xin, MA Shijun, et al. Dynamic beam hopping of multi-beam satellite based on genetic algorithm[C]. 2020 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom), Exeter, UK, 2020: 1364–1370.
[6]	LEI Jiang, HAN Zhu, VÁZQUEZ-CASTRO M Á, et al. Secure satellite communication systems design with individual secrecy rate constraints[J]. IEEE Transactions on Information Forensics and Security, 2011, 6(3): 661–671. doi: 10.1109/TIFS.2011.2148716
[7]	HAN Han, ZHENG Xueqiang, HUANG Qinfei, et al. QoS-equilibrium slot allocation for beam hopping in broadband satellite communication systems[J]. Wireless Networks, 2015, 21(8): 2617–2630. doi: 10.1007/S11276-015-0934-Z
[8]	LIZARRAGA J, ANGELETTI P, ALAGHA N, et al. Flexibility performance in advanced Ka-band multibeam satellites[C]. 2014 IEEE International Vacuum Electronics Conference, Monterey, USA, 2014: 45–46.
[9]	ALEGRE R, ALAGHA N, and VÁZQUEZ-CASTRO M A. Heuristic algorithms for flexible resource allocation in beam hopping multi-beam satellite systems[C]. The 29th AIAA International Communications Satellite Systems Conference, Nara, Japan, 2011: 6–20.
[10]	SHI Shengchao, LI Guangxia, LI Zhiqiang, et al. Joint power and bandwidth allocation for beam-hopping user downlinks in smart gateway multibeam satellite systems[J]. International Journal of Distributed Sensor Networks, 2017, 13(5): 155014771770946.
[11]	LEI Lei, LAGUNAS E, YUAN Yaxiong, et al. Deep learning for beam hopping in multibeam satellite systems[C]. The 2020 IEEE 91st Vehicular Technology Conference, Antwerp, Belgium, 2020: 1–5.
[12]	LEI Lei, LAGUNAS E, YUAN Yaxiong, et al. Beam illumination pattern design in satellite networks: Learning and optimization for efficient beam hopping[J]. IEEE Access, 2020, 8: 136655–136667. doi: 10.1109/ACCESS.2020.3011746
[13]	International Telecommunication Union-Radio(ITU-R). Rec. ITU-R S. 1528 Satellite antenna radiation patterns for non-geostationary orbit satellite antennas operating in the fixed-satellite service below 30 GHz[S]. 2001.
[14]	管令进. 基于深度强化学习的异构云无线接入网资源分配算法研究[D]. [硕士论文], 重庆邮电大学, 2020. GUAN Lingjin. Deep reinforcement learning-based resource allocation algorithm research for heterogeneous cloud access network[D]. [Master dissertation], Chongqing University of Posts and Telecommunications, 2020.
[15]	王艺鹏. 多波束卫星通信系统中的动态波束调度技术研究[D]. [硕士论文], 北京邮电大学, 2019. WANG Yipeng. Research on dynamic beam scheduling technology in multi-beam satellite communication system[D]. [Master dissertation], Beijing University of Posts and Telecommunications, 2019.
[16]	JUSTESEN N, BONTRAGER P, TOGELIUS J, et al. Deep learning for video game playing[J]. IEEE Transactions on Games, 2020, 12(1): 1–20. doi: 10.1109/TG.2019.2896986
[17]	陈前斌, 管令进, 李子煜, 等. 基于深度强化学习的异构云无线接入网自适应无线资源分配算法[J]. 电子与信息学报, 2020, 42(6): 1468–1477. doi: 10.11999/JEIT190511 CHEN Qianbin, GUAN Lingjin, LI Ziyu, et al. Deep reinforcement learning-based adaptive wireless resource allocation algorithm for heterogeneous cloud wireless access network[J]. Journal of Electronics &Information Technology, 2020, 42(6): 1468–1477. doi: 10.11999/JEIT190511
[18]	ŞEN S Y and ÖZKURT N. Convolutional neural network hyperparameter tuning with adam optimizer for ECG classification[C]. 2020 Innovations in Intelligent Systems and Applications Conference, Istanbul, Turkey, 2020: 1–6.
[19]	KOUSHI A M, HU Fei, and KUMAR S. Intelligent spectrum management based on transfer actor-critic learning for rateless transmissions in cognitive radio networks[J]. IEEE Transactions on Mobile Computing, 2018, 17(5): 1204–1215. doi: 10.1109/TMC.2017.2744620
[20]	唐伦, 贺小雨, 王晓, 等. 基于迁移演员-评论家学习的服务功能链部署算法[J]. 电子与信息学报, 2020, 42(11): 2671–2679. doi: 10.11999/JEIT190542 TANG Lun, HE Xiaoyu, WANG Xiao, et al. Deployment algorithm of service function chain based on transfer actor-critic learning[J]. Journal of Electronics &Information Technology, 2020, 42(11): 2671–2679. doi: 10.11999/JEIT190542
[21]	PRATT S R, RAINES R A, FOSSA C E, et al. An operational and performance overview of the IRIDIUM low earth orbit satellite system[J]. IEEE Communications Surveys, 1999, 2(2): 2–10. doi: 10.1109/COMST.1999.5340513
[22]	HU Xin, ZHANG Yuchen, LIAO Xianglai, et al. Dynamic beam hopping method based on multi-objective deep reinforcement learning for next generation satellite broadband systems[J]. IEEE Transactions on Broadcasting, 2020, 66(3): 630–646. doi: 10.1109/TBC.2019.2960940