Resource Management Based on Deep Reinforcement Learning for UAV Communication Considering Power-Consumption Outage
-
摘要: 最新研究表明,高速传输导致的手机温度变化会影响相应的传输性能。针对高速传输下未考虑与手机温度有关的能耗中断而导致传输性能降低的问题,该文提出一种基于深度强化学习的资源管理方案去考虑无人机(UAV)通信场景下的能耗中断。首先,给出无人机通信的网络模型与智能手机热传递模型的分析;其次,将能耗中断的影响以约束条件的形式整合到无人机场景的优化问题中,并通过联合考虑带宽分配、功率分配和轨迹设计优化系统吞吐量;最后,采用马尔可夫决策过程描述相应的优化问题并通过名为归一化优势函数的深度强化学习算法求解。仿真表明,所提方案能有效提升系统吞吐量并得到合理的无人机飞行轨迹。Abstract: Recent research has demonstrated that the temperature variation of smartphone caused by high data rate transmission could affect the corresponding performance on transmission. Considering the problem of performance degradation on transmission caused by the ignorance of the power-consumption outage which is related with the temperature of smartphone, a deep reinforcement learning based resource management scheme is proposed to consider the power-consumption outage for Unmanned Aerial Vehicle (UAV) communication scenario. Firstly, the analysis for the network model of UAV communication and heat transfer model in smartphone is established. Then, the influence of power-consumption outage is integrated into the optimization problem of UAV scenario in the form of constraint, and the system throughput is optimized via the joint consideration of bandwidth allocation, power allocation and trajectory design. Finally, Markov decision process is adopted to depict the problem and the optimization target is achieved by a deep reinforcement learning algorithm named normalized advantage function. Simulation results manifest that the proposed scheme can effectively enhance the system throughput and achieve appropriate trajectory of UAV.
-
算法1 求解问题P1的NAF算法 输入:主网络Q的参数集v,目标网络ˆQ的参数集v−=v,经验
池X=∅,计数器t=0,T=0输出:动作向量a(t) Repeat: 获得当前时隙状态st,tstart=t Repeat: 产生随机噪声向量Nt 选择当前时隙动作at=α'(st|vt)+Nt 执行动作a(t)并获得即时奖励rt和下一时隙状态st+1 将经验(st,at,rt,st+1)存储到经验池X 从经验池X中随机抽样包含M条经验的Mini-batch 对于经验m计算:ym=rm+γˆV(sm+1|v−t) 计算损失函数:
L(vt)=1MM∑m=1(ym−Q(sm,am|vt))2使用梯度下降法对主网络进行更新:
vt:=vt−α∇L(vt)t:=t+1,每隔Y个时隙更新目标网络v−t=vt Until t==tstart+~T T:=T+1 Until T>Tmax 表 1 仿真参数
参数 值 参数 值 参数 值 ˜T(回合) 30 Ic 102 G(dBi) 10 M(经验) 128 η 0.59 fCF(GHz) 40 Y(时隙) 300 FBP 3 Hu(m) 70 NTx(天线) 64 FAP 4 Tenv(K) 298 NRx(天线) 4 B(GHz) 1 Tsur0(K) 303 θ π/2 mc(g) 1 KBP 8×107 χNLos 2.4 Ptx(W) 5 KAP 6×107 ψNLoS 5.27 d0(m) 5 PLNA(mW) 24.3 χLos 2 L(mm) 2 N0(dBm/Hz) −174 ψLoS 5.3 D(mm) 1 k1(W/(m⋅K)) 401 α1 0.1 A(cm2) 1 k2(W/(m⋅K)) 130 α2 0.2 Dmax(m) 90 hair(W/(m2⋅K)) 26.3 λ 0.3 Tmax(回合) 600 cchip(J/(kg⋅K)) 1030 -
[1] GERACI G, GARCIA-RODRIGUE A, AZARI M M, et al. What will the future of UAV cellular communications be? A flight from 5G to 6G[J]. IEEE Communications Surveys & Tutorials, 2022, 24(3): 1304–1335. doi: 10.1109/COMST.2022.3171135 [2] YANG Jing, GE Xiaohu, THOMPSON J, et al. Power-consumption outage in beyond fifth generation mobile communication systems[J]. IEEE Transactions on Wireless Communications, 2021, 20(2): 897–910. doi: 10.1109/TWC.2020.3029051 [3] GARIMELLA S V, PERSOONS T, WEIBEL J A, et al. Electronics thermal management in information and communications technologies: Challenges and future directions[J]. IEEE Transactions on Components, Packaging and Manufacturing Technology, 2017, 7(8): 1191–1205. doi: 10.1109/TCPMT.2016.2603600 [4] CHIRIAC V, MOLLOY S, ANDERSON J, et al. A figure of merit for mobile device thermal management[C]. The 15th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm), Las Vegas, USA, 2016: 1393–1397. [5] BHAT G, GUMUSSOY S, and OGRAS U Y. Power and thermal analysis of commercial mobile platforms: Experiments and case studies[C]. 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), Florence, Italy, 2019: 144–149. [6] ARNOMO S A, SIMANJUNTAK P, and NUR SADIKAN S F. Overheating analysis of mobile phone temperature based on multitasking process[C]. 2021 International Conference on Computer Science and Engineering (IC2SE), Padang, Indonesia, 2021: 1–6. [7] MAMMELA A and ANTTONEN A. Why will computing power need particular attention in future wireless devices[J]. IEEE Circuits and Systems Magazine, 2017, 17(1): 12–26. doi: 10.1109/MCAS.2016.2642679 [8] YANG Jing, GE Xiaohu, and ZHONG Yi. How much of wireless rates can smartphones support in 5G networks?[J]. IEEE Network, 2019, 33(3): 122–129. doi: 10.1109/MNET.2018.1800025 [9] 陈新颖, 盛敏, 李博, 等. 面向6G的无人机通信综述[J]. 电子与信息学报, 2022, 44(3): 781–789. doi: 10.11999/JEIT210789CHEN Xinying, SHENG Min, LI Bo, et al. Survey on unmanned aerial vehicle communications for 6G[J]. Journal of Electronics &Information Technology, 2022, 44(3): 781–789. doi: 10.11999/JEIT210789 [10] ZHAN Cheng and HUANG Renjie. Energy efficient adaptive video streaming with rotary-wing UAV[J]. IEEE Transactions on Vehicular Technology, 2020, 69(7): 8040–8044. doi: 10.1109/TVT.2020.2993303 [11] CHEN Yan, ZHANG Hangjing, and HU Yang. Optimal power and bandwidth allocation for multiuser video streaming in UAV relay networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69(6): 6644–6655. doi: 10.1109/TVT.2020.2985061 [12] FU Xiuhua, DING Tian, KADOCH M, et al. Uplink performance analysis of UAV cellular communications with power control[C]. 2020 International Wireless Communications and Mobile Computing (IWCMC), Limassol, Cyprus, 2020: 676–679. [13] LIU Xiao, LIU Yuanwei, CHEN Yue, et al. Trajectory design and power control for multi-UAV assisted wireless networks: A machine learning approach[J]. IEEE Transactions on Vehicular Technology, 2019, 68(8): 7957–7969. doi: 10.1109/TVT.2019.2920284 [14] CHEN Mingzhe, SAAD W, and YIN Changchuan. Echo-liquid state deep learning for 360 content transmission and caching in wireless VR networks with cellular-connected UAVs[J]. IEEE Transactions on Communications, 2019, 67(9): 6386–6400. doi: 10.1109/TCOMM.2019.2917440 [15] GOLDSMITH A. Wireless Communications[M]. Cambridge, USA: Cambridge University Press, 2005: 78–79. [16] ZHAO Pengtao, TIAN Hui, CHEN K C, et al. Context-aware TDD configuration and resource allocation for mobile edge computing[J]. IEEE Transactions on Communications, 2020, 68(2): 1118–1131. doi: 10.1109/tcomm.2019.2952580 [17] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529–533. doi: 10.1038/nature14236 [18] GU Shixiang, LILLICRAP T, SUTSKEVER I, et al. Continuous deep Q-learning with model-based acceleration[C]. The 33rd International Conference on International Conference on Machine Learning, New York, USA, 2016: 2829–2838. 期刊类型引用(3)
1. 邓韬,刘哲潮,汪华章,何磊. 多源声发射信号混合重叠组稀疏分类研究. 计量学报. 2024(01): 64-72 . 百度学术
2. 李晓东,刘冠男,付宁,乔立岩. 异常值干扰下的加权中值截取相位检索算法. 宇航总体技术. 2024(03): 44-49 . 百度学术
3. 王世元,史春芬,蒋云翔,王文月,钱国兵. 基于q梯度的仿射投影算法及其稳态均方收敛分析. 电子与信息学报. 2018(10): 2402-2407 . 本站查看
其他类型引用(4)
-