LoRa网络中基于深度强化学习的信息年龄优化

程克非; 陈彩蝶; 罗佳; 陈前斌

doi:10.11999/JEIT240404

LoRa网络中基于深度强化学习的信息年龄优化

doi: 10.11999/JEIT240404

程克非¹,
陈彩蝶¹,
罗佳^{1, 2, ,},
陈前斌²

1.
重庆邮电大学网络空间安全与信息法学院重庆 400065
2.
重庆邮电大学通信与信息工程学院移动通信技术重点实验室重庆 400065

基金项目: 重庆市教委科学技术研究项目(KJQN202400643)

详细信息

作者简介:
程克非：男，博士生导师，研究方向为无线通信网络、云计算与大数据、嵌入式系统及应用、网络空间安全等

陈彩蝶：女，硕士生，研究方向为LoRa物联网

罗佳：男，讲师，博士，研究方向为下一代无线通信网络、人工智能、区块链等

陈前斌：男，教授，博士生导师，研究方向为个人通信、多媒体信息处理与传输、异构蜂窝网络等

通讯作者:
罗佳　s220802003@stu.cqupt.edu.cn

中图分类号: TN929.5
计量
- 文章访问数: 48
- HTML全文浏览量: 17
- PDF下载量: 0
- 被引次数: 0
出版历程
- 收稿日期: 2024-04-04
- 修回日期: 2025-01-08
- 网络出版日期: 2025-01-25

Optimizing Age of Information in LoRa Networks via Deep Reinforcement Learning

1.
School of Cyber Security and Information law, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.
Key Laboratory of Mobile Communication Technology, School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Funds: The Science and Technology Research Program of Chongqing Municipal Education Commission (KJQN202400643)

摘要

摘要: 信息年龄(AoI)是信息新鲜度的衡量指标，针对时间敏感的物联网，最小化AoI显得尤为重要。该文基于LoRa网络的智能交通环境，分析Slot-Aloha协议下的AoI优化策略。该文建立了Slot-Aloha协议下数据包之间传输碰撞和等待时间的系统模型，并通过分析指出，在LoRa上行传输过程中，随着数据包数量增多，AoI主要受到数据包碰撞影响。为克服优化问题中动作空间过大导致难以实现有效求解的问题，该文采用连续动作空间映射离散动作空间的方式，使用柔性动作-评价 (SAC)算法对LoRa网络下的AoI进行优化。仿真结果显示，SAC算法优于传统算法与传统深度强化学习算法，可有效降低网络的平均AoI。
- 信息年龄 /
- LoRa /
- 柔性动作-评价算法 /
- 深度强化学习 /
- 优化策略
Abstract: Age of Information (AoI) is a measure of information freshness. For the time-sensitive Internet of Things, minimizing AoI is particularly important. This paper analyzes the AoI optimization strategy under Slot-Aloha protocol in an intelligent transportation environment based on LoRa network. This paper establishes a system model of transmission collisions and waiting time between packets under the Slot-Aloha protocol, and points out through analysis that during the LoRa uplink transmission process, as the number of packets increases, AoI is mainly affected by packet collisions. In order to overcome the problem that the action space is too large, which makes it difficult to achieve effective solutions, this paper adopts the method of mapping the continuous action space to the discrete action space, and uses the Soft Actor-Critical (SAC) algorithm to optimize AoI under LoRa network. Simulation results show that the SAC algorithm is superior to traditional algorithms and traditional deep reinforcement learning algorithms, and can effectively reduce the average AoI of the network. Objective With the rapid development of intelligent transportation systems, the real-time and accuracy of traffic data have become particularly important, especially in the transmission systems of traffic monitoring cameras and other equipment. Long-distance Low-power R adio frequency network (LoRa) has become an important technology for connecting sensors in the field of intelligent transportation due to its advantages of low power consumption, high coverage and long-distance communication. However, in an urban environment, LoRa networks face problems such as data collisions that may occur frequently when devices send data, which may affect the timeliness of information, which in turn affects the effectiveness of traffic management decisions. Therefore, how to optimize the timeliness of data packets in the LoRa network and improve the communication efficiency of the system has become a key issue. The research of this paper aims to solve the problem of how to effectively optimize AoI in LoRa networks, especially under the slotted Aloha protocol, to study the impact of factors such as packet collisions and over-the-air transmission time on AoI. On this basis, this paper proposes an optimization method based on deep reinforcement learning, using the Soft Actor-Critic algorithm to optimize AoI, in order to achieve lower latency and higher data transmission success rate in an intelligent transportation environment where data is frequently transmitted., thereby improving the overall performance of the system and the real-time nature of information transmission, and meeting the needs of intelligent transportation for information freshness. Method Based on the requirements for information freshness in intelligent transportation scenarios, this paper studies the optimization problem of packet AoI in LoRa networks under the slotted Aloha protocol. Aiming at the frequent data transmission in LoRa network, a system model based on LoRa packet collision is established, focusing on analyzing the impact of packet collision and over-the-air transmission time under the slotted Aloha protocol on AoI in LoRa network, providing theoretical support for improving information transmission efficiency. Considering that the temporal evolution of AoI is Markov, this paper models the optimization problem as a Markov Decision Process (MDP) and uses the SAC algorithm in deep reinforcement learning to solve it. Results and Discussions This paper analyzes the change of AoI during collision (Fig. 2), and establishes a collision model during transmission of each data packet (Fig. 4). The simulation results show that the SAC algorithm is better than the TD algorithm and the traditional algorithm (Fig. 6). As the number of terminals increases, the system average AoI increases (Fig. 7), and the change of the system average AoI under different time slots for SAC and TD3 algorithms (Fig. 8). Conclusions In view of the lack of research on AoI in LoRa networks, this paper studies the AoI optimization problem of LoRa uplink packet transmission based on the intelligent traffic management environment, and proposes a packet collision model under the slotted Aloha protocol. The greedy algorithm and SAC algorithm are used to optimize AoI respectively. Simulation results show that the greedy algorithm is better than the traditional deep reinforcement learning algorithm and worse than the SAC algorithm. SAC algorithm can effectively improve the AoI optimization problem in LoRa networks. In addition, this paper only considers AoI optimization problems in the network and does not jointly consider issues such as energy consumption and packet loss rate. In view of this deficiency, future research can further consider the balance between energy consumption, packet loss rate, and AoI optimization to reduce energy consumption and packet loss rate. In addition, this paper has not yet covered the research of heterogeneous scenarios. In a transmission environment where LoRa networks coexist with other communication technologies (such as Wi-Fi, Bluetooth, NB-IoT, etc.), interoperability, data consistency, and network management between different communication protocols and device types will bring new challenges. By conducting AoI optimization research in heterogeneous transmission environments, the performance and reliability of LoRa networks in complex application scenarios such as intelligent traffic management can be further improved.
- Age of Information (AoI) /
- LoRa /
- Soft Actor-Critic algorithm (SAC) /
- Deep reinforcement learning /
- Optimization strategy

HTML全文

图 1 系统模型

下载: 全尺寸图片幻灯片

图 2 基于时隙Aloha的数据包AoI变化情况

下载: 全尺寸图片幻灯片

图 3 数据包传输情况

下载: 全尺寸图片幻灯片

图 4 各终端数据包传输情况

下载: 全尺寸图片幻灯片

图 5 状态转移图

下载: 全尺寸图片幻灯片

图 6 不同算法的收敛曲线

下载: 全尺寸图片幻灯片

图 7 不同终端数量下的AoI

下载: 全尺寸图片幻灯片

图 8 TD3算法和SAC算法在不同时隙长度$ {T}_{\mathrm{s}\mathrm{l}} $下平均AoI变化

下载: 全尺寸图片幻灯片

表 1 空中传输时间

SF	7	8	9	10	11	12
$ {T}^{\mathrm{a}} $ (ms)	73.1	128	227.6	409.6	744.7	1365.3

下载: 导出CSV

表 2 实验参数值

参数名	值
信道数量($ c $)	2
终端数量($ N $)	12
SF数量	6
编码率($ \mathrm{C}\mathrm{R} $)	4/5
带宽($ \mathrm{B}\mathrm{W} $)	125 kHz
数据包大小($ {L}_{\mathrm{d}} $)	50 byte
step总数($ {T}_{\mathrm{s}\mathrm{t}} $)	500
时隙长度($ {T}_{\mathrm{s}\mathrm{l}} $)	500 ms

下载: 导出CSV

参考文献(21)

[1]	KAUL S, YATES R, and GRUTESER M. Real-time status: How often should one update?[C]. IEEE INFOCOM, Orlando, USA, 2012: 2731–2735. doi: 10.1109/INFCOM.2012.6195689.
[2]	INOUE Y, MASUYAMA H, TAKINE T, et al. A general formula for the stationary distribution of the age of information and its application to single-server queues[J]. IEEE Transactions on Information Theory, 2019, 65(12): 8305–8324. doi: 10.1109/TIT.2019.2938171.
[3]	BEDEWY A M, SUN Yin, and SHROFF N B. Minimizing the age of information through queues[J]. IEEE Transactions on Information Theory, 2019, 65(8): 5215–5232. doi: 10.1109/TIT.2019.2912159.
[4]	HE Qing, YUAN Di, and EPHREMIDES A. Optimal link scheduling for age minimization in wireless systems[J]. IEEE Transactions on Information Theory, 2018, 64(7): 5381–5394. doi: 10.1109/TIT.2017.2746751.
[5]	KADOTA I, SINHA A, UYSAL-BIYIKOGLU E, et al. Scheduling policies for minimizing age of information in broadcast wireless networks[J]. IEEE/ACM Transactions on Networking, 2018, 26(6): 2637–2650. doi: 10.1109/TNET.2018.2873606.
[6]	WU Beining, CAI Zhengkun, WU Wei, et al. AoI-aware resource management for smart health via deep reinforcement learning[J]. IEEE Access, 2023, 11: 81180–81195. doi: 10.1109/ACCESS.2023.3299340.
[7]	PENG Kai, XIAO Peiyun, WANG Shangguang, et al. AoI-aware partial computation offloading in IIoT with edge computing: A deep reinforcement learning based approach[J]. IEEE Transactions on Cloud Computing, 2023, 11(4): 3766–3777. doi: 10.1109/TCC.2023.3328614.
[8]	WANG Hao, LIU Chi, YANG Haoming, et al. Ensuring threshold AoI for UAV-assisted mobile crowdsensing by multi-agent deep reinforcement learning with transformer[J]. IEEE/ACM Transactions on Networking, 2024, 32(1): 566–581. doi: 10.1109/TNET.2023.3289172.
[9]	WANG Zhuoyao, XU Xiaokang, and ZHAO Jin. Spreading factor allocation and rate adaption for minimizing age of information in LoRaWAN[C]. 2022 IEEE 24th International Conference on High Performance Computing & Communications; 8th International Conference on Data Science & Systems; 20th International Conference on Smart City; 8th International Conference on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), Hainan, China, 2022: 482–489. doi: 10.1109/HPCC-DSS-SmartCity-DependSys57074.2022.00092.
[10]	CUOMO F, CAMPO M, CAPONI A, et al. EXPLoRa: Extending the performance of LoRa by suitable spreading factor allocations[C]. Proceedings of the 13th International Conference on Wireless and Mobile Computing, Networking and Communications, Rome, Italy, 2017: 1–8. doi: 10.1109/WiMOB.2017.8115779.
[11]	GEORGIOU O and RAZA U. Low power wide area network analysis: Can LoRa scale?[J]. IEEE Wireless Communications Letters, 2017, 6(2): 162–165. doi: 10.1109/LWC.2016.2647247.
[12]	HAMDI R, QARAQE M, and ALTHUNIBAT S. Dynamic spreading factor assignment in LoRa wireless networks[C]. 2020 IEEE International Conference on Communications (ICC), Dublin, Ireland, 2020: 1–5. doi: 10.1109/ICC40277.2020.9149243.
[13]	FARHAD A, KIM D H, STHAPIT P, et al. Interference-aware spreading factor assignment scheme for the massive LoRaWAN network[C]. International Conference on Electronics, Information, and Communications, Auckland, New Zealand, 2019: 1–2. doi: 10.23919/ELINFOCOM.2019.8706416.
[14]	FARHAD A, KIM D H, and PYUN J Y. Resource allocation to massive internet of things in LoRaWANs[J]. Sensors, 2020, 20(9): 2645. doi: 10.3390/s20092645.
[15]	BELTRAMELLI L, MAHMOOD A, ÖSTERBERG P, et al. LoRa beyond aloha: An investigation of alternative random access protocols[J]. IEEE Transactions on Industrial Informatics, 2021, 17(5): 3544–3554. doi: 10.1109/TII.2020.2977046.
[16]	WANG Jiwen, YU Jihong, CHEN Xiaoming, et al. Age of information for frame slotted aloha[J]. IEEE Transactions on Communications, 2023, 71(4): 2121–2135. doi: 10.1109/TCOMM.2023.3244214.
[17]	CHEIKH I, SABIR E, AOUAMI R, et al. Throughput-delay tradeoffs for slotted-aloha-based LoRaWAN networks[C]. International Conference on Wireless Communications and Mobile Computing, Harbin City, China, 2021: 2020–2025. doi: 10.1109/IWCMC51323.2021.9498969.
[18]	HAMDI R, BACCOUR E, ERBAD A, et al. LoRa-RL: Deep reinforcement learning for resource management in hybrid energy LoRa wireless networks[J]. IEEE Internet of Things Journal, 2022, 9(9): 6458–6476. doi: 10.1109/JIOT.2021.3110996.
[19]	HONG Shengguang, YAO Fang, ZHANG Fengyun, et al. Reinforcement learning approach for SF allocation in LoRa network[J]. IEEE Internet of Things Journal, 2023, 10(20): 18259–18272. doi: 10.1109/JIOT.2023.3279429.
[20]	WARET A, KANEKO M, GUITTON A, et al. LoRa throughput analysis with imperfect spreading factor orthogonality[J]. IEEE Wireless Communications Letters, 2019, 8(2): 408–411. doi: 10.1109/LWC.2018.2873705.
[21]	HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]. Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, 2018: 1856–1865.