A Multi-user Computation Offloading Optimization Model and Algorithm Based on Deep Reinforcement Learning

LI Zhihua; YU Zili

doi:10.11999/JEIT230445

Volume 46 Issue 4

Apr. 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2024 > 46(4): 1321-1332

LI Zhihua, YU Zili. A Multi-user Computation Offloading Optimization Model and Algorithm Based on Deep Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2024, 46(4): 1321-1332. doi: 10.11999/JEIT230445

Citation:

LI Zhihua, YU Zili. A Multi-user Computation Offloading Optimization Model and Algorithm Based on Deep Reinforcement Learning[J]. Journal of Electronics & Information Technology, 2024, 46(4): 1321-1332. doi: 10.11999/JEIT230445

Citation:

PDF( 3653 KB)

A Multi-user Computation Offloading Optimization Model and Algorithm Based on Deep Reinforcement Learning

doi: 10.11999/JEIT230445

LI Zhihua^,,
YU Zili

School of Artificial Intelligence and Computer, Jiangnan University, Wuxi 214122, China

Funds: The Ministry of Industry and Information Technology Manufacturing Project (ZH-XZ-180004), The Fundamental Research Funds for the Central Universities (JUSRP211A41, JUSRP42003)

Received Date: 2023-05-18
Rev Recd Date: 2023-11-03

Available Online: 2023-11-14

Publish Date: 2024-04-24

Abstract

Abstract

In Mobile Edge Computing (MEC) intensive deployment scenarios, the uncertainty of edge server load can easily cause edge server overload, leading to a significant increase in delay and energy consumption during the computation offloading process. In response to this issue, a multi-user computation offloading optimization model and algorithm based on Deep Deterministic Policy Gradient (DDPG) is proposed. Firstly, considering the balance optimization of delay and energy consumption, a utility function is established to maximize system utility as the optimization objective, and the computational offloading problem is transformed into a mixed integer nonlinear programming problem. Then, in response to the problem of large state space and coexistence of discrete and continuous variables in the action space, the DDPG deep reinforcement learning algorithm is discretized and improved. Based on this, a multi-user computation offloading optimization method is proposed. Finally, this method is used to solve nonlinear programming problems. The simulation experimental results show that compared with existing algorithms, the proposed method can effectively reduce the probability of edge server overload and has good stability.
- Mobile Edge Computing(MEC),
- Computation offloading,
- Deep reinforcement learning,
- Resource allocation

FullText(HTML)

References(28)

References

[1]	ZHOU Zhi, CHEN Xu, LI En, et al. Edge intelligence: Paving the last mile of artificial intelligence with edge computing[J]. Proceedings of the IEEE, 2019, 107(8): 1738–1762. doi: 10.1109/JPROC.2019.2918951.
[2]	GERARDS M E T, HURINK J L, and KUPER J. On the interplay between global DVFS and scheduling tasks with precedence constraints[J]. IEEE Transactions on Computers, 2015, 64(6): 1742–1754. doi: 10.1109/TC.2014.2345410.
[3]	SADATDIYNOV K, CUI Laizhong, ZHANG Lei, et al. A review of optimization methods for computation offloading in edge computing networks[J]. Digital Communications and Networks, 2023, 9(2): 450–461. doi: 10.1016/j.dcan.2022.03.003.
[4]	SUN Jiannan, GU Qing, ZHENG Tao, et al. Joint optimization of computation offloading and task scheduling in vehicular edge computing networks[J]. IEEE Access, 2020, 8: 10466–10477. doi: 10.1109/ACCESS.2020.2965620.
[5]	LIU Hui, NIU Zhaocheng, DU Junzhao, et al. Genetic algorithm for delay efficient computation offloading in dispersed computing[J]. Ad Hoc Networks, 2023, 142: 103109. doi: 10.1016/j.adhoc.2023.103109.
[6]	ALAMEDDINE H A, SHARAFEDDINE S, SEBBAH S, et al. Dynamic task offloading and scheduling for low-latency IoT services in multi-access edge computing[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(3): 668–682. doi: 10.1109/JSAC.2019.2894306.
[7]	BI Suzhi, HUANG Liang, and ZHANG Y J A. Joint optimization of service caching placement and computation offloading in mobile edge computing systems[J]. IEEE Transactions on Wireless Communications, 2020, 19(7): 4947–4963. doi: 10.1109/TWC.2020.2988386.
[8]	YI Changyan, CAI Jun, and SU Zhou. A multi-user mobile computation offloading and transmission scheduling mechanism for delay-sensitive applications[J]. IEEE Transactions on Mobile Computing, 2020, 19(1): 29–43. doi: 10.1109/TMC.2019.2891736.
[9]	MITSIS G, TSIROPOULOU E E, and PAPAVASSILIOU S. Price and risk awareness for data offloading decision-making in edge computing systems[J]. IEEE Systems Journal, 2022, 16(4): 6546–6557. doi: 10.1109/JSYST.2022.3188997.
[10]	ZHANG Kaiyuan, GUI Xiaolin, REN Dewang, et al. Optimal pricing-based computation offloading and resource allocation for blockchain-enabled beyond 5G networks[J]. Computer Networks, 2022, 203: 108674. doi: 10.1016/j.comnet.2021.108674.
[11]	TONG Zhao, DENG Xin, MEI Jing, et al. Stackelberg game-based task offloading and pricing with computing capacity constraint in mobile edge computing[J]. Journal of Systems Architecture, 2023, 137: 102847. doi: 10.1016/j.sysarc.2023.102847.
[12]	张祥俊, 伍卫国, 张弛, 等. 面向移动边缘计算网络的高能效计算卸载算法[J]. 软件学报, 2023, 34(2): 849–867. doi: 10.13328/j.cnki.jos.006417. ZHANG Xiangjun, WU Weiguo, ZHANG Chi, et al. Energy-efficient computing offloading algorithm for mobile edge computing network[J]. Journal of Software, 2023, 34(2): 849–867. doi: 10.13328/j.cnki.jos.006417.
[13]	YAO Liang, XU Xiaolong, BILAL M, et al. Dynamic edge computation offloading for internet of vehicles with deep reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(11): 12991–12999. doi: 10.1109/TITS.2022.3178759.
[14]	SADIKI A, BENTAHAR J, DSSOULI R, et al. Deep reinforcement learning for the computation offloading in MIMO-based Edge Computing[J]. Ad Hoc Networks, 2023, 141: 103080. doi: 10.1016/j.adhoc.2022.103080.
[15]	TANG Ming and WONG V W S. Deep reinforcement learning for task offloading in mobile edge computing systems[J]. IEEE Transactions on Mobile Computing, 2022, 21(6): 1985–1997. doi: 10.1109/TMC.2020.3036871.
[16]	CHENG Nan, LYU Feng, QUAN Wei, et al. Space/aerial-assisted computing offloading for IoT applications: A learning-based approach[J]. IEEE Journal on Selected Areas in Communications, 2019, 37(5): 1117–1129. doi: 10.1109/JSAC.2019.2906789.
[17]	ZHOU Huan, JIANG Kai, LIU Xuxun, et al. Deep reinforcement learning for energy-efficient computation offloading in mobile-edge computing[J]. IEEE Internet of Things Journal, 2022, 9(2): 1517–1530. doi: 10.1109/JIOT.2021.3091142.
[18]	WANG Yunpeng, FANG Weiwei, DING Yi, et al. Computation offloading optimization for UAV-assisted mobile edge computing: A deep deterministic policy gradient approach[J]. Wireless Networks, 2021, 27(4): 2991–3006. doi: 10.1007/s11276-021-02632-z.
[19]	ALE L, ZHANG Ning, FANG Xiaojie, et al. Delay-aware and energy-efficient computation offloading in mobile-edge computing using deep reinforcement learning[J]. IEEE Transactions on Cognitive Communications and Networking, 2021, 7(3): 881–892. doi: 10.1109/TCCN.2021.3066619.
[20]	DAI Yueyue, XU Du, ZHANG Ke, et al. Deep reinforcement learning for edge computing and resource allocation in 5G beyond[C]. The IEEE 19th International Conference on Communication Technology, Xian, China, 2019: 866–870. doi: 10.1109/ICCT46805.2019.8947146.
[21]	3GPP. TR 36.814 v9.0. 0. Further advancements for E-UTRA physical layer aspects[S]. 2010.
[22]	WANG Yanting, SHENG Min, WANG Xijun, et al. Mobile-edge computing: Partial computation offloading using dynamic voltage scaling[J]. IEEE Transactions on Communications, 2016, 64(10): 4268–4282. doi: 10.1109/TCOMM.2016.2599530.
[23]	ZHANG Ke, MAO Yuming, LENG Supeng, et al. Energy-efficient offloading for mobile edge computing in 5G heterogeneous networks[J]. IEEE Access, 2016, 4: 5896–5907. doi: 10.1109/ACCESS.2016.2597169.
[24]	ZHANG Lianhong, ZHOU Wenqi, XIA Junjuan, et al. DQN-based mobile edge computing for smart Internet of vehicle[J]. EURASIP Journal on Advances in Signal Processing, 2022, 2022(1): 45. doi: 10.1186/s13634-022-00876-1.
[25]	WANG Jin, HU Jia, MIN Geyong, et al. Dependent task offloading for edge computing based on deep reinforcement learning[J]. IEEE Transactions on Computers, 2022, 71(10): 2449–2461. doi: 10.1109/TC.2021.3131040.
[26]	SUTTON R S and BARTO A G. Reinforcement Learning: An Introduction[M]. 2nd ed. Cambridge: A Bradford Book, 2018: 47–50.
[27]	LIU Y C and HUANG Chiyu. DDPG-based adaptive robust tracking control for aerial manipulators with decoupling approach[J]. IEEE Transactions on Cybernetics, 2022, 52(8): 8258–8271. doi: 10.1109/TCYB.2021.3049555.
[28]	HU Shihong and LI Guanghui. Dynamic request scheduling optimization in mobile edge computing for IoT applications[J]. IEEE Internet of Things Journal, 2020, 7(2): 1426–1437. doi: 10.1109/JIOT.2019.2955311.