Virtual Network Function Migration Optimization Algorithm Based on Deep Deterministic Policy Gradient
-
摘要:
针对NFV/SDN架构下，服务功能链(SFC)的资源需求动态变化引起的虚拟网络功能(VNF)迁移优化问题，该文提出一种基于深度强化学习的VNF迁移优化算法。首先，在底层CPU、带宽资源和SFC端到端时延约束下，建立基于马尔可夫决策过程(MDP)的随机优化模型，该模型通过迁移VNF来联合优化网络能耗和SFC端到端时延。其次，由于状态空间和动作空间是连续值集合，提出一种基于深度确定性策略梯度(DDPG)的VNF智能迁移算法，从而得到近似最优的VNF迁移策略。仿真结果表明，该算法可以实现网络能耗和SFC端到端时延的折中，并提高物理网络的资源利用率。
Abstract:To solve the problem of Virtual Network Function (VNF) migration optimization, which is caused by the dynamic change of resource requirements of Service Function Chain (SFC) under Network Function Virtualization/ Software Defined Network (NFV/SDN) architecture, a VNF migration optimization algorithm is proposed based on deep reinforcement learning. Firstly, based on the underlying CPU, bandwidth resources and SFC end-to-end delay constraints, a Markov Decision Process (MDP) based stochastic optimization model is established. This model is used to optimize jointly network energy consumption and SFC end-to-end delay by migrating VNF. Secondly, since the state space and action space of this paper are continuous value sets, a VNF intelligent migration algorithm based on Deep Deterministic Policy Gradient (DDPG) is proposed to obtain an approximate optimal VNF migration strategy. The simulation results show that the algorithm can achieve the compromise between network energy consumption and SFC end-to-end delay, and improve the resource utilization of the physical network.
-
表 1 基于DDPG的迁移策略训练算法
输入：DDPG参数：回合数$M$，训练次数$T$，训练样本长度$L$，评判家网络学习率${l_c}$，行动者学习率${l_a}$，折扣因子$\mu $，软更新因子$\tau $，经验
回放池大小$B$，最小样本长度$N$，高斯噪声$n$输出：策略${\pi} $ 1. 初始化经验回放池$H$ 2. 随机初始化行动者网络参数$({\theta ^{\pi} },{\theta ^{{{\pi} '}}})$和评判家网络参数$({\theta ^Q},{\theta ^{{Q'}}})$ 3. for ${\rm{episode}} = 1,2, ··· ,M$ do 4. 初始化环境$s(0)$ 5. for $t = 1,2, ··· ,T$ do 6. 根据当前策略得到动作，增添随机噪声进行探索：$a(t) = {\pi} (s(t)|{\theta ^{\pi} }) + n$ 7. if C1～C10约束满足 then 8. 采取动作$a(t)$，得到状态$s(t + 1)$，根据式(11)得到$r(t)$ 9. if 经验回放池$H$没有溢出 then 10. 将$(s(t),a(t),r(t),s(t + 1))$存储到经验池中 11. else 12. 用$(s(t),a(t),r(t),s(t + 1))$随机替代存入经验池的集合 13. 随机选择$N$个集合构成样本：$(s(i),a(i),r(i),s(i + 1)),\forall i = 1,2, ··· ,N$ 14. 通过目标评判家网络得到$Q(s(i + 1),a(i + 1)|{\theta ^{{Q'}}})$，然后根据式(15)得到损失函数$L({\theta ^Q})$ 15. 通过式(16)来更新估计评判家网络参数 16. 从估计评判家网络得到${Q_{{\theta ^Q}}}(s(i),a(i))$，通过式(19)得到策略梯度${\nabla _{{\theta ^{\pi} }}}J({{\pi} _{{\theta ^{\pi} }}})$ 17. 通过式(20)更新估计行动者网络的参数 18. 通过式(22)软更新目标评判家网络和目标行动者网络的参数 19. end if 20. end if 21. end for 22. end for 
下载: 导出CSV
表 仿真参数
仿真参数 值 仿真参数 值 通用服务器数量 12 服务器CPU容量(MIPS) Uniform[250, 300] VNF集合长度(个) Uniform[2, 5] 链路带宽容量(Mbps) Uniform[100, 200] ${r_b}$ 0.2 服务器CPU总能耗(W) Uniform[170, 230] ${r_c}$ 0.2 服务器待机时的能耗(W) Uniform[90, 120] SFC最长时延限制(ms) 30 服务器状态切换能耗(W) Uniform[25, 35] 软更新因子 0.01 虚拟链路带宽资源(Mbps) Uniform[5, 10] 折扣因子 0.99 VNF CPU资源需求(MIPS) Uniform[10, 20]
下载: 导出CSV
-
唐伦, 周钰, 杨友超, 等. 5G网络切片场景中基于预测的虚拟网络功能动态部署算法[J]. 电子与信息学报, 2019, 41(9): 2071–2078. doi: 10.11999/JEIT180894TANG Lun, ZHOU Yu, YANG Youchao, et al. Virtual network function dynamic deployment algorithm based on prediction for 5g network slicing[J]. Journal of Electronics &Information Technology, 2019, 41(9): 2071–2078. doi: 10.11999/JEIT180894 唐伦, 杨恒, 马润琳, 等. 基于5G接入网络的多优先级虚拟网络功能迁移开销与网络能耗联合优化算法[J]. 电子与信息学报, 2019, 41(9): 2079–2086. doi: 10.11999/JEIT180906TANG Lun, YANG Heng, MA Runlin, et al. Multi-priority based joint optimization algorithm of virtual network function migration cost and network energy consumption[J]. Journal of Electronics &Information Technology, 2019, 41(9): 2079–2086. doi: 10.11999/JEIT180906 程国振. 基于元能力的网络功能组合关键技术研究[D]. [博士论文], 解放军信息工程大学, 2015.CHENG Guozhen. Research on the key technologies of function composition based on atomic capability[D]. [Ph. D. dissertation], Information Engineering University, 2015. CHO D, TAHERI J, ZOMAYA A Y, et al. Real-time virtual network function (VNF) migration toward low network latency in cloud environments[C]. The 10th IEEE International Conference on Cloud Computing (CLOUD), Honolulu, USA, 2017: 798–801. doi: 10.1109/CLOUD.2017.118. XIA Jing, CAI Zhiping, and XU Ming. Optimized virtual network functions migration for NFV[C]. The 22nd IEEE International Conference on Parallel and Distributed Systems (ICPADS), Wuhan, China, 2016: 340–246. doi: 10.1109/ICPADS.2016.0053. GHARBAOUI M, CONTOLI C, DAVOLI G, et al. Demonstration of latency-aware and self-adaptive service chaining in 5G/SDN/NFV infrastructures[C]. 2018 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), Verona, Italy, 2018: 1–2. doi: 10.1109/NFV-SDN.2018.8725645. 唐伦, 赵培培, 赵国繁, 等. 基于QoS保障的服务功能链动态部署算法[J]. 北京邮电大学学报, 2018, 41(6): 90–96. doi: 10.13190/j.jbupt.2018-013TANG Lun, ZHAO Peipei, ZHAO Guofan, et al. Dynamic deployment algorithm for service function chaining with QoS guarantee[J]. Journal of Beijing University of Posts and Telecommunications, 2018, 41(6): 90–96. doi: 10.13190/j.jbupt.2018-013 ERAMO V, AMMAR M, and LAVACCA F G. Migration energy aware reconfigurations of virtual network function instances in NFV architectures[J]. IEEE Access, 2017, 5: 4927–4938. doi: 10.1109/ACCESS.2017.2685437 LI Han, GAO Hui, LÜ Tiejun, et al. Deep q-learning based dynamic resource allocation for self-powered ultra-dense networks[C]. 2018 IEEE International Conference on Communications Workshops (ICC Workshops), Kansas City, USA, 2018: 1–6. doi: 10.1109/ICCW.2018.8403505. YE Junhong and ZHANG Y J A. DRAG: Deep reinforcement learning based base station activation in heterogeneous networks[J]. IEEE Transactions on Mobile Computing, 2020, 19(9): 2076–2087. doi: 10.1109/TMC.2019.2922602 CHU Man, LIAO Xuewen, LI Hang, et al. Power control in energy harvesting multiple access system with reinforcement learning[J]. IEEE Internet of Things Journal, 2019, 6(5): 9175–9186. doi: 10.1109/JIOT.2019.2928837 LI Han, LÜ Tiejun, and ZHANG Xuewei. Deep deterministic policy gradient based dynamic power control for self-powered ultra-dense networks[C]. 2018 IEEE Globecom Workshops (GC Wkshps), Abu Dhabi, United Arab Emirates, 2018: 1–6. doi: 10.1109/GLOCOMW.2018.8644157. ERAMO V, MIUCCI E, AMMAR M, et al. An approach for service function chain routing and virtual function network instance migration in network function virtualization architectures[J]. IEEE/ACM Transactions on Networking, 2017, 25(4): 2008–2025. doi: 10.1109/TNET.2017.2668470 -
计量
- 文章访问数: 1274
- HTML全文浏览量: 566
- PDF下载量: 131
- 被引次数: 0
下载:
