基于深度确定性策略梯度的虚拟网络功能迁移优化算法

唐伦; 贺兰钦; 谭颀; 陈前斌

doi:10.11999/JEIT190921

基于深度确定性策略梯度的虚拟网络功能迁移优化算法

doi: 10.11999/JEIT190921

1.
重庆邮电大学通信与信息工程学院重庆 400065
2.
重庆邮电大学移动通信技术重点实验室重庆 400065

基金项目: 国家自然科学基金(62071078)，重庆市教委科学技术研究项目(KJZD-M20180601)，重庆市重大主题专项项目 (cstc2019jscx-zdztzxX0006)

详细信息

作者简介:
唐伦：男，1973年生，教授，博士，主要研究方向为下一代无线通信网络、异构蜂窝网络、软件定义无线网络等

贺兰钦：男，1995年生，硕士生，研究方向为5G网络切片，机器学习算法

谭颀：女，1995年生，硕士生，研究方向为5G网络切片、资源分配、随机优化理论

陈前斌：男，1967年生，教授，博士生导师，主要研究方向为个人通信、多媒体信息处理与传输、异构蜂窝网络等

通讯作者:
贺兰钦　719097886@qq.com

中图分类号: TN929.5
计量
- 文章访问数: 1137
- HTML全文浏览量: 460
- PDF下载量: 119
- 被引次数: 0
出版历程
- 收稿日期: 2019-11-15
- 修回日期: 2020-11-02
- 网络出版日期: 2020-12-09
- 刊出日期: 2021-02-23

Virtual Network Function Migration Optimization Algorithm Based on Deep Deterministic Policy Gradient

1.
School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.
Key Laboratory of Mobile Communications Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Funds: The National Natural Science Foundation of China (62071078), The Science and Technology Research Program of Chongqing Municipal Education Commission (KJZD-M20180601), The Major Theme Special Projects of Chongqing (cstc2019jscx-zdztzxX0006)

摘要

摘要:
针对NFV/SDN架构下，服务功能链(SFC)的资源需求动态变化引起的虚拟网络功能(VNF)迁移优化问题，该文提出一种基于深度强化学习的VNF迁移优化算法。首先，在底层CPU、带宽资源和SFC端到端时延约束下，建立基于马尔可夫决策过程(MDP)的随机优化模型，该模型通过迁移VNF来联合优化网络能耗和SFC端到端时延。其次，由于状态空间和动作空间是连续值集合，提出一种基于深度确定性策略梯度(DDPG)的VNF智能迁移算法，从而得到近似最优的VNF迁移策略。仿真结果表明，该算法可以实现网络能耗和SFC端到端时延的折中，并提高物理网络的资源利用率。
- 虚拟网络功能 /
- 深度强化学习 /
- SFC端到端时延 /
- 网络能耗
Abstract:
To solve the problem of Virtual Network Function (VNF) migration optimization, which is caused by the dynamic change of resource requirements of Service Function Chain (SFC) under Network Function Virtualization/ Software Defined Network (NFV/SDN) architecture, a VNF migration optimization algorithm is proposed based on deep reinforcement learning. Firstly, based on the underlying CPU, bandwidth resources and SFC end-to-end delay constraints, a Markov Decision Process (MDP) based stochastic optimization model is established. This model is used to optimize jointly network energy consumption and SFC end-to-end delay by migrating VNF. Secondly, since the state space and action space of this paper are continuous value sets, a VNF intelligent migration algorithm based on Deep Deterministic Policy Gradient (DDPG) is proposed to obtain an approximate optimal VNF migration strategy. The simulation results show that the algorithm can achieve the compromise between network energy consumption and SFC end-to-end delay, and improve the resource utilization of the physical network.
- Virtual Network Function (VNF) /
- Deep reinforcement learning /
- SFC end-to-end delay /
- Network energy consumption

HTML全文

图 1 系统模型

下载: 全尺寸图片幻灯片

图 2 不同评判家学习率下的损失函数值

下载: 全尺寸图片幻灯片

图 3 不同评判家学习率下的累积奖励值

下载: 全尺寸图片幻灯片

图 4 权重对系统总时延的影响

下载: 全尺寸图片幻灯片

图 5 权重对系统总能耗的影响

下载: 全尺寸图片幻灯片

图 6 系统总时延对比

下载: 全尺寸图片幻灯片

图 7 网络能耗对比

下载: 全尺寸图片幻灯片

图 8 计算资源利用率

下载: 全尺寸图片幻灯片

表 1 基于DDPG的迁移策略训练算法

输入：DDPG参数：回合数$M$，训练次数$T$，训练样本长度$L$，评判家网络学习率${l_c}$，行动者学习率${l_a}$，折扣因子$\mu $，软更新因子$\tau $，经验　回放池大小$B$，最小样本长度$N$，高斯噪声$n$
输出：策略${\pi} $
1. 　初始化经验回放池$H$
2. 　随机初始化行动者网络参数$({\theta ^{\pi} },{\theta ^{{{\pi} '}}})$和评判家网络参数$({\theta ^Q},{\theta ^{{Q'}}})$
3. 　for ${\rm{episode}} = 1,2, ··· ,M$ do
4. 　　初始化环境$s(0)$
5. 　　for $t = 1,2, ··· ,T$ do
6. 　　　根据当前策略得到动作，增添随机噪声进行探索：$a(t) = {\pi} (s(t)\|{\theta ^{\pi} }) + n$
7. 　　　if C1～C10约束满足 then
8. 　　　　采取动作$a(t)$，得到状态$s(t + 1)$，根据式(11)得到$r(t)$
9. 　　　　if 经验回放池$H$没有溢出 then
10. 　　　　　将$(s(t),a(t),r(t),s(t + 1))$存储到经验池中
11. 　　　　else
12. 　　　　　用$(s(t),a(t),r(t),s(t + 1))$随机替代存入经验池的集合
13. 　　　　　随机选择$N$个集合构成样本：$(s(i),a(i),r(i),s(i + 1)),\forall i = 1,2, ··· ,N$
14. 　　　　　通过目标评判家网络得到$Q(s(i + 1),a(i + 1)\|{\theta ^{{Q'}}})$，然后根据式(15)得到损失函数$L({\theta ^Q})$
15. 　　　　　通过式(16)来更新估计评判家网络参数
16. 　　　　　从估计评判家网络得到${Q_{{\theta ^Q}}}(s(i),a(i))$，通过式(19)得到策略梯度${\nabla _{{\theta ^{\pi} }}}J({{\pi} _{{\theta ^{\pi} }}})$
17. 　　　　　通过式(20)更新估计行动者网络的参数
18. 　　　　　通过式(22)软更新目标评判家网络和目标行动者网络的参数
19. 　　　　end if
20. 　　　end if
21. 　　end for
22. 　end for

下载: 导出CSV

表仿真参数

仿真参数	值	仿真参数	值
通用服务器数量	12	服务器CPU容量(MIPS)	Uniform[250, 300]
VNF集合长度(个)	Uniform[2, 5]	链路带宽容量(Mbps)	Uniform[100, 200]
${r_b}$	0.2	服务器CPU总能耗(W)	Uniform[170, 230]
${r_c}$	0.2	服务器待机时的能耗(W)	Uniform[90, 120]
SFC最长时延限制(ms)	30	服务器状态切换能耗(W)	Uniform[25, 35]
软更新因子	0.01	虚拟链路带宽资源(Mbps)	Uniform[5, 10]
折扣因子	0.99	VNF CPU资源需求(MIPS)	Uniform[10, 20]

下载: 导出CSV

参考文献(13)

唐伦, 周钰, 杨友超, 等. 5G网络切片场景中基于预测的虚拟网络功能动态部署算法[J]. 电子与信息学报, 2019, 41(9): 2071–2078. doi: 10.11999/JEIT180894

TANG Lun, ZHOU Yu, YANG Youchao, et al. Virtual network function dynamic deployment algorithm based on prediction for 5g network slicing[J]. Journal of Electronics &Information Technology, 2019, 41(9): 2071–2078. doi: 10.11999/JEIT180894

唐伦, 杨恒, 马润琳, 等. 基于5G接入网络的多优先级虚拟网络功能迁移开销与网络能耗联合优化算法[J]. 电子与信息学报, 2019, 41(9): 2079–2086. doi: 10.11999/JEIT180906

TANG Lun, YANG Heng, MA Runlin, et al. Multi-priority based joint optimization algorithm of virtual network function migration cost and network energy consumption[J]. Journal of Electronics &Information Technology, 2019, 41(9): 2079–2086. doi: 10.11999/JEIT180906

程国振. 基于元能力的网络功能组合关键技术研究[D]. [博士论文], 解放军信息工程大学, 2015.

CHENG Guozhen. Research on the key technologies of function composition based on atomic capability[D]. [Ph. D. dissertation], Information Engineering University, 2015.

CHO D, TAHERI J, ZOMAYA A Y, et al. Real-time virtual network function (VNF) migration toward low network latency in cloud environments[C]. The 10th IEEE International Conference on Cloud Computing (CLOUD), Honolulu, USA, 2017: 798–801. doi: 10.1109/CLOUD.2017.118.

XIA Jing, CAI Zhiping, and XU Ming. Optimized virtual network functions migration for NFV[C]. The 22nd IEEE International Conference on Parallel and Distributed Systems (ICPADS), Wuhan, China, 2016: 340–246. doi: 10.1109/ICPADS.2016.0053.

GHARBAOUI M, CONTOLI C, DAVOLI G, et al. Demonstration of latency-aware and self-adaptive service chaining in 5G/SDN/NFV infrastructures[C]. 2018 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN), Verona, Italy, 2018: 1–2. doi: 10.1109/NFV-SDN.2018.8725645.

唐伦, 赵培培, 赵国繁, 等. 基于QoS保障的服务功能链动态部署算法[J]. 北京邮电大学学报, 2018, 41(6): 90–96. doi: 10.13190/j.jbupt.2018-013

TANG Lun, ZHAO Peipei, ZHAO Guofan, et al. Dynamic deployment algorithm for service function chaining with QoS guarantee[J]. Journal of Beijing University of Posts and Telecommunications, 2018, 41(6): 90–96. doi: 10.13190/j.jbupt.2018-013

ERAMO V, AMMAR M, and LAVACCA F G. Migration energy aware reconfigurations of virtual network function instances in NFV architectures[J]. IEEE Access, 2017, 5: 4927–4938. doi: 10.1109/ACCESS.2017.2685437

LI Han, GAO Hui, LÜ Tiejun, et al. Deep q-learning based dynamic resource allocation for self-powered ultra-dense networks[C]. 2018 IEEE International Conference on Communications Workshops (ICC Workshops), Kansas City, USA, 2018: 1–6. doi: 10.1109/ICCW.2018.8403505.

YE Junhong and ZHANG Y J A. DRAG: Deep reinforcement learning based base station activation in heterogeneous networks[J]. IEEE Transactions on Mobile Computing, 2020, 19(9): 2076–2087. doi: 10.1109/TMC.2019.2922602

CHU Man, LIAO Xuewen, LI Hang, et al. Power control in energy harvesting multiple access system with reinforcement learning[J]. IEEE Internet of Things Journal, 2019, 6(5): 9175–9186. doi: 10.1109/JIOT.2019.2928837

LI Han, LÜ Tiejun, and ZHANG Xuewei. Deep deterministic policy gradient based dynamic power control for self-powered ultra-dense networks[C]. 2018 IEEE Globecom Workshops (GC Wkshps), Abu Dhabi, United Arab Emirates, 2018: 1–6. doi: 10.1109/GLOCOMW.2018.8644157.

ERAMO V, MIUCCI E, AMMAR M, et al. An approach for service function chain routing and virtual function network instance migration in network function virtualization architectures[J]. IEEE/ACM Transactions on Networking, 2017, 25(4): 2008–2025. doi: 10.1109/TNET.2017.2668470

施引文献

资源附件(0)

访问统计