高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于Q-Learning算法的毫微微小区功率控制算法

李云 唐英 刘涵霄

李云, 唐英, 刘涵霄. 基于Q-Learning算法的毫微微小区功率控制算法[J]. 电子与信息学报, 2019, 41(11): 2557-2564. doi: 10.11999/JEIT181191
引用本文: 李云, 唐英, 刘涵霄. 基于Q-Learning算法的毫微微小区功率控制算法[J]. 电子与信息学报, 2019, 41(11): 2557-2564. doi: 10.11999/JEIT181191
Yun LI, Ying TANG, Hanxiao LIU. Power Control Algorithm Based on Q-Learning in Femtocell[J]. Journal of Electronics & Information Technology, 2019, 41(11): 2557-2564. doi: 10.11999/JEIT181191
Citation: Yun LI, Ying TANG, Hanxiao LIU. Power Control Algorithm Based on Q-Learning in Femtocell[J]. Journal of Electronics & Information Technology, 2019, 41(11): 2557-2564. doi: 10.11999/JEIT181191

基于Q-Learning算法的毫微微小区功率控制算法

doi: 10.11999/JEIT181191
基金项目: 国家自然科学基金(61671096),重庆市研究生科研创新项目(CYS17220),重庆市“科技创新领军人才支持计划”(CSTCCXLJRC201710),重庆市基础科学与前沿技术研究项目(cstc2017jcyjBX0005),重庆市留学人员创业创新支持计划
详细信息
    作者简介:

    李云:男,1974年生,教授,博士生导师,主要研究领域为无线移动通信

    唐英:女,1993年生,硕士生,研究方向为异构蜂窝无线网络

    刘涵霄:男,1994年生,硕士生,研究方向为异构蜂窝无线网络

    通讯作者:

    唐英 17749963914@163.com

  • 中图分类号: TN92

Power Control Algorithm Based on Q-Learning in Femtocell

Funds: The National Natural Science Foundation of China (61671096), The Chongqing Research and Innovation Program of Graduated Students (CYS17220), The Chongqing Science and Technology Innovation Leadership Talent Support Program (CSTCCXLJRC201710), The Chongqing Research Program of Basic Science and Frontier Technology (cstc2017jcyjBX0005), The Chongqing Overseas Students Entrepreneurship and Innovation Support Plan
  • 摘要: 该文研究macro-femto异构蜂窝网络中移动用户的功率控制问题,首先建立了以最小接收信号信干噪比为约束条件,最大化毫微微小区的总能效为目标的优化模型;然后提出了基于Q-Learning算法的毫微微小区集中式功率控制(PCQL)算法,该算法基于强化学习,能在没有准确信道状态信息的情况下,实现对小区内所有用户终端的发射功率统一调整。仿真结果表明该算法能实现对用户终端的功率有效控制,提升系统能效。
  • 图  1  异构蜂窝网络模型

    图  2  代理自主学习过程

    图  3  小区用户数为4时,系统能效对比

    图  4  小区用户数为4时,系统吞吐量对比

    图  5  系统能效与用户数的关系

    图  6  系统吞吐量与用户数的关系

    图  7  信道状态信息存在估计误差时,系统能效与用户数的关系

    图  8  信道状态信息存在估计误差时,系统吞吐量与用户数的关系

    图  9  能效优化的算法运行时间对比

    图  10  吞吐量优化的算法运行时间对比

    表  1  基于Q-Learning算法的毫微微小区功率控制算法(PCQL)

     输入:W, ${n_0}$, $P_{b,\mu }^{\rm{c}} $, ${\rm{SINR}}_{b,\mu }^{\min }$, $p_{b,\mu }^{{\rm{max}}}$, $\gamma $, $\alpha $, $T\;$, $\varepsilon $,动作空间${A_b}$;
     输出:${{\text{π}}^ * }$, $p_{b,\mu }^*$($\mu \in {U_b}$);
     定义:${\text{k}}$表示代理选取的动作;${\rm{SINR}}_{b,\mu }^{{\rm{real}}}$表示${u_{b,\mu }}$与基站$b$通信时 的实际信干噪比;
     $Q\left( {{{\text{s}}_b},{{\text{a}}_b}} \right) = 0$, ${\text{π}}\left( {{{\text{s}}_b},{{\text{a}}_b}} \right) = \frac{1}{{\left| {{A_b}\left( {{{\text{s}}_b}} \right)} \right|}}$, $\text{s}_b^t = \text{s}_b^0$;
     for $t = 0,1, ·\!·\!· ,T\;$ do
     若rand()<$\varepsilon $,从${A_b}$中随机选动作${\text{k}}$;否则${\text{k}} \!=\! \mathop {\arg \max }\limits_{{\text{a}}_b^t} \!Q\left( {{\text{s}}_b^t,{\text{a}}_b^t} \right)$;
     根据式(1)确定${\rm{SINR}}_{b,\mu }^{{\rm{real}}}$;
     for $\mu = 1,2, ·\!·\!· ,{N_b}$ do
     若${\rm{SINR}}_{b,\mu }^{{\mathop{\rm real}\nolimits} } \ge {\rm{SINR}}_{b,\mu }^{\min }$,那么${\lambda _{b,\mu }} = 1$;否则${\lambda _{b,\mu }} = 0$;
     end for;
     根据式(7)计算采取动作${\text{a}}_b^t = {\text{k}}$所带来的奖赏值${\Re _b}\left( {{\text{s}}_b^t,{\text{a}}_b^t} \right)$;
     ${\text{a}}_b^{t + 1} = {\text{π}}\left( {{\text{s}}_b^{t + 1}} \right)$;
     ${\rm Q}\left( { { {\text{s} } }_b^t,{ {\text{a} } }_b^t} \right) \leftarrow {\rm Q}\left( { { {\text{s} } }_b^t,{ {\text{a} } }_b^t} \right) + \alpha ( {\Re _b}\left( { { {\text{s} } }_b^t,{ {\text{a} } }_b^t} \right) \!+\! \gamma \mathop {\max}\limits_{ {\rm{a} }_b^{t + 1} } \left( { {\rm Q}\left( { { {\text{s} } }_b^{t + 1},{ {\text{a} } }_b^{t + 1} } \right)} \right)$  $\left.- {{\rm Q}\left( {{{\text{s}}}_b^t,{{\text{a}}}_b^t} \right)} \right)$;
     ${\text{s}}_b^t \leftarrow {\text{s}}_b^{t + 1}$;
     end for;
     ${{\text{π}}^ * }\left( {{{\text{s}}_b}} \right) = \mathop {\arg \max }\limits_{{{\text{a}}_b}} Q\left( {{{\text{s}}_b},{{\text{a}}_b}} \right),\forall {{\text{s}}_b} \in S$.
    下载: 导出CSV

    表  2  主要的仿真参数

    参数名称参数值
    MBS/FBS1个/4个
    MUE/FUE最大的发射功率37 dBm/30 dBm
    MBS/FBS覆盖范围半径250 m/50 m
    ${{\rm{SINR}} _{b,\mu }}^{\min }$–9 dB
    固定的电路功耗100 mW
    信道带宽10 MHz
    高斯白噪声的功率谱密度${10^{ - 11}}$ W/Hz
    下载: 导出CSV
  • LÓPEZ-PÉREZ D, DING M, CLAUSSEN H, et al. Towards 1 Gbps/UE in cellular systems: understanding ultra-dense small cell deployments[J]. IEEE Communications Surveys & Tutorials, 2015, 17(4): 2078–2101. doi: 10.1109/COMST.2015.2439636
    YUNAS S F, VALKAMA M, and NIEMELÄ J. Spectral and energy efficiency of ultra-dense networks under different deployment strategies[J]. IEEE Communications Magazine, 2015, 53(1): 90–100. doi: 10.1109/MCOM.2015.7010521
    MARTOLIA D, SATHYA V, RANGISETTI A K, et al. Enhancing performance of victim macro users via joint ABSF and dynamic power control in LTE HetNets[C]. The Twenty-third National Conference on Communications, Chennai, India, 2017: 1–6.
    SHIN D and CHOI S. Dynamic power control for balanced data traffic with coverage in femtocell networks[C]. The 8th International Wireless Communications and Mobile Computing Conference, Limassol, Cyprus, 2012: 648–653.
    ZHANG Jinzhu, HONG Peilin, XUE Kaiping, et al. A novel power control scheme for femtocell in heterogeneous networks[C]. 2012 IEEE Consumer Communications and Networking Conference, Las Vegas, USA, 2012: 802–806.
    PAN Zhenni, MEGUMI, SAITOU, et al. Neuron control-based power adjustment scheme for sleep two-tier cellular networks[C]. 2014 IEEE Wireless Communications and Networking Conference, Istanbul, Turkey, 2014: 3201–3206.
    ZHOU Tianqing, LIU Zunxiong, ZHAO Junhui, et al. Joint user association and power control for load balancing in downlink heterogeneous cellular networks[J]. IEEE Transactions on Vehicular Technology, 2018, 67(3): 2582–2593. doi: 10.1109/TVT.2017.2768574
    MARTIN-VEGA F J, GOMEZ G, AGUAYO-TORRES M C, et al. Analytical modeling of interference aware power control for the uplink of heterogeneous cellular networks[J]. IEEE Transactions on Wireless Communications, 2016, 15(10): 6742–6757. doi: 10.1109/TWC.2016.2588469
    ZHANG Jing, LIAO Yan, and XIN Yili. Uplink power control for heterogeneous small cell networks[C]. 2016 IEEE 83rd Vehicular Technology Conference, Nanjing, China, 2016: 1–5.
    WANG Min, GAO Hui, and LV Tiejun. Energy-efficient user association and power control in the heterogeneous network[J]. IEEE Access, 2017, 5: 5059–5068. doi: 10.1109/ACCESS.2017.2690305
    ZHANG Jing, XIANG Lin, NG D W K, et al. Energy efficiency evaluation of multi-tier cellular uplink transmission under maximum power constraint[J]. IEEE Transactions on Wireless Communications, 2017, 16(11): 7092–7107. doi: 10.1109/TWC.2017.2739142
    PAN Zhenni and SHIMAMOTO S. Cell sizing based energy optimization in joint macro-femto deployments via sleep activation[C]. 2013 IEEE Wireless Communications and Networking Conference, Shanghai, China, 2013: 4765–4770.
    SHIFAT A S M Z, CHOWDHURY M Z, and JANG Y M. Game-based approach for QoS provisioning and interference management in heterogeneous networks[J]. IEEE Access, 2018, 6: 10208–10220. doi: 10.1109/ACCESS.2017.2704094
    MISHRA S and MURTHY C S R. Increasing energy efficiency via transmit power spreading in dense femto cell networks[J]. IEEE Systems Journal, 2018, 12(1): 971–980. doi: 10.1109/JSYST.2016.2573845
    GURUACHARYA S, NIYATO D, KIM D I, et al. Hierarchical competition for downlink power allocation in OFDMA femtocell networks[J]. IEEE Transactions on Wireless Communications, 2013, 12(4): 1543–1553. doi: 10.1109/TWC.2013.022213.120016
    WANG Haining, WANG Jiaheng, and DING Zhi. Distributed power control in a two-tier heterogeneous network[J]. IEEE Transactions on Wireless Communications, 2015, 14(12): 6509–6523. doi: 10.1109/TWC.2015.2456055
    MAO Tingli, FENG Gang, LIANG Liang, et al. Energy-efficient power control for macro-femto networks[C]. The 22nd Wireless and Optical Communication Conference, Chongqing, China, 2013: 122–125.
    MAO Tingli, FENG Gang, LIANG Liang, et al. Distributed energy-efficient power control for macro-femto networks[J]. IEEE Transactions on Vehicular Technology, 2016, 65(2): 718–731. doi: 10.1109/TVT.2015.2402618
    LAI Weisheng, CHANG T H, and LEE T S. Joint power and admission control for spectral and energy efficiency maximization in heterogeneous OFDMA networks[J]. IEEE Transactions on Wireless Communications, 2016, 15(5): 3531–3547. doi: 10.1109/TWC.2016.2522958
    LOODARICHEH R A, MALLICK S, BHARGAVA V K. Energy-efficient resource allocation for OFDMA cellular networks with user cooperation and QoS provisioning[J]. IEEE Transactions on Wireless Communications, 2014, 13(11): 6132–6146. doi: 10.1109/TWC.2014.2329877
    GHADIMI E, CALABRESE F D, PETERS G, et al. A reinforcement learning approach to power control and rate adaptation in cellular networks[C]. 2017 IEEE International Conference on Communications, Paris, France, 2017: 1–7.
    周志华. 机器学习[M]. 北京: 清华大学出版社, 2016: 372–390.

    ZHOU Zhihua. Machine Learning[M]. Beijing: Tsinghua University Press, 2016: 372–390.
  • 加载中
图(10) / 表(2)
计量
  • 文章访问数:  3321
  • HTML全文浏览量:  1462
  • PDF下载量:  105
  • 被引次数: 0
出版历程
  • 收稿日期:  2018-12-28
  • 修回日期:  2019-04-10
  • 网络出版日期:  2019-05-21
  • 刊出日期:  2019-11-01

目录

    /

    返回文章
    返回