Advanced Search
Volume 44 Issue 2
Feb.  2022
Turn off MathJax
Article Contents
KUANG Junqian, ZHAO Chang, YANG Liu, WANG Haifeng, QIAN Hua. An Outlier Cleaning Algorithm Based on Deep Learning[J]. Journal of Electronics & Information Technology, 2022, 44(2): 507-513. doi: 10.11999/JEIT201097
Citation: KUANG Junqian, ZHAO Chang, YANG Liu, WANG Haifeng, QIAN Hua. An Outlier Cleaning Algorithm Based on Deep Learning[J]. Journal of Electronics & Information Technology, 2022, 44(2): 507-513. doi: 10.11999/JEIT201097

An Outlier Cleaning Algorithm Based on Deep Learning

doi: 10.11999/JEIT201097
Funds:  The National Natural Science Foundation of China (61971286) , The National Key Research and Development Program of China (2020YFB2205603), The Science and Technology Commission Foundation of Shanghai (19DZ1204300)
  • Received Date: 2020-12-30
  • Rev Recd Date: 2021-07-21
  • Available Online: 2021-11-09
  • Publish Date: 2022-02-25
  • The use of appropriate abnormal data cleaning algorithms in the Internet of Things (IoT) can greatly improve data quality. Statistical methods or clustering methods are utilized to clean anomalies in Spatio-temporal data. However, these methods require additional prior knowledge, which will incur additional computational overhead for the sink node. In this paper, in line with the low-rank sparse matrix decomposition model, a fast anomaly cleaning algorithm based on a deep neural network is proposed to solve the Spatio-temporal data cleaning problem in IoT. Both the Spatio-temporal correlation of sensing data and the abnormal values' sparsity are considered in an optimization problem. The Iterative Shrinkage-Thresholding Algorithm (ISTA) is used to solve it. Then the ISTA is unfolded into a fixed-length deep neural network. The real-world dataset’s experimental results show that the proposed method can automatically update the thresholds faster and more accurately than the traditional ISTA.
  • loading
  • [1]
    蒋俊正, 杨杰, 欧阳缮. 一种新的无线传感器网络中异常节点检测定位算法[J]. 电子与信息学报, 2018, 40(10): 2358–2364. doi: 10.11999/JEIT171207

    JIANG Junzheng, YANG Jie, and OUYANG Shan. Novel method for outlier nodes detection and localization in wireless sensor networks[J]. Journal of Electronics &Information Technology, 2018, 40(10): 2358–2364. doi: 10.11999/JEIT171207
    [2]
    郭志懋, 周傲英. 数据质量和数据清洗研究综述[J]. 软件学报, 2002, 13(11): 2076–2082.

    GUO Zhimao and ZHOU Aoying. Research on data quality and data cleaning: A survey[J]. Journal of Software, 2002, 13(11): 2076–2082.
    [3]
    YU Tianqi, WANG Xianbin, and SHAMI A. Recursive principal component analysis-based data outlier detection and sensor data aggregation in IoT systems[J]. IEEE Internet of Things Journal, 2017, 4(6): 2207–2216. doi: 10.1109/JIOT.2017.2756025
    [4]
    KUMAR V and KHOSLA C. Data cleaning-a thorough analysis and survey on unstructured data[C]. The 8th International Conference on Cloud Computing, Data Science & Engineering, Noida, India, 2018: 305–309.
    [5]
    DIAO Yinglong, LIU Keyan, MENG Xiaoli, et al. A big data online cleaning algorithm based on dynamic outlier detection[C]. 2015 International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery, Xi'an, China, 2015: 230–234.
    [6]
    田江, 顾宏. 孤立点一类支持向量机算法研究[J]. 电子与信息学报, 2010, 32(6): 1284–1288. doi: 10.3724/SP.J.1146.2009.00861

    TIAN Jiang and GU Hong. Outlier one class support vector machines[J]. Journal of Electronics &Information Technology, 2010, 32(6): 1284–1288. doi: 10.3724/SP.J.1146.2009.00861
    [7]
    ZOU Zhuping, XIE Yulai, HUANG Kai, et al. A docker container anomaly monitoring system based on optimized isolation forest[J]. IEEE Transactions on Cloud Computing, To be published. doi: 10.1109/TCC.2019.2935724.
    [8]
    ZHOU Zihan, LI Xiaodong, WRIGHT J, et al. Stable principal component pursuit[C]. 2010 IEEE International Symposium on Information Theory, Austin, USA, 2010: 1518–1522.
    [9]
    XU Yichu, DU Bo, ZHANG Liangpei, et al. A low-rank and sparse matrix decomposition-based dictionary reconstruction and anomaly extraction framework for hyperspectral anomaly detection[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(7): 1248–1252. doi: 10.1109/LGRS.2019.2943861
    [10]
    DAUBECHIES I, DEFRISE M, and DE MOL C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint[J]. Communications on Pure and Applied Mathematics, 2004, 57(11): 1413–1457. doi: 10.1002/cpa.20042
    [11]
    BIOUCAS-DIAS J M and FIGUEIREDO M A T. A new TwIST: Two-step iterative shrinkage/thresholding algorithms for image restoration[J]. IEEE Transactions on Image Processing, 2007, 16(12): 2992–3004. doi: 10.1109/TIP.2007.909319
    [12]
    CANDES E J, WAKIN M B, and BOYD S. Enhancing sparsity by reweighted l1 minimization[J]. Journal of Fourier Analysis and Applications, 2008, 14(5): 877–905.
    [13]
    ELAD M. Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing[M]. New York: Springer, 2010: 185–200.
    [14]
    CHENG Jie, YE Qiang, JIANG Hongbo, et al. STCDG: An efficient data gathering algorithm based on matrix completion for wireless sensor networks[J]. IEEE Transactions on Wireless Communications, 2013, 12(2): 850–861. doi: 10.1109/TWC.2012.121412.120148
    [15]
    李鹏, 王建新, 曹建农. 无线传感器网络中基于压缩感知和GM(1, 1)的异常检测方案[J]. 电子与信息学报, 2015, 37(7): 1586–1590. doi: 10.11999/JEIT141219

    LI Peng, WANG Jianxin, and CAO Jiannong. Abnormal event detection scheme based on compressive sensing and GM (1, 1) in wireless sensor networks[J]. Journal of Electronics &Information Technology, 2015, 37(7): 1586–1590. doi: 10.11999/JEIT141219
    [16]
    LIU Jing and RAO B D. Robust PCA via 0-1 regularization[J]. IEEE Transactions on Signal Processing, 2019, 67(2): 535–549. doi: 10.1109/TSP.2018.2883924
    [17]
    RAHMANI M and ATIA G K. High dimensional low rank plus sparse matrix decomposition[J]. IEEE Transactions on Signal Processing, 2017, 65(8): 2004–2019. doi: 10.1109/TSP.2017.2649482
    [18]
    ORTIZ-RODRIGUEZ J M and VEGA-CARRILLO H R. A neutron spectra unfolding code, based on iterative procedures, designed under LabVIEW environment[C]. 2012 IEEE Ninth Electronics, Robotics and Automotive Mechanics Conference, Cuernavaca, Mexico, 2012: 315–319.
    [19]
    GIRYES R, ELDAR Y C, BRONSTEIN A M, et al. Tradeoffs between convergence speed and reconstruction accuracy in inverse problems[J]. IEEE Transactions on Signal Processing, 2018, 66(7): 1676–1690. doi: 10.1109/TSP.2018.2791945
    [20]
    YANG Yang, SUN Jian, LI Huibin, et al. ADMM-CSNet: A deep learning approach for image compressive sensing[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(3): 521–538. doi: 10.1109/TPAMI.2018.2883941
    [21]
    CHEN Yunjin, WEI Yu, and POCK T. On learning optimized reaction diffusion processes for effective image restoration[C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition, Boston, USA, 2015: 5261–5269.
    [22]
    SOLOMON O, COHEN R, ZHANG Yi, et al. Deep unfolded robust PCA with application to clutter suppression in ultrasound[J]. IEEE Transactions on Medical Imaging, 2020, 39(4): 1051–1063. doi: 10.1109/TMI.2019.2941271
    [23]
    Intel Berkeley Research Lab. Intel lab data[EB/OL]. http://db.lcs.mit.edu/labdata/labdata.html, 2019.
    [24]
    苏凤阁. 大纳伦河流域修正后的温度和降水数据集(1951–2016)[R]. 国家青藏高原科学数据中心, 2019. doi: 10.11888/Hydro.tpdc.270216.

    SU Fengge. Revised dataset of temperature and precipitation in the Greater Naren River Basin (1951–2016)[R]. National Tibetan Plateau Data Center, 2019. doi: 10.11888/Hydro.tpdc.270216.
  • 加载中

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(7)  / Tables(1)

    Article Metrics

    Article views (1638) PDF downloads(235) Cited by()
    Proportional views
    Related

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return