Optimization in Capsule Network Based on Mutual Information Autoencoder and Variational Routing

Jingyi BAO; Ning XU; Yunhao SHANG; Xin CHU

doi:10.11999/JEIT201094

Volume 43 Issue 11

Nov. 2021

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2021 > 43(11): 3309-3318

Jingyi BAO, Ning XU, Yunhao SHANG, Xin CHU. Optimization in Capsule Network Based on Mutual Information Autoencoder and Variational Routing[J]. Journal of Electronics & Information Technology, 2021, 43(11): 3309-3318. doi: 10.11999/JEIT201094

Citation:

Jingyi BAO, Ning XU, Yunhao SHANG, Xin CHU. Optimization in Capsule Network Based on Mutual Information Autoencoder and Variational Routing[J]. Journal of Electronics & Information Technology, 2021, 43(11): 3309-3318. doi: 10.11999/JEIT201094

Citation:

PDF( 3469 KB)

Optimization in Capsule Network Based on Mutual Information Autoencoder and Variational Routing

doi: 10.11999/JEIT201094

1.
Changzhou Institute of Technology, Changzhou 213032, China
2.
Hohai University Changzhou Campus, Changzhou 213022, China

Funds: The National Natural Science Foundation of China (61872199), The Fundamental Research Funds for the Central Universities (B210202083)

Received Date: 2020-12-30
Rev Recd Date: 2021-07-01

Available Online: 2021-07-08

Publish Date: 2021-11-23

Abstract

Abstract

Capsule network is a new type of network model which is different from convolutional neural network. This paper attempts to improve its generalization and accuracy. Firstly, variational routing is used to alleviate the problem of classic routing that is highly dependent on prior information and can easily lead to model overfitting. By using the Gaussian Mixture Model (GMM) to fit the low-level matrix capsule and using the variational method to fit the approximation distribution, the error of the maximum likelihood point estimation is avoided, and the confidence calculation is used to improve the generalization performance; Secondly, considering that the actual data is mostly untagged or difficult to label, a capsule autoencoder with mutual information evaluation criterion is constructed to achieve effective selection of feature parameters. That is, by introducing a local encoder, only the most effective features in the capsule for identifying and classifying the original input are retained, which reduces the computational burden of the network while improving the accuracy of classification and recognition at the same time. The method in this paper is compared and tested on datasets such as MNIST, FashionMNIST, CIFAR-10, and CIFAR-100. The experimental results show that the performance of the proposed method is significantly improved compared with the classic capsule network.
- Capsule network,
- Variational routing,
- Capsule autoencoder based on mutual information

FullText(HTML)

References(22)

References

[1]	SABOUR S, FROSST N, and HINTON G E. Dynamic routing between capsules[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 3856–3866.
[2]	HINTON G E, SABOUR S, and FROSST N. Matrix capsules with EM routing[C]. International Conference on Learning Representations, Vancouver, Canada, 2018.
[3]	GOLHANI K, BALASUNDRAM S K, VADAMALAI G, et al. A review of neural networks in plant disease detection using hyperspectral data[J]. Information Processing in Agriculture, 2018, 5(3): 354–371. doi: 10.1016/j.inpa.2018.05.002
[4]	PAOLETTI M E, HAUT J M, FERNANDEZ-BELTRAN R, et al. Capsule networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(4): 2145–2160. doi: 10.1109/TGRS.2018.2871782
[5]	CHU Xin, XU Ning, LIU Xiaofeng, et al. Research on capsule network optimization structure by variable route planning[C]. 2019 IEEE International Conference on Real-time Computing and Robotics (RCAR), Irkutsk, Russia, 2019: 858–861.
[6]	AUBERT G and VESE L. A variational method in image recovery[J]. SIAM Journal on Numerical Analysis, 1997, 34(5): 1948–1979. doi: 10.1137/S003614299529230X
[7]	李速, 齐翔林, 胡宏, 等. 功能柱结构神经网络模型中的同步振荡现象[J]. 中国科学C辑, 2004, 34(4): 385–394. doi: 10.3321/j.issn:1006-9259.2004.04.012
[8]	MOON T K. The expectation-maximization algorithm[J]. IEEE Signal Processing Magazine, 1996, 13(6): 47–60. doi: 10.1109/79.543975
[9]	西广成. 基于平均场理论逼近的神经网络[J]. 电子学报, 1995(8): 62–64. doi: 10.3321/j.issn:0372-2112.1995.08.016 XI Guangcheng. Neural network based on mean-field theory approximation[J]. Acta Electronica Sinica, 1995(8): 62–64. doi: 10.3321/j.issn:0372-2112.1995.08.016
[10]	BISHOP C M. Pattern Recognition and Machine Learning[M]. New York: Springer, 2006: 293–355.
[11]	GÖRÜR D and RASMUSSEN C E. Dirichlet process Gaussian mixture models: Choice of the base distribution[J]. Journal of Computer Science and Technology, 2010, 25(4): 653–664. doi: 10.1007/s11390-010-9355-8
[12]	SHRIBERG E, FERRER L, KAJAREKAR S, et al. Modeling prosodic feature sequences for speaker recognition[J]. Speech Communication, 2005, 46(3/4): 455–472.
[13]	HJELM R D, FEDOROV A, LAVOIE-MARCHILDON S, et al. Learning deep representations by mutual information estimation and maximization[C]. 7th International Conference on Learning Representations, New Orleans, USA, 2019: 1–24.
[14]	BELGHAZI M I, RAJESWAR S, BARATIN A, et al. MINE: Mutual information neural estimation[J]. arXiv: 1801.04062, 2018: 531–540.
[15]	徐峻岭, 周毓明, 陈林, 等. 基于互信息的无监督特征选择[J]. 计算机研究与发展, 2012, 49(2): 372–382. XU Junling, ZHOU Yuming, CHEN Lin, et al. An unsupervised feature selection approach based on mutual information[J]. Journal of Computer Research and Development, 2012, 49(2): 372–382.
[16]	姚志均, 刘俊涛, 周瑜, 等. 基于对称KL距离的相似性度量方法[J]. 华中科技大学学报: 自然科学版, 2011, 39(11): 1–4, 38. YAO Zhijun, LIU Juntao, ZHOU Yu, et al. Similarity measure method using symmetric KL divergence[J]. Journal of Huazhong University of Science and Technology:Nature Science, 2011, 39(11): 1–4, 38.
[17]	PATHAK D, KRÄHENBÜHL P, DONAHUE J, et al. Context encoders: Feature learning by inpainting[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 2536–2544.
[18]	KRIZHEVSKY A and HINTON G E. Learning multiple layers of features from tiny images[R]. Technical report, 2009.
[19]	LECUN Y, CORTES C, and BURGES C J C. MNIST handwritten digit database. 2010[OL]. http://yann.lecun.com/exdb/mnist, 2010, 7: 23.
[20]	XIAO H, RASUL K, and VOLLGRAF R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv: 1708.07747, 2017.
[21]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 770–778.
[22]	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 2818–2826.