A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans

ZHOU Peng; LI Changyong; BU Yuxin; ZHOU Zhinuo; WANG Chunsheng; SHEN Hongbin; PAN Xiaoyong

doi:10.11999/JEIT231365

Volume 46 Issue 5

May 2024

Turn off MathJax

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2024 > 46(5): 1853-1864

ZHOU Peng, LI Changyong, BU Yuxin, ZHOU Zhinuo, WANG Chunsheng, SHEN Hongbin, PAN Xiaoyong. A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans[J]. Journal of Electronics & Information Technology, 2024, 46(5): 1853-1864. doi: 10.11999/JEIT231365

Citation:

ZHOU Peng, LI Changyong, BU Yuxin, ZHOU Zhinuo, WANG Chunsheng, SHEN Hongbin, PAN Xiaoyong. A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans[J]. Journal of Electronics & Information Technology, 2024, 46(5): 1853-1864. doi: 10.11999/JEIT231365

Citation:

ZHOU Peng, LI Changyong, BU Yuxin, ZHOU Zhinuo, WANG Chunsheng, SHEN Hongbin, PAN Xiaoyong. A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans[J]. Journal of Electronics & Information Technology, 2024, 46(5): 1853-1864. doi: 10.11999/JEIT231365

PDF( 1114 KB)

A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans

doi: 10.11999/JEIT231365

ZHOU Peng^{1, 2, 3},
LI Changyong³,
BU Yuxin^{1, 2},
ZHOU Zhinuo³,
WANG Chunsheng^{1, 2},
SHEN Hongbin³,
PAN Xiaoyong^{3
,
,}

1.
Second Institute of Oceanography, Ministry of Natural Resources, Hangzhou 310012, China
2.
Key Laboratory of Marine Ecosystem Dynamics, Ministry of Natural Resources, Hangzhou 310012, China
3.
Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, and Key Laboratory of System Control and Information Processing, Ministry of Education of China, Shanghai 200240, China

Funds: The National Key Research and Development Program of China (2023YFC2811502), The Oceanic Interdisciplinary Program of Shanghai Jiao Tong University (SL2022ZD108, SL2021MS005)

Received Date: 2023-12-11
Rev Recd Date: 2024-03-29

Available Online: 2024-04-12

Publish Date: 2024-05-10

Abstract

Abstract

Understanding the species composition, abundance and temporal and spatial distribution of fish on a global scale will help their biodiversity conservation. Underwater image acquisition is one of the main means to survey fish species diversity, but image data analysis is time-consuming and labor-intensive. Since 2015, a series of progress has been made in updating the datasets of marine fish images and optimizing the algorithm of deep learning models, but the performance of fine-grained classification is still insufficient, and the production practice application of research results is relatively weak. Therefore, the need for automated fish image classification in marine investigations is firstly studied. Then a comprehensive introduction to fish image datasets and deep learning algorithm applications is provided, and the main challenges and the corresponding solutions are analyzed. Finally, the importance of automated classification of marine fish images for related image information processing research is discussed, and its prospects in the field of marine monitoring are summarized.
- Deep learning,
- Marine fish,
- Biodiversity,
- Few shots,
- Image classification

FullText(HTML)

References(64)

References

[1]	KUITER R H and DEBELIUS H. World atlas of marine fishes[M]. 2nd ed. Frankfurt: IKAN-Unterwasserarchiv, 2006.
[2]	ALLEN G R. Field Guide to Marine Fishes of Tropical Australia and South-East Asia[M]. Western Australian Museum, 2009.
[3]	陈大刚, 张美昭. 中国海洋鱼类上[M]. 青岛: 中国海洋大学出版社, 2015. CHEN Dagang and ZHANG Meizhao. Marine Fishes of China[M]. Qingdao: China Ocean University Press, 2015.
[4]	陈大刚, 张美昭. 中国海洋鱼类下[M]. 青岛: 中国海洋大学出版社, 2015. CHEN Dagang and ZHANG Meizhao. Marine Fishes of China[M]. Qingdao: China Ocean University Press, 2015.
[5]	陈大刚, 张美昭. 中国海洋鱼类中[M]. 青岛: 中国海洋大学出版社, 2015. CHEN Dagang and ZHANG Meizhao. Marine Fishes of China[M]. Qingdao: China Ocean University Press, 2015.
[6]	LI Xiu, SHANG Min, QIN Hongwei, et al. Fast accurate fish detection and recognition of underwater images with Fast R-CNN[C]. MTS/IEEE Washington, Washington, USA, 2015: 1–5. doi: 10.23919/OCEANS.2015.7404464.
[7]	VILLON S, CHAUMONT M, SUBSOL G, et al. Coral reef fish detection and recognition in underwater videos by supervised machine learning: Comparison between Deep Learning and HOG+SVM methods[C]. The 17th International Conference on Advanced Concepts for Intelligent Vision Systems, Lecce, Italy, 2016: 160–171. doi: 10.1007/978-3-319-48680-2_15.
[8]	JÄGER J, SIMON M, DENZLER J, et al. Croatian fish dataset: Fine-grained classification of fish species in their natural habitat[C]. Machine Vision of Animals and their Behaviour Workshop 2015, 2015: 2. doi: 10.5244/C.29.MVAB.6.
[9]	SALEH A, LARADJI I H, KONOVALOV D A, et al. A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis[J]. Scientific Reports, 2020, 10(1): 14671. doi: 10.1038/s41598-020-71639-x.
[10]	LIN Yuan, CHU Zhaoqi, KORHONEN J, et al. Fast accurate fish recognition with deep learning based on a domain-specific large-scale fish dataset[C]. The 29th International Conference on Multimedia Modeling, Bergen, Norway, 2023: 515–526. doi: 10.1007/978-3-031-27077-2_40.
[11]	ZHUANG Peiqin, WANG Yali, and QIAO Yu. Wildfish++: A comprehensive fish benchmark for multimedia research[J]. IEEE Transactions on Multimedia, 2021, 23: 3603–3617. doi: 10.1109/TMM.2020.3028482.
[12]	KHAN F F, LI Xiang, TEMPLE A J, et al. FishNet: A large-scale dataset and benchmark for fish recognition, detection, and functional trait prediction[C]. 2023 IEEE/CVF International Conference on Computer Vision, Paris, France, 2023: 20439–20449. doi: 10.1109/ICCV51070.2023.01874.
[13]	QIN Hongwei, LI Xiu, LIANG Jian, et al. DeepFish: Accurate underwater live fish recognition with a deep architecture[J]. Neurocomputing, 2016, 187: 49–58. doi: 10.1016/j.neucom.2015.10.122.
[14]	LI Xiu, TANG Youhua, and GAO Tingwei. Deep but lightweight neural networks for fish detection[C]. OCEANS 2017 - Aberdeen, Aberdeen, UK, 2017: 1–5. doi: 10.1109/OCEANSE.2017.8084961.
[15]	RATHI D, JAIN S, and INDU S. Underwater fish species classification using convolutional neural network and deep learning[C]. The 2017 Ninth International Conference on Advances in Pattern Recognition, Bangalore, India, 2017: 1–6. doi: 10.1109/ICAPR.2017.8593044.
[16]	TAMOU A B, BENZINOU A, NASREDDINE K, et al. Underwater live fish recognition by deep learning[C]. The 8th International Conference on Image and Signal Processing, Cherbourg, France, 2018: 275–283. doi: 10.1007/978-3-319-94211-7_30.
[17]	QIU Chenchen, ZHANG Shaoyong, WANG Chao, et al. Improving transfer learning and squeeze- and-excitation networks for small-scale fine-grained fish image classification[J]. IEEE Access, 2018, 6: 78503–78512. doi: 10.1109/ACCESS.2018.2885055.
[18]	SIDDIQUI S A, SALMAN A, MALIK M I, et al. Automatic fish species classification in underwater videos: Exploiting pre-trained deep neural network models to compensate for limited labelled data[J]. ICES Journal of Marine Science, 2018, 75(1): 374–389. doi: 10.1093/icesjms/fsx109.
[19]	ZHUANG Peiqin, WANG Yali, and QIAO Yu. WildFish: A large benchmark for fish recognition in the wild[C]. The 26th ACM international conference on Multimedia. Seoul, Republic of Korea, 2018: 1301–1039. doi: 10.1145/3240508.3240616.
[20]	孙东洋, 刘辉, 张纪红, 等. 基于深度卷积神经网络的海洋牧场岩礁性生物图像分类[J]. 海洋与湖沼, 2021, 52(5): 1160–1169. doi: 10.11693/hyhz20210100005. SUN Dongyang, LIU Hui, ZHANG Jihong, et al. Classification of reef biological images of marine ranch based on deep convolution neural network[J]. Oceanologia et Limnologia Sinica, 2021, 52(5): 1160–1169. doi: 10.11693/hyhz20210100005.
[21]	IQBAL M A, WANG Zhijie, ALI Z A, et al. Automatic fish species classification using deep convolutional neural networks[J]. Wireless Personal Communications, 2021, 116(2): 1043–1053. doi: 10.1007/s11277-019-06634-1.
[22]	BOOM B J, HUANG P X, HE Jiyin, et al. Supporting ground-truth annotation of image datasets using clustering[C]. The 21st International Conference on Pattern Recognition, Tsukuba, Japan, 2012: 1542–1545.
[23]	KAY J and MERRIFIELD M. The fishnet open images database: A dataset for fish detection and fine-grained categorization in fisheries[J]. arXiv: 2106.09178, 2021.
[24]	TORISAWA S, KADOTA M, KOMEYAMA K, et al. A digital stereo-video camera system for three-dimensional monitoring of free-swimming Pacific bluefin tuna, Thunnus orientalis, cultured in a net cage[J]. Aquatic Living Resources, 2011, 24(2): 107–112. doi: 10.1051/alr/2011133.
[25]	MENG Lin, HIRAYAMA T, and OYANAGI S. Underwater-drone with panoramic camera for automatic fish recognition based on deep learning[J]. IEEE Access, 2018, 6: 17880–17886. doi: 10.1109/ACCESS.2018.2820326.
[26]	PLATT J C. Sequential minimal optimization: A fast algorithm for training support vector machines[R]. MSR-TR-98-14, 1998.
[27]	HUANG P X, BOOM B J, and FISHER R B. Hierarchical classification with reject option for live fish recognition[J]. Machine Vision and Applications, 2015, 26(1): 89–102. doi: 10.1007/s00138-014-0641-2.
[28]	CHUANG Mengche, HWANG J N, and WILLIAMS K. A feature learning and object recognition framework for underwater fish images[J]. IEEE Transactions on Image Processing, 2016, 25(4): 1862–1872. doi: 10.1109/TIP.2016.2535342.
[29]	ISLAM S M M, BANI S I, and GHOSH R. Content-based fish classification using combination of machine learning methods[J]. International Journal of Information Technology and Computer Science (IJITCS), 2021, 13(8): 62–68. doi: 10.5815/ijitcs.2021.01.05.
[30]	OU Liguo, LIU Bilin, CHEN Xinjun, et al. Automatic classification of the phenotype textures of three Thunnus species based on the machine learning SVM algorithm[J]. Canadian Journal of Fisheries and Aquatic Sciences, 2023, 80(8): 1221–1236. doi: 10.1139/cjfas-2022-0270.
[31]	LUAN Jing, ZHANG Chongliang, XU Binduo, et al. The predictive performances of random forest models with limited sample size and different species traits[J]. Fisheries Research, 2020, 227: 105534. doi: 10.1016/j.fishres.2020.105534.
[32]	MAMPITIYA L I, NALMI R, and RATHNAYAKE N. Performance comparison of sea fish species classification using hybrid and supervised machine learning algorithms[C]. 2022 Moratuwa Engineering Research Conference, Moratuwa, Sri Lanka, 2022: 1–6. doi: 10.1109/MERCon55799.2022.9906206.
[33]	LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324. doi: 10.1109/5.726791.
[34]	SARIGÜL M and AVCI M. Comparison of different deep structures for fish classification[J]. International Journal of Computer Theory and Engineering, 2017, 9(5): 362–366. doi: 10.7763/IJCTE.2017.V9.1167.
[35]	DEEP B V and DASH R. Underwater fish species recognition using deep learning techniques[C]. The 2019 6th International Conference on Signal Processing and Integrated Networks, Noida, India, 2019: 665–669. doi: 10.1109/SPIN.2019.8711657.
[36]	KONOVALOV D A, SALEH A, BRADLEY M, et al. Underwater fish detection with weak multi-domain supervision[C]. 2019 International Joint Conference on Neural Networks, Budapest, Hungary, 2019: 1–8. doi: 10.1109/IJCNN.2019.8851907.
[37]	ZHAO Zhenxi, LIU Yang, SUN Xudong, et al. Composited fishnet: Fish detection and species recognition from low-quality underwater videos[J]. IEEE Transactions on Image Processing, 2021, 30: 4719–4734. doi: 10.1109/TIP.2021.3074738.
[38]	ZENG Lingcai, SUN Bing, and ZHU Daqi. Underwater target detection based on Faster R-CNN and adversarial occlusion network[J]. Engineering Applications of Artificial Intelligence, 2021, 100: 104190. doi: 10.1016/j.engappai.2021.104190.
[39]	HONG KHAI T, ABDULLAH S N H S, HASAN M K, et al. Underwater fish detection and counting using mask regional convolutional neural network[J]. Water, 2022, 14(2): 222. doi: 10.3390/w14020222.
[40]	SONG Pinhao, LI Pengteng, DAI Linhui, et al. Boosting R-CNN: Reweighting R-CNN samples by RPN’s error for underwater object detection[J]. Neurocomputing, 2023, 530: 150–164. doi: 10.1016/j.neucom.2023.01.088.
[41]	XU Wenwei and MATZNER S. Underwater fish detection using deep learning for water power applications[C]. 2018 International Conference on Computational Science and Computational Intelligence, Las Vegas, USA, 2018: 313–318. doi: 10.1109/CSCI46756.2018.00067.
[42]	HU Xuelong, LIU Yang, ZHAO Zhengxi, et al. Real-time detection of uneaten feed pellets in underwater images for aquaculture using an improved YOLO-V4 network[J]. Computers and Electronics in Agriculture, 2021, 185: 106135. doi: 10.1016/j.compag.2021.106135.
[43]	AL MUKSIT A, HASAN F, HASAN BHUIYAN EMON F, et al. YOLO-Fish: A robust fish detection model to detect fish in realistic underwater environment[J]. Ecological Informatics, 2022, 72: 101847. doi: 10.1016/j.ecoinf.2022.101847.
[44]	LI Jianyuan, LIU Chunna, LU Xiaochun, et al. CME-YOLOv5: An efficient object detection network for densely spaced fish and small targets[J]. Water, 2022, 14(15): 2412. doi: 10.3390/w14152412.
[45]	ALABA S Y, NABI M M, SHAH C, et al. Class-aware fish species recognition using deep learning for an imbalanced dataset[J]. Sensors, 2022, 22(21): 8268. doi: 10.3390/s22218268.
[46]	MNIH V, HEESS N, GRAVES A, et al. Recurrent models of visual attention[C]. The 27th International Conference on Neural Information Processing Systems, Cambridge, USA, 2014: 2204–2212. doi: 10.5555/2969033.2969073.
[47]	ZHANG Wenbo, WU Chaoyi, and BAO Zhenshan. DPANet: Dual pooling-aggregated attention network for fish segmentation[J]. IET Computer Vision, 2022, 16(1): 67–82. doi: 10.1049/cvi2.12065.
[48]	GUPTA S, MUKHERJEE P, CHAUDHURY S, et al. DFTNet: Deep fish tracker with attention mechanism in unconstrained marine environments[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5016213. doi: 10.1109/TIM.2021.3109731.
[49]	LI Shanmin, PAN Bei, CHENG Yuanshun, et al. Underwater fish object detection based on attention mechanism improved ghost-YOLOv5[C]. The 2022 7th International Conference on Intelligent Computing and Signal Processing, Xi’an, China, 2022: 599–603. doi: 10.1109/ICSP54964.2022.9778582.
[50]	CHEN Lulu, ZANG Zhaoxiang, HUANG Tianxing, et al. Marine fish object detection based on YOLOv5 and attention mechanism[C]. 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta), Haikou, China, 2022: 1252–1258. doi: 10.1109/SmartWorld-UIC-ATC-ScalCom-DigitalTwin-PriComp-Metaverse56740.2022.00185.
[51]	HAN Xu, ZHANG Zhengyan, DING Ning, et al. Pre-trained models: Past, present and future[J]. AI Open, 2021, 2: 225–250. doi: 10.1016/j.aiopen.2021.08.002.
[52]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 6000–6010. doi: 10.5555/3295222.3295349.
[53]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]. The 9th International Conference on Learning Representations, 2021.
[54]	SALEH A, SHEAVES M, JERRY D, et al. Transformer-based Self-Supervised Fish Segmentation in Underwater Videos[J]. arXiv: 2206.05390, 2022.
[55]	SALEH A, JONES D, JERRY D, et al. A lightweight Transformer-based model for fish landmark detection[J]. arXiv: 2209.05777, 2022.
[56]	LIU Yang, AN Dong, REN Yinjie, et al. DP-FishNet: Dual-path Pyramid Vision Transformer-based underwater fish detection network[J]. Expert Systems with Applications, 2024, 238: 122018. doi: 10.1016/j.eswa.2023.122018.
[57]	GONG Bo, DAI Kanyuan, SHAO Ji, et al. Fish-TViT: A novel fish species classification method in multi water areas based on transfer learning and vision transformer[J]. Heliyon, 2023, 9(6): e16761. doi: 10.1016/j.heliyon.2023.e16761.
[58]	WANG Yaqing, YAO Quanming, KWOK J T, et al. Generalizing from a few examples: A survey on few-shot learning[J]. ACM Computing Surveys, 2020, 53(3): 63. doi: 10.1145/3386252.
[59]	VILLON S, IOVAN C, MANGEAS M, et al. Automatic underwater fish species classification with limited data using few-shot learning[J]. Ecological Informatics, 2021, 63: 101320. doi: 10.1016/j.ecoinf.2021.101320.
[60]	GUO Zonghui, ZHANG Liqiang, JIANG Yufeng, et al. Few-shot fish image generation and classification[C]. Global Oceans 2020: Singapore – U. S. Gulf Coast, Biloxi, USA, 2020: 1–6. doi: 10.1109/IEEECONF38699.2020.9389005.
[61]	LIU Feng, DING Hao, LI Daihui, et al. Few-shot learning with data enhancement and transfer learning for underwater target recognition[C]. 2021 OES China Ocean Acoustics, Harbin, China, 2021: 992–994. doi: 10.1109/COA50123.2021.9519853.
[62]	GONG Longqin, HU Zhuhua, and ZHOU Xiaoyi. A few samples underwater fish tracking method based on semi-supervised and attention mechanism[C]. The 2022 6th International Conference on Robotics, Control and Automation, Xiamen, China, 2022: 18–22. doi: 10.1109/ICRCA55033.2022.9828911.
[63]	ZHAI Jiping, HAN Lu, XIAO Ying, et al. Few-shot fine-grained fish species classification via sandwich attention CovaMNet[J]. Frontiers in Marine Science, 2023, 10: 1149186. doi: 10.3389/fmars.2023.1149186.
[64]	WEI Xiushen, SONG Yizhe, MAC AODHA O, et al. Fine-grained image analysis with deep learning: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 8927–8948. doi: 10.1109/TPAMI.2021.3126648.