高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

全球尺度下的海洋鱼类图像智能分类研究进展

周鹏 李昌永 步雨馨 周芷诺 王春生 沈红斌 潘小勇

周鹏, 李昌永, 步雨馨, 周芷诺, 王春生, 沈红斌, 潘小勇. 全球尺度下的海洋鱼类图像智能分类研究进展[J]. 电子与信息学报. doi: 10.11999/JEIT231365
引用本文: 周鹏, 李昌永, 步雨馨, 周芷诺, 王春生, 沈红斌, 潘小勇. 全球尺度下的海洋鱼类图像智能分类研究进展[J]. 电子与信息学报. doi: 10.11999/JEIT231365
ZHOU Peng, LI Changyong, BU Yuxin, ZHOU Zhinuo, WANG Chunsheng, SHEN Hongbin, PAN Xiaoyong. A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT231365
Citation: ZHOU Peng, LI Changyong, BU Yuxin, ZHOU Zhinuo, WANG Chunsheng, SHEN Hongbin, PAN Xiaoyong. A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT231365

全球尺度下的海洋鱼类图像智能分类研究进展

doi: 10.11999/JEIT231365
基金项目: 国家重点研发计划(2023YFC2811502),上海交通大学深蓝计划(SL2022ZD108,SL2021MS005)
详细信息
    作者简介:

    周鹏:男,高级工程师,研究方向为海洋生物学与生物信息学研究

    李昌永:男,博士生,研究方向为模式识别与生物信息学

    步雨馨:女,硕士生,研究方向为海洋生物学

    周芷诺:女,本科生,研究方向为模式识别与生物信息学

    王春生:男,研究员,研究方向为海洋生态学

    沈红斌:男,教授,研究方向为识别与生物信息学

    潘小勇:男,长聘教轨副教授,研究方向为图像处理与模式识别研究

    通讯作者:

    潘小勇 2008xypan@sjtu.edu.cn

  • 中图分类号: TN081; TN911.7

A Review of the Artificial Intelligence-based Image Classification of Fishes in the Global Oceans

Funds: The National Key Research and Development Program of China (2023YFC2811502), The Oceanic Interdisciplinary Program of Shanghai Jiao Tong University (SL2022ZD108, SL2021MS005)
  • 摘要: 在全球尺度上了解鱼类物种组成、丰度及时空分布等,将有助于其生物多样保护。水下图像采集是获取鱼类物种多样性数据的主要调查手段之一,但图像信息分析工作耗时耗力。2015年以来,海洋鱼类图像数据集更新和深度学习模型算法优化等方面取得了一系列进展,但细粒度分类表现仍显不足,研究成果的生产实践应用相对薄弱。因此,该文首先分析海洋相关行业对鱼类自动化图像分类的需求,然后综合介绍鱼类图像数据集和深度学习算法应用,并分析了所面临的小样本下的细粒度分析等主要挑战及相应解决方法。最后探讨了基于深度学习的海洋鱼类图像自动化分类对相关图像信息处理研究及应用平台对未来在生态环境监测等海洋相关产业领域的重要性及其前景。该文旨在为快速了解基于深度学习的海洋鱼类图像自动化分类的研究背景、进展和未来方向的工作者提供相关信息。
  • 图  1  鱼类检测算法发展时序图

    图  2  全球海洋鱼类图像分类实现及应用示意图

    表  1  海洋鱼类多样性调查的主要海洋产业需求及示例场景

    海洋产业 行业需求 功能场景示例
    海洋资源开发利用 重大工程生态环境影响评价/监测 核电对鱼类多样性影响
    海洋油气、海底矿产等资源开发 采矿对鱼类多样性影响
    海洋生态环境保护 海洋污染治理评估 微塑料等污染对鱼类影响
    海洋保护区设立及划区依据 鱼类多样性及地理分布调查
    海洋防灾减灾 赤潮、绿潮、海洋酸化 灾害对鱼类影响
    全球气候变化 厄尔尼诺和拉尼娜现象对鱼类多样性影响;鱼类迁移变化
    海洋安全保障 海洋环境安全保障 核污水对鱼类影响
    港口航运 外来鱼类物种入侵监测
    海洋科学认知 海底深部探测 鱼类新物种发现
    公众普及 海洋鱼类识别
    海洋渔业 海洋捕捞 鱼类资源调查与评估
    海水养殖 环境污染对鱼类影响
    滨海旅游 海岸带可持续发展 污染影响评估;濒危鱼类识别
    健康与安全 危险物种识别,如有毒虾虎鱼
    下载: 导出CSV

    表  2  深度学习在鱼类图像自动化识别中的应用实例

    年份 数据源 物种数 原始图像张数 准确率(%) 参考文献
    2015 ImageCLEF 12 24 277 81.4 [6]
    2015 Croatian fish dataset 12 794 66.78 [9]
    2016 Fish4Knowledge 23 27 370 98.64 [17]
    2017 ImageCLEF 12 24 277 89.95 [18]
    2017 Fish4Knowledge 23 27 142 96.29 [19]
    2018 Fish4Knowledge 23 27 000 99.45 [20]
    2018 Croatian fish dataset 12 794 83.92 [21]
    2018 自建图集 16 1 647 94.30 [22]
    2018 WildFish 1 000 54 459 74.70 [23]
    2020 DeepFish 20 39 766 99.00 [10]
    2020 WildFish++ 2 348 103 034 74.70 * [13]
    2021 自建图集 15 23 211 99.23 [24]
    2021 QUT dataset 6 1 334 90.48 [25]
    2023 OceanFish 136 63 622 97.12 [12]
    2023 FishNet 17 357 94 532 61.38 [14]
    注:* 仅针对选取出的易混淆的22对鱼类,训练集含1 668张图片,测试集含1 320张图片。
       包括鱼类11种、棘皮动物类3种和甲壳动物类1种。
       在科一级分类水平上,共463科。
    下载: 导出CSV

    表  3  主要问题、原因分析及解决方案

    问题原因解决方案
    识别准确率低训练数据集小扩大数据集
    算法需要优化使用和优化小样本算法
    使用流程复杂需数学、编程等专业知识利用网络平台提高易用性
    应用范围不广受操作系统、区域限制
    下载: 导出CSV
  • [1] KUITER R H and DEBELIUS H. World atlas of marine fishes[M]. 2nd ed. Frankfurt: IKAN-Unterwasserarchiv, 2006.
    [2] ALLEN G R. Field Guide to Marine Fishes of Tropical Australia and South-East Asia[M]. Western Australian Museum, 2009.
    [3] 陈大刚, 张美昭. 中国海洋鱼类 上[M]. 青岛: 中国海洋大学出版社, 2015.

    CHEN Dagang and ZHANG Meizhao. Marine Fishes of China[M]. Qingdao: China Ocean University Press, 2015.
    [4] 陈大刚, 张美昭. 中国海洋鱼类 下[M]. 青岛: 中国海洋大学出版社, 2015.

    CHEN Dagang and ZHANG Meizhao. Marine Fishes of China[M]. Qingdao: China Ocean University Press, 2015.
    [5] 陈大刚, 张美昭. 中国海洋鱼类 中[M]. 青岛: 中国海洋大学出版社, 2015.

    CHEN Dagang and ZHANG Meizhao. Marine Fishes of China[M]. Qingdao: China Ocean University Press, 2015.
    [6] LI Xiu, SHANG Min, QIN Hongwei, et al. Fast accurate fish detection and recognition of underwater images with Fast R-CNN[C]. MTS/IEEE Washington, Washington, USA, 2015: 1–5. doi: 10.23919/OCEANS.2015.7404464.
    [7] VILLON S, CHAUMONT M, SUBSOL G, et al. Coral reef fish detection and recognition in underwater videos by supervised machine learning: Comparison between Deep Learning and HOG+SVM methods[C]. The 17th International Conference on Advanced Concepts for Intelligent Vision Systems, Lecce, Italy, 2016: 160–171. doi: 10.1007/978-3-319-48680-2_15.
    [8] BOOM B J, HUANG P X, HE Jiyin, et al. Supporting ground-truth annotation of image datasets using clustering[C]. The 21st International Conference on Pattern Recognition, Tsukuba, Japan, 2012: 1542–1545.
    [9] JÄGER J, SIMON M, DENZLER J, et al. Croatian fish dataset: Fine-grained classification of fish species in their natural habitat[C]. Machine Vision of Animals and their Behaviour Workshop 2015, 2015: 2. doi: 10.5244/C.29.MVAB.6.
    [10] SALEH A, LARADJI I H, KONOVALOV D A, et al. A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis[J]. Scientific Reports, 2020, 10(1): 14671. doi: 10.1038/s41598-020-71639-x.
    [11] KAY J and MERRIFIELD M. The fishnet open images database: A dataset for fish detection and fine-grained categorization in fisheries[J]. arXiv: 2106.09178, 2021.
    [12] LIN Yuan, CHU Zhaoqi, KORHONEN J, et al. Fast accurate fish recognition with deep learning based on a domain-specific large-scale fish dataset[C]. The 29th International Conference on Multimedia Modeling, Bergen, Norway, 2023: 515–526. doi: 10.1007/978-3-031-27077-2_40.
    [13] ZHUANG Peiqin, WANG Yali, and QIAO Yu. Wildfish++: A comprehensive fish benchmark for multimedia research[J]. IEEE Transactions on Multimedia, 2021, 23: 3603–3617. doi: 10.1109/TMM.2020.3028482.
    [14] KHAN F F, LI Xiang, TEMPLE A J, et al. FishNet: A large-scale dataset and benchmark for fish recognition, detection, and functional trait prediction[C]. 2023 IEEE/CVF International Conference on Computer Vision, Paris, France, 2023: 20439–20449. doi: 10.1109/ICCV51070.2023.01874.
    [15] TORISAWA S, KADOTA M, KOMEYAMA K, et al. A digital stereo-video camera system for three-dimensional monitoring of free-swimming Pacific bluefin tuna, Thunnus orientalis, cultured in a net cage[J]. Aquatic Living Resources, 2011, 24(2): 107–112. doi: 10.1051/alr/2011133.
    [16] MENG Lin, HIRAYAMA T, and OYANAGI S. Underwater-drone with panoramic camera for automatic fish recognition based on deep learning[J]. IEEE Access, 2018, 6: 17880–17886. doi: 10.1109/ACCESS.2018.2820326.
    [17] QIN Hongwei, LI Xiu, LIANG Jian, et al. DeepFish: Accurate underwater live fish recognition with a deep architecture[J]. Neurocomputing, 2016, 187: 49–58. doi: 10.1016/j.neucom.2015.10.122.
    [18] LI Xiu, TANG Youhua, and GAO Tingwei. Deep but lightweight neural networks for fish detection[C]. OCEANS 2017 - Aberdeen, Aberdeen, UK, 2017: 1–5. doi: 10.1109/OCEANSE.2017.8084961.
    [19] RATHI D, JAIN S, and INDU S. Underwater fish species classification using convolutional neural network and deep learning[C]. The 2017 Ninth International Conference on Advances in Pattern Recognition, Bangalore, India, 2017: 1–6. doi: 10.1109/ICAPR.2017.8593044.
    [20] TAMOU A B, BENZINOU A, NASREDDINE K, et al. Underwater live fish recognition by deep learning[C]. The 8th International Conference on Image and Signal Processing, Cherbourg, France, 2018: 275–283. doi: 10.1007/978-3-319-94211-7_30.
    [21] QIU Chenchen, ZHANG Shaoyong, WANG Chao, et al. Improving transfer learning and squeeze- and-excitation networks for small-scale fine-grained fish image classification[J]. IEEE Access, 2018, 6: 78503–78512. doi: 10.1109/ACCESS.2018.2885055.
    [22] SIDDIQUI S A, SALMAN A, MALIK M I, et al. Automatic fish species classification in underwater videos: Exploiting pre-trained deep neural network models to compensate for limited labelled data[J]. ICES Journal of Marine Science, 2018, 75(1): 374–389. doi: 10.1093/icesjms/fsx109.
    [23] ZHUANG Peiqin, WANG Yali, and QIAO Yu. WildFish: A large benchmark for fish recognition in the wild[C]. The 26th ACM international conference on Multimedia. Seoul, Republic of Korea, 2018: 1301–1039. doi: 10.1145/3240508.3240616.
    [24] 孙东洋, 刘辉, 张纪红, 等. 基于深度卷积神经网络的海洋牧场岩礁性生物图像分类[J]. 海洋与湖沼, 2021, 52(5): 1160–1169. doi: 10.11693/hyhz20210100005.

    SUN Dongyang, LIU Hui, ZHANG Jihong, et al. Classification of reef biological images of marine ranch based on deep convolution neural network[J]. Oceanologia et Limnologia Sinica, 2021, 52(5): 1160–1169. doi: 10.11693/hyhz20210100005.
    [25] IQBAL M A, WANG Zhijie, ALI Z A, et al. Automatic fish species classification using deep convolutional neural networks[J]. Wireless Personal Communications, 2021, 116(2): 1043–1053. doi: 10.1007/s11277-019-06634-1.
    [26] PLATT J C. Sequential minimal optimization: A fast algorithm for training support vector machines[R]. MSR-TR-98-14, 1998.
    [27] HUANG P X, BOOM B J, and FISHER R B. Hierarchical classification with reject option for live fish recognition[J]. Machine Vision and Applications, 2015, 26(1): 89–102. doi: 10.1007/s00138-014-0641-2.
    [28] CHUANG Mengche, HWANG J N, and WILLIAMS K. A feature learning and object recognition framework for underwater fish images[J]. IEEE Transactions on Image Processing, 2016, 25(4): 1862–1872. doi: 10.1109/TIP.2016.2535342.
    [29] ISLAM S M M, BANI S I, and GHOSH R. Content-based fish classification using combination of machine learning methods[J]. International Journal of Information Technology and Computer Science (IJITCS), 2021, 13(8): 62–68. doi: 10.5815/ijitcs.2021.01.05.
    [30] OU Liguo, LIU Bilin, CHEN Xinjun, et al. Automatic classification of the phenotype textures of three Thunnus species based on the machine learning SVM algorithm[J]. Canadian Journal of Fisheries and Aquatic Sciences, 2023, 80(8): 1221–1236. doi: 10.1139/cjfas-2022-0270.
    [31] LUAN Jing, ZHANG Chongliang, XU Binduo, et al. The predictive performances of random forest models with limited sample size and different species traits[J]. Fisheries Research, 2020, 227: 105534. doi: 10.1016/j.fishres.2020.105534.
    [32] MAMPITIYA L I, NALMI R, and RATHNAYAKE N. Performance comparison of sea fish species classification using hybrid and supervised machine learning algorithms[C]. 2022 Moratuwa Engineering Research Conference, Moratuwa, Sri Lanka, 2022: 1–6. doi: 10.1109/MERCon55799.2022.9906206.
    [33] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324. doi: 10.1109/5.726791.
    [34] SARIGÜL M and AVCI M. Comparison of different deep structures for fish classification[J]. International Journal of Computer Theory and Engineering, 2017, 9(5): 362–366. doi: 10.7763/IJCTE.2017.V9.1167.
    [35] DEEP B V and DASH R. Underwater fish species recognition using deep learning techniques[C]. The 2019 6th International Conference on Signal Processing and Integrated Networks, Noida, India, 2019: 665–669. doi: 10.1109/SPIN.2019.8711657.
    [36] KONOVALOV D A, SALEH A, BRADLEY M, et al. Underwater fish detection with weak multi-domain supervision[C]. 2019 International Joint Conference on Neural Networks, Budapest, Hungary, 2019: 1–8. doi: 10.1109/IJCNN.2019.8851907.
    [37] ZHAO Zhenxi, LIU Yang, SUN Xudong, et al. Composited fishnet: Fish detection and species recognition from low-quality underwater videos[J]. IEEE Transactions on Image Processing, 2021, 30: 4719–4734. doi: 10.1109/TIP.2021.3074738.
    [38] ZENG Lingcai, SUN Bing, and ZHU Daqi. Underwater target detection based on Faster R-CNN and adversarial occlusion network[J]. Engineering Applications of Artificial Intelligence, 2021, 100: 104190. doi: 10.1016/j.engappai.2021.104190.
    [39] HONG KHAI T, ABDULLAH S N H S, HASAN M K, et al. Underwater fish detection and counting using mask regional convolutional neural network[J]. Water, 2022, 14(2): 222. doi: 10.3390/w14020222.
    [40] SONG Pinhao, LI Pengteng, DAI Linhui, et al. Boosting R-CNN: Reweighting R-CNN samples by RPN’s error for underwater object detection[J]. Neurocomputing, 2023, 530: 150–164. doi: 10.1016/j.neucom.2023.01.088.
    [41] XU Wenwei and MATZNER S. Underwater fish detection using deep learning for water power applications[C]. 2018 International Conference on Computational Science and Computational Intelligence, Las Vegas, USA, 2018: 313–318. doi: 10.1109/CSCI46756.2018.00067.
    [42] HU Xuelong, LIU Yang, ZHAO Zhengxi, et al. Real-time detection of uneaten feed pellets in underwater images for aquaculture using an improved YOLO-V4 network[J]. Computers and Electronics in Agriculture, 2021, 185: 106135. doi: 10.1016/j.compag.2021.106135.
    [43] AL MUKSIT A, HASAN F, HASAN BHUIYAN EMON F, et al. YOLO-Fish: A robust fish detection model to detect fish in realistic underwater environment[J]. Ecological Informatics, 2022, 72: 101847. doi: 10.1016/j.ecoinf.2022.101847.
    [44] LI Jianyuan, LIU Chunna, LU Xiaochun, et al. CME-YOLOv5: An efficient object detection network for densely spaced fish and small targets[J]. Water, 2022, 14(15): 2412. doi: 10.3390/w14152412.
    [45] ALABA S Y, NABI M M, SHAH C, et al. Class-aware fish species recognition using deep learning for an imbalanced dataset[J]. Sensors, 2022, 22(21): 8268. doi: 10.3390/s22218268.
    [46] MNIH V, HEESS N, GRAVES A, et al. Recurrent models of visual attention[C]. The 27th International Conference on Neural Information Processing Systems, Cambridge, USA, 2014: 2204–2212. doi: 10.5555/2969033.2969073.
    [47] ZHANG Wenbo, WU Chaoyi, and BAO Zhenshan. DPANet: Dual pooling-aggregated attention network for fish segmentation[J]. IET Computer Vision, 2022, 16(1): 67–82. doi: 10.1049/cvi2.12065.
    [48] GUPTA S, MUKHERJEE P, CHAUDHURY S, et al. DFTNet: Deep fish tracker with attention mechanism in unconstrained marine environments[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 5016213. doi: 10.1109/TIM.2021.3109731.
    [49] LI Shanmin, PAN Bei, CHENG Yuanshun, et al. Underwater fish object detection based on attention mechanism improved ghost-YOLOv5[C]. The 2022 7th International Conference on Intelligent Computing and Signal Processing, Xi’an, China, 2022: 599–603. doi: 10.1109/ICSP54964.2022.9778582.
    [50] CHEN Lulu, ZANG Zhaoxiang, HUANG Tianxing, et al. Marine fish object detection based on YOLOv5 and attention mechanism[C]. 2022 IEEE Smartworld, Ubiquitous Intelligence & Computing, Scalable Computing & Communications, Digital Twin, Privacy Computing, Metaverse, Autonomous & Trusted Vehicles (SmartWorld/UIC/ScalCom/DigitalTwin/PriComp/Meta), Haikou, China, 2022: 1252–1258. doi: 10.1109/SmartWorld-UIC-ATC-ScalCom-DigitalTwin-PriComp-Metaverse56740.2022.00185.
    [51] HAN Xu, ZHANG Zhengyan, DING Ning, et al. Pre-trained models: Past, present and future[J]. AI Open, 2021, 2: 225–250. doi: 10.1016/j.aiopen.2021.08.002.
    [52] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 6000–6010. doi: 10.5555/3295222.3295349.
    [53] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]. The 9th International Conference on Learning Representations, 2021.
    [54] SALEH A, SHEAVES M, JERRY D, et al. Transformer-based Self-Supervised Fish Segmentation in Underwater Videos[J]. arXiv: 2206.05390, 2022.
    [55] SALEH A, JONES D, JERRY D, et al. A lightweight Transformer-based model for fish landmark detection[J]. arXiv: 2209.05777, 2022.
    [56] LIU Yang, AN Dong, REN Yinjie, et al. DP-FishNet: Dual-path Pyramid Vision Transformer-based underwater fish detection network[J]. Expert Systems with Applications, 2024, 238: 122018. doi: 10.1016/j.eswa.2023.122018.
    [57] GONG Bo, DAI Kanyuan, SHAO Ji, et al. Fish-TViT: A novel fish species classification method in multi water areas based on transfer learning and vision transformer[J]. Heliyon, 2023, 9(6): e16761. doi: 10.1016/j.heliyon.2023.e16761.
    [58] WANG Yaqing, YAO Quanming, KWOK J T, et al. Generalizing from a few examples: A survey on few-shot learning[J]. ACM Computing Surveys, 2020, 53(3): 63. doi: 10.1145/3386252.
    [59] VILLON S, IOVAN C, MANGEAS M, et al. Automatic underwater fish species classification with limited data using few-shot learning[J]. Ecological Informatics, 2021, 63: 101320. doi: 10.1016/j.ecoinf.2021.101320.
    [60] GUO Zonghui, ZHANG Liqiang, JIANG Yufeng, et al. Few-shot fish image generation and classification[C]. Global Oceans 2020: Singapore – U. S. Gulf Coast, Biloxi, USA, 2020: 1–6. doi: 10.1109/IEEECONF38699.2020.9389005.
    [61] LIU Feng, DING Hao, LI Daihui, et al. Few-shot learning with data enhancement and transfer learning for underwater target recognition[C]. 2021 OES China Ocean Acoustics, Harbin, China, 2021: 992–994. doi: 10.1109/COA50123.2021.9519853.
    [62] GONG Longqin, HU Zhuhua, and ZHOU Xiaoyi. A few samples underwater fish tracking method based on semi-supervised and attention mechanism[C]. The 2022 6th International Conference on Robotics, Control and Automation, Xiamen, China, 2022: 18–22. doi: 10.1109/ICRCA55033.2022.9828911.
    [63] ZHAI Jiping, HAN Lu, XIAO Ying, et al. Few-shot fine-grained fish species classification via sandwich attention CovaMNet[J]. Frontiers in Marine Science, 2023, 10: 1149186. doi: 10.3389/fmars.2023.1149186.
    [64] WEI Xiushen, SONG Yizhe, MAC AODHA O, et al. Fine-grained image analysis with deep learning: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 8927–8948. doi: 10.1109/TPAMI.2021.3126648.
  • 加载中
图(2) / 表(3)
计量
  • 文章访问数:  52
  • HTML全文浏览量:  31
  • PDF下载量:  7
  • 被引次数: 0
出版历程
  • 收稿日期:  2023-12-11
  • 修回日期:  2024-03-29
  • 网络出版日期:  2024-04-12

目录

    /

    返回文章
    返回