高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

利用跨模态轻量级YOLOv5模型的PET/CT肺部肿瘤检测

周涛 叶鑫宇 刘凤珍 陆惠玲

周涛, 叶鑫宇, 刘凤珍, 陆惠玲. 利用跨模态轻量级YOLOv5模型的PET/CT肺部肿瘤检测[J]. 电子与信息学报, 2024, 46(2): 624-632. doi: 10.11999/JEIT230052
引用本文: 周涛, 叶鑫宇, 刘凤珍, 陆惠玲. 利用跨模态轻量级YOLOv5模型的PET/CT肺部肿瘤检测[J]. 电子与信息学报, 2024, 46(2): 624-632. doi: 10.11999/JEIT230052
ZHOU Tao, YE Xinyu, LIU Fengzhen, LU Huiling. CL-YOLOv5: PET/CT Lung Cancer Detection With Cross-modal Lightweight YOLOv5 Model[J]. Journal of Electronics & Information Technology, 2024, 46(2): 624-632. doi: 10.11999/JEIT230052
Citation: ZHOU Tao, YE Xinyu, LIU Fengzhen, LU Huiling. CL-YOLOv5: PET/CT Lung Cancer Detection With Cross-modal Lightweight YOLOv5 Model[J]. Journal of Electronics & Information Technology, 2024, 46(2): 624-632. doi: 10.11999/JEIT230052

利用跨模态轻量级YOLOv5模型的PET/CT肺部肿瘤检测

doi: 10.11999/JEIT230052
基金项目: 国家自然科学基金(62062003),宁夏自然科学基金(2022AAC03149),宁夏回族自治区重点研发计划(2020BEB04022)
详细信息
    作者简介:

    周涛:男,教授,博士生导师,研究方向为医学图像处理、计算机辅助诊断、模式识别

    叶鑫宇:男,硕士生,研究方向为医学图像处理、计算机辅助诊断

    刘凤珍:女,硕士生,研究方向为医学图像处理、计算机辅助诊断

    陆惠玲:女,教授,研究方向为医学图像分析处理、机器学习

    通讯作者:

    叶鑫宇 3303626778@qq.com

  • 中图分类号: TP391.41

CL-YOLOv5: PET/CT Lung Cancer Detection With Cross-modal Lightweight YOLOv5 Model

Funds: The National Natural Science Foundation of China (62062003), Ningxia Natural Science Foundation Project (2022AAC03149), Key Research and Development Projects of Ningxia Autonomous Region(2020BEB04022)
  • 摘要: 多模态医学图像可在同一病灶处提供更多语义信息,针对跨模态语义相关性未充分考虑和模型复杂度过高的问题,该文提出基于跨模态轻量级YOLOv5(CL-YOLOv5)的肺部肿瘤检测模型。首先,提出学习正电子发射型断层显像(PET)、 计算机断层扫描(CT)和PET/CT不同模态语义信息的3分支网络;然后,设计跨模态交互式增强块充分学习多模态语义相关性,余弦重加权计算Transformer高效学习全局特征关系,交互式增强网络提取病灶的能力;最后,提出双分支轻量块, 激活函数簇(ACON)瓶颈结构降低参数同时增加网络深度和鲁棒性,另一分支为密集连接的递进重参卷积,特征传递达到最大化,递进空间交互高效地学习多模态特征。在肺部肿瘤PET/CT多模态数据集中,该文模型获得94.76% mAP最优性能和3238 s最高效率,以及0.81 M参数量,较YOLOv5s和EfficientDet-d0降低7.7倍和5.3倍,多模态对比实验中总体上优于现有的先进方法,消融实验和热力图可视化进一步验证。
  • 图  1  CL-YOLOv5整体框架

    图  2  递进重参卷积结构

    图  3  双分支轻量块的结构

    图  4  跨模态交互式增强块的结构

    图  5  已配准的PET, CT和PET/CT图像

    图  6  消融实验的可视化结果

    图  7  不同模型在肺部肿瘤PET/CT多模态数据集上的检测结果

    图  8  不同模型的PR曲线

    图  9  不同模型的F1曲线

    图  10  肺部肿瘤影像和模型热力图

    表  1  在肺部肿瘤PET/CT多模态数据集上的消融实验对比结果

    实验添加的模块参数量计算量精度召回率mAPF1FPS总时间(s)
    YOLOv5s7.06M5.24G0.9416±1.20.8965±1.40.9221±1.50.9185±1.4102.633661
    1+递进重参卷积2.77M2.23G0.9514±1.20.9108±1.30.9402±1.40.9306±1.3124.153457
    2+双分支轻量块473.09K310.69M0.9566±1.10.9160±1.20.9448±1.30.9359±1.2149.343049
    3+两模态CT717.04K600.92M0.9609±1.10.9186±1.20.9486±1.20.9393±1.2141.503215
    4+两模态PET717.04K600.92M0.9595±1.10.9326±1.00.9507±1.10.9458±1.1143.133169
    5+3模态717.04K600.92M0.9652±0.90.9354±0.90.9558±1.00.9501±0.9142.353182
    6注意力814.39K673.06M0.9729±0.70.9476±0.80.9651±0.70.9603±0.7138.473238
    下载: 导出CSV

    表  2  不同模型在肺部肿瘤PET/CT多模态数据集上的对比结果

    检测模型参数量计算量精度召回率mAPF1FPS总时间(s)
    R-FCN(Res101-FPN)[2]50.80M60.51G0.8947±1.20.8839±1.40.9013±1.50.8893±1.415.338010
    SSD512(VGG16)[2]23.75M87.63G0.8467±1.60.8398±2.10.8540±2.10.8433±2.034.624133
    EfficientDet-d0[18]4.31M2.58G0.8934±1.30.8719±1.50.8962±1.70.8825±1.526.244474
    YOLOv4l[4]63.96M45.28G0.9374±1.10.8926±1.40.9162±1.50.9146±1.463.995516
    YOLOv5l[5]46.65M36.56G0.9495±1.10.8968±1.30.9307±1.30.9244±1.371.944968
    TPH-YOLOv5[19]40.83M36.26G0.9523±1.20.9142±1.30.9408±1.40.9329±1.338.786795
    PP-PicoDet-l[3]1.18M4.59G0.9342±1.40.8873±1.70.9131±1.80.9101±1.6109.553601
    NanoDet-Plus-m[20]1.19M1.20G0.9431±1.30.8987±1.60.9264±1.60.9204±1.5117.183435
    Poly-YOLO[4]6.16M7.01G0.9478±1.10.9101±1.40.9378±1.40.9286±1.369.124491
    YOLOv7l[5]37.19M33.64G0.9558±0.90.9237±1.20.9476±1.20.9395±1.273.814712
    YOLOv8l[19]43.63M52.93G0.9592±0.90.9287±1.10.9514±1.20.9437±1.156.125956
    CL-YOLOv50.81M0.67G0.9729±0.70.9476±0.80.9651±0.70.9603±0.7138.473238
    下载: 导出CSV

    表  3  多模态检测模型的对比结果

    检测模型精度召回率mAPF1
    ConvNet[7]0.94880.91970.9392±1.30.9340±1.3
    BIRANet[8]0.95190.92110.9417±1.30.9362±1.2
    MVDNet[9]0.95870.92820.9508±1.10.9432±1.0
    ProbEn[10]0.96230.93100.9543±0.90.9464±0.9
    CL-YOLOv50.97290.94760.9651±0.70.9603±0.7
    下载: 导出CSV
  • [1] MIRANDA D, THENKANIDIYOOR V, and DINESH D A. Review on approaches to concept detection in medical images[J]. Biocybernetics and Biomedical Engineering, 2022, 42(2): 453–462. doi: 10.1016/j.bbe.2022.02.012.
    [2] 周涛, 刘赟璨, 陆惠玲, 等. ResNet及其在医学图像处理领域的应用: 研究进展与挑战[J]. 电子与信息学报, 2022, 44(1): 149–167. doi: 10.11999/JEIT210914.

    ZHOU Tao, LIU Yuncan, LU Huiling, et al. ResNet and its application to medical image processing: Research progress and challenges[J]. Journal of Electronics &Information Technology, 2022, 44(1): 149–167. doi: 10.11999/JEIT210914.
    [3] YU Guanghua, CHANG Qinyao, LV Wengyu, et al. PP-PicoDet: A better real-time object detector on mobile devices[J]. arXiv: 2111.00902, 2021.
    [4] HURTIK P, MOLEK V, HULA J, et al. Poly-YOLO: Higher speed, more precise detection and instance segmentation for YOLOv3[J]. Neural Computing and Applications, 2022, 34(10): 8275–8290. doi: 10.1007/s00521-021-05978-9.
    [5] WANG C Y, BOCHKOVSKIY A, and LIAO H Y M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[J]. arXiv: 2207.02696, 2022.
    [6] 刘政怡, 段群涛, 石松, 等. 基于多模态特征融合监督的RGB-D图像显著性检测[J]. 电子与信息学报, 2020, 42(4): 997–1004. doi: 10.11999/JEIT190297.

    LIU Zhenyi, DUAN Quntao, SHI Song, et al. RGB-D image saliency detection based on multi-modal feature-fused supervision[J]. Journal of Electronics &Information Technology, 2020, 42(4): 997–1004. doi: 10.11999/JEIT190297.
    [7] ASVADI A, GARROTE L, PREMEBIDA C, et al. Real-time deep convnet-based vehicle detection using 3d-lidar reflection intensity data[C]. ROBOT 2017: Third Iberian Robotics Conference, Sevilla, Spain, 2017: 475–486.
    [8] YADAV R, VIERLING A, and BERNS K. Radar+ RGB fusion for robust object detection in autonomous vehicle[C]. The 2020 IEEE International Conference on Image Processing, Abu Dhabi, United Arab Emirates, 2020: 1986–1990.
    [9] QIAN Kun, ZHU Shilin, ZHANG Xinyu, et al. Robust multimodal vehicle detection in foggy weather using complementary Lidar and radar signals[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 444–453.
    [10] CHEN Yiting, SHI Jinghao, YE Zelin, et al. Multimodal object detection via probabilistic ensembling[C]. 17th European Conference on Computer Vision, Tel Aviv, Israel, 2022: 139–158.
    [11] HERMESSI H, MOURALI O, and ZAGROUBA E. Multimodal medical image fusion review: Theoretical background and recent advances[J]. Signal Processing, 2021, 183: 108036. doi: 10.1016/j.sigpro.2021.108036.
    [12] MOKNI R, GARGOURI N, DAMAK A, et al. An automatic computer-aided diagnosis system based on the multimodal fusion of breast cancer (MF-CAD)[J]. Biomedical Signal Processing and Control, 2021, 69: 102914. doi: 10.1016/j.bspc.2021.102914.
    [13] RUBINSTEIN E, SALHOV M, NIDAM-LESHEM M, et al. Unsupervised tumor detection in dynamic PET/CT imaging of the prostate[J]. Medical Image Analysis, 2019, 55: 27–40. doi: 10.1016/j.media.2019.04.001.
    [14] MING Yue, DONG Xiying, ZHAO Jihuai, et al. Deep learning-based multimodal image analysis for cervical cancer detection[J]. Methods, 2022, 205: 46–52. doi: 10.1016/j.ymeth.2022.05.004.
    [15] QIN Ruoxi, WANG Zhenzhen, JIANG Lingyun, et al. Fine-grained lung cancer classification from PET and CT images based on multidimensional attention mechanism[J]. Complexity, 2020, 2020: 6153657. doi: 10.1155/2020/6153657.
    [16] DIRKS I, KEYAERTS M, NEYNS B, et al. Computer-aided detection and segmentation of malignant melanoma lesions on whole-body 18F-FDG PET/CT using an interpretable deep learning approach[J]. Computer Methods and Programs in Biomedicine, 2022, 221: 106902. doi: 10.1016/j.cmpb.2022.106902.
    [17] CAO Siyuan, YU Beinan, LUO Lun, et al. PCNet: A structure similarity enhancement method for multispectral and multimodal image registration[J]. Information Fusion, 2023, 94: 200–214. doi: 10.1016/j.inffus.2023.02.004.
    [18] TAN Mingxing, PANG Ruoming, and LE Q V. EfficientDet: Scalable and efficient object detection[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, USA, 2020: 10778–10787.
    [19] LI Chuyi, LI Lulu, GENG Yifei, et al. YOLOv6 v3.0: A full-scale reloading[J]. arXiv: 2301.05586, 2023.
    [20] LI Dongyang and ZHAI Junyong. A real-time vehicle window positioning system based on nanodet[C]. 2022 Chinese Intelligent Systems Conference, Singapore, 2022: 697–705.
  • 加载中
图(10) / 表(3)
计量
  • 文章访问数:  510
  • HTML全文浏览量:  130
  • PDF下载量:  106
  • 被引次数: 0
出版历程
  • 收稿日期:  2023-02-14
  • 修回日期:  2023-05-05
  • 网络出版日期:  2023-05-16
  • 刊出日期:  2024-02-10

目录

    /

    返回文章
    返回