面向深度伪造泛化检测的层次化原型学习与共享子空间分解方法

彭舒凡; 芦天亮; 和椿皓; 张璐; 赵凯

doi:10.11999/JEIT260426

面向深度伪造泛化检测的层次化原型学习与共享子空间分解方法

doi: 10.11999/JEIT260426 cstr: 32379.14.JEIT260426

1.
中国人民公安大学信息网络安全学院北京 100038
2.
人工智能与智慧警务应用北京市重点实验室北京 100038

基金项目: 国家社会科学基金重大项目(25&ZD238)

详细信息

作者简介:
彭舒凡：男，博士生，研究方向为多媒体取证，邮箱　2020211623@stu.ppsuc.edu.cn

芦天亮：男，教授，研究方向为多媒体取证、人工智能安全，邮箱　lutianliang@ppsuc.edu.cn

和椿皓：男，博士生，研究方向为多媒体取证

张璐：男，博士生，研究方向为多媒体取证

赵凯：男，博士生，研究方向为多媒体取证

通讯作者:
芦天亮　lutianliang@ppsuc.edu.cn

中图分类号: TN911.73; TP309.2
计量
- 文章访问数: 16
- HTML全文浏览量: 3
- PDF下载量: 0
- 被引次数: 0
出版历程
- 收稿日期: 2026-04-10
- 修回日期: 2026-08-10
- 录用日期: 2026-08-10
- 网络出版日期: 2026-08-15

Hierarchical Prototype Learning with Shared Subspace Factorization for Generalizable Deepfake Detection

1.
School of Information Network Security, People’s Public Security University of China, Beijing 100038, China
2.
Beijing Key Laboratory of Artificial Intelligence and Smart Policing Applications, Beijing 100038, China

Funds: Major Program of the National Social Science Fund of China (25&ZD238)

摘要

摘要: 深度伪造检测模型在未知伪造方法、跨数据集和复杂退化场景中易受分布偏移影响。训练数据中的伪造样本既包含由生成方式、数据来源和后处理条件引起的模式差异，也包含跨潜在模式稳定存在的共享判别结构。针对二者难以协同建模的问题，提出层次化原型学习与共享子空间分解方法(HPL-SF)。该方法依次完成类内差异观测、共享结构分解和样本级结构利用。训练阶段，多个特异伪造原型在真伪二分类监督下形成差异化响应，以表征伪造类内部的潜在子模式。随后，从原型矩阵的主导低秩方向中分解共享伪造子空间，并聚合各原型的共享投影，构造共性伪造原型。推理阶段，固定真实原型、共性伪造原型和共享伪造子空间，并根据测试样本对两个语义原型的相对响应，自适应融合原始表征与共享投影表征。在跨数据集、跨伪造类型和扩散人脸伪造评估中，受试者工作特征曲线下面积的平均值分别达到91.67%、92.41%和84.39%。原型分配、共享—特异成分和测试时权重分析进一步支持了上述三个阶段之间的递进关系。
- 深度伪造检测 /
- 深度伪造泛化检测 /
- 层次化原型学习 /
- 共享子空间分解 /
- 测试时表征适配
Abstract: Objective Deepfake detectors often exhibit substantial performance degradation when applied to unseen manipulation methods, cross-dataset distribution shifts, diffusion-generated faces, or common image degradations. This limitation is critical in forensic applications because the generation process, data source, and post-processing history of a questioned sample are usually unknown. Existing approaches often represent the entire fake class using a single feature center, despite the heterogeneous patterns produced by different generation methods, source datasets, and processing conditions. As a result, transferable forensic evidence may become entangled with mode-specific artifacts, weakening generalization to unknown domains. To address this issue, a hierarchical prototype learning framework with shared subspace factorization (HPL-SF) is proposed. The framework exploits within-class diversity to estimate a low-rank structure shared across latent fake modes and adaptively uses this structure for individual test samples. Methods A pretrained DINOv2 Vision Transformer (ViT-L/14) is used as the backbone. Its original parameters are frozen, and low-rank adaptation modules are inserted into the query and value mappings of the self-attention layers for parameter-efficient training. All features and prototypes are L2-normalized so that cosine similarity is used consistently for prototype assignment, binary classification, and test-time representation adaptation. HPL-SF comprises three successive stages (Fig. 1). First, one real prototype and multiple mode-specific fake prototypes are maintained in the normalized feature space. Each training sample is softly assigned to the fake prototypes according to its cosine similarity to each prototype, and the weighted prototypes are aggregated into a sample-adaptive fake representation. The resulting response distribution provides an observable representation of latent within-class modes without requiring forgery-source labels. A binary classification loss, a sample–prototype contrastive loss, and a prototype-diversity loss are jointly minimized to separate real and fake samples, improve sample–prototype alignment, and prevent the fake prototypes from collapsing into a single direction. Second, the normalized mode-specific fake prototypes are arranged as a prototype matrix. Singular value decomposition is applied to this matrix, and the largest gap between consecutive singular values determines the dimension of the shared fake subspace. Each fake prototype is decomposed into a projection within this subspace and an orthogonal residual. Only the shared projections are aggregated to update the shared fake prototype, whereas the residuals retain mode-dependent information and are excluded from this update. The real and shared fake prototypes are updated using exponential moving averages. Gradients are not propagated through subspace construction, dimension selection, or the updates of these semantic prototypes. Third, the backbone, adaptation modules, prototypes, and subspace basis are fixed during inference. A test feature is compared with the real and shared fake prototypes, and their relative responses determine a sample-specific adaptation weight. The original feature is blended with its projection onto the shared subspace, normalized again, and classified according to its cosine similarities to the two semantic prototypes. Thus, the estimated shared structure can be exploited on a per-sample basis without updating model parameters during testing. Results and Discussions The method is evaluated under cross-dataset, cross-forgery-type, diffusion-forgery, repeated-run, image-degradation, ablation, and mechanism-analysis protocols. Across seven unseen datasets, HPL-SF achieves the highest area under the receiver operating characteristic curve (AUC) on every dataset and an average AUC of 91.67%, exceeding the second-highest average by 2.10 percentage points (Table 1). When trained on FaceForensics++ and directly evaluated on the Diffusion Facial Forgery dataset, HPL-SF obtains the highest AUC on the text-to-image, image-to-image, face-swapping, and face-editing subsets, with an average AUC of 84.39% (Table 2). In four cross-forgery-type settings on FaceForensics++, the average accuracy and AUC reach 85.29% and 92.41%, respectively, both ranking first among the compared methods. In the high-quality-compression setting where DeepFakes is held out for testing, HPL-SF trails the best method by only 0.91 percentage points in accuracy and 0.39 percentage points in AUC (Table 3). Repeated experiments with multiple random seeds yield the highest mean values for all four aggregate metrics, with standard deviations ranging from 0.38 to 0.62 percentage points (Table 4). Under five severity levels of compression, blur, and noise on the Deepfake Detection Challenge Preview dataset, HPL-SF achieves the best or joint-best AUC at most severity levels and remains comparatively stable under moderate and severe degradations (Fig. 2). All six ablated variants perform worse than the complete model on the four unseen test sets, indicating that mode-specific prototype learning, sample–prototype contrast, prototype diversity, shared subspace factorization, and test-time representation adaptation make complementary contributions (Fig. 3). Performance generally improves as the number of mode-specific fake prototypes increases and begins to plateau when the number reaches 10. The largest spectral gap occurs between the fifth and sixth singular values; accordingly, the shared dimension is set to 5. With this setting, HPL-SF attains a diffusion-forgery AUC of 84.39% ± 0.62%, compared with 79.68% for direct mean aggregation and 81.56% ± 0.80% without test-time representation adaptation (Figs. 4(a)–4(c)). Linear-probe results further show that the shared component provides stronger discrimination in unknown domains while retaining less domain-identifying information than the mode-specific residual, supporting the intended separation of transferable and domain-related cues (Fig. 4(d)). The t-distributed stochastic neighbor embedding visualization reveals clearer real–fake separation and closer same-class distributions across data sources (Fig. 5). Prototype-allocation statistics indicate differentiated yet balanced use of the fake prototypes. The adaptation weights are higher for fake samples, particularly for correctly detected fake samples, whereas misclassified samples lie closer to the balanced-response line (Fig. 6). The remaining false positives are mainly associated with low resolution, compression, filters, or occlusion, whereas false negatives usually contain weak or high-quality forgery traces (Fig. 7). Conclusions HPL-SF organizes generalizable deepfake detection as a progression from observing within-class variation to estimating and exploiting shared structure. The experimental evidence indicates that separating shared projections from mode-dependent residuals provides transferable decision information under dataset shifts, unseen forgery types, diffusion-generated manipulations, and common image degradations. The framework requires no parameter updates during testing and adaptively exploits the estimated shared structure based on each sample’s relative responses to the real and shared fake prototypes. Nevertheless, errors remain when degradations in real images resemble forgery artifacts or when high-quality forgeries contain only weak traces. Future work will extend the framework to temporal prototypes for video, multimodal forensic evidence, and adaptive discrimination under broader open-world conditions.
- Deepfake detection /
- Generalizable deepfake detection /
- Hierarchical prototype learning /
- Shared subspace factorization /
- Test-time representation adaptation

HTML全文

图 1 本文方法的整体框架

下载: 全尺寸图片幻灯片

图 2 不同方法在压缩、模糊和噪声退化条件下的AUC比较

下载: 全尺寸图片幻灯片

图 3 不同模块的消融实验结果

下载: 全尺寸图片幻灯片

图 4 参数与共享子空间分析

下载: 全尺寸图片幻灯片

图 5 不同方法特征分布的t-SNE可视化结果

下载: 全尺寸图片幻灯片

图 6 原型分配与测试时表征适配行为分析

下载: 全尺寸图片幻灯片

图 7 HPL-SF的典型失败案例

下载: 全尺寸图片幻灯片

表 4 统一数据与评测协议下可复现方法的多随机种子比较

方法	跨数据集平均AUC	跨伪造类型平均ACC	跨伪造类型平均AUC	DiFF平均AUC
UCF†^[1]	86.36±0.65	82.56±0.78	88.67±0.62	79.71±0.90
RINE†^[4]	86.94±0.52	82.66±0.61	89.19±0.55	80.52±0.78
D³†^[5]	89.45±0.58	80.33±0.70	87.10±0.66	80.08±0.82
DINOv2+LoRA	84.22±0.48	80.10±0.62	86.96±0.56	74.83±0.85
HPL-SF	91.67±0.38	85.29±0.48	92.41±0.42	84.39±0.62

下载: 导出CSV

表 1 七个未见数据集上的跨数据集评估结果

方法	发表来源	CDF	WDF	DFDC-P	DFDC	DFD	DFR	FFIW	平均结果
UIA-ViT^[17]	ECCV 2022	88.85	-	79.54	-	-	-	-	-
F2Trans^[18]	TIFS 2023	89.87	-	-	76.15	-	-	-	-
TALL^[19]	ICCV 2023	86.58	-	-	74.10	-	-	-	-
FA-ViT^[20]	TCSVT 2025	93.83	84.32	85.41	78.32	94.88	98.01	92.22	89.57
DSRL^[21]	IJCV 2024	84.56	77.31	75.64	77.99	92.32	97.23	88.69	84.82
DCL*^[22]	AAAI 2022	88.24	76.21	77.96	74.19	92.62	95.94	85.28	84.35
SBI*^[2]	CVPR 2022	84.54	70.54	85.75	75.91	80.28	85.56	84.65	81.03
LSDA*^[23]	CVPR 2024	92.59	77.12	84.06	76.62	94.65	91.08	86.28	86.06
CFM*^[24]	TIFS 2023	90.79	82.23	81.12	71.81	94.79	97.42	85.89	86.29
UCF†^[1]	ICCV 2023	91.78	80.57	77.13	78.46	88.37	95.46	92.75	86.36
RINE†^[4]	ECCV 2024	87.29	86.74	80.03	76.58	90.61	94.47	92.85	86.94
D³†^[5]	CVPR 2025	92.38	85.65	84.32	80.41	93.09	96.73	93.56	89.45
DINOv2+LoRA	Baseline	86.73	80.43	79.10	75.12	88.32	91.51	88.34	84.22
HPL-SF	Ours	94.83	88.01	87.34	83.47	95.62	98.14	94.29	91.67

下载: 导出CSV

表 3 FF++上的跨伪造类型评估结果

方法	留出DF (HQ)		留出DF (LQ)		留出F2F (HQ)		留出F2F (LQ)		平均结果
方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
EfficientNet^[25]	82.40	91.11	67.60	75.30	63.32	80.10	61.41	67.40	68.68	78.48
MLGD^[26]	84.21	91.82	67.15	73.12	63.46	77.10	58.12	61.70	68.24	75.94
LTW^[27]	85.60	92.70	69.15	75.60	65.60	80.20	65.70	72.40	71.51	80.23
DCL^[22]	87.70	94.90	75.90	83.82	68.40	82.93	67.85	75.07	74.96	84.18
M2TR^[28]	81.07	94.91	74.29	84.85	55.71	76.99	66.43	71.70	69.38	82.11
F3Net^[29]	83.57	94.95	77.50	85.77	61.07	81.20	64.64	73.70	71.70	83.91
F2Trans^[18]	92.86	98.92	82.14	88.77	86.07	94.08	70.36	77.73	82.86	89.88
FA-ViT^[20]	92.86	98.10	81.28	87.89	82.57	91.20	67.91	76.84	81.16	88.51
UCF†^[1]	94.59	98.35	78.74	85.39	86.02	94.95	70.87	75.97	82.56	88.67
RINE†^[4]	93.06	98.32	80.77	86.39	85.24	92.81	71.58	79.23	82.66	89.19
D³†^[5]	92.47	98.03	77.89	86.25	81.52	89.17	69.43	74.96	80.33	87.10
DINOv2+LoRA	90.84	95.76	79.38	86.37	81.64	89.28	68.55	76.42	80.10	86.96
HPL-SF	93.68	98.53	85.69	92.46	87.26	95.15	74.52	83.50	85.29	92.41

下载: 导出CSV

表 2 DiFF数据集上的评估结果

方法	训练集	测试子集				平均结果
方法	训练集	T2I	I2I	FS	FE	平均结果
DINOv2+LoRA	FF++	79.48	72.81	82.68	64.36	74.83
UCF†^[1]		82.95	81.38	86.05	68.44	79.71
RINE†^[4]		84.62	79.18	88.74	69.52	80.52
D³†^[5]		83.08	80.54	86.72	69.98	80.08
HPL-SF		88.44	85.56	90.17	73.39	84.39

下载: 导出CSV

参考文献(30)

[1]	YAN Zhiyuan, ZHANG Yong, FAN Yanbo, et al. UCF: Uncovering common features for generalizable deepfake detection[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023: 22355–22366. doi: 10.1109/ICCV51070.2023.02048.
[2]	SHIOHARA K and YAMASAKI T. Detecting deepfakes with self-blended images[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 18699–18708. doi: 10.1109/CVPR52688.2022.01816.
[3]	王艳, 孙钦东, 荣东柱, 等. 伪影间共性机理驱动的多域感知社交网络深度伪造视频检测[J]. 电子与信息学报, 2024, 46(9): 3713–3721. doi: 10.11999/JEIT240025. WANG Yan, SUN Qindong, RONG Dongzhu, et al. Deepfake video detection on social networks using multi-domain aware driven by common mechanism analysis between artifacts[J]. Journal of Electronics & Information Technology, 2024, 46(9): 3713–3721. doi: 10.11999/JEIT240025.
[4]	KOUTLIS C and PAPADOPOULOS S. Leveraging representations from intermediate encoder-blocks for synthetic image detection[C]. Proceedings of 18th European Conference on Computer Vision – ECCV 2024, Milan, Italy, 2024: 394–411. doi: 10.1007/978-3-031-73220-1_23.
[5]	YANG Yongqi, QIAN Zhihao, ZHU Ye, et al. D³: Scaling up deepfake detection by learning from discrepancy[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2025: 23850–23859. doi: 10.1109/CVPR52734.2025.02221.
[6]	TRINH L, TSANG M, RAMBHATLA S, et al. Interpretable and trustworthy deepfake detection via dynamic prototypes[C]. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, USA, 2021: 1972–1982. doi: 10.1109/WACV48630.2021.00202.
[7]	ZOU Zheng, PENG Dunlu, ZHAO Yu, et al. TTP-AP: Test-time projection of augmented prototypes for generalized deepfake detection[J]. Knowledge-Based Systems, 2025, 330: 114710. doi: 10.1016/j.knosys.2025.114710.
[8]	徐延杰, 孙浩, 林秦杰, 等. 残差子空间原型学习约束的SAR目标类增量识别[J]. 电子与信息学报, 2025, 47(12): 4838–4850. doi: 10.11999/JEIT251007. XU Yanjie, SUN Hao, LIN Qinjie, et al. Residual subspace prototype constraint for SAR target class-incremental recognition[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4838–4850. doi: 10.11999/JEIT251007.
[9]	OQUAB M, DARCET T, MOUTAKANNI T, et al. DINOv2: Learning robust visual features without supervision[J]. Transactions on Machine Learning Research, 2024. (查阅网上资料, 未找到本条文献卷期和页码信息, 请确认).
[10]	孙昆阳, 姚睿, 祝汉城, 等. 一种测试时间自适应的夜间图像辅助波束预测方法[J]. 电子与信息学报, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530. SUN Kunyang, YAO Rui, ZHU Hancheng, et al. A test-time adaptive method for nighttime image-aided beam prediction[J]. Journal of Electronics & Information Technology, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530.
[11]	RÖSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: Learning to detect manipulated facial images[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019: 1–11. doi: 10.1109/ICCV.2019.00009.
[12]	LI Yuezun, YANG Xin, SUN Pu, et al. Celeb-DF: A large-scale challenging dataset for deepfake forensics[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 3204–3213. doi: 10.1109/CVPR42600.2020.00327.
[13]	ZI Bojia, CHANG Minghao, CHEN Jingjing, et al. WildDeepfake: A challenging real-world dataset for deepfake detection[C]. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, USA, 2020: 2382–2390. doi: 10.1145/3394171.3413769.
[14]	JIANG Liming, LI Ren, WU Wayne, et al. DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 2886–2895. doi: 10.1109/CVPR42600.2020.00296.
[15]	CHENG H, GUO Yangyang, WANG Tianyi, et al. Diffusion facial forgery detection[C]. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia, 2024: 5939–5948. doi: 10.1145/3664647.3680797.
[16]	JIANG Yuming, HUANG Ziqi, PAN Xingang, et al. Talk-to-Edit: Fine-grained facial editing via dialog[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada, 2021: 13779–13788. doi: 10.1109/ICCV48922.2021.01354.
[17]	ZHUANG Wanyi, CHU Qi, TAN Zhentao, et al. UIA-ViT: Unsupervised inconsistency-aware method based on vision transformer for face forgery detection[C]. Proceedings of 17th European Conference on Computer Vision – ECCV 2022, Tel Aviv, Israel, 2022: 391–407. doi: 10.1007/978-3-031-20065-6_23.
[18]	MIAO Changtao, TAN Zichang, CHU Qi, et al. F²Trans: High-frequency fine-grained transformer for face forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 1039–1051. doi: 10.1109/TIFS.2022.3233774.
[19]	XU Yuting, LIANG Jian, JIA Gengyun, et al. TALL: Thumbnail layout for deepfake video detection[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023: 22601–22611. doi: 10.1109/ICCV51070.2023.02071.
[20]	LUO Anwei, CAI Rizhao, KONG Chenqi, et al. Forgery-aware adaptive learning with vision transformer for generalized face forgery detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35(5): 4116–4129. doi: 10.1109/TCSVT.2024.3522091.
[21]	CAO Junyi, ZHANG Keyue, YAO Taiping, et al. Towards unified defense for face forgery and spoofing attacks via dual space reconstruction learning[J]. International Journal of Computer Vision, 2024, 132(12): 5862–5887. doi: 10.1007/S11263-024-02151-2.
[22]	SUN Ke, YAO Taiping, CHEN Shen, et al. Dual contrastive learning for general face forgery detection[C]. Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022: 2316–2324. doi: 10.1609/aaai.v36i2.20130. (查阅网上资料,未找到本条文献出版地信息,请确认).
[23]	YAN Zhiyuan, LUO Yuhao, LYU Siwei, et al. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 8984–8994. doi: 10.1109/CVPR52733.2024.00858.
[24]	LUO Anwei, KONG Chenqi, HUANG Jiwu, et al. Beyond the prior forgery knowledge: Mining critical clues for general face forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 1168–1182. doi: 10.1109/TIFS.2023.3332218.
[25]	TAN Mingxing and LE Q. EfficientNet: Rethinking model scaling for convolutional neural networks[C]. Proceedings of the 36th International Conference on Machine Learning, Long Beach, USA, 2019: 6105–6114.
[26]	LI Da, YANG Yongxin, SONG Yizhe, et al. Learning to generalize: Meta-learning for domain generalization[C]. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, USA, 2018: 3490–3497. doi: 10.1609/aaai.v32i1.11596.
[27]	SUN Ke, LIU Hong, YE Qixiang, et al. Domain general face forgery detection by learning to weight[C]. Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021: 2638–2646. doi: 10.1609/aaai.v35i3.16367. (查阅网上资料,未找到本条文献出版地信息,请确认).
[28]	WANG Junke, WU Zuxuan, OUYANG Wenhao, et al. M2TR: Multi-modal multi-scale transformers for deepfake detection[C]. Proceedings of the 2022 International Conference on Multimedia Retrieval, Newark, USA, 2022: 615–623. doi: 10.1145/3512527.3531415.
[29]	QIAN Yuyang, YIN Guojun, SHENG Lu, et al. Thinking in frequency: Face forgery detection by mining frequency-aware clues[C]. Proceedings of 16th European Conference on Computer Vision – ECCV 2020, Glasgow, UK, 2020: 86–103. doi: 10.1007/978-3-030-58610-2_6.
[30]	WU Haiwei, ZHOU Jiantao, TIAN Jinyu, et al. Robust image forgery detection over online social network shared images[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 13430–13439. doi: 10.1109/CVPR52688.2022.01308.