面向深度伪造泛化检测的层次化原型学习与共享子空间分解方法

彭舒凡; 芦天亮; 和椿皓; 张璐; 赵凯

doi:10.11999/JEIT260426

面向深度伪造泛化检测的层次化原型学习与共享子空间分解方法

doi: 10.11999/JEIT260426 cstr: 32379.14.JEIT260426

1.
中国人民公安大学信息网络安全学院北京 100038
2.
人工智能与智慧警务应用北京市重点实验室北京 100038

基金项目: 国家社会科学基金重大项目(25&ZD238)

详细信息

作者简介:
彭舒凡：男，博士生，研究方向为多媒体取证，邮箱　2020211623@stu.ppsuc.edu.cn

芦天亮：男，教授，研究方向为多媒体取证、人工智能安全，邮箱　lutianliang@ppsuc.edu.cn

和椿皓：男，博士生，研究方向为多媒体取证

张璐：男，博士生，研究方向为多媒体取证

赵凯：男，博士生，研究方向为多媒体取证

通讯作者:
芦天亮　lutianliang@ppsuc.edu.cn

中图分类号: TN911.73; TP309.2
计量
- 文章访问数: 190
- HTML全文浏览量: 46
- PDF下载量: 11
- 被引次数: 0
出版历程
- 收稿日期: 2026-04-10
- 修回日期: 2026-08-10
- 录用日期: 2026-08-10
- 网络出版日期: 2026-08-15

Hierarchical Prototype Learning with Shared Subspace Factorization for Generalizable Deepfake Detection

1.
School of Information Network Security, People’s Public Security University of China, Beijing 100038, China
2.
Beijing Key Laboratory of Artificial Intelligence and Smart Policing Applications, Beijing 100038, China

Funds: Major Program of the National Social Science Fund of China (25&ZD238)

摘要

摘要: 深度伪造检测模型在未知伪造方法、跨数据集和复杂退化场景中易受分布偏移影响。训练数据中的伪造样本既包含由生成方式、数据来源和后处理条件引起的模式差异，也包含跨潜在模式稳定存在的共享判别结构。针对二者难以协同建模的问题，提出层次化原型学习与共享子空间分解方法(HPL-SF)。该方法依次完成类内差异观测、共享结构分解和样本级结构利用。训练阶段，多个特异伪造原型在真伪二分类监督下形成差异化响应，以表征伪造类内部的潜在子模式。随后，从原型矩阵的主导低秩方向中分解共享伪造子空间，并聚合各原型的共享投影，构造共性伪造原型。推理阶段，固定真实原型、共性伪造原型和共享伪造子空间，并根据测试样本对两个语义原型的相对响应，自适应融合原始表征与共享投影表征。在跨数据集、跨伪造类型和扩散人脸伪造评估中，受试者工作特征曲线下面积的平均值分别达到91.67%、92.41%和84.39%。原型分配、共享—特异成分和测试时权重分析进一步支持了上述三个阶段之间的递进关系。
- 深度伪造检测 /
- 深度伪造泛化检测 /
- 层次化原型学习 /
- 共享子空间分解 /
- 测试时表征适配
Abstract: Objective Deepfake detectors often exhibit performance degradation when applied to unseen manipulation methods, cross-dataset distribution shifts, diffusion-generated faces, and common image degradations. In forensic applications, the generation process, data source, and post-processing conditions of a questioned sample are usually unknown. Existing methods often represent the fake class with a single feature center, although different generation methods, data sources, and processing conditions produce heterogeneous patterns. Transferable forensic cues may therefore be mixed with mode-specific artifacts, limiting generalization to unknown domains. To address this problem, a hierarchical prototype learning framework with shared subspace factorization (HPL-SF) is proposed. Within-class diversity is first used to observe latent fake modes, followed by estimation of a low-rank structure shared across these modes and sample-level exploitation of the shared structure during inference. Methods A pretrained DINOv2 Vision Transformer (ViT-L/14) is used as the backbone. Its original parameters are frozen, and Low-Rank Adaptation (LoRA) modules are inserted into the query and value mappings of the self-attention layers for parameter-efficient training. All features and prototypes are L2-normalized, and cosine similarity is used for prototype assignment, binary classification, and test-time representation adaptation. HPL-SF consists of three successive stages (Fig. 1). First, one real prototype and multiple mode-specific fake prototypes are maintained in the normalized feature space. Each training sample is softly assigned to the fake prototypes according to its cosine similarity to each prototype, and the weighted prototypes are aggregated to obtain a sample-adaptive fake representation. The response distribution thus provides an observable representation of latent within-class modes without using forgery-source labels. A binary classification loss, a sample–prototype contrastive loss, and a prototype diversity loss are jointly optimized to separate real and fake samples, improve sample–prototype alignment, and prevent prototype collapse. Second, the normalized mode-specific fake prototypes are arranged into a prototype matrix. Singular Value Decomposition (SVD) is applied to this matrix, and the largest gap between consecutive singular values is used to determine the dimension of the shared fake subspace. Each fake prototype is then decomposed into a projection onto the shared fake subspace and a mode-specific residual. Only the shared projections are aggregated to update the shared fake prototype, whereas the residuals retain mode-specific information and are excluded from this update. The real prototype and shared fake prototype are updated using Exponential Moving Average (EMA). Gradients are not propagated through subspace construction, dimension selection, or semantic prototype updates. Third, the backbone, LoRA modules, prototypes, and subspace basis are fixed during inference. A test feature is compared with the real and shared fake prototypes, and their relative responses determine a sample-specific adaptation weight. The original feature is blended with its projection onto the shared fake subspace and then L2-normalized. The resulting feature is classified according to its cosine similarities to the two semantic prototypes. Thus, the estimated shared structure is exploited on a sample-by-sample basis without updating model parameters during testing. Results and Discussions The method is evaluated under cross-dataset, cross-forgery-type, diffusion-forgery, repeated-run, image-degradation, ablation, and mechanism-analysis protocols. Across seven unseen datasets, HPL-SF achieves the highest area under the receiver operating characteristic curve (AUC) on all seven datasets, with an average AUC of 91.67%, which is 2.10 percentage points higher than the second-highest average (Table 1). When trained on FaceForensics++ and evaluated on the Diffusion Facial Forgery dataset, HPL-SF achieves the highest AUC on the text-to-image, image-to-image, face-swapping, and face-editing subsets, with an average AUC of 84.39% (Table 2). Across four cross-forgery-type settings on FaceForensics++, the average accuracy and AUC reach 85.29% and 92.41%, respectively, both ranking first among the compared methods. When DeepFakes under high-quality compression is held out for testing, HPL-SF trails the best-performing method by only 0.91 percentage points in accuracy and 0.39 percentage points in AUC (Table 3). Repeated experiments with multiple random seeds yield the highest mean values for all four aggregate metrics, with standard deviations ranging from 0.38 to 0.62 percentage points (Table 4). Under five severity levels of compression, blur, and noise on the Deepfake Detection Challenge Preview dataset, HPL-SF achieves the best or joint-best AUC at most severity levels and remains relatively stable under moderate and severe degradations (Fig. 2). All six ablated variants perform worse than the complete HPL-SF model on the four unseen test sets, indicating complementary contributions from mode-specific prototype learning, sample–prototype contrastive loss, prototype diversity loss, shared subspace factorization, and test-time representation adaptation (Fig. 3). Performance generally increases as the number of mode-specific fake prototypes increases and plateaus when the number reaches 10. The largest spectral gap occurs between the fifth and sixth singular values, yielding a shared subspace dimension of 5. With this setting, HPL-SF achieves a diffusion-forgery AUC of 84.39% ± 0.62%, compared with 79.68% for direct mean aggregation and 81.56% ± 0.80% without test-time representation adaptation (Figs. 4(a)–4(c)). Linear-probe analysis further shows that the shared component provides stronger discrimination in unknown domains and weaker domain-identifying capability than the mode-specific residual, supporting the separation of transferable and domain-related cues (Fig. 4(d)). The t-distributed stochastic neighbor embedding (t-SNE) visualization shows clearer real–fake separation and closer distributions of same-class samples from different data sources (Fig. 5). Prototype allocation statistics show differentiated responses across fake types, with prototype usage remaining close to the uniform baseline. The adaptation weights are higher for fake samples, particularly for correctly classified fake samples, whereas misclassified samples show responses closer to the balanced-response line (Fig. 6). False positives are mainly associated with low resolution, compression, filters, or occlusion, whereas false negatives usually contain high-quality forgeries or weak forgery traces (Fig. 7). Conclusions HPL-SF organizes generalizable deepfake detection into three successive stages: observation of within-class diversity, estimation of shared structure, and sample-level exploitation of that structure. Experimental results show that separating shared projections from mode-specific residuals provides transferable discriminative cues under dataset shifts, unseen forgery types, diffusion-generated manipulations, and common image degradations. The framework requires no parameter updates during testing and adaptively exploits the shared structure according to each sample’s relative responses to the real and shared fake prototypes. Errors remain for degraded real images whose artifacts resemble forgery traces and for high-quality forgeries with weak traces. Future work will extend the framework to temporal prototypes for video, multimodal forensic evidence, and adaptive discrimination in broader open-world settings.
- Deepfake detection /
- Generalizable deepfake detection /
- Hierarchical prototype learning /
- Shared subspace factorization /
- Test-time representation adaptation

HTML全文

图 1 本文方法的整体框架

下载: 全尺寸图片幻灯片

图 2 不同方法在压缩、模糊和噪声退化条件下的AUC比较

下载: 全尺寸图片幻灯片

图 3 不同模块的消融实验结果

下载: 全尺寸图片幻灯片

图 4 参数与共享子空间分析

下载: 全尺寸图片幻灯片

图 5 不同方法特征分布的t-SNE可视化结果

下载: 全尺寸图片幻灯片

图 6 原型分配与测试时表征适配行为分析

下载: 全尺寸图片幻灯片

图 7 HPL-SF的典型失败案例

下载: 全尺寸图片幻灯片

表 4 统一数据与评测协议下可复现方法的多随机种子比较

方法	跨数据集平均AUC	跨伪造类型平均ACC	跨伪造类型平均AUC	DiFF平均AUC
UCF†^[1]	86.36±0.65	82.56±0.78	88.67±0.62	79.71±0.90
RINE†^[4]	86.94±0.52	82.66±0.61	89.19±0.55	80.52±0.78
D³†^[5]	89.45±0.58	80.33±0.70	87.10±0.66	80.08±0.82
DINOv2+LoRA	84.22±0.48	80.10±0.62	86.96±0.56	74.83±0.85
HPL-SF	91.67±0.38	85.29±0.48	92.41±0.42	84.39±0.62

下载: 导出CSV

表 1 七个未见数据集上的跨数据集评估结果

方法	发表来源	CDF	WDF	DFDC-P	DFDC	DFD	DFR	FFIW	平均结果
UIA-ViT^[17]	ECCV 2022	88.85	-	79.54	-	-	-	-	-
F2Trans^[18]	TIFS 2023	89.87	-	-	76.15	-	-	-	-
TALL^[19]	ICCV 2023	86.58	-	-	74.10	-	-	-	-
FA-ViT^[20]	TCSVT 2025	93.83	84.32	85.41	78.32	94.88	98.01	92.22	89.57
DSRL^[21]	IJCV 2024	84.56	77.31	75.64	77.99	92.32	97.23	88.69	84.82
DCL*^[22]	AAAI 2022	88.24	76.21	77.96	74.19	92.62	95.94	85.28	84.35
SBI*^[2]	CVPR 2022	84.54	70.54	85.75	75.91	80.28	85.56	84.65	81.03
LSDA*^[23]	CVPR 2024	92.59	77.12	84.06	76.62	94.65	91.08	86.28	86.06
CFM*^[24]	TIFS 2023	90.79	82.23	81.12	71.81	94.79	97.42	85.89	86.29
UCF†^[1]	ICCV 2023	91.78	80.57	77.13	78.46	88.37	95.46	92.75	86.36
RINE†^[4]	ECCV 2024	87.29	86.74	80.03	76.58	90.61	94.47	92.85	86.94
D³†^[5]	CVPR 2025	92.38	85.65	84.32	80.41	93.09	96.73	93.56	89.45
DINOv2+LoRA	Baseline	86.73	80.43	79.10	75.12	88.32	91.51	88.34	84.22
HPL-SF	Ours	94.83	88.01	87.34	83.47	95.62	98.14	94.29	91.67

下载: 导出CSV

表 3 FF++上的跨伪造类型评估结果

方法	留出DF (HQ)		留出DF (LQ)		留出F2F (HQ)		留出F2F (LQ)		平均结果
方法	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC	ACC	AUC
EfficientNet^[25]	82.40	91.11	67.60	75.30	63.32	80.10	61.41	67.40	68.68	78.48
MLGD^[26]	84.21	91.82	67.15	73.12	63.46	77.10	58.12	61.70	68.24	75.94
LTW^[27]	85.60	92.70	69.15	75.60	65.60	80.20	65.70	72.40	71.51	80.23
DCL^[22]	87.70	94.90	75.90	83.82	68.40	82.93	67.85	75.07	74.96	84.18
M2TR^[28]	81.07	94.91	74.29	84.85	55.71	76.99	66.43	71.70	69.38	82.11
F3Net^[29]	83.57	94.95	77.50	85.77	61.07	81.20	64.64	73.70	71.70	83.91
F2Trans^[18]	92.86	98.92	82.14	88.77	86.07	94.08	70.36	77.73	82.86	89.88
FA-ViT^[20]	92.86	98.10	81.28	87.89	82.57	91.20	67.91	76.84	81.16	88.51
UCF†^[1]	94.59	98.35	78.74	85.39	86.02	94.95	70.87	75.97	82.56	88.67
RINE†^[4]	93.06	98.32	80.77	86.39	85.24	92.81	71.58	79.23	82.66	89.19
D³†^[5]	92.47	98.03	77.89	86.25	81.52	89.17	69.43	74.96	80.33	87.10
DINOv2+LoRA	90.84	95.76	79.38	86.37	81.64	89.28	68.55	76.42	80.10	86.96
HPL-SF	93.68	98.53	85.69	92.46	87.26	95.15	74.52	83.50	85.29	92.41

下载: 导出CSV

表 2 DiFF数据集上的评估结果

方法	训练集	测试子集				平均结果
方法	训练集	T2I	I2I	FS	FE	平均结果
DINOv2+LoRA	FF++	79.48	72.81	82.68	64.36	74.83
UCF†^[1]		82.95	81.38	86.05	68.44	79.71
RINE†^[4]		84.62	79.18	88.74	69.52	80.52
D³†^[5]		83.08	80.54	86.72	69.98	80.08
HPL-SF		88.44	85.56	90.17	73.39	84.39

下载: 导出CSV

参考文献(30)

[1]	YAN Zhiyuan, ZHANG Yong, FAN Yanbo, et al. UCF: Uncovering common features for generalizable deepfake detection[C]. The IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023: 22355–22366. doi: 10.1109/ICCV51070.2023.02048.
[2]	SHIOHARA K and YAMASAKI T. Detecting deepfakes with self-blended images[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 18699–18708. doi: 10.1109/CVPR52688.2022.01816.
[3]	王艳, 孙钦东, 荣东柱, 等. 伪影间共性机理驱动的多域感知社交网络深度伪造视频检测[J]. 电子与信息学报, 2024, 46(9): 3713–3721. doi: 10.11999/JEIT240025. WANG Yan, SUN Qindong, RONG Dongzhu, et al. Deepfake video detection on social networks using multi-domain aware driven by common mechanism analysis between artifacts[J]. Journal of Electronics & Information Technology, 2024, 46(9): 3713–3721. doi: 10.11999/JEIT240025.
[4]	KOUTLIS C and PAPADOPOULOS S. Leveraging representations from intermediate encoder-blocks for synthetic image detection[C]. The 18th European Conference on Computer Vision – ECCV 2024, Milan, Italy, 2024: 394–411. doi: 10.1007/978-3-031-73220-1_23.
[5]	YANG Yongqi, QIAN Zhihao, ZHU Ye, et al. D³: Scaling up deepfake detection by learning from discrepancy[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2025: 23850–23859. doi: 10.1109/CVPR52734.2025.02221.
[6]	TRINH L, TSANG M, RAMBHATLA S, et al. Interpretable and trustworthy deepfake detection via dynamic prototypes[C]. The IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, USA, 2021: 1972–1982. doi: 10.1109/WACV48630.2021.00202.
[7]	ZOU Zheng, PENG Dunlu, ZHAO Yu, et al. TTP-AP: Test-time projection of augmented prototypes for generalized deepfake detection[J]. Knowledge-Based Systems, 2025, 330: 114710. doi: 10.1016/j.knosys.2025.114710.
[8]	徐延杰, 孙浩, 林秦杰, 等. 残差子空间原型学习约束的SAR目标类增量识别[J]. 电子与信息学报, 2025, 47(12): 4838–4850. doi: 10.11999/JEIT251007. XU Yanjie, SUN Hao, LIN Qinjie, et al. Residual subspace prototype constraint for SAR target class-incremental recognition[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4838–4850. doi: 10.11999/JEIT251007.
[9]	OQUAB M, DARCET T, MOUTAKANNI T, et al. DINOv2: Learning robust visual features without supervision[J]. Transactions on Machine Learning Research, 2024.
[10]	孙昆阳, 姚睿, 祝汉城, 等. 一种测试时间自适应的夜间图像辅助波束预测方法[J]. 电子与信息学报, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530. SUN Kunyang, YAO Rui, ZHU Hancheng, et al. A test-time adaptive method for nighttime image-aided beam prediction[J]. Journal of Electronics & Information Technology, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530.
[11]	RÖSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: Learning to detect manipulated facial images[C]. The IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019: 1–11. doi: 10.1109/ICCV.2019.00009.
[12]	LI Yuezun, YANG Xin, SUN Pu, et al. Celeb-DF: A large-scale challenging dataset for deepfake forensics[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 3204–3213. doi: 10.1109/CVPR42600.2020.00327.
[13]	ZI Bojia, CHANG Minghao, CHEN Jingjing, et al. WildDeepfake: A challenging real-world dataset for deepfake detection[C]. The 28th ACM International Conference on Multimedia, Seattle, USA, 2020: 2382–2390. doi: 10.1145/3394171.3413769.
[14]	JIANG Liming, LI Ren, WU Wayne, et al. DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 2886–2895. doi: 10.1109/CVPR42600.2020.00296.
[15]	CHENG H, GUO Yangyang, WANG Tianyi, et al. Diffusion facial forgery detection[C]. The 32nd ACM International Conference on Multimedia, Melbourne, Australia, 2024: 5939–5948. doi: 10.1145/3664647.3680797.
[16]	JIANG Yuming, HUANG Ziqi, PAN Xingang, et al. Talk-to-Edit: Fine-grained facial editing via dialog[C]. The IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada, 2021: 13779–13788. doi: 10.1109/ICCV48922.2021.01354.
[17]	ZHUANG Wanyi, CHU Qi, TAN Zhentao, et al. UIA-ViT: Unsupervised inconsistency-aware method based on vision transformer for face forgery detection[C]. The 17th European Conference on Computer Vision – ECCV 2022, Tel Aviv, Israel, 2022: 391–407. doi: 10.1007/978-3-031-20065-6_23.
[18]	MIAO Changtao, TAN Zichang, CHU Qi, et al. F²Trans: High-frequency fine-grained transformer for face forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 1039–1051. doi: 10.1109/TIFS.2022.3233774.
[19]	XU Yuting, LIANG Jian, JIA Gengyun, et al. TALL: Thumbnail layout for deepfake video detection[C]. The IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023: 22601–22611. doi: 10.1109/ICCV51070.2023.02071.
[20]	LUO Anwei, CAI Rizhao, KONG Chenqi, et al. Forgery-aware adaptive learning with vision transformer for generalized face forgery detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35(5): 4116–4129. doi: 10.1109/TCSVT.2024.3522091.
[21]	CAO Junyi, ZHANG Keyue, YAO Taiping, et al. Towards unified defense for face forgery and spoofing attacks via dual space reconstruction learning[J]. International Journal of Computer Vision, 2024, 132(12): 5862–5887. doi: 10.1007/S11263-024-02151-2.
[22]	SUN Ke, YAO Taiping, CHEN Shen, et al. Dual contrastive learning for general face forgery detection[C]. The Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022: 2316–2324. doi: 10.1609/aaai.v36i2.20130.
[23]	YAN Zhiyuan, LUO Yuhao, LYU Siwei, et al. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 8984–8994. doi: 10.1109/CVPR52733.2024.00858.
[24]	LUO Anwei, KONG Chenqi, HUANG Jiwu, et al. Beyond the prior forgery knowledge: Mining critical clues for general face forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 1168–1182. doi: 10.1109/TIFS.2023.3332218.
[25]	TAN Mingxing and LE Q. EfficientNet: Rethinking model scaling for convolutional neural networks[C]. The 36th International Conference on Machine Learning, Long Beach, USA, 2019: 6105–6114.
[26]	LI Da, YANG Yongxin, SONG Yizhe, et al. Learning to generalize: Meta-learning for domain generalization[C]. The Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, USA, 2018: 3490–3497. doi: 10.1609/aaai.v32i1.11596.
[27]	SUN Ke, LIU Hong, YE Qixiang, et al. Domain general face forgery detection by learning to weight[C]. The Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021: 2638–2646. doi: 10.1609/aaai.v35i3.16367.
[28]	WANG Junke, WU Zuxuan, OUYANG Wenhao, et al. M2TR: Multi-modal multi-scale transformers for deepfake detection[C]. The 2022 International Conference on Multimedia Retrieval, Newark, USA, 2022: 615–623. doi: 10.1145/3512527.3531415.
[29]	QIAN Yuyang, YIN Guojun, SHENG Lu, et al. Thinking in frequency: Face forgery detection by mining frequency-aware clues[C]. The 16th European Conference on Computer Vision – ECCV 2020, Glasgow, UK, 2020: 86–103. doi: 10.1007/978-3-030-58610-2_6.
[30]	WU Haiwei, ZHOU Jiantao, TIAN Jinyu, et al. Robust image forgery detection over online social network shared images[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 13430–13439. doi: 10.1109/CVPR52688.2022.01308.