Hierarchical Prototype Learning with Shared Subspace Factorization for Generalizable Deepfake Detection

PENG Shufan; LU Tianliang; HE Chunhao; ZHANG Lu; ZHAO Kai

doi:10.11999/JEIT260426

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

PENG Shufan, LU Tianliang, HE Chunhao, ZHANG Lu, ZHAO Kai. Hierarchical Prototype Learning with Shared Subspace Factorization for Generalizable Deepfake Detection[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260426

Citation:

PENG Shufan, LU Tianliang, HE Chunhao, ZHANG Lu, ZHAO Kai. Hierarchical Prototype Learning with Shared Subspace Factorization for Generalizable Deepfake Detection[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260426

Citation:

PDF( 3893 KB)

Hierarchical Prototype Learning with Shared Subspace Factorization for Generalizable Deepfake Detection

doi: 10.11999/JEIT260426 cstr: 32379.14.JEIT260426

1.
School of Information Network Security, People’s Public Security University of China, Beijing 100038, China
2.
Beijing Key Laboratory of Artificial Intelligence and Smart Policing Applications, Beijing 100038, China

Funds: Major Program of the National Social Science Fund of China (25&ZD238)

Received Date: 2026-04-10
Accepted Date: 2026-08-10
Rev Recd Date: 2026-08-10

Available Online: 2026-08-15

Abstract

Abstract

Objective Deepfake detectors often exhibit substantial performance degradation when applied to unseen manipulation methods, cross-dataset distribution shifts, diffusion-generated faces, or common image degradations. This limitation is critical in forensic applications because the generation process, data source, and post-processing history of a questioned sample are usually unknown. Existing approaches often represent the entire fake class using a single feature center, despite the heterogeneous patterns produced by different generation methods, source datasets, and processing conditions. As a result, transferable forensic evidence may become entangled with mode-specific artifacts, weakening generalization to unknown domains. To address this issue, a hierarchical prototype learning framework with shared subspace factorization (HPL-SF) is proposed. The framework exploits within-class diversity to estimate a low-rank structure shared across latent fake modes and adaptively uses this structure for individual test samples. Methods A pretrained DINOv2 Vision Transformer (ViT-L/14) is used as the backbone. Its original parameters are frozen, and low-rank adaptation modules are inserted into the query and value mappings of the self-attention layers for parameter-efficient training. All features and prototypes are L2-normalized so that cosine similarity is used consistently for prototype assignment, binary classification, and test-time representation adaptation. HPL-SF comprises three successive stages (Fig. 1). First, one real prototype and multiple mode-specific fake prototypes are maintained in the normalized feature space. Each training sample is softly assigned to the fake prototypes according to its cosine similarity to each prototype, and the weighted prototypes are aggregated into a sample-adaptive fake representation. The resulting response distribution provides an observable representation of latent within-class modes without requiring forgery-source labels. A binary classification loss, a sample–prototype contrastive loss, and a prototype-diversity loss are jointly minimized to separate real and fake samples, improve sample–prototype alignment, and prevent the fake prototypes from collapsing into a single direction. Second, the normalized mode-specific fake prototypes are arranged as a prototype matrix. Singular value decomposition is applied to this matrix, and the largest gap between consecutive singular values determines the dimension of the shared fake subspace. Each fake prototype is decomposed into a projection within this subspace and an orthogonal residual. Only the shared projections are aggregated to update the shared fake prototype, whereas the residuals retain mode-dependent information and are excluded from this update. The real and shared fake prototypes are updated using exponential moving averages. Gradients are not propagated through subspace construction, dimension selection, or the updates of these semantic prototypes. Third, the backbone, adaptation modules, prototypes, and subspace basis are fixed during inference. A test feature is compared with the real and shared fake prototypes, and their relative responses determine a sample-specific adaptation weight. The original feature is blended with its projection onto the shared subspace, normalized again, and classified according to its cosine similarities to the two semantic prototypes. Thus, the estimated shared structure can be exploited on a per-sample basis without updating model parameters during testing. Results and Discussions The method is evaluated under cross-dataset, cross-forgery-type, diffusion-forgery, repeated-run, image-degradation, ablation, and mechanism-analysis protocols. Across seven unseen datasets, HPL-SF achieves the highest area under the receiver operating characteristic curve (AUC) on every dataset and an average AUC of 91.67%, exceeding the second-highest average by 2.10 percentage points (Table 1). When trained on FaceForensics++ and directly evaluated on the Diffusion Facial Forgery dataset, HPL-SF obtains the highest AUC on the text-to-image, image-to-image, face-swapping, and face-editing subsets, with an average AUC of 84.39% (Table 2). In four cross-forgery-type settings on FaceForensics++, the average accuracy and AUC reach 85.29% and 92.41%, respectively, both ranking first among the compared methods. In the high-quality-compression setting where DeepFakes is held out for testing, HPL-SF trails the best method by only 0.91 percentage points in accuracy and 0.39 percentage points in AUC (Table 3). Repeated experiments with multiple random seeds yield the highest mean values for all four aggregate metrics, with standard deviations ranging from 0.38 to 0.62 percentage points (Table 4). Under five severity levels of compression, blur, and noise on the Deepfake Detection Challenge Preview dataset, HPL-SF achieves the best or joint-best AUC at most severity levels and remains comparatively stable under moderate and severe degradations (Fig. 2). All six ablated variants perform worse than the complete model on the four unseen test sets, indicating that mode-specific prototype learning, sample–prototype contrast, prototype diversity, shared subspace factorization, and test-time representation adaptation make complementary contributions (Fig. 3). Performance generally improves as the number of mode-specific fake prototypes increases and begins to plateau when the number reaches 10. The largest spectral gap occurs between the fifth and sixth singular values; accordingly, the shared dimension is set to 5. With this setting, HPL-SF attains a diffusion-forgery AUC of 84.39% ± 0.62%, compared with 79.68% for direct mean aggregation and 81.56% ± 0.80% without test-time representation adaptation (Figs. 4(a)–4(c)). Linear-probe results further show that the shared component provides stronger discrimination in unknown domains while retaining less domain-identifying information than the mode-specific residual, supporting the intended separation of transferable and domain-related cues (Fig. 4(d)). The t-distributed stochastic neighbor embedding visualization reveals clearer real–fake separation and closer same-class distributions across data sources (Fig. 5). Prototype-allocation statistics indicate differentiated yet balanced use of the fake prototypes. The adaptation weights are higher for fake samples, particularly for correctly detected fake samples, whereas misclassified samples lie closer to the balanced-response line (Fig. 6). The remaining false positives are mainly associated with low resolution, compression, filters, or occlusion, whereas false negatives usually contain weak or high-quality forgery traces (Fig. 7). Conclusions HPL-SF organizes generalizable deepfake detection as a progression from observing within-class variation to estimating and exploiting shared structure. The experimental evidence indicates that separating shared projections from mode-dependent residuals provides transferable decision information under dataset shifts, unseen forgery types, diffusion-generated manipulations, and common image degradations. The framework requires no parameter updates during testing and adaptively exploits the estimated shared structure based on each sample’s relative responses to the real and shared fake prototypes. Nevertheless, errors remain when degradations in real images resemble forgery artifacts or when high-quality forgeries contain only weak traces. Future work will extend the framework to temporal prototypes for video, multimodal forensic evidence, and adaptive discrimination under broader open-world conditions.
- Deepfake detection,
- Generalizable deepfake detection,
- Hierarchical prototype learning,
- Shared subspace factorization,
- Test-time representation adaptation

FullText(HTML)

References(30)

References

[1]	YAN Zhiyuan, ZHANG Yong, FAN Yanbo, et al. UCF: Uncovering common features for generalizable deepfake detection[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023: 22355–22366. doi: 10.1109/ICCV51070.2023.02048.
[2]	SHIOHARA K and YAMASAKI T. Detecting deepfakes with self-blended images[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 18699–18708. doi: 10.1109/CVPR52688.2022.01816.
[3]	王艳, 孙钦东, 荣东柱, 等. 伪影间共性机理驱动的多域感知社交网络深度伪造视频检测[J]. 电子与信息学报, 2024, 46(9): 3713–3721. doi: 10.11999/JEIT240025. WANG Yan, SUN Qindong, RONG Dongzhu, et al. Deepfake video detection on social networks using multi-domain aware driven by common mechanism analysis between artifacts[J]. Journal of Electronics & Information Technology, 2024, 46(9): 3713–3721. doi: 10.11999/JEIT240025.
[4]	KOUTLIS C and PAPADOPOULOS S. Leveraging representations from intermediate encoder-blocks for synthetic image detection[C]. Proceedings of 18th European Conference on Computer Vision – ECCV 2024, Milan, Italy, 2024: 394–411. doi: 10.1007/978-3-031-73220-1_23.
[5]	YANG Yongqi, QIAN Zhihao, ZHU Ye, et al. D³: Scaling up deepfake detection by learning from discrepancy[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2025: 23850–23859. doi: 10.1109/CVPR52734.2025.02221.
[6]	TRINH L, TSANG M, RAMBHATLA S, et al. Interpretable and trustworthy deepfake detection via dynamic prototypes[C]. Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), Waikoloa, USA, 2021: 1972–1982. doi: 10.1109/WACV48630.2021.00202.
[7]	ZOU Zheng, PENG Dunlu, ZHAO Yu, et al. TTP-AP: Test-time projection of augmented prototypes for generalized deepfake detection[J]. Knowledge-Based Systems, 2025, 330: 114710. doi: 10.1016/j.knosys.2025.114710.
[8]	徐延杰, 孙浩, 林秦杰, 等. 残差子空间原型学习约束的SAR目标类增量识别[J]. 电子与信息学报, 2025, 47(12): 4838–4850. doi: 10.11999/JEIT251007. XU Yanjie, SUN Hao, LIN Qinjie, et al. Residual subspace prototype constraint for SAR target class-incremental recognition[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4838–4850. doi: 10.11999/JEIT251007.
[9]	OQUAB M, DARCET T, MOUTAKANNI T, et al. DINOv2: Learning robust visual features without supervision[J]. Transactions on Machine Learning Research, 2024. (查阅网上资料, 未找到本条文献卷期和页码信息, 请确认).
[10]	孙昆阳, 姚睿, 祝汉城, 等. 一种测试时间自适应的夜间图像辅助波束预测方法[J]. 电子与信息学报, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530. SUN Kunyang, YAO Rui, ZHU Hancheng, et al. A test-time adaptive method for nighttime image-aided beam prediction[J]. Journal of Electronics & Information Technology, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530.
[11]	RÖSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: Learning to detect manipulated facial images[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019: 1–11. doi: 10.1109/ICCV.2019.00009.
[12]	LI Yuezun, YANG Xin, SUN Pu, et al. Celeb-DF: A large-scale challenging dataset for deepfake forensics[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 3204–3213. doi: 10.1109/CVPR42600.2020.00327.
[13]	ZI Bojia, CHANG Minghao, CHEN Jingjing, et al. WildDeepfake: A challenging real-world dataset for deepfake detection[C]. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, USA, 2020: 2382–2390. doi: 10.1145/3394171.3413769.
[14]	JIANG Liming, LI Ren, WU Wayne, et al. DeeperForensics-1.0: A large-scale dataset for real-world face forgery detection[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 2886–2895. doi: 10.1109/CVPR42600.2020.00296.
[15]	CHENG H, GUO Yangyang, WANG Tianyi, et al. Diffusion facial forgery detection[C]. Proceedings of the 32nd ACM International Conference on Multimedia, Melbourne, Australia, 2024: 5939–5948. doi: 10.1145/3664647.3680797.
[16]	JIANG Yuming, HUANG Ziqi, PAN Xingang, et al. Talk-to-Edit: Fine-grained facial editing via dialog[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, Canada, 2021: 13779–13788. doi: 10.1109/ICCV48922.2021.01354.
[17]	ZHUANG Wanyi, CHU Qi, TAN Zhentao, et al. UIA-ViT: Unsupervised inconsistency-aware method based on vision transformer for face forgery detection[C]. Proceedings of 17th European Conference on Computer Vision – ECCV 2022, Tel Aviv, Israel, 2022: 391–407. doi: 10.1007/978-3-031-20065-6_23.
[18]	MIAO Changtao, TAN Zichang, CHU Qi, et al. F²Trans: High-frequency fine-grained transformer for face forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 1039–1051. doi: 10.1109/TIFS.2022.3233774.
[19]	XU Yuting, LIANG Jian, JIA Gengyun, et al. TALL: Thumbnail layout for deepfake video detection[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Paris, France, 2023: 22601–22611. doi: 10.1109/ICCV51070.2023.02071.
[20]	LUO Anwei, CAI Rizhao, KONG Chenqi, et al. Forgery-aware adaptive learning with vision transformer for generalized face forgery detection[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35(5): 4116–4129. doi: 10.1109/TCSVT.2024.3522091.
[21]	CAO Junyi, ZHANG Keyue, YAO Taiping, et al. Towards unified defense for face forgery and spoofing attacks via dual space reconstruction learning[J]. International Journal of Computer Vision, 2024, 132(12): 5862–5887. doi: 10.1007/S11263-024-02151-2.
[22]	SUN Ke, YAO Taiping, CHEN Shen, et al. Dual contrastive learning for general face forgery detection[C]. Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022: 2316–2324. doi: 10.1609/aaai.v36i2.20130. (查阅网上资料,未找到本条文献出版地信息,请确认).
[23]	YAN Zhiyuan, LUO Yuhao, LYU Siwei, et al. Transcending forgery specificity with latent space augmentation for generalizable deepfake detection[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 8984–8994. doi: 10.1109/CVPR52733.2024.00858.
[24]	LUO Anwei, KONG Chenqi, HUANG Jiwu, et al. Beyond the prior forgery knowledge: Mining critical clues for general face forgery detection[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 1168–1182. doi: 10.1109/TIFS.2023.3332218.
[25]	TAN Mingxing and LE Q. EfficientNet: Rethinking model scaling for convolutional neural networks[C]. Proceedings of the 36th International Conference on Machine Learning, Long Beach, USA, 2019: 6105–6114.
[26]	LI Da, YANG Yongxin, SONG Yizhe, et al. Learning to generalize: Meta-learning for domain generalization[C]. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, USA, 2018: 3490–3497. doi: 10.1609/aaai.v32i1.11596.
[27]	SUN Ke, LIU Hong, YE Qixiang, et al. Domain general face forgery detection by learning to weight[C]. Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021: 2638–2646. doi: 10.1609/aaai.v35i3.16367. (查阅网上资料,未找到本条文献出版地信息,请确认).
[28]	WANG Junke, WU Zuxuan, OUYANG Wenhao, et al. M2TR: Multi-modal multi-scale transformers for deepfake detection[C]. Proceedings of the 2022 International Conference on Multimedia Retrieval, Newark, USA, 2022: 615–623. doi: 10.1145/3512527.3531415.
[29]	QIAN Yuyang, YIN Guojun, SHENG Lu, et al. Thinking in frequency: Face forgery detection by mining frequency-aware clues[C]. Proceedings of 16th European Conference on Computer Vision – ECCV 2020, Glasgow, UK, 2020: 86–103. doi: 10.1007/978-3-030-58610-2_6.
[30]	WU Haiwei, ZHOU Jiantao, TIAN Jinyu, et al. Robust image forgery detection over online social network shared images[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 13430–13439. doi: 10.1109/CVPR52688.2022.01308.