ByteHOI：码流域人-物交互识别基础模型

李方成; 吴科君; 刘琼; 杨铀

doi:10.11999/JEIT261062

ByteHOI：码流域人-物交互识别基础模型

doi: 10.11999/JEIT261062 cstr: 32379.14.JEIT261062

华中科技大学电子信息与通信学院武汉 430074

基金项目: 国家自然科学基金(62501246)

详细信息

作者简介:
李方成：男，硕士生，研究方向为多模态大模型应用，邮箱　lifangcheng2002@163.com

吴科君：男，讲师，研究方向为多模态大模型、字节语言模型等，邮箱　wukejun_hust@163.com

刘琼：女，教授，研究方向为视频智能编码、三维立体视频等

杨铀：男，教授，研究方向为计算成像、具身智能、图像智能处理等

通讯作者:
吴科君　wukejun_hust@163.com

中图分类号: TP391.41
计量
- 文章访问数: 10
- HTML全文浏览量: 0
- PDF下载量: 0
- 被引次数: 0
出版历程
- 修回日期: 2026-10-16
- 录用日期: 2026-10-16
- 网络出版日期: 2026-10-29

ByteHOI: Human-Object Interaction Recognition Foundation Model in Bitstream Domain

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China

Funds: The National Natural Science Foundation of China (62501246)

摘要

摘要: 人物交互(Human-Object Interaction, HOI)识别旨在识别图像中人与物体之间的交互关系，现有方法通常需要将压缩图像解码为像素后再进行识别，而显式像素重建可能增加隐私敏感场景中的内容暴露风险。为此，该文提出码流域人物交互识别(Bitstream HOI Recognition, BHOIR)，旨在直接从压缩图像的原始字节序列中识别交互关系，模型推理过程中无需解码和重建像素。针对该任务，该文提出ByteHOI基础模型框架，直接对图像码流进行建模理解。为缓解交互类别组合空间庞大及长尾分布带来的学习困难，ByteHOI引入动作-物体组合解耦分类，将交互预测分解为动作与物体两个语义子空间，并通过组合得到最终的交互类别得分；进一步设计字节-文本语义对齐分类器，利用可学习投影将码流特征映射至预训练视觉-语言模型的文本语义空间，并基于固定语义原型进行余弦相似度分类。实验结果表明，ByteHOI在HICO和V-COCO数据集上的默认mAP分别达到27.34%和21.38%，并在精确率、召回率和F1值等指标上显著优于对比方法。此外，ByteHOI在受损码流下仍展现出较强的识别鲁棒性，进一步验证了码流域人物交互识别的有效性，为隐私敏感场景下的应用提供了新的思路。
- 人物交互识别 /
- 码流域理解 /
- 隐私保护
Abstract: Objective Human-Object Interaction (HOI) recognition aims to identify semantic interactions between people and objects and is an important task in high-level scene understanding, with applications in behavior analysis, human-machine interaction, assisted driving, and intelligent monitoring. Existing HOI methods are generally developed in the pixel domain, where compressed images must first be decoded before feature extraction and interaction prediction. This explicit pixel reconstruction may increase the exposure of sensitive visual content in privacy-sensitive scenarios and also makes recognition dependent on successful decoding. Since images are commonly stored and transmitted as compressed bitstreams, it is meaningful to investigate whether high-level interaction semantics can be recognized directly from bitstreams without reconstructing pixels. To this end, the Bitstream Human-Object Interaction Recognition (BHOIR) task is formulated, which directly takes the raw byte sequence of a compressed image as input and predicts interaction categories without pixel decoding or reconstruction. BHOIR is more challenging than conventional byte-level image classification because HOI categories have a large compositional label space and a severe long-tailed distribution. For example, HICO contains 600 interaction categories composed of 117 actions and 80 objects, with many combinations having limited training samples. Moreover, bitstream features do not explicitly correspond to high-level concepts such as actions and objects. Therefore, BHOIR requires effective exploitation of HOI compositional structure and enhanced semantic modeling of byte-level representations. Methods A bitstream-domain HOI recognition framework named ByteHOI is developed for BHOIR, consisting of bitstream feature extraction and interaction classification (Fig. 1). ByteFormer-Tiny is adopted as the encoder. Each JPEG image is represented by its raw byte sequence and truncated or zero-padded to 50,000 bytes. The bytes are converted into embeddings, reduced by a one-dimensional convolutional module, combined with positional embeddings, and processed by 12 window-based Transformer blocks. Global average pooling then produces a 192-dimensional bitstream representation without pixel decoding or reconstruction. Two components are introduced for interaction classification. First, action-object compositional decoupling decomposes HOI prediction into separate action and object classification branches, and interaction scores are reconstructed according to predefined action-object combinations. This enables categories sharing the same action or object to share supervision and alleviates the sparsity of rare interaction combinations. Second, a byte-text semantic alignment classifier introduces fixed semantic prototypes generated by the pretrained CLIP ViT-B/32 text encoder. Action and object prompts, such as “a person riding” and “a horse”, are encoded offline and L2-normalized. Two learnable projection networks map the 192-dimensional bitstream feature into separate 512-dimensional action and object semantic spaces. Classification logits are obtained from cosine similarities between projected features and fixed text prototypes, with a temperature coefficient of 0.01. Binary cross-entropy losses are applied to both branches, and final HOI scores are obtained by combining the corresponding action and object predictions. ByteHOI is trained for 50 epochs using AdamW with a batch size of 64. The initial learning rates of the bitstream encoder and classification module are set to $ 1\times {10}^{-5} $ and $ 5\times {10}^{-4} $, respectively. Results and Discussions Experiments are conducted on the HICO and V-COCO datasets to evaluate the effectiveness and robustness of ByteHOI. General byte-level models, including MambaByte, MBLM, MegaByte, and bGPT, achieve Default mAP values below 2% on HICO, showing that generic byte-sequence modeling is insufficient for complex interaction recognition. ByteFormer, which benefits from visual byte-level pretraining, achieves 16.95% Default mAP, whereas ByteHOI improves the performance to 27.34%. Consistent improvements are observed across low-, medium-, and high-frequency interaction categories, particularly in the low-frequency group (Table 1). On V-COCO, the Default mAP is improved from 16.71% to 21.38%, and consistent gains are also obtained in Macro-Precision, Macro-Recall, and Macro-F1 on both datasets, demonstrating the cross-dataset effectiveness of the proposed framework (Table 2). Ablation experiments further confirm the contributions of the two key designs. On HICO, action-object compositional decoupling increases the Default mAP from 16.95% to 24.81%, while the introduction of CLIP semantic knowledge and cosine-based byte-text alignment further improves the performance to 27.34%. Similar trends are observed on V-COCO (Tables 3 and 4). Robustness is further evaluated by randomly flipping bits in the compressed data segment. Although some pixel-domain models achieve higher recognition accuracy on intact inputs, their performance decreases sharply as bitstream corruption increases because damaged files may suffer severe visual degradation or fail to decode. In contrast, ByteHOI maintains relatively stable performance. At a bit error rate of 0.01, it still achieves 24.82% and 19.72% mAP on HICO and V-COCO, respectively, corresponding to decreases of only 2.52 and 1.66 percentage points from the intact-bitstream setting (Fig. 2). Additional experiments show that ByteHOI is relatively insensitive to different JPEG encoder implementations and file-header modifications, whereas substantial changes in JPEG quality factor lead to more evident performance degradation, indicating that compression quality has a greater influence on bitstream distribution (Tables 5 and 6; Fig. 3). Conclusions The BHOIR task is formulated to investigate whether complex human-object interactions can be recognized directly from compressed image bitstreams without reconstructing pixel images during model inference. ByteHOI is developed to address two major difficulties in this setting. Action-object compositional decoupling is used to exploit the inherent compositional structure of HOI labels and alleviate the learning difficulty caused by sparse and long-tailed interaction categories, while byte-text semantic alignment is introduced to establish explicit associations between bitstream features and high-level action and object semantics using fixed CLIP text prototypes. Experimental results on HICO and V-COCO demonstrate that ByteHOI consistently outperforms the compared byte-level methods and achieves Default mAP values of 27.34% and 21.38%, respectively. The effectiveness of the proposed components is further supported by ablation studies. Bitstream corruption experiments show that ByteHOI maintains relatively stable recognition performance even when corrupted files become difficult or impossible to decode through conventional pixel-domain pipelines. By avoiding explicit reconstruction of visible pixel images during recognition, the proposed method reduces the exposure of pixel-level visual content in the recognition pipeline while preserving task-related interaction semantics. This privacy-related benefit is limited to the recognition process and does not constitute encryption or a formal privacy guarantee. The current study focuses on JPEG bitstreams. Future work will extend bitstream-domain HOI recognition to other image formats, including PNG, WebP, and AVIF, and will investigate unified bitstream representations with stronger adaptability across different encoding formats.
- Human-Object Interaction (HOI) recognition /
- Bitstream domain understanding /
- Privacy preservation

HTML全文

图 1 ByteHOI基础模型整体框架

下载: 全尺寸图片幻灯片

图 2 码流随机比特损坏下不同模型的识别性能对比：(a)HICO(b)V-COCO

下载: 全尺寸图片幻灯片

图 3 不同编码质量因子下ByteHOI在HICO和V-COCO数据集上的识别性能

下载: 全尺寸图片幻灯片

表 1 不同方法在HICO码流数据集上的主要实验结果

模型	默认mAP(%)				已知目标mAP(%)
模型	低频类	中频类	高频类	总体	低频类	中频类	高频类	总体
MambaByte	0.31	0.79	1.72	0.94	0.87	1.92	3.72	2.17
MBLM	0.22	0.51	1.31	0.68	0.58	1.36	3.79	1.91
MegaByte	0.43	0.92	2.01	1.12	1.12	2.28	4.28	2.56
bGPT	0.67	1.08	2.48	1.41	1.52	3.02	6.50	3.68
ByteFormer	3.53	10.73	36.61	16.95	26.96	35.57	69.21	43.92
ByteHOI	14.20	21.76	46.05	27.34	31.11	39.80	71.45	47.45

下载: 导出CSV

表 2 ByteFormer与ByteHOI在HICO和V-COCO数据集上的多指标比较

数据集	模型	默认mAP(%)	已知目标mAP(%)	Macro-P(%)	Macro-R (%)	Macro-F1 (%)
HICO	ByteFormer	16.95	43.92	15.51	5.54	7.18
HICO	ByteHOI	27.34	47.45	28.53	9.95	13.25
V-COCO	ByteFormer	16.71	37.94	20.57	8.43	10.72
V-COCO	ByteHOI	21.38	38.67	24.87	11.70	14.11

下载: 导出CSV

表 3 ByteHOI在HICO数据集上的组件消融结果

变体	组合解耦	CLIP语义	余弦对齐	默认 mAP(%)	已知目标mAP(%)	Macro-P(%)	Macro-R (%)	Macro-F1 (%)
(a)	×	×	×	16.95	43.92	15.51	5.54	7.18
(b)	√	×	×	24.81	45.23	26.06	8.50	11.44
(c)	√	√	×	25.12	45.65	26.91	8.80	11.88
(d)	√	√	√	27.34	47.45	28.53	9.95	13.25

下载: 导出CSV

表 4 ByteHOI在V-COCO数据集上的组件消融结果

变体	组合解耦	CLIP语义	余弦对齐	默认mAP(%)	已知目标mAP(%)	Macro-P(%)	Macro-R (%)	Macro-F1 (%)
(a)	×	×	×	16.71	37.94	20.57	8.43	10.72
(b)	√	×	×	19.35	38.16	20.08	8.52	10.45
(c)	√	√	×	20.68	38.22	22.38	9.54	11.63
(d)	√	√	√	21.38	38.67	24.87	11.70	14.11

下载: 导出CSV

表 5 不同编码器对ByteHOI识别性能的影响

JPEG编码器	默认mAP(%)	已知目标mAP(%)	Macro-P(%)	Macro-R(%)	Macro-F1(%)
Pillow	27.34	47.45	28.53	9.95	13.25
OpenCV	27.46	47.49	28.71	9.99	13.30
libjpeg-turbo	27.46	47.49	28.71	9.99	13.30

下载: 导出CSV

表 6 文件头信息对ByteHOI识别性能的影响

数据集	文件结构	默认mAP (%)	已知目标mAP (%)	Macro-P (%)	Macro-R (%)	Macro-F1 (%)
HICO	原始码流	27.34	47.45	28.53	9.95	13.25
HICO	文件头置零	27.15	47.38	28.10	10.01	13.30
HICO	去除文件头	26.44	46.70	28.80	9.85	13.25
V-COCO	原始码流	21.38	38.67	24.87	11.70	14.11
V-COCO	文件头置零	21.39	38.70	24.44	11.38	13.64
V-COCO	去除文件头	20.80	38.75	24.29	10.83	12.96

下载: 导出CSV

参考文献(18)

[1]	HAN Geng, ZHAO Jiachen, ZHANG Lele, et al. A survey of human-object interaction detection with deep learning[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2025, 9(1): 3–26. doi: 10.1109/TETCI.2024.3518613.
[2]	袁霖, 武雁尚, 张力元, 等. 面向人脸识别可信应用的隐私保护计算研究综述[J]. 电子与信息学报, 2026, 48(4): 1549–1568. doi: 10.11999/JEIT251063. YUAN Lin, WU Yanshang, ZHANG Liyuan, et al. Privacy-preserving computation in trustworthy face recognition: A comprehensive survey[J]. Journal of Electronics & Information Technology, 2026, 48(4): 1549–1568. doi: 10.11999/JEIT251063.
[3]	HORTON M, MEHTA S, FARHADI A, et al. Bytes are all you need: Transformers operating directly on file bytes[J]. Transactions on Machine Learning Research, 2024, 2024. (查阅网上资料, 未找到对应的期页码信息, 请确认).
[4]	孙中华, 吴双, 贾克斌, 等. 基于对比学习的动作识别研究综述[J]. 电子与信息学报, 2025, 47(8): 2473–2485. doi: 10.11999/JEIT250131. SUN Zhonghua, WU Shuang, JIA Kebin, et al. A review on action recognition based on contrastive learning[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2473–2485. doi: 10.11999/JEIT250131.
[5]	TAMURA M, OHASHI H, and YOSHINAGA T. QPIC: Query-based pairwise human-object interaction detection with image-wide contextual information[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 10410–10419. doi: 10.1109/CVPR46437.2021.01027.
[6]	KIM B, LEE J, KANG J, et al. HOTR: End-to-end human-object interaction detection with transformers[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 74–83. doi: 10.1109/CVPR46437.2021.00014.
[7]	KIM S, JUNG D, and CHO M. Locality-aware zero-shot human-object interaction detection[C]. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2025: 20190–20200. doi: 10.1109/CVPR52734.2025.01880.
[8]	YU Lili, SIMIG D, FLAHERTY C, et al. MEGABYTE: Predicting million-byte sequences with multiscale transformers[C]. Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, USA, 2023: 3447.
[9]	WANG Junxiong, GANGAVARAPU T, YAN J N, et al. MambaByte: Token-free selective state space model[C]. Proceedings of the 1st Conference on Language Modeling, Philadelphia, PA, USA, 2024.
[10]	EGLI E, MANICA M, and BORN J. Multiscale byte language models a hierarchical architecture for causal million-length sequence modeling[EB/OL]. https://arxiv.org/abs/2502.14553, 2025.
[11]	WU Shangda, TAN Xu, WANG Zili, et al. Beyond language models: Byte models are digital world simulators[EB/OL]. https://arxiv.org/abs/2402.19155, 2024.
[12]	武辉, 赵妍, 张培溶, 等. 视觉-语言基础模型驱动的遥感图文检索研究进展[J]. 电子与信息学报, 2026, 48(8): 3229–3246. doi: 10.11999/JEIT260189. WU Hui, ZHAO Yan, ZHANG Peirong, et al. Recent advances in remote sensing image-text retrieval driven by vision-language foundation models[J]. Journal of Electronics & Information Technology, 2026, 48(8): 3229–3246. doi: 10.11999/JEIT260189.
[13]	LIN Lin, ZHANG Sihao, FU Song, et al. FD-LLM: Large language model for fault diagnosis of complex equipment[J]. Advanced Engineering Informatics, 2025, 65: 103208. doi: 10.1016/j.aei.2025.103208.
[14]	WU Kejun, WANG Shen, GAO Jianjun, et al. LLMArk: Instance-aware foundation model for flood risk assessment[J]. Information Fusion, 2026, 135: 104440. doi: 10.1016/j.inffus.2026.104440.
[15]	LAN Xiang, WU Feng, HE Kai, et al. GEM: Empowering MLLM for grounded ECG understanding with time series and images[C]. Proceedings of the 39th Conference on Neural Information Processing Systems, San Diego, USA, 2025: 104720–104754. doi: 10.52202/085713-3158.
[16]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.
[17]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]. Proceedings of the 9th International Conference on Learning Representations, 2021. (查阅网上资料, 未找到对应的出版地信息, 请确认).
[18]	ZHANG Aixi, LIAO Yue, LIU Si, et al. Mining the benefits of two-stage and one-stage HOI detection[C]. Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021: 17209–17220. (查阅网上资料, 未找到对应的出版地信息, 请确认).