ByteHOI: Human-Object Interaction Recognition Foundation Model in Bitstream Domain

LI Fangcheng; WU Kejun; LIU Qiong; YANG You

doi:10.11999/JEIT261062

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

LI Fangcheng, WU Kejun, LIU Qiong, YANG You. ByteHOI: Human-Object Interaction Recognition Foundation Model in Bitstream Domain[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT261062

Citation:

LI Fangcheng, WU Kejun, LIU Qiong, YANG You. ByteHOI: Human-Object Interaction Recognition Foundation Model in Bitstream Domain[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT261062

Citation:

PDF( 1514 KB)

ByteHOI: Human-Object Interaction Recognition Foundation Model in Bitstream Domain

doi: 10.11999/JEIT261062 cstr: 32379.14.JEIT261062

School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan 430074, China

Funds: The National Natural Science Foundation of China (62501246)

Accepted Date: 2026-10-16
Rev Recd Date: 2026-10-16

Available Online: 2026-10-29

Abstract

Abstract

Objective Human-Object Interaction (HOI) recognition aims to identify semantic interactions between people and objects and is an important task in high-level scene understanding, with applications in behavior analysis, human-machine interaction, assisted driving, and intelligent monitoring. Existing HOI methods are generally developed in the pixel domain, where compressed images must first be decoded before feature extraction and interaction prediction. This explicit pixel reconstruction may increase the exposure of sensitive visual content in privacy-sensitive scenarios and also makes recognition dependent on successful decoding. Since images are commonly stored and transmitted as compressed bitstreams, it is meaningful to investigate whether high-level interaction semantics can be recognized directly from bitstreams without reconstructing pixels. To this end, the Bitstream Human-Object Interaction Recognition (BHOIR) task is formulated, which directly takes the raw byte sequence of a compressed image as input and predicts interaction categories without pixel decoding or reconstruction. BHOIR is more challenging than conventional byte-level image classification because HOI categories have a large compositional label space and a severe long-tailed distribution. For example, HICO contains 600 interaction categories composed of 117 actions and 80 objects, with many combinations having limited training samples. Moreover, bitstream features do not explicitly correspond to high-level concepts such as actions and objects. Therefore, BHOIR requires effective exploitation of HOI compositional structure and enhanced semantic modeling of byte-level representations. Methods A bitstream-domain HOI recognition framework named ByteHOI is developed for BHOIR, consisting of bitstream feature extraction and interaction classification (Fig. 1). ByteFormer-Tiny is adopted as the encoder. Each JPEG image is represented by its raw byte sequence and truncated or zero-padded to 50,000 bytes. The bytes are converted into embeddings, reduced by a one-dimensional convolutional module, combined with positional embeddings, and processed by 12 window-based Transformer blocks. Global average pooling then produces a 192-dimensional bitstream representation without pixel decoding or reconstruction. Two components are introduced for interaction classification. First, action-object compositional decoupling decomposes HOI prediction into separate action and object classification branches, and interaction scores are reconstructed according to predefined action-object combinations. This enables categories sharing the same action or object to share supervision and alleviates the sparsity of rare interaction combinations. Second, a byte-text semantic alignment classifier introduces fixed semantic prototypes generated by the pretrained CLIP ViT-B/32 text encoder. Action and object prompts, such as “a person riding” and “a horse”, are encoded offline and L2-normalized. Two learnable projection networks map the 192-dimensional bitstream feature into separate 512-dimensional action and object semantic spaces. Classification logits are obtained from cosine similarities between projected features and fixed text prototypes, with a temperature coefficient of 0.01. Binary cross-entropy losses are applied to both branches, and final HOI scores are obtained by combining the corresponding action and object predictions. ByteHOI is trained for 50 epochs using AdamW with a batch size of 64. The initial learning rates of the bitstream encoder and classification module are set to $ 1\times {10}^{-5} $ and $ 5\times {10}^{-4} $, respectively. Results and Discussions Experiments are conducted on the HICO and V-COCO datasets to evaluate the effectiveness and robustness of ByteHOI. General byte-level models, including MambaByte, MBLM, MegaByte, and bGPT, achieve Default mAP values below 2% on HICO, showing that generic byte-sequence modeling is insufficient for complex interaction recognition. ByteFormer, which benefits from visual byte-level pretraining, achieves 16.95% Default mAP, whereas ByteHOI improves the performance to 27.34%. Consistent improvements are observed across low-, medium-, and high-frequency interaction categories, particularly in the low-frequency group (Table 1). On V-COCO, the Default mAP is improved from 16.71% to 21.38%, and consistent gains are also obtained in Macro-Precision, Macro-Recall, and Macro-F1 on both datasets, demonstrating the cross-dataset effectiveness of the proposed framework (Table 2). Ablation experiments further confirm the contributions of the two key designs. On HICO, action-object compositional decoupling increases the Default mAP from 16.95% to 24.81%, while the introduction of CLIP semantic knowledge and cosine-based byte-text alignment further improves the performance to 27.34%. Similar trends are observed on V-COCO (Tables 3 and 4). Robustness is further evaluated by randomly flipping bits in the compressed data segment. Although some pixel-domain models achieve higher recognition accuracy on intact inputs, their performance decreases sharply as bitstream corruption increases because damaged files may suffer severe visual degradation or fail to decode. In contrast, ByteHOI maintains relatively stable performance. At a bit error rate of 0.01, it still achieves 24.82% and 19.72% mAP on HICO and V-COCO, respectively, corresponding to decreases of only 2.52 and 1.66 percentage points from the intact-bitstream setting (Fig. 2). Additional experiments show that ByteHOI is relatively insensitive to different JPEG encoder implementations and file-header modifications, whereas substantial changes in JPEG quality factor lead to more evident performance degradation, indicating that compression quality has a greater influence on bitstream distribution (Tables 5 and 6; Fig. 3). Conclusions The BHOIR task is formulated to investigate whether complex human-object interactions can be recognized directly from compressed image bitstreams without reconstructing pixel images during model inference. ByteHOI is developed to address two major difficulties in this setting. Action-object compositional decoupling is used to exploit the inherent compositional structure of HOI labels and alleviate the learning difficulty caused by sparse and long-tailed interaction categories, while byte-text semantic alignment is introduced to establish explicit associations between bitstream features and high-level action and object semantics using fixed CLIP text prototypes. Experimental results on HICO and V-COCO demonstrate that ByteHOI consistently outperforms the compared byte-level methods and achieves Default mAP values of 27.34% and 21.38%, respectively. The effectiveness of the proposed components is further supported by ablation studies. Bitstream corruption experiments show that ByteHOI maintains relatively stable recognition performance even when corrupted files become difficult or impossible to decode through conventional pixel-domain pipelines. By avoiding explicit reconstruction of visible pixel images during recognition, the proposed method reduces the exposure of pixel-level visual content in the recognition pipeline while preserving task-related interaction semantics. This privacy-related benefit is limited to the recognition process and does not constitute encryption or a formal privacy guarantee. The current study focuses on JPEG bitstreams. Future work will extend bitstream-domain HOI recognition to other image formats, including PNG, WebP, and AVIF, and will investigate unified bitstream representations with stronger adaptability across different encoding formats.
- Human-Object Interaction (HOI) recognition,
- Bitstream domain understanding,
- Privacy preservation

FullText(HTML)

References(18)

References

[1]	HAN Geng, ZHAO Jiachen, ZHANG Lele, et al. A survey of human-object interaction detection with deep learning[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2025, 9(1): 3–26. doi: 10.1109/TETCI.2024.3518613.
[2]	袁霖, 武雁尚, 张力元, 等. 面向人脸识别可信应用的隐私保护计算研究综述[J]. 电子与信息学报, 2026, 48(4): 1549–1568. doi: 10.11999/JEIT251063. YUAN Lin, WU Yanshang, ZHANG Liyuan, et al. Privacy-preserving computation in trustworthy face recognition: A comprehensive survey[J]. Journal of Electronics & Information Technology, 2026, 48(4): 1549–1568. doi: 10.11999/JEIT251063.
[3]	HORTON M, MEHTA S, FARHADI A, et al. Bytes are all you need: Transformers operating directly on file bytes[J]. Transactions on Machine Learning Research, 2024, 2024. (查阅网上资料, 未找到对应的期页码信息, 请确认).
[4]	孙中华, 吴双, 贾克斌, 等. 基于对比学习的动作识别研究综述[J]. 电子与信息学报, 2025, 47(8): 2473–2485. doi: 10.11999/JEIT250131. SUN Zhonghua, WU Shuang, JIA Kebin, et al. A review on action recognition based on contrastive learning[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2473–2485. doi: 10.11999/JEIT250131.
[5]	TAMURA M, OHASHI H, and YOSHINAGA T. QPIC: Query-based pairwise human-object interaction detection with image-wide contextual information[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 10410–10419. doi: 10.1109/CVPR46437.2021.01027.
[6]	KIM B, LEE J, KANG J, et al. HOTR: End-to-end human-object interaction detection with transformers[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2021: 74–83. doi: 10.1109/CVPR46437.2021.00014.
[7]	KIM S, JUNG D, and CHO M. Locality-aware zero-shot human-object interaction detection[C]. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, USA, 2025: 20190–20200. doi: 10.1109/CVPR52734.2025.01880.
[8]	YU Lili, SIMIG D, FLAHERTY C, et al. MEGABYTE: Predicting million-byte sequences with multiscale transformers[C]. Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans, USA, 2023: 3447.
[9]	WANG Junxiong, GANGAVARAPU T, YAN J N, et al. MambaByte: Token-free selective state space model[C]. Proceedings of the 1st Conference on Language Modeling, Philadelphia, PA, USA, 2024.
[10]	EGLI E, MANICA M, and BORN J. Multiscale byte language models a hierarchical architecture for causal million-length sequence modeling[EB/OL]. https://arxiv.org/abs/2502.14553, 2025.
[11]	WU Shangda, TAN Xu, WANG Zili, et al. Beyond language models: Byte models are digital world simulators[EB/OL]. https://arxiv.org/abs/2402.19155, 2024.
[12]	武辉, 赵妍, 张培溶, 等. 视觉-语言基础模型驱动的遥感图文检索研究进展[J]. 电子与信息学报, 2026, 48(8): 3229–3246. doi: 10.11999/JEIT260189. WU Hui, ZHAO Yan, ZHANG Peirong, et al. Recent advances in remote sensing image-text retrieval driven by vision-language foundation models[J]. Journal of Electronics & Information Technology, 2026, 48(8): 3229–3246. doi: 10.11999/JEIT260189.
[13]	LIN Lin, ZHANG Sihao, FU Song, et al. FD-LLM: Large language model for fault diagnosis of complex equipment[J]. Advanced Engineering Informatics, 2025, 65: 103208. doi: 10.1016/j.aei.2025.103208.
[14]	WU Kejun, WANG Shen, GAO Jianjun, et al. LLMArk: Instance-aware foundation model for flood risk assessment[J]. Information Fusion, 2026, 135: 104440. doi: 10.1016/j.inffus.2026.104440.
[15]	LAN Xiang, WU Feng, HE Kai, et al. GEM: Empowering MLLM for grounded ECG understanding with time series and images[C]. Proceedings of the 39th Conference on Neural Information Processing Systems, San Diego, USA, 2025: 104720–104754. doi: 10.52202/085713-3158.
[16]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, USA, 2016: 770–778. doi: 10.1109/CVPR.2016.90.
[17]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]. Proceedings of the 9th International Conference on Learning Representations, 2021. (查阅网上资料, 未找到对应的出版地信息, 请确认).
[18]	ZHANG Aixi, LIAO Yue, LIU Si, et al. Mining the benefits of two-stage and one-stage HOI detection[C]. Proceedings of the 35th International Conference on Neural Information Processing Systems, 2021: 17209–17220. (查阅网上资料, 未找到对应的出版地信息, 请确认).