SkipSync: Accelerating Instruction Sampling for LLM Workloads

CAI Luoshan; ZHOU Yaoyang; WANG Kaifan; LIU Tianyi; SUN Ninghui; BAO Yungang

doi:10.11999/JEIT260397

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2025 >

CAI Luoshan, ZHOU Yaoyang, WANG Kaifan, LIU Tianyi, SUN Ninghui, BAO Yungang. SkipSync: Accelerating Instruction Sampling for LLM Workloads[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260397

Citation:

CAI Luoshan, ZHOU Yaoyang, WANG Kaifan, LIU Tianyi, SUN Ninghui, BAO Yungang. SkipSync: Accelerating Instruction Sampling for LLM Workloads[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260397

Citation:

PDF( 3474 KB)

SkipSync: Accelerating Instruction Sampling for LLM Workloads

doi: 10.11999/JEIT260397 cstr: 32379.14.JEIT260397

CAI Luoshan^{1, 2},
ZHOU Yaoyang³,
WANG Kaifan^{1, 2},
LIU Tianyi^{1, 2},
SUN Ninghui^{1, 2},
BAO Yungang^{1, 2
,
,}

1.
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China
3.
Beijing Institute of Open Source Chip, Beijing 100080, China

Funds: The National Natural Science Foundation of China (62090022, 62090023), The Innovation Funding of ICT, CAS (E661150), Beijing Natural Science Foundation-Xiaomi Joint Innovation Fund (L2603004)

Received Date: 2026-04-07
Accepted Date: 2026-07-28
Rev Recd Date: 2026-07-22

Available Online: 2026-08-07

Abstract

Abstract

Objective The rapid evolution of Large Language Models (LLMs) has increased the demand for efficient design and evaluation of domain-specific accelerators. Sampling-based performance evaluation methods reduce the cost of cycle-accurate simulation but rely on Instruction Set Simulators (ISS) to execute complete workloads for profiling and checkpoint generation. In LLM inference scenarios, frequent changes in model architectures, inference frameworks, and accelerator instruction sets prevent checkpoint reuse and substantially increase the overhead of ISS-based functional simulation. This overhead has become a major bottleneck in the evaluation workflow. Existing ISS acceleration methods, such as hardware virtualization and Dynamic Binary Translation (DBT), either require target and host systems to share the same Instruction Set Architecture (ISA) or have high implementation complexity, making them difficult to apply to rapidly evolving LLM workloads. Therefore, a flexible and efficient ISS acceleration method is needed to support fast and accurate sampling-based performance evaluation of LLM accelerators. Methods An ISS acceleration method, SkipSync, is proposed for instruction sampling of LLM inference workloads. The key observation is that the control flow of most LLM operators is independent of runtime tensor values and is determined by static parameters. Based on this observation, the Skip mechanism is introduced to bypass the execution stage of accelerator instructions within selected operators while preserving instruction fetch and decode to maintain sampling accuracy. To ensure correct subsequent execution, a lightweight Host-Simulator Synchronization mechanism is further introduced to synchronize host-computed results back to the ISS. Lightweight synchronization primitives and custom instructions are designed to integrate SkipSync into existing sampling-based performance evaluation workflows with minimal implementation effort. Results and Discussions SkipSync is implemented on QEMU and supports RISC-V vector and matrix extensions as accelerator instruction sets. Experimental results show that SkipSync substantially reduces ISS functional simulation overhead while preserving sampling accuracy. Compared with the baseline, average speedups of 4.64× and 5.51× are achieved in the Prefill and Decode stages, respectively (Fig. 5), by eliminating the dominant execution cost of vector and matrix instructions, which accounts for 78.67% of runtime in the Prefill stage and 81.95% in the Decode stage (Table 1). SkipSync outperforms SIMD_DBT (Fig. 6) and provides better support for newly introduced accelerator instructions. The synchronization mechanism introduces limited overhead, with an average runtime increase of only 8.5% relative to ideal Skip-only execution (Fig. 8). Thus, synchronization does not negate the performance gains from the Skip mechanism. SkipSync also maintains high sampling accuracy, with an average sampling error of 2.55%, and the sampling results are nearly identical to those of baseline QEMU execution (Fig. 9). Conclusions An extensible ISS acceleration method, SkipSync, is presented for sampling-based performance evaluation of LLM inference workloads. By combining the Skip mechanism with Host-Simulator Synchronization, SkipSync effectively alleviates the ISS bottleneck in LLM sampling workflows while preserving execution correctness and sampling accuracy. Experimental results demonstrate substantial speedups, low synchronization overhead, and negligible effects on sampling accuracy. The proposed method provides a practical solution for efficient accelerator evaluation in rapidly evolving LLM inference systems. Future work will extend SkipSync to more complex LLM inference optimization scenarios and explore automatic identification of skippable operators and insertion of synchronization primitives to further improve usability.
- Large Language Model (LLM),
- Workload sampling,
- Instruction Set Simulator(ISS)

FullText(HTML)

References(25)

References

[1]	SHERWOOD T, PERELMAN E, HAMERLY G, et al. Automatically characterizing large scale program behavior[J]. ACM SIGPLAN Notices, 2002, 37(10): 45–57. doi: 10.1145/605432.605403.
[2]	WUNDERLICH R E, WENISCH T F, FALSAFI B, et al. SMARTS: Accelerating microarchitecture simulation via rigorous statistical sampling[C]. The 2003 30th Annual International Symposium on Computer Architecture (ISCA), San Diego, USA, 2003: 84–95. doi: 10.1109/ISCA.2003.1206991.
[3]	BELLARD F. QEMU, a fast and portable dynamic translator[C]. The Annual Conference on USENIX Annual Technical Conference, Anaheim, USA, 2005: 41–46.
[4]	BINKERT N, BECKMANN B, BLACK G, et al. The gem5 simulator[J]. ACM SIGARCH Computer Architecture News, 2011, 39(2): 1–7. doi: 10.1145/2024716.2024718.
[5]	SANDBERG A, NIKOLERIS N, CARLSON T E, et al. Full speed ahead: Detailed architectural simulation at near-native speed[C]. 2015 IEEE International Symposium on Workload Characterization (IISWC), Atlanta, USA, 2015: 183–192. doi: 10.1109/IISWC.2015.29.
[6]	FU Shengyu, HONG Dingyong, LIU Yuping, et al. Efficient and retargetable SIMD translation in a dynamic binary translator[J]. Software: Practice and Experience, 2018, 48(6): 1312–1330. doi: 10.1002/spe.2573.
[7]	JIANG Jinhu, LIANG Chaoyi, DONG Rongchao, et al. A system-level dynamic binary translator using automatically-learned translation rules[C]. 2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), Edinburgh, UK, 2024: 423–434. doi: 10.1109/CGO57630.2024.10444850.
[8]	卢迪, 王振发. 脉动阵列协同层融合的卷积神经网络加速器设计[J]. 电子与信息学报, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867. LU Di and WANG Zhenfa. Design of a CNN accelerator based on systolic array collaboration with inter-layer fusion[J]. Journal of Electronics & Information Technology, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867.
[9]	李天阳, 张帆, 王松, 等. 基于FPGA的卷积神经网络和视觉Transformer通用加速器[J]. 电子与信息学报, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713. LI Tianyang, ZHANG Fan, WANG Song, et al. FPGA-based unified accelerator for convolutional neural network and vision Transformer[J]. Journal of Electronics & Information Technology, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713.
[10]	于斌, 闵玉新, 张自豪, 等. 基于RISC-V指令扩展的双线性对协处理器设计[J]. 电子与信息学报, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367. YU Bin, MIN Yuxin, ZHANG Zihao, et al. Design of a bilinear pairing coprocessor based on RISC-V instruction extension[J]. Journal of Electronics & Information Technology, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367.
[11]	KIM H, YE Gaohan, WANG Nachuan, et al. Exploiting Intel Advanced Matrix Extensions (AMX) for large language model inference[J]. IEEE Computer Architecture Letters, 2024, 23(1): 117–120. doi: 10.1109/LCA.2024.3397747.
[12]	ILIESCU D A and PETROGALLI F. Arm scalable vector extension and application to machine learning[R]. 2017.
[13]	PARK D and EGGER B. Improving throughput-oriented LLM inference with CPU computations[C]. The 2024 International Conference on Parallel Architectures and Compilation Techniques (PACT), Long Beach, USA, 2024: 233–245. doi: 10.1145/3656019.3676949.
[14]	LUO Xinhao, LIU Zihan, ZHOU Yangjie, et al. ClusterFusion: Expanding operator fusion scope for LLM inference via cluster-level collective primitive[C]. 39th Conference on Neural Information Processing Systems, San Diago, USA, 2025.
[15]	SHAH J, BIKSHANDI G, ZHANG Ying, et al. Flashattention-3: Fast and accurate attention with asynchrony and low-precision[C]. The 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 2193.
[16]	LIU Zihan, LUO Xinhao, GUO Junxian, et al. VQ-LLM: High-performance code generation for vector quantization augmented LLM inference[C]. 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), Las Vegas, USA, 2025: 1496–1509. doi: 10.1109/HPCA61900.2025.00112.
[17]	HAN Chenji, LI Xinyu, XUE Feng, et al. MultiPoint: Enabling scalable pre-silicon performance evaluation for multi-task workloads[J]. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2024, 4(3): 100189. doi: 10.1016/j.tbench.2025.100189.
[18]	LIU Changxi, SABU A, CHAUDHARI A, et al. Pac-Sim: Simulation of multi-threaded workloads using intelligent, live sampling[J]. ACM Transactions on Architecture and Code Optimization, 2024, 21(4): 81. doi: 10.1145/3680548.
[19]	LIU Changxi, SUN Yifan, and CARLSON T E. Photon: A fine-grained sampled simulation methodology for GPU workloads[C]. 2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO), Toronto, Canada, 2023: 1227–1241. doi: 10.1145/3613424.3623773.
[20]	LIU Yuping, HONG Dingyong, WU Jan-jan, et al. Exploiting SIMD asymmetry in ARM-to-x86 dynamic binary translation[J]. ACM Transactions on Architecture and Code Optimization, 2019, 16(1): 2. doi: 10.1145/3301488.
[21]	RISC-V International. RISCVarchive/RISCV-V-spec[EB/OL]. https://github.com/riscv/riscv-v-spec. 2026.
[22]	XUANTIE-RV. RISC-V matrix extension specification proposal V0.6. 0[EB/OL]. https://github.com/XUANTIE-RV/riscv-matrix-extension-spec, 2026.
[23]	LEVIATHAN Y, KALMAN M, and MATIAS Y. Fast inference from transformers via speculative decoding[C]. The 40th International Conference on Machine Learning (ICML), Honolulu, USA, 2023: 795.
[24]	QEMU. QEMU v9.0. 0[EB/OL]. https://github.com/qemu/qemu/releases/tag/v9.0.0, 2024.
[25]	GRYGIER A. Llama. cpp[EB/OL]. https://github.com/ggml-org/llama.cpp, 2026.