SkipSync：面向大模型负载的指令采样加速方法

蔡洛姗; 周耀阳; 王凯帆; 刘天义; 孙凝晖; 包云岗

doi:10.11999/JEIT260397

SkipSync：面向大模型负载的指令采样加速方法

doi: 10.11999/JEIT260397 cstr: 32379.14.JEIT260397

蔡洛姗^{1, 2},
周耀阳³,
王凯帆^{1, 2},
刘天义^{1, 2},
孙凝晖^{1, 2},
包云岗^{1, 2, ,}

1.
中国科学院计算技术研究所处理器芯片全国重点实验室北京 100190
2.
中国科学院大学北京 100049
3.
北京开源芯片研究院北京 100080

基金项目: (Grant No. 62090022, 62090023)，中国科学院计算所创新课题 (E661150)，北京市自然科学基金-小米创新联合基金(L2603004)

详细信息

作者简介:
蔡洛姗：女，博士生，研究方向为开源处理器微架构、AI加速器芯片敏捷设计与评估

周耀阳：男，博士，研究方向为可扩展处理器设计、负载采样和性能评测方法

王凯帆：男，博士生，研究方向为计算机体系结构、开源处理器敏捷性能分析

刘天义：男，博士，副研究员，研究方向为计算机系统结构、开源处理器敏捷设计、云渲染和AI基础设施

孙凝晖：男，博士，中国工程院院士，CCF会士，研究方向为计算机系统结构、高性能计算

包云岗：男，博士，研究员，研究方向为数据中心体系结构、处理器芯片敏捷设计方法论、开源处理器芯片生态

通讯作者:
包云岗　baoyg@ict.ac.cn

中图分类号: TN47; TP391.9
计量
- 文章访问数: 251
- HTML全文浏览量: 137
- PDF下载量: 14
- 被引次数: 0
出版历程
- 收稿日期: 2026-04-07
- 修回日期: 2026-07-22
- 录用日期: 2026-07-28
- 网络出版日期: 2026-08-07

SkipSync: Accelerating Instruction Sampling for LLM Workloads

CAI Luoshan^{1, 2},
ZHOU Yaoyang³,
WANG Kaifan^{1, 2},
LIU Tianyi^{1, 2},
SUN Ninghui^{1, 2},
BAO Yungang^{1, 2
, ,}

1.
National Key Laboratory of Processor Chips, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China
3.
Beijing Institute of Open Source Chip, Beijing 100080, China

Funds: The National Natural Science Foundation of China (Grant No. 62090022, 62090023), The Innovation Funding of ICT, CAS (E661150), Beijing Natural Science Foundation-Xiaomi Joint Innovation Fund (L2603004)

摘要

摘要: 指令采样是一种广泛应用于硬件设计评估的技术，能够在加快评估速度的同时保证评估精度。然而在大语言模型(LLM)推理场景下，模型结构、软件框架和专用加速器指令等软硬件技术快速演进，使得每一次加速器迭代评估时都需要在指令集模拟器(ISS)上重新执行完整的应用负载以获得最新指令流完成采样。此外，加速器指令在ISS中通常以解释执行的方式运行，导致指令采样时间长，因此ISS上的执行效率逐渐成为评估流程的主要瓶颈。针对上述问题，该文提出一种面向LLM推理负载的指令采样加速方法SkipSync。该方法基于如下观察：LLM算子内部的程序控制流通常与张量计算结果无关，因此可以在这类算子中跳过高开销的加速器指令执行过程，从而提升ISS的执行速度。同时为保障程序的后续执行与采样准确性，SkipSync通过轻量级数据同步机制用宿主机的正确计算结果校正模拟器上的运行状态。基于QEMU指令集模拟器的实验结果表明，在多种模型规模及推理阶段下，SkipSync可获得平均5.08倍的加速效果，显著提升了基于指令采样的LLM加速器性能评估效率，且不影响采样的准确性。另外，该方法无需针对新型加速器指令进行定制实现，具有良好的可扩展性，能够有效支撑快速演进背景下的LLM加速器性能评估与设计迭代。
- 大语言模型 /
- 采样评估方法 /
- 指令集模拟器
Abstract: Objective The rapid evolution of Large Language Models (LLMs) has increased the demand for efficient design and evaluation of domain-specific accelerators. Sampling-based performance evaluation methods reduce the cost of cycle-accurate simulation but rely on Instruction Set Simulators (ISS) to execute complete workloads for profiling and checkpoint generation. In LLM inference scenarios, frequent changes in model architectures, inference frameworks, and accelerator instruction sets prevent checkpoint reuse and substantially increase the overhead of ISS-based functional simulation. This overhead has become a major bottleneck in the evaluation workflow. Existing ISS acceleration methods, such as hardware virtualization and Dynamic Binary Translation (DBT), either require target and host systems to share the same Instruction Set Architecture (ISA) or have high implementation complexity, making them difficult to apply to rapidly evolving LLM workloads. Therefore, a flexible and efficient ISS acceleration method is needed to support fast and accurate sampling-based performance evaluation of LLM accelerators. Methods An ISS acceleration method, SkipSync, is proposed for instruction sampling of LLM inference workloads. The key observation is that the control flow of most LLM operators is independent of runtime tensor values and is determined by static parameters. Based on this observation, the Skip mechanism is introduced to bypass the execution stage of accelerator instructions within selected operators while preserving instruction fetch and decode to maintain sampling accuracy. To ensure correct subsequent execution, a lightweight Host–Simulator Synchronization mechanism is further introduced to synchronize host-computed results back to the ISS. Lightweight synchronization primitives and custom instructions are designed to integrate SkipSync into existing sampling-based performance evaluation workflows with minimal implementation effort. Results and Discussions SkipSync is implemented on QEMU and supports RISC-V vector and matrix extensions as accelerator instruction sets. Experimental results show that SkipSync substantially reduces ISS functional simulation overhead while preserving sampling accuracy. Compared with the baseline, average speedups of 4.64× and 5.51× are achieved in the Prefill and Decode stages, respectively (Fig. 5), by eliminating the dominant execution cost of vector and matrix instructions, which accounts for 78.67% of runtime in the Prefill stage and 81.95% in the Decode stage (Table 1). SkipSync outperforms SIMD_DBT (Fig. 6) and provides better support for newly introduced accelerator instructions. The synchronization mechanism introduces limited overhead, with an average runtime increase of only 8.5% relative to ideal Skip-only execution (Fig. 8). Thus, synchronization does not negate the performance gains from the Skip mechanism. SkipSync also maintains high sampling accuracy, with an average sampling error of 2.55%, and the sampling results are nearly identical to those of baseline QEMU execution (Fig. 9). Conclusions An extensible ISS acceleration method, SkipSync, is presented for sampling-based performance evaluation of LLM inference workloads. By combining the Skip mechanism with Host–Simulator Synchronization, SkipSync effectively alleviates the ISS bottleneck in LLM sampling workflows while preserving execution correctness and sampling accuracy. Experimental results demonstrate substantial speedups, low synchronization overhead, and negligible effects on sampling accuracy. The proposed method provides a practical solution for efficient accelerator evaluation in rapidly evolving LLM inference systems. Future work will extend SkipSync to more complex LLM inference optimization scenarios and explore automatic identification of skippable operators and insertion of synchronization primitives to further improve usability.
- Large Language Model (LLM) /
- Workload sampling /
- Instruction Set Simulator(ISS)

HTML全文

图 1 LLM加速器设计：流程瓶颈从性能仿真转向功能仿真

下载: 全尺寸图片幻灯片

图 2 跳过指令执行的加速方法示意图

下载: 全尺寸图片幻灯片

图 3 主机-模拟器间同步流程

下载: 全尺寸图片幻灯片

图 4 SkipSync 采样加速流程总览

下载: 全尺寸图片幻灯片

图 5 SkipSync对比原始QEMU在RISC-V向量和矩阵扩展中的延迟与加速比

下载: 全尺寸图片幻灯片

图 6 用户态模式下SkipSync对比SIMD_DBT在ARM NEON扩展中的延迟与加速比

下载: 全尺寸图片幻灯片

图 7 全系统模式下SkipSync对比SIMD_DBT理想上界的加速比

下载: 全尺寸图片幻灯片

图 8 Skip vs Skip+Sync归一化运行时间对比

下载: 全尺寸图片幻灯片

图 9 SimPoint采样误差对比

下载: 全尺寸图片幻灯片

表 1 QEMU运行LLM负载的时间占比分析(%)

操作类别	向量/矩阵访存	向量/矩阵计算	标量指令执行	其他开销
Prefill阶段	34.93	43.74	7.53	13.80
Decode阶段	68.55	13.40	4.02	14.03

下载: 导出CSV

表 2 LLM算子控制流特性分析

算子类型	是否控制流独立	原因说明
Matmul	√	仅由矩阵尺寸和分块方式决定
LayerNorm/RMSNorm	√	循环次数固定
Softmax	√	Reduction 范围由张量维度决定
SiLU/GELU/ReLU	√	逐元素计算范围固定
Add/Mul/Residual Add	√	逐元素计算范围固定
Top-k Sampling	×	Top-k选择由运行时数据决定
Dynamic Sparse Attention	×	需要根据实际数据确定非零元素位置

下载: 导出CSV

表 3 多核系统下SkipSync的加速比

阶段	核心数量
阶段	1	2	4	8	16
Prefill阶段	4.43	4.51	4.45	4.68	4.59
Decode阶段	5.48	5.63	5.26	5.66	5.46

下载: 导出CSV

参考文献(25)

[1]	SHERWOOD T, PERELMAN E, HAMERLY G, et al. Automatically characterizing large scale program behavior[J]. ACM SIGPLAN Notices, 2002, 37(10): 45–57. doi: 10.1145/605432.605403.
[2]	WUNDERLICH R E, WENISCH T F, FALSAFI B, et al. SMARTS: Accelerating microarchitecture simulation via rigorous statistical sampling[C]. The 2003 30th Annual International Symposium on Computer Architecture (ISCA), San Diego, USA, 2003: 84–95. doi: 10.1109/ISCA.2003.1206991.
[3]	BELLARD F. QEMU, a fast and portable dynamic translator[C]. The Annual Conference on USENIX Annual Technical Conference, Anaheim, USA, 2005: 41–46.
[4]	BINKERT N, BECKMANN B, BLACK G, et al. The gem5 simulator[J]. ACM SIGARCH Computer Architecture News, 2011, 39(2): 1–7. doi: 10.1145/2024716.2024718.
[5]	SANDBERG A, NIKOLERIS N, CARLSON T E, et al. Full speed ahead: Detailed architectural simulation at near-native speed[C]. 2015 IEEE International Symposium on Workload Characterization (IISWC), Atlanta, USA, 2015: 183–192. doi: 10.1109/IISWC.2015.29.
[6]	FU Shengyu, HONG Dingyong, LIU Yuping, et al. Efficient and retargetable SIMD translation in a dynamic binary translator[J]. Software: Practice and Experience, 2018, 48(6): 1312–1330. doi: 10.1002/spe.2573.
[7]	JIANG Jinhu, LIANG Chaoyi, DONG Rongchao, et al. A system-level dynamic binary translator using automatically-learned translation rules[C]. 2024 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), Edinburgh, UK, 2024: 423–434. doi: 10.1109/CGO57630.2024.10444850.
[8]	卢迪, 王振发. 脉动阵列协同层融合的卷积神经网络加速器设计[J]. 电子与信息学报, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867. LU Di and WANG Zhenfa. Design of a CNN accelerator based on systolic array collaboration with inter-layer fusion[J]. Journal of Electronics & Information Technology, 2026, 48(3): 1280–1291. doi: 10.11999/JEIT250867.
[9]	李天阳, 张帆, 王松, 等. 基于FPGA的卷积神经网络和视觉Transformer通用加速器[J]. 电子与信息学报, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713. LI Tianyang, ZHANG Fan, WANG Song, et al. FPGA-based unified accelerator for convolutional neural network and vision Transformer[J]. Journal of Electronics & Information Technology, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713.
[10]	于斌, 闵玉新, 张自豪, 等. 基于RISC-V指令扩展的双线性对协处理器设计[J]. 电子与信息学报, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367. YU Bin, MIN Yuxin, ZHANG Zihao, et al. Design of a bilinear pairing coprocessor based on RISC-V instruction extension[J]. Journal of Electronics & Information Technology, 2025, 47(9): 3137–3145. doi: 10.11999/JEIT250367.
[11]	KIM H, YE Gaohan, WANG Nachuan, et al. Exploiting Intel Advanced Matrix Extensions (AMX) for large language model inference[J]. IEEE Computer Architecture Letters, 2024, 23(1): 117–120. doi: 10.1109/LCA.2024.3397747.
[12]	ILIESCU D A and PETROGALLI F. Arm scalable vector extension and application to machine learning[R]. 2017.
[13]	PARK D and EGGER B. Improving throughput-oriented LLM inference with CPU computations[C]. The 2024 International Conference on Parallel Architectures and Compilation Techniques (PACT), Long Beach, USA, 2024: 233–245. doi: 10.1145/3656019.3676949.
[14]	LUO Xinhao, LIU Zihan, ZHOU Yangjie, et al. ClusterFusion: Expanding operator fusion scope for LLM inference via cluster-level collective primitive[C]. 39th Conference on Neural Information Processing Systems, San Diago, USA, 2025.
[15]	SHAH J, BIKSHANDI G, ZHANG Ying, et al. Flashattention-3: Fast and accurate attention with asynchrony and low-precision[C]. The 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 2193.
[16]	LIU Zihan, LUO Xinhao, GUO Junxian, et al. VQ-LLM: High-performance code generation for vector quantization augmented LLM inference[C]. 2025 IEEE International Symposium on High Performance Computer Architecture (HPCA), Las Vegas, USA, 2025: 1496–1509. doi: 10.1109/HPCA61900.2025.00112.
[17]	HAN Chenji, LI Xinyu, XUE Feng, et al. MultiPoint: Enabling scalable pre-silicon performance evaluation for multi-task workloads[J]. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2024, 4(3): 100189. doi: 10.1016/j.tbench.2025.100189.
[18]	LIU Changxi, SABU A, CHAUDHARI A, et al. Pac-Sim: Simulation of multi-threaded workloads using intelligent, live sampling[J]. ACM Transactions on Architecture and Code Optimization, 2024, 21(4): 81. doi: 10.1145/3680548.
[19]	LIU Changxi, SUN Yifan, and CARLSON T E. Photon: A fine-grained sampled simulation methodology for GPU workloads[C]. 2023 56th IEEE/ACM International Symposium on Microarchitecture (MICRO), Toronto, Canada, 2023: 1227–1241. doi: 10.1145/3613424.3623773.
[20]	LIU Yuping, HONG Dingyong, WU Jan-jan, et al. Exploiting SIMD asymmetry in ARM-to-x86 dynamic binary translation[J]. ACM Transactions on Architecture and Code Optimization, 2019, 16(1): 2. doi: 10.1145/3301488.
[21]	RISC-V International. RISCVarchive/RISCV-V-spec[EB/OL]. https://github.com/riscv/riscv-v-spec. 2026.
[22]	XUANTIE-RV. RISC-V matrix extension specification proposal V0.6. 0[EB/OL]. https://github.com/XUANTIE-RV/riscv-matrix-extension-spec, 2026.
[23]	LEVIATHAN Y, KALMAN M, and MATIAS Y. Fast inference from transformers via speculative decoding[C]. The 40th International Conference on Machine Learning (ICML), Honolulu, USA, 2023: 795.
[24]	QEMU. QEMU v9.0. 0[EB/OL]. https://github.com/qemu/qemu/releases/tag/v9.0.0, 2024.
[25]	GRYGIER A. Llama. cpp[EB/OL]. https://github.com/ggml-org/llama.cpp, 2026.