QuadPipe: A Low-Error and High-Throughput Softmax Accelerator for Large Model Inference on FPGA
-
摘要: 随着大模型推理在云端与边缘场景的规模化部署，FPGA凭借可重构计算、高资源效率与低推理时延等优势，已成为补充GPU的异构加速方案。然而，在FPGA上实现高效Softmax仍面临三重瓶颈：长序列分母累加引发舍入误差积累与数值漂移，固定并行度流水线在变长向量尾拍对齐时利用率显著下降，以及访存带宽与计算密度失衡制约系统吞吐。针对上述问题，该文提出QuadPipe Softmax加速器，以3项设计协同应对：(1)Q48.47格式96位扩展精度累加器抑制长序列分母误差传播；(2)基于负无穷填充的硬件级边界无关处理，消除变长向量的软件预补齐开销；(3)X-Y双维批处理命令调度与多级反压闭环，统一计算核心与AXI接口的吞吐节拍。在Xilinx Kintex-7 325T开发板上，QuadPipe以仅占器件总逻辑4.72%的LUT与8个DSP的轻量级资源占用，实现了59周期流水线深度、每周期4个结果的稳态吞吐与150 MHz下600 M elements/s的峰值吞吐。板级实测MAE维持在10–11～10–10量级，Q48.47扩展累加使长序列分母误差较标准FP32累加降低87.59%；硬件边界无关处理使非对齐向量有效吞吐较软件预补齐最高提升6.49%。该设计在统一流水线框架内协同应对了长序列数值稳定性、变长输入处理与系统吞吐匹配3项工程挑战，为FPGA平台上大模型推理的Softmax加速提供了一套高效的系统方案。Abstract:
Objective With the large-scale deployment of large model inference in cloud and edge scenarios, FPGA has emerged as an important heterogeneous acceleration platform that complements GPU due to its reconfigurable computing capability, high resource efficiency, and low inference latency. However, efficient Softmax implementation on FPGAs still faces three major challenges: (i) rounding-error accumulation and numerical drift caused by long-sequence denominator summation, (ii) significant utilization degradation of fixed-parallelism pipelines when processing misaligned tails of variable-length vectors, and (iii) throughput limitations caused by the imbalance between memory bandwidth and computational density. Methods This paper proposes QuadPipe, a high-throughput Softmax hardware pipeline that integrates three complementary design strategies. (1) A Q48.47-format 96-bit extended-precision accumulator is introduced into the denominator summation path. This design reduces the upper bound of per-addition rounding error from O(2–24) under FP32 to O(2–47), thereby suppressing error propagation during long-sequence denominator accumulation. (2) A hardware-level boundary-agnostic processing mechanism based on negative-infinity padding is developed. By exploiting the property $ {e}^{-\mathrm{\infty }}=0 $, the mechanism automatically inserts NEG_INF into invalid tail slots within the compute core, eliminating software-side pre-padding overhead for variable-length vectors. (3) An X-Y dual-dimensional batch command scheduler and a multi-level backpressure control loop are designed to coordinate throughput between the compute core and the AXI interface. The proposed cross-precision data path, consisting of INT8 input loading, FP32 computation, and FP32 output writeback, enables continuous batch processing of up to Y=65 536 vectors while reducing input bandwidth requirements to one quarter of those of the FP32 scheme. Results and Discussions The proposed QuadPipe accelerator is implemented on a Xilinx Kintex-7 325T development board. At 150 MHz, it achieves a 59-cycle pipeline depth, a steady-state throughput of 4 FP32 results per cycle, and a peak throughput of 600 M elements/s, while consuming only 4.72% of the device LUT resources and 8 DSPs (Table 1). The Figure Of Merit (FOM) reaches 1.14×106 bit·s–1·(LUT+FF)–1 (Fig. 1 and Table 1). Board-level measurements show that the Mean Absolute Error (MAE) remains within the 10–11～10–10 range across all tested sequence lengths (Table 2). At N=2048, the Q48.47 extended-precision accumulation reduces the denominator error by 87.59% compared with standard FP32 accumulation (Table 2). Compared with the software pre-padding scheme, the hardware boundary-agnostic processing mechanism improves the effective element throughput of misaligned vectors by up to 6.49% at N=1 025 and reduces batch-processing latency by up to 5.98% (Table 3). Conclusions The proposed QuadPipe accelerator integrates extended-precision accumulation, boundary-agnostic processing, and dual-dimensional batch scheduling within a unified pipeline framework to address three major challenges in FPGA-based Softmax acceleration: long-sequence numerical stability, variable-length input processing, and system throughput matching. The experimental results demonstrate that QuadPipe achieves low numerical error, efficient resource utilization, and reliable system-level integration, providing a feasible hardware solution for large model inference on FPGA platforms. -
表 1 FPGA Softmax加速器横向对比
工作 年份 目标器件 精度w 并行度n fmax(MHz) LUT FF FOM DSP BRAM 方法特征 Zhu[11] 2020 Zc706 定点可调(16) 1 500 395 498 8.96×106 0 0 精度可调分段 Gao[4] 2020 KC705 定点16 1 154 2229 224 1.00×106 - - Taylor 近似 Koca[15] 2023 ZCU102 定点16 8 476 909 333 4.91×107 0 - adder/shifter 近似 HyFT[12] 2024 Xc7z030 FP16 8 645 1172 1361 3.26×107 - - 混合数值格式 DB-Attn[28] 2025 Alveo U280 DBFP(m8,e5) 1024 455 10872 3743 5.10×108 - - 动态块浮点+DH-LUT QuadPipe 2026 Kintex-7 325T INT8→FP32 4 150 9455 7453 1.14×106 8 14 两遍式 + Q48.47 累加 
下载: 导出CSV
表 2 不同累加路径在长序列上相对FP64参考的平均绝对误差(MAE)
向量长度N RTL MAE
均值RTL MAE
最大值FP32 MAE
均值FP32 MAE
最大值acc96 proxy
均值acc96 proxy
最大值RTL是否
通过10–6128 $ 2.92\times {10}^{-10} $ $ 3.56\times {10}^{-10} $ $ 5.94\times {10}^{-10} $ $ 1.34\times {10}^{-9} $ $ 8.33\times {10}^{-11} $ $ 1.00\times {10}^{-10} $ 是 512 $ 1.14\times {10}^{-10} $ $ 1.56\times {10}^{-10} $ $ 2.18\times {10}^{-10} $ $ 3.49\times {10}^{-10} $ $ 2.58\times {10}^{-11} $ $ 2.94\times {10}^{-11} $ 是 1024 $ 4.32\times {10}^{-11} $ $ 5.61\times {10}^{-11} $ $ 2.93\times {10}^{-10} $ $ 5.57\times {10}^{-10} $ $ 1.08\times {10}^{-11} $ $ 1.10\times {10}^{-11} $ 是 2048 $ 2.47\times {10}^{-11} $ $ 3.34\times {10}^{-11} $ $ 1.99\times {10}^{-10} $ $ 3.63\times {10}^{-10} $ $ 6.49\times {10}^{-12} $ $ 7.02\times {10}^{-12} $ 是
下载: 导出CSV
表 3 变长序列场景下 hw_auto_pad 与 sw_pre_pad 的系统性能对比
长度N 模式 启动长度 额外填充元素 批处理时延(ns) 注入间隙(ns) 有效元素吞吐(elem·ns–1) 对齐后吞吐(elem·ns–1) 127 hw_auto_pad 127 0 35240198 2264013 5.80×10–5 5.80×10–5 127 sw_pre_pad 128 1 33452189 2632016 6.10×10–5 6.10×10–5 511 hw_auto_pad 511 0 35402200 4038026 2.31×10–4 2.31×10–4 511 sw_pre_pad 512 1 37452211 5684035 2.18×10–4 2.19×10–4 1025 hw_auto_pad 1025 0 40040225 7226039 4.10×10–4 4.10×10–4 1025 sw_pre_pad 1028 3 42588240 7170043 3.85×10–4 3.86×10–4 2049 hw_auto_pad 2049 0 49498279 14624083 6.62×10–4 6.62×10–4 2049 sw_pre_pad 2052 3 49896282 14902086 6.57×10–4 6.58×10–4
下载: 导出CSV
-
[1] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]. The 31st International Conference on Neural Information Processing Systems, Long Beach, USA, 2017: 5998–6008. [2] YUAN Bo. Efficient hardware architecture of softmax layer in deep neural network[C]. The 29th IEEE International System-on-Chip Conference, Seattle, USA, 2016: 323–326. doi: 10.1109/SOCC.2016.7905501. [3] WANG Meiqi, LU Siyuan, ZHU Danyang, et al. A high-speed and low-complexity architecture for softmax function in deep learning[C]. 2018 IEEE Asia Pacific Conference on Circuits and Systems, Chengdu, China, 2018: 223–226. doi: 10.1109/APCCAS.2018.8605654. [4] GAO Yue, LIU Weiqiang, and LOMBARDI F. Design and implementation of an approximate softmax layer for deep neural networks[C]. 2020 IEEE International Symposium on Circuits and Systems, Seville, Spain, 2020: 1–5. doi: 10.1109/ISCAS45731.2020.9180870. [5] KIM J, KIM S, CHOI K, et al. Hardware-efficient softmax architecture with bit-wise exponentiation and reciprocal calculation[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2024, 71(10): 4574–4585. doi: 10.1109/TCSI.2024.3443270. [6] TARASOV I and POTEKHIN D. Calculation of activation functions in FPGA-based neuroprocessors using the CORDIC algorithm[C]. The 11th International Conference on High-Performance Computing Systems and Technologies in Scientific Research, Automation of Control and Production, Barnaul, Russia, 2022: 16–30. doi: 10.1007/978-3-030-94141-3_2. [7] SUDHA J, HANUMANTHARAJU M C, VENKATESWARULU V, et al. A novel method for computing exponential function using CORDIC algorithm[J]. Procedia Engineering, 2012, 30: 519–528. doi: 10.1016/j.proeng.2012.01.893. [8] STEVENS J R, VENKATESAN R, DAI S, et al. Softermax: Hardware/software co-design of an efficient softmax for transformers[C]. The 58th ACM/IEEE Design Automation Conference, San Francisco, USA, 2021: 469–474. doi: 10.1109/DAC18074.2021.9586134. [9] ZHANG Huan, ZHANG Yonggang, PENG Lele, et al. Base-2 softmax function: Suitability for training and efficient hardware implementation[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2022, 69(9): 3605–3618. doi: 10.1109/TCSI.2022.3175534. [10] SPAGNOLO F, PERRI S, and CORSONELLO P. Aggressive approximation of the softmax function for power-efficient hardware implementations[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2022, 69(3): 1652–1656. doi: 10.1109/TCSII.2021.3120495. [11] ZHU Danyang, LU Siyuan, WANG Meiqi, et al. Efficient precision-adjustable architecture for softmax function in deep learning[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2020, 67(12): 3382–3386. doi: 10.1109/TCSII.2020.3002564. [12] XIA Tianhua and ZHANG Saiqian. Hyft: A reconfigurable softmax accelerator with hybrid numeric format for both training and inference[C]. The 29th ACM/IEEE International Symposium on Low Power Electronics and Design, Newport Beach, USA, 2024. doi: 10.1145/3665314.3670816. [13] WU Yuanchen, XIE Zhiheng, PAN Hongbing, et al. MBS: A high-precision approximation method for softmax and efficient hardware implementation[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2025, 72(7): 3366–3375. doi: 10.1109/TCSI.2025.3559069. [14] CUI Zhanhao, MA Yuhao, CHENG Yuan, et al. TEA-SPS: A tiny and efficient architecture for softmax with parallelism and sparsity adaptability[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2026, 73(2): 1245–1258. doi: 10.1109/TCSI.2025.3610210. [15] KOCA N A, DO A T, and CHANG C H. Hardware-efficient softmax approximation for self-attention networks[C]. 2023 IEEE International Symposium on Circuits and Systems, Monterey, USA, 2023. doi: 10.1109/ISCAS46773.2023.10181465. [16] DAO T, FU D Y, ERMON S, et al. FLASHATTENTION: Fast and memory-efficient exact attention with IO-awareness[C]. The 36th International Conference on Neural Information Processing Systems, New Orleans, USA, 2022: 1189. [17] YE Wenhua, ZHOU Xu, ZHOU J, et al. Accelerating attention mechanism on FPGAs based on efficient reconfigurable systolic array[J]. ACM Transactions on Embedded Computing Systems, 2023, 22(6): 93. doi: 10.1145/3549937. [18] KABIR E, KABIR A, DOWNEY A R J, et al. FAMOUS: Flexible accelerator for the attention mechanism of transformer on Ultrascale+ FPGAs[C]. 2024 International Conference on Field Programmable Technology, Sydney, Australia, 2024. doi: 10.1109/ICFPT64416.2024.11113430. [19] ISLAMOGLU G, SCHERER M, PAULIN G, et al. ITA: An energy-efficient attention and softmax accelerator for quantized transformers[C]. 2023 IEEE/ACM International Symposium on Low Power Electronics and Design, Vienna, Austria, 2023. doi: 10.1109/ISLPED58423.2023.10244348. [20] LI Tianyang, ZHANG Fan, FAN Xitian, et al. Unified accelerator for attention and convolution in inference based on FPGA[C]. 2023 IEEE International Symposium on Circuits and Systems, Monterey, USA, 2023. doi: 10.1109/ISCAS46773.2023.10182145. [21] 李天阳, 张帆, 王松, 等. 基于FPGA的卷积神经网络和视觉Transformer通用加速器[J]. 电子与信息学报, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713.LI Tianyang, ZHANG Fan, WANG Song, et al. FPGA-based unified accelerator for convolutional neural network and vision Transformer[J]. Journal of Electronics & Information Technology, 2024, 46(6): 2663–2672. doi: 10.11999/JEIT230713. [22] 姜小波, 邓晗珂, 莫志杰, 等. 规则压缩模型和灵活架构的Transformer加速器设计[J]. 电子与信息学报, 2024, 46(3): 1079–1088. doi: 10.11999/JEIT230188.JIANG Xiaobo, DENG Hanke, MO Zhijie, et al. Design of Transformer accelerator with regular compression model and flexible architecture[J]. Journal of Electronics & Information Technology, 2024, 46(3): 1079–1088. doi: 10.11999/JEIT230188. [23] ZENG Shulin, LIU Jun, DAI Guohao, et al. FlightLLM: Efficient large language model inference with a complete mapping flow on FPGAs[C]. 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, Monterey, USA, 2024. doi: 10.1145/3626202.3637562. [24] CHEN Hongzheng, ZHANG Jiahao, DU Yixiao, et al. Understanding the potential of FPGA-based spatial acceleration for large language model inference[J]. ACM Transactions on Reconfigurable Technology and Systems, 2025, 18(1): 5. doi: 10.1145/3656177. [25] LI Cong, YIN Yihan, WU Xintong, et al. H2-LLM: Hardware-dataflow co-exploration for heterogeneous hybrid-bonding-based low-batch LLM inference[C]. The 52nd Annual International Symposium on Computer Architecture, Tokyo, Japan, 2025. doi: 10.1145/3695053.3731008. [26] 王泽昊, 朱振华, 谢童欣, 等. 混合专家大语言模型的系统与架构优化技术综述[J]. 电子与信息学报, 2025, 47(11): 4055–4078. doi: 10.11999/JEIT250407.WANG Zehao, ZHU Zhenhua, XIE Tongxin, et al. A survey on system and architecture optimization techniques for mixture-of-experts large language models[J]. Journal of Electronics & Information Technology, 2025, 47(11): 4055–4078. doi: 10.11999/JEIT250407. [27] 林志平, 肖亮, 陈宏毅, 等. 面向大语言模型的抗干扰协同推断技术[J]. 电子与信息学报, 2025, 47(11): 4572–4582. doi: 10.11999/JEIT250675.LIN Zhiping, XIAO Liang, CHEN Hongyi, et al. Collaborative inference for large language models against jamming attacks[J]. Journal of Electronics & Information Technology, 2025, 47(11): 4572–4582. doi: 10.11999/JEIT250675. [28] WANG Hui, CHENG Yuan, HAN Xiaomeng, et al. Pushing the limits of BFP on narrow precision LLM inference[C]. The 39th AAAI Conference on Artificial Intelligence, Philadelphia, USA, 2025: 21099–21107. doi: 10.1609/aaai.v39i20.35407. -
下载:
图(4) / 表(3)
计量
- 文章访问数: 165
- HTML全文浏览量: 42
- PDF下载量: 8
- 被引次数: 0
