Optimized Design of Non-Transparent Bridge for Heterogeneous Interconnects in Hyper-converged Infrastructure
-
摘要: 为提升超融合(HCI)系统内异构域跨域的传输性能和稳定性，该文提出一种支持双传输模式的非透明桥(NTB)数据通路架构(D-MNTBA)。通过融合所提旁路架构下的快速传输模式和传统架构(TDPA)下的稳定传输模式，NTB能够结合HCI数据特性与跨域需求进行分流传输报文。通过对地址转换和ID转换进行硬件级优化，NTB中地址转换可支持更复杂的转换方案，并最大限度地压缩了ID转换时间。实验结果表明，在所构建的HCI环境中，D-MNTBA的最大带宽及吞吐量分别可达
1500 MB/s和1.36 GB/s，ID转换时间降低至71 ns。相较于以太网卡，其带宽及吞吐量分别提升了约19.0%和40.2%。对比PEX8748，其ID转换时间缩短了约34.9%，带宽及吞吐量分别提升了约27.1%和51.1%，且系统稳定性更强，可有效支撑HCI中异构域的跨域传输。Abstract:Objective The integration of heterogeneous computing resource clusters into modern Hyper-Converged Infrastructure (HCI) systems imposes stricter performance requirements in latency, bandwidth, throughput, and cross-domain transmission stability. Traditional HCI systems primarily rely on the Ethernet TCP/IP protocol, which exhibits inherent limitations, including low bandwidth efficiency, high latency, and limited throughput. Existing PCIe Switch products typically employ Non-Transparent Bridges (NTBs) for conventional dual-system connections or intra-server communication; however, they do not meet the performance demands of heterogeneous cross-domain transmission within HCI environments. To address this limitation, a novel Dual-Mode Non-Transparent Bridge Architecture (D-MNTBA) is proposed to support dual transmission modes. D-MNTBA combines a fast transmission mode via a bypass mechanism with a stable transmission mode derived from the Traditional Data Path Architecture (TDPA), thereby aligning with the data characteristics and cross-domain streaming demands of HCI systems. Hardware-level enhancements in address and ID translation schemes enable D-MNTBA to support more complex mappings while minimizing translation latency. These improvements increase system stability and effectively support the cross-domain transmission of heterogeneous data in HCI systems. Methods To overcome the limitations of traditional single-pass architectures and the bypass optimizations of the TDPA, the proposed D-MNTBA incorporates both a fast transmission path and a stable transmission path. This dual-mode design enables the NTB to leverage the data characteristics of HCI systems for telegram-based streaming, thereby reducing dependence on intermediate protocols and data format conversions. The stable transmission mode ensures reliable message delivery, while the fast transmission mode—enhanced through hardware-level optimizations in address and ID translation—supports high-real-time cross-domain communication. This combination improves overall transmission performance by reducing both latency and system overhead. To meet the low-latency demands of the bypass transmission path, the architecture implements hardware-level enhancements to the address and ID conversion modules. The address translation module is expanded with a larger lookup table, allowing for more complex and flexible mapping schemes. This enhancement enables efficient utilization of non-contiguous and fragmented address spaces without compromising performance. Simultaneously, the ID conversion module is optimized through multiple conversion strategies and streamlined logic, significantly reducing the time required for ID translation. Results and Discussions Address translation in the proposed D-MNTBA is validated through emulation within a constructed HCI environment. The simulation log for indirect address translation shows no errors or deadlocks, and successful hits are observed on BAR2/3. During dual-host disk access, packet header addresses and payload content remain consistent, with no packet loss detected ( Fig. 14 ), indicating that indirect address translation is accurately executed under D-MNTBA. ID conversion performance is evaluated by comparing the proposed architecture with the TDPA implemented in the PEX8748 chip. The switch based on D-MNTBA exhibits significantly shorter ID conversion times. A maximum reduction of approximately 34.9% is recorded, with an ID conversion time of 71 ns for a 512-Byte payload (Fig. 15 ). These findings suggest that the ID function mapping method adopted in D-MNTBA effectively reduces conversion latency and enhances system performance. Throughput stability is assessed under sustained heavy traffic with payloads ranging from 256 to 2 048 Bytes. The maximum throughputs of D-MNTBA, the Ethernet card, and PEX8748 are measured at 1.36 GB/s, 0.97 GB/s, and 0.9 GB/s, respectively (Fig. 16 ). Compared to PEX8748 and the Ethernet architecture, D-MNTBA improves throughput by approximately 51.1% and 40.2%, respectively, and shows the slowest degradation trend, reflecting superior stability in heterogeneous cross-domain transmission. Bandwidth comparison reveals that D-MNTBA outperforms TDPA and the Ethernet card, with bandwidth improvements of approximately 27.1% and 19.0%, respectively (Fig. 17 ). These results highlight the significant enhancement in cross-domain transmission performance achieved by the proposed architecture in heterogeneous environments.Conclusions This study proposes a Dual-Mode D-MNTBA to address the challenges of heterogeneous interconnection in HCI systems. By integrating a fast transmission path enabled by a bypass architecture with the stable transmission path of the TDPA, D-MNTBA accommodates the specific data characteristics of cross-domain transmission in heterogeneous environments and enables efficient message routing. D-MNTBA enhances transmission stability while improving system-wide performance, offering robust support for high-real-time cross-domain transmission in HCI. It also reduces latency and overhead, thereby improving overall transmission efficiency. Compared with existing transmission schemes, D-MNTBA achieves notable gains in performance, making it a suitable solution for the demands of heterogeneous domain interconnects in HCI systems. However, the architectural enhancements, particularly the bypass design and associated optimizations, increase logic resource utilization and power consumption. Future work should focus on refining hardware design, layout, and wiring strategies to reduce logic complexity and resource consumption without compromising performance. -
表 1 path_select模块路选规则
报文类型 规则 传输模式 MWr/MRd 命中Bar0,Bar0/1 稳定传输 MWr/MRd 命中Bar2, Bar3, Bar4, Bar5, Bar4/5且path_mux=1 快速传输 MRd 命中Expansion ROM 稳定传输 MWr 命中Expansion ROM 丢弃 CPL/CPLD DMA使能且path_mux=1 快速传输 Credit(信用值) 原有数据通路信用量缺失 快速传输 
下载: 导出CSV
表 2 ID转换模式控制方式
寄存器模式 条目格式 Lut_format=0
(不支持function num检查)[15:8]bus num [7:3]device num [2]reserved [1]no snoop ctrl(1:清除；0:不清除) [0]entry en [15:8]bus num Lut_format=1
(支持function num检查)[7:3]device num [2:0]function num No snoop ctrl保证每一比特与每个entry一一对应
下载: 导出CSV
表 3 HCI异构互连场景资源部署情况
编号 资源名称 型号 数量 1 处理器 Intel i9-13900K 1 2 处理器 飞腾嵌入式：2000-4 1 3 处理器 龙芯2K30000 1 4 GPU显卡 NVIDIA-T1000 1 5 GPU显卡 华为升腾310B 1 6 GPU显卡 景嘉微JM930 1 7 GPU显卡 天数智芯智凯100 1 8 FPGA板卡 亚科19p 3 板卡 PEX8748 3 9 万兆以太网卡 Intelx520 3 10 NVMe SSD盘 联想SL7000-M 1 11 NVMe SSD盘 致钛TiPlus7100-M.2-1TB 1 12 NVMe SSD盘 Intel P4510-U.2-1TB 1 13 NVMe SSD盘 致钛PC005 Active-M.2-1TB 1 14 操作系统 Ubuntu 20.04.6 1 15 操作系统 Windows11 1
下载: 导出CSV
表 4 32 bit-64 bit间接地址转换测试用例描述
用例描述 通过准则 命中BAR2进行32-64
间接地址转换；
报文类型、地址长度
随机；(1)验证环境对比地址转换通过、包头地址、payload比对通过；
(2)验证环境监视过程中np_fifo未满，
无死锁；
(3)验证环境进行包计数的预期；
(4)用例仿真log，打印关键信息。
下载: 导出CSV
1 AA模式主备倒换测试
输入：定义初始默认角色为active, global_role = “active”；
倒换时间failover_start = 6；恢复时间failover_step2 = $(expr
$failover_start + 3)；重启时间failover_step3 = $(expr
$failover_start + 4)输出：主备倒换完成指示信号 (1) /*AA模式接管方操作，等待接管完成*/ (2) aa_passive_failover_strart() (3) echo_trace“passive start failover”；//客户端需要重启，
等待接管完成(4) aa_stop_client； (5) aa_passive_failover_step2()//
下载:
