Benchmarking Methodology for AI Training Network Fabrics

1.1. Requirements Language

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

1.2. Scope and Applicability

This document applies to Ethernet-based AI training backend network fabrics employing RoCEv2 and/or UEC Ultra Ethernet Transport (UET) protocols. The scope includes leaf-spine (2-tier Clos) and leaf-spine-superspine (3-tier Clos) topologies.¶

InfiniBand fabrics are explicitly out of scope, though many KPIs defined herein may be adapted for IB benchmarking by future documents. The DUT is the network fabric itself (the collection of switches and interconnecting links), not individual accelerators or host NICs, though host-side configuration MUST be documented as it materially affects results.¶

The methodology is designed for controlled laboratory environments per the BMWG charter; it is NOT intended for production network measurement.¶

1.3. Relationship to Existing BMWG Work

3.1. Reference Fabric Topologies

Three reference topologies are defined. All tests MUST specify which topology is used. Results obtained under different topologies are NOT directly comparable without normalization.¶

+--------+ +--------+ +--------+ +--------+
| Spine1 | | Spine2 | | Spine3 | | SpineN |
+--++--+  +--++--+  +--++--+  +--++--+
   ||         ||         ||         ||
   ||   Full Mesh Interconnect       ||
   ||   (ECMP / DLB / Spray)        ||
   ||         ||         ||         ||
+--++--+  +--++--+  +--++--+  +--++--+
| Leaf 1 | | Leaf 2 | | Leaf 3 | | Leaf N |
+--++--+  +--++--+  +--++--+  +--++--+
   ||         ||         ||         ||
[GPU/XPU] [GPU/XPU] [GPU/XPU] [GPU/XPU]
Hosts w/  Hosts w/  Hosts w/  Hosts w/
RoCEv2 NIC            RoCEv2 NIC

                       SPINE LAYER
+--------+ +--------+ +--------+ +--------+
| Spine1 | | Spine2 | | Spine3 | | SpineN |
+--+--+--+ +--+--+--+ +--+--+--+ +--+--+--+
 |     Full Mesh Interconnect (ECMP/Spray)  |
+--+--+--+ +--+--+--+ +--+--+--+ +--+--+--+
| Rail-0 | | Rail-1 | | Rail-2 | | Rail-7 |  RAIL (LEAF) LAYER
|  Leaf  | |  Leaf  | |  Leaf  | |  Leaf  |  one switch per NIC
+--+--+--+ +--+--+--+ +--+--+--+ +--+--+--+
  |   |       |   |       |   |      |   |
NIC-0 NIC-0 NIC-1 NIC-1 NIC-2 NIC-2 NIC-7 NIC-7
  |   |       |   |       |   |      |   |
+--------+ +--------+ +--------+ +--------+
| Host A | | Host B | | Host C | | Host D |  GPU HOSTS
| GPU[0] | | GPU[0] | | GPU[0] | | GPU[0] |  (each host has
| GPU[1] | | GPU[1] | | GPU[1] | | GPU[1] |   8 NICs, one
|  ...   | |  ...   | |  ...   | |  ...   |   per rail)
| GPU[7] | | GPU[7] | | GPU[7] | | GPU[7] |
+--------+ +--------+ +--------+ +--------+
|<------ Rail-0 ------->|         |<-Rail-7->|

3.2. Device Under Test (DUT) Identification

3.3. Traffic Generator Requirements

4.1. Primary KPIs

4.2. Secondary KPIs

4.3. Fabric Health Indicators

5.1. Baseline Throughput

Objective: Determine the maximum sustainable RDMA Write throughput through the DUT fabric at each tested message size.¶

Procedure:¶

• Configure N host pairs, each establishing Q Queue Pairs per pair¶

• Initiate RDMA Write operations and measure aggregate goodput¶

• Test MUST run for at least 60 seconds at each rate¶

• Binary search per [RFC2544] Section 26.1 SHOULD be used¶

• Message sizes: 64B, 256B, 1KB, 4KB, 64KB, 256KB, 1MB, 4MB¶

• QP counts: 1, 4, 16, 32 per src-dst pair¶

• Test both unidirectional and bidirectional traffic¶

Reporting: Report aggregate throughput (Tbps), per-port utilization (%), and throughput efficiency (measured/theoretical). Present as table indexed by message size x QP count, and as graph (message size on X-axis).¶

5.2. Latency Characterization

Objective: Determine one-way and round-trip RDMA latency distribution at the throughput rate from Section 5.1.¶

Procedure:¶

• Inject tagged frames at 60s into a 120s stream (per [RFC2544] Section 26.2)¶

• Nanosecond-precision timestamping¶

• MUST report: min, mean, P50, P95, P99, P99.9, max¶

• Repeat at least 20 times; report averages¶

• Test under both zero-load (single QP) and loaded (full fabric utilization) conditions¶

Reporting: Tabulate latency statistics per message size. Provide histogram and CDF plot. Report latency increase factor (loaded/unloaded).¶

5.3. Back-to-Back Burst Absorption

Objective: Characterize the DUT fabric's ability to absorb back-to-back RDMA bursts without loss, extending [RFC9004] methodology for RoCEv2.¶

Procedure:¶

• Transmit bursts at line rate with minimum inter-frame gap¶

• Increase burst length until first frame loss is detected¶

• Test incast ratios: 2:1, 4:1, 8:1, 16:1, 32:1¶

• Repeat at least 50 times per burst length¶

Reporting: Report burst absorption capacity (frames and bytes) for each message size and incast ratio. Plot burst capacity vs. incast ratio.¶

6.1. UET Throughput by Transport Service

Objective: Determine maximum sustainable throughput under each UET transport service (ROD, RUD, RUDI, UUD) and compare to RoCEv2 RC/UC on the same DUT fabric.¶

Procedure: Use UEC 1.0-compliant NICs; establish PDCs; use libfabric fi_write. Apply binary search ([RFC2544] Section 26.1). Vary PDC counts: 1, 4, 16, 32. A parallel RoCEv2 test series MUST be executed. Both unidirectional and bidirectional configurations MUST be tested.¶

Reporting template:¶

6.2. UET Latency Characterization

Objective: Measure latency distribution for UET transport services; quantify differential vs. RoCEv2, with particular attention to connectionless PDC establishment overhead.¶

Procedure: Measure latency for: (a) steady-state PDC transfers; (b) first-packet latency (PDC + first data packet, measuring "data before handshake"); (c) zero-load baseline. Test ROD and RUD separately to isolate reordering-related latency.¶

Reporting: Tabulate latency statistics per (transport_service, message_size, load_condition) tuple. Plot latency CDF for UET ROD, UET RUD, and RoCEv2 RC side-by-side.¶

6.3. Packet Spray Efficacy Under UET RUD

Objective: Quantify the load balancing improvement achieved by UET's native per-packet spray with RUD, which eliminates the receiver reorder buffer constraint.¶

Procedure: Test four configurations:¶

• UET RUD + packet spray¶

• UET ROD + packet spray¶

• RoCEv2 RC + packet spray¶

• RoCEv2 RC + standard ECMP (baseline)¶

Measure MMR, JFI, out-of-order delivery rate, retransmission rate, and effective goodput. Vary ECMP paths: 4, 8, 16, 32.¶

Reporting template:¶

• UET RUD SHOULD achieve zero host-visible reordering despite per-packet spray because the transport layer natively tolerates unordered delivery.¶

6.4. UET Congestion Control Benchmarks

Objective: Evaluate UET's dual-sided (sender + receiver) congestion control under N:1 incast conditions vs. RoCEv2 DCQCN.¶

Procedure: Measure: (a) incast throughput at N = {2, 4, 8, 16, 32, 64}; (b) convergence time after doubling active senders (until all flows within 10% of fair share); (c) PFC avoidance with PFC disabled on the DUT; (d) receiver credit utilization.¶

Reporting: Tabulate incast throughput, convergence time, peak queue depth, PFC event count, and packet drop rate for UET vs. DCQCN per incast ratio. Critical differentiator: report whether UET achieves zero application-visible loss without PFC.¶

6.5. Link Layer Enhancement Benchmarks

Objective: Measure performance impact of optional link-layer enhancements: LLR, Packet Trimming (PT), and CBFC.¶

Procedure:¶

• (a) LLR Retry Latency: inject controlled bit errors; measure LLR retry latency (expected sub-microsecond per hop) vs. transport-layer retransmission (~10-100us RTT). Run with 80% background load.¶

• (b) Packet Trimming Effectiveness: configure 2:1 oversubscription bottleneck; measure time from congestion onset to first retransmission request, bandwidth saved vs. full-packet drops.¶

• (c) CBFC vs. PFC: identical N:1 (N=32) incast scenarios; measure head-of-line blocking duration (CBFC is per-destination, PFC is per-priority), pause propagation hops, and throughput of non-congested flows.¶

Reporting: Before/after comparison table for each enhancement. Note which features are hardware-supported vs. software-emulated.¶

6.6. UET Collective Communication Performance

Objective: Measure collective communication (AllReduce, AlltoAll, AllGather) performance over UET and compare to RoCEv2.¶

Procedure: Execute the collective benchmark suite from Section 9 over UET RUD transport using a UEC-compliant collective library. Same accelerator count, message sizes, and fabric topology. Run UET RUD + packet spray as primary; UET ROD + ECMP as secondary baseline.¶

Reporting template:¶

6.7. UET PDC Scalability and Connection Setup Rate

Objective: Measure PDC establishment rate and maximum concurrent PDC count vs. RoCEv2 QP-based connections.¶

Procedure: (a) PDC establishment rate: initiate PDC creation to M = {100, 1000, 10000, 100000} remote endpoints. (b) Data-before-handshake: measure first-byte latency for UET vs. RoCEv2 RDMA Write. (c) Maximum concurrent PDC count: scale until per-PDC throughput drops below 90% of single-PDC rate. The UEC specification targets up to 1 million endpoints.¶

7.1. ECN Marking Accuracy and Threshold

Objective: Verify that the DUT marks packets with ECN CE at the configured threshold with correct granularity.¶

Procedure: Configure threshold T on DUT egress queue. Verify: (a) no packets marked below T; (b) 100% marked above maximum threshold; (c) appropriate WRED/RED probability ramp between thresholds. Test thresholds: low (~100KB), medium (~1MB), high (~5MB).¶

Reporting: Plot ECN marking probability vs. instantaneous queue depth. Report measured threshold accuracy (deviation from configured).¶

7.2. PFC Behavior Under Incast

Objective: Characterize DUT's PFC generation behavior under N:1 incast conditions.¶

Procedure: Generate N:1 incast at 100% line rate, N = {2, 4, 8, 16, 32, 64}. Measure PFC PAUSE frame count/sec per hop, PFC PAUSE duration per port, PFC storm onset, and end-to-end throughput. Test SHOULD verify correct headroom sizing and PFC watchdog effectiveness.¶

7.3. DCQCN Convergence Time

Objective: Measure time for DCQCN to converge to fair-share rate after congestion onset.¶

Procedure: Establish M flows through a common bottleneck. At T0, inject additional M flows (creating 2:1 oversubscription). Measure time until all 2M flows achieve rates within 10% of fair share. Repeat for M = {4, 16, 64, 256}. Vary DCQCN parameters and report sensitivity.¶

7.4. PFC Storm and Deadlock Resilience

Objective: Verify the DUT does not enter PFC deadlock or sustained PFC storm under adversarial traffic.¶

Procedure: Generate cyclic traffic patterns known to cause PFC deadlocks. Run for 300 seconds. The DUT MUST demonstrate resilience via PFC watchdog or architectural immunity (e.g., VOQ-based scheduling).¶

8.1. ECMP Entropy and Polarization

Objective: Quantify traffic polarization under standard ECMP hashing for AI training flow patterns.¶

Procedure: Configure standard 5-tuple ECMP. Generate traffic with Q = {1, 4, 8, 16, 32} QPs per src-dst pair. Measure per-link utilization, MMR, and JFI. Test with and without BTH-aware hashing. Repeat for fabric sizes of 8, 16, 32, and 64 leaf switches.¶

8.2. Dynamic Load Balancing (Flowlet)

Objective: Evaluate DUT's flowlet-based DLB performance and compare to baseline ECMP.¶

Procedure: Configure vendor-specific DLB (document algorithm type). Generate traffic with Q=4 QPs. Measure MMR, JFI, per-link utilization, out-of-order rate. Vary flowlet gap timer and report sensitivity.¶

8.3. Packet Spraying

Objective: Evaluate DUT's per-packet spraying performance and quantify the utilization vs. reordering tradeoff.¶

Procedure: Configure per-packet load balancing. Measure MMR (expected ~1.0), JFI (expected ~1.0), out-of-order rate, and RDMA retransmission impact. If the DUT provides an in-fabric reorder buffer, document per Appendix B.¶

8.4. Jain Fairness Index Measurement

Objective: Single-number summary of load balancing quality comparable across all strategies.¶

Formula:¶

JFI = (Sum LinkTx_i)^2 / (N x Sum LinkTx_i^2)

where LinkTx_i = transmitted traffic on fabric link i, N = total parallel links. Range: 1/N (worst) to 1.0 (perfect).¶

Reporting: Report JFI for each load balancing strategy. Provide bar chart comparing ECMP, DLB, and packet spray.¶

9.1. AllReduce Benchmark

Objective: Measure fabric performance during AllReduce operations -- the dominant collective for gradient synchronization in data-parallel training.¶

Procedure:¶

• Message sizes: 1MB, 8MB, 64MB, 256MB, 1GB, 4GB¶

• Accelerator counts: N = {8, 16, 32, 64, 128, 256, 512, 1024}¶

• Execute at least 100 iterations per (message_size, N) pair¶

• Report average, P50, P95, P99 BusBW¶

• Run under each load balancing strategy (ECMP, DLB, spray)¶

9.2. AlltoAll Benchmark

Objective: Measure fabric performance during AlltoAll operations -- used in Mixture-of-Experts (MoE) models and expert parallelism.¶

Procedure: Execute AlltoAll with same parameters as Section 9.1. AlltoAll creates the worst-case congestion scenario: every accelerator simultaneously sends to every other. Report JCT per iteration -- the most sensitive indicator of fabric congestion management quality.¶

9.3. AllGather Benchmark

Objective: Measure fabric performance during AllGather operations -- used for parameter distribution in tensor-parallel and pipeline-parallel training.¶

Procedure: Execute AllGather with same parameter sweeps as Section 9.1. Measure BusBW, JCT, and fabric health indicators.¶

9.4. Collective Communication Library Bus Bandwidth Summary

Reporting template:¶

10.1. Synthetic JCT Under Controlled Conditions

Objective: Measure JCT for a defined synthetic workload with a known computation-to-communication ratio to isolate fabric-induced overhead.¶

Procedure: Define a synthetic training iteration as:¶

1. Computation phase of C milliseconds (simulated sleep or GPU compute kernel)¶

2. Communication phase: AllReduce of S bytes across N accelerators¶

Roofline JCT = Iterations x (C + S x algo_factor / NIC_line_rate)
JCT Ratio    = Measured_JCT / Roofline_JCT

• JCT Ratio < 1.05 = excellent fabric performance; > 1.15 = significant fabric-induced overhead.¶

10.2. MLPerf-Aligned JCT

Objective: Measure JCT using MLPerf Training benchmark workloads [MLPERF] to enable comparison with published industry results.¶

Procedure: Execute MLPerf Training closed-division workloads (e.g., BERT, ResNet, GPT-3 175B) per MLPerf submission rules. Simultaneously capture all fabric KPIs from Section 4. Report time-to-train and/or tokens-per-second.¶

10.3. Multi-Tenant JCT Interference

Objective: Quantify JCT impact when multiple training jobs share the same fabric.¶

Procedure: Configure two or more independent training jobs. Jobs SHOULD overlap in spine-layer link usage. Measure baseline JCT (isolated) and contention JCT (simultaneous).¶

JCT Interference Factor = Contention_JCT / Baseline_JCT

Test with spine link overlap: 0%, 25%, 50%, 75%.¶

11.1. Fabric Scale Limits

Objective: Determine the maximum fabric scale at which the DUT maintains acceptable KPI performance.¶

Procedure: Progressively increase active accelerator endpoints from N=64 to maximum topology support while running AllReduce (Section 9.1, S=1GB). At each scale point record JCT Ratio, BusBW, ECN ratio, PFC count, CPU and memory utilization. Also measure BGP/routing convergence time after clearing all adjacencies (analogous to [EVPN-BENCH] Sections 3.10, 3.11, 4.9, 4.10).¶

11.2. Link Failure Convergence

Objective: Measure traffic disruption and JCT impact when a fabric link fails during active training.¶

Procedure: With the fabric fully loaded (AllReduce, N=128, S=1GB), administratively fail a spine uplink. Measure:¶

• Duration of packet loss¶

• Packets lost¶

• JCT overhead for the failure iteration vs. steady state¶

• Time for load balancing mechanism to redistribute flows¶

Repeat for: leaf uplink failure, spine switch failure, superspine link failure (if applicable). Test under each load balancing strategy.¶

11.3. Zero-Impact Failover Measurement

Objective: Verify vendor claims of zero-impact or sub-microsecond failover.¶

Procedure: Execute Section 11.2 with nanosecond-precision measurement. A failure is considered "zero-impact" if the measured JCT for the failure iteration is within the P99 JCT of steady-state iterations.¶

12.1. 24-Hour Sustained Load

Objective: Verify DUT fabric stability under sustained AI training load over an extended period, following the methodology pattern from [EVPN-BENCH] Sections 3.12, 4.11.¶

Procedure: Configure DUT at maximum validated scale from Section 11.1. Generate bidirectional collective communication traffic (alternating AllReduce and AlltoAll). Run continuously for 24 hours. Sample all KPIs from Section 4 every 60 seconds.¶

There SHOULD NOT be any memory leaks, crashes, or CPU spikes. Any anomaly MUST be reported with timestamp and duration.¶

Reporting: Time-series plots of JCT Ratio, BusBW, ECN ratio, PFC count, CPU, and memory over the 24-hour period. Report standard deviation of JCT Ratio (stability metric).¶

12.2. Resource Leak Detection

Objective: Detect memory leaks, handle exhaustion, or gradual performance degradation in DUT software.¶

Procedure: Record per-process memory usage at T=0, T=1h, T=6h, T=12h, T=24h. Compute linear regression slope of memory usage over time. A slope exceeding 1MB/hour for any process indicates a potential memory leak and MUST be reported. Also monitor forwarding-plane counter wraparounds and hardware table occupancy trends.¶

16.1. Normative References

[RFC1242]

Bradner, S., "Benchmarking Terminology for Network Interconnection Devices", RFC 1242, DOI 10.17487/RFC1242, July 1991, <https://www.rfc-editor.org/rfc/rfc1242>.

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC2544]

Bradner, S. and J. McQuaid, "Benchmarking Methodology for Network Interconnect Devices", RFC 2544, DOI 10.17487/RFC2544, March 1999, <https://www.rfc-editor.org/rfc/rfc2544>.

[RFC2889]

Mandeville, R. and J. Perser, "Benchmarking Methodology for LAN Switching Devices", RFC 2889, DOI 10.17487/RFC2889, August 2000, <https://www.rfc-editor.org/rfc/rfc2889>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

[RFC8238]

Avramov, L. and J. Rapp, "Data Center Benchmarking Terminology", RFC 8238, DOI 10.17487/RFC8238, August 2017, <https://www.rfc-editor.org/rfc/rfc8238>.

[RFC8239]

Avramov, L. and J. Rapp, "Data Center Benchmarking Methodology", RFC 8239, DOI 10.17487/RFC8239, August 2017, <https://www.rfc-editor.org/rfc/rfc8239>.

[RFC9004]

Morton, A., "Updates for the Back-to-Back Frame Benchmark in RFC 2544", RFC 9004, DOI 10.17487/RFC9004, May 2021, <https://www.rfc-editor.org/rfc/rfc9004>.

[UEC-1.0]

Ultra Ethernet Consortium, "Ultra Ethernet Transport (UET) Specification 1.0", June 2025, <https://ultraethernet.org>.

16.2. Informative References

[DCQCN-PAPER]

Zhu, Y., "Congestion Control for Large-Scale RDMA Deployments", DOI ACM SIGCOMM 2015, 2015, <https://doi.org/ACM SIGCOMM 2015>.

[EVPN-BENCH]

Jacob, S. and K. Tiruveedhula, "Benchmarking Methodology for EVPN and PBB-EVPN", Work in Progress, Internet-Draft, draft-ietf-bmwg-evpntest-11, August 2023, <https://datatracker.ietf.org/doc/html/draft-ietf-bmwg-evpntest-11>.

[LIBFABRIC]

OpenFabrics Interfaces Working Group, "libfabric: Open Fabric Interfaces", n.d., <https://ofiwg.github.io/libfabric/>.

[LLM-BENCH]

Gaikwad, et al, "Benchmarking Methodology for Large Language Model Serving", Work in Progress, Internet-Draft, draft-gaikwad-llm-benchmarking-methodology-00, January 2026, <https://datatracker.ietf.org/doc/html/draft-gaikwad-llm-benchmarking-methodology-00>.

[META-ROCE]

Gangidi, A., "RDMA over Ethernet for Distributed AI Training at Meta Scale", DOI ACM SIGCOMM 2024, 2024, <https://doi.org/ACM SIGCOMM 2024>.

[MLPERF]

MLCommons, "MLPerf Training Benchmark Suite", n.d., <https://mlcommons.org>.

[NCCL]

NVIDIA, "NVIDIA Collective Communications Library (NCCL)", n.d., <https://developer.nvidia.com/nccl>.

[RFC6815]

Bradner, S., Dubray, K., McQuaid, J., and A. Morton, "Applicability Statement for RFC 2544: Use on Production Networks Considered Harmful", RFC 6815, DOI 10.17487/RFC6815, November 2012, <https://www.rfc-editor.org/rfc/rfc6815>.

D.1. Key Differences from RoCEv2

1. UDP Destination Port: UET uses port 4793 vs. RoCEv2 port 4791.¶

2. Entropy Value: Explicit entropy field for ECMP path selection. Test equipment MUST vary this field to achieve uniform path distribution.¶

3. Transport Service Indicator: Header encodes transport service (ROD/RUD/RUDI/UUD). Tests MUST set this to match the service being benchmarked.¶

4. PDC Identifier: Connectionless PDC ID replaces RoCEv2's Destination QP. Test equipment MUST track PDC lifecycle for accurate measurement.¶

5. Layered Sub-Headers: UET uses four sub-layers (SES, PDS, CMS, TSS) with variable-length headers. Implementations MUST follow [UEC-1.0] Section 4 for wire format details.¶

6. Optional Link Layer Headers: When LLR, Packet Trimming, or PRI features are enabled, additional link-layer framing may be present. Test equipment MUST be configured to recognize and parse these.¶