Packet Throughput

Throughput graphs are generated based on the results data obtained from the CSIT-2206 test jobs. In order to verify benchmark results repeatibility selected, CSIT performance tests are executed multiple times (target: 10 times) on each physical testbed type. Box-and-Whisker plots are used to display variations in measured throughput values.

Lists of tests selected for multiple execution and graphing are captured per testbed type in test_select_list_{testbed_type}.md files.

Graphs are split into sections as follows:

  1. Header 1: VPP packet path and lookup types

    • L2 Ethernet Switching: L2 bridge-doman, L2 cross-connect and L2 patch

    • IPv4 Routing: IPv4 routing with /32 prefixes

    • IPv6 Routing: IPv6 routing with /128 prefixes

    • SRv6 Routing: SRv6 with IPv6 routing

    • IPv4 Tunnels: IPv4 overlay tunnels

    • KVM VMs vhost-user: KVM VMs connected over virtio and vhost-user interfaces

    • LXC/DRC Container Memif: Linux containers and Docker containers connected over Memif interfaces

    • IPsec IPv4 Routing: IPsec encryption/decryption with IPv4 routing

  2. Header 2: testbeds and NIC models

    • section name format:

      • {testbed_type}-{nic_model}

    • testbed_type:

      • 2n-icx: 2-node Xeon Icelake

      • 3n-icx: 3-node Xeon Icelake

      • 2n-aws: 2-node AWS

      • 2n-skx: 2-node Xeon Skylake

      • 3n-skx: 3-node Xeon Skylake

      • 2n-clx: 2-node Xeon Cascade Lake

      • 2n-zn2: 2-node AMD Zen2

      • 3n-alt: 2-node Arm Altra

      • 3n-tsh: 3-node Arm TaiShan

      • 2n-tx2: 2-node Arm ThunderX2

      • 2n-dnv: 2-node Atom Denverton

      • 3n-dnv: 3-node Atom Denverton

    • nic_model:

      • xxv710: xxv710 2p25GE Intel (Fortville)

      • x710: x710 4p10GE Intel (Fortville)

      • xl710: xl710 2p40GE Intel (Fortville)

      • x520: x520 2p10GE Intel (Niantic)

      • x553: x553 2p10GE Intel (Niantic)

      • cx556a: cx556a-edat 2p100GE Mellanox ConnectX5

      • e810cq: E810-2CQDA2 2p100GE Intel Columbiaville

  3. Header 3: test group names

    • section name format:

      • {frame_size}-{worker_thread_core_cfg}-{vpp_functionality}-{vpp_lookup_type}-{baseline_scale}-{nic_driver}

    • frame_size:

      • 64b: 64 byte frames, smallest frame size for untagged IPv4 packets

      • 78b: 78 byte frames, smallest frame size for untagged IPv6 packets

      • 114b: VXLAN encapsulated L2 frames

      • imix: a sequence of (7x64B, 4x570, 1x1518) byte frames

    • worker_thread_core_cfg:

      • 1t1c: 1 worker thread on 1 core, hyper-threading not used

      • 2t1c: 2 worker threads on 1 core, hyper-threading used

    • vpp_functionality (optional):

      • features: including input-acl, output-acl, macip-iacl, nat44

      • srv6: srv6 encap/decap, proxy

      • link-bonding: L2 link aggregation with 1 or 2 bonded links

      • ipsec: IPsec encryption/decryption with different ciphers

      • vts: Virtual Topology System specific tests

    • vpp_lookup_type:

      • l2switching, ip4routing, ip6routing, ip4tunnel, vhost, memif

    • baseline_scale:

      • base: baseline tests with less than 10 forwarding entries

      • scale: scale tests with up to 2 million forwarding entries

      • base-scale: both baseline and scale tests grouped together

    • nic_driver:

      • avf: VPP native avf driver for Intel Fortville NICs

      • i40e: dpdk poll mode driver for Intel Fortville NICs

      • ixgbe: dpdk poll mode driver for Intel Niantic NICs

For each test case, Box-and-Whisker plots show the quartiles (Min, 1st quartile / 25th percentile, 2nd quartile / 50th percentile / mean, 3rd quartile / 75th percentile, Max) across collected data set. Outliers are plotted as individual points.

Additional information about graph data:

  1. Graph Title: describes tested packet path, testbed topology, processor model, NIC model, packet size, number of cores and threads used by data plane workers and indication of VPP DUT configuration.

  2. X-axis Labels: indices of individual test suites as listed in Graph Legend.

  3. Y-axis Labels: measured Packets Per Second [pps] throughput values.

  4. Graph Legend: lists X-axis indices with associated CSIT test suites executed to generate graphed test results.

  5. Hover Information: lists minimum, first quartile, median, third quartile, and maximum. If either type of outlier is present the whisker on the appropriate side is taken to 1.5×IQR from the quartile (the “inner fence”) rather than the max or min, and individual outlying data points are displayed as unfilled circles (for suspected outliers) or filled circles (for outliers). (The “outer fence” is 3×IQR from the quartile.)

Note

Test results are stored in build logs from FD.io vpp performance job 2n-icx, build logs from FD.io vpp performance job 3n-icx, build logs from FD.io vpp performance job 2n-aws, build logs from FD.io vpp performance job 2n-skx, build logs from FD.io vpp performance job 3n-skx, build logs from FD.io vpp performance job 2n-clx, build logs from FD.io vpp performance job 2n-zn2, build logs from FD.io vpp performance job 3n-alt, build logs from FD.io vpp performance job 3n-tsh, build logs from FD.io vpp performance job 2n-tx2, build logs from FD.io vpp performance job 2n-dnv and build logs from FD.io vpp performance job 3n-dnv with RF result files csit-vpp-perf-2206-*.zip archived here. Required per test case data set size is 10, but for VPP tests the actual size varies per test case and is <=10.