1.1. OverviewΒΆ

This is the Fast Data I/O Project (FD.io) Continuous System Integration and Testing (CSIT) project report for CSIT rls1801_2 system testing of VPP-18.01.2 release.

This is the full html version, there is also a reduced PDF version of this report.

The report describes CSIT functional and performance tests and their continuous execution delivered in CSIT rls1801_2. A high-level overview is provided for each CSIT test environment running in LF FD.io Continuous Performance Labs. This is followed by summary of all executed tests against the VPP-18.01.2 release and associated FD.io projects and sub-systems (Honeycomb, DPDK, NSH_SFC), CSIT rls1801_2 release notes, result highlights and known issues discovered in CSIT. More detailed description of each environment, pointers to CSIT test code documentation and detailed test resuls with links to the source data files are also provided.

CSIT rls1801_2 report contains following main sections and sub-sections:

  1. Introduction - general introduction to CSIT project; Overview - this section; CSIT Test Naming - CSIT general naming convention for test suites and test cases, important to recognize the high-level test content and interpret reported results; General Notes - general notes related to this report.
  2. VPP Performance Tests - VPP performance tests executed in physical FD.io testbeds; Overview - tested topologies, test coverage and naming specifics, methodology for multi-core, packet throughput and latency, and KVM VM vhost tests; CSIT Release Notes - changes in CSIT rls1801_2, added tests, performance changes, environment or methodology changes, known CSIT issues; Packet Throughput Graphs and Packet Latency Graphs - plotted NDR, PDR throughput and latency results from multiple test job executions; Test Environment - environment description; Documentation - CSIT source code documentation for VPP performance tests.
  3. VPP Functional Tests - VPP functional tests executed in virtual FD.io testbeds; Overview - tested virtual topologies, test coverage and naming specifics; CSIT Release Notes - changes in CSIT rls1801_2, added tests, environment or methodology changes, known CSIT issues, tests to be added; Test Environment - environment description ; Documentation - source code documentation for VPP functional tests.
  4. Honeycomb Functional Tests - Honeycomb functional tests executed in virtual FD.io testbeds; Overview - tested virtual topologies, test coverage and naming specifics; CSIT Release Notes - changes in CSIT rls1801_2, added tests, environment or methodology changes, known CSIT issues; Test Environment - environment description ; Documentation - source code documentation for Honeycomb functional tests.
  5. NSH_SFC Functional Tests - NSH_SFC functional tests executed in virtual FD.io testbeds; Overview - tested virtual topologies, test coverage and naming specifics; CSIT Release Notes - changes in CSIT rls1801_2, added tests, environment or methodology changes, known CSIT issues; Test Environment - environment description ; Documentation - source code documentation for NSH_SFC functional tests.
  6. Detailed Test Results - auto-generated results from CSIT jobs executions using CSIT Robot Framework output files as source data; VPP Performance Results, VPP Functional Results, Honeycomb Functional Results, VPPtest Functional Results.
  7. Test Configuration - auto-generated DUT configuration data from CSIT jobs executions using CSIT Robot Framework output files as source data; VPP Performance Test Configs, VPP Functional Test Configs.
  8. Test Operational Data - auto-generated DUT operational data from CSIT jobs executions using CSIT Robot Framework output files as source data; VPP Performance Operational Data.
  9. CSIT Framework Documentation - description of the overall CSIT framework design hierarchy, CSIT test naming convention, followed by description of Presentation and Analytics Layer (PAL) introduced in CSIT rls1801_2.