Testing VPP =========== As of this writing, the vpp source tree includes over 1,000 unit test vectors. Best practices prior to pushing patches for code review: make sure that all of the “make test” test vectors pass. We attempt to maintain the top-level “make test-help” command so that it accurately describes all of the “make test” options. Examples -------- Basic test run, all test vectors, single-vpp instance, optimized image: :: $ make test 10-way parallel basic test run: :: $ make TEST_JOBS=10 test Run a specific test suite (mpls, in this case): :: $ make TEST=test_mpls test Run a specific test suite, debug image, pause prior to running the test suite; attach to the vpp image in gdb: :: $ make TEST=xxx DEBUG=gdb test-debug Detailed Documentation ---------------------- Current “make test-help” output: :: $ make test-help Running tests: test - build and run (basic) functional tests test-debug - build and run (basic) functional tests (debug build) test-all - build and run functional and extended tests test-all-debug - build and run functional and extended tests (debug build) retest - run functional tests retest-debug - run functional tests (debug build) retest-all - run functional and extended tests retest-all-debug - run functional and extended tests (debug build) test-cov - generate code coverage report for test framework test-gcov - build and run functional tests (gcov build) test-wipe - wipe (temporary) files generated by unit tests test-wipe-cov - wipe code coverage report for test framework test-wipe-papi - rebuild vpp_papi sources test-wipe-all - wipe (temporary) files generated by unit tests, and coverage test-shell - enter shell with test environment test-shell-debug - enter shell with test environment (debug build) test-checkstyle - check PEP8 compliance for test framework test-refresh-deps - refresh the Python dependencies for the tests Arguments controlling test runs: V=[0|1|2] - set test verbosity level 0=ERROR, 1=INFO, 2=DEBUG TEST_JOBS=[|auto] - use at most parallel python processes for test execution, if auto, set to number of available cpus (default: 1) MAX_VPP_CPUS=[|auto]- use at most cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto) CACHE_OUTPUT=[0|n|no] - disable cache VPP stdout/stderr and log as one block after test finishes (default: yes) FAILFAST=[1|y|yes] - fail fast if 1, otherwise complete all tests TIMEOUT= - fail test suite if any single test takes longer than (in seconds) to finish (default: 600) RETRIES= - retry failed tests times DEBUG= - set VPP debugging kind DEBUG=core - detect coredump and load it in gdb on crash DEBUG=gdb - allow easy debugging by printing VPP PID and waiting for user input before running and tearing down a testcase DEBUG=gdbserver - run gdb inside a gdb server, otherwise same as above DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb) STEP=[1|y|yes] - enable stepping through a testcase (for testcase debugging) SANITY=[0|n|no] - disable sanity import of vpp-api/sanity vpp run before running tests EXTENDED_TESTS=[1|y|yes] - run extended tests TEST= - filter the set of tests: by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py by wildcard - wildcard filter is .., each can be replaced by '*' e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name TEST='bfd.*.*' is equivalent to above example of filter by file-suffix TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes VARIANT= - specify which march node variant to unit test e.g. VARIANT=skx test the skx march variants e.g. VARIANT=icl test the icl march variants COREDUMP_SIZE= - pass as unix { coredump-size } argument to vpp e.g. COREDUMP_SIZE=4g COREDUMP_SIZE=unlimited COREDUMP_COMPRESS=[1|y|yes] - compress core files if not debugging them EXTERN_TESTS= - path to out-of-tree test_.py files containing test cases EXTERN_PLUGINS= - path to out-of-tree plugins to be loaded by vpp under test EXTERN_COV_DIR= - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report PROFILE=[1|y|yes] - enable profiling of test framework via cProfile module PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime) PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout) TEST_DEBUG=[1|y|yes] - enable debugging of the test framework itself (expert) API_FUZZ=[1|y|yes] - enable VPP api fuzz testing RND_SEED= - Seed RND with given seed Starting VPP in GDB for use with DEBUG=attach: test-start-vpp-in-gdb - start VPP in gdb (release) test-start-vpp-debug-in-gdb - start VPP in gdb (debug) Creating test code coverage report: test-cov - generate code coverage report for test framework test-wipe-cov - wipe code coverage report for test framework Verifying code-style: test-checkstyle - check PEP8 compliance