Testing VPP =========== As of this writing, the vpp source tree includes over 1,000 unit test vectors. Best practices prior to pushing patches for code review: make sure that all of the “make test” test vectors pass. We attempt to maintain the top-level “make test-help” command so that it accurately describes all of the “make test” options. Examples -------- Basic test run, all test vectors, single-vpp instance, optimized image: :: $ make test 10-way parallel basic test run: :: $ make TEST_JOBS=10 test Run a specific test suite (mpls, in this case): :: $ make TEST=test_mpls test Run a specific test suite, debug image, pause prior to running the test suite; attach to the vpp image in gdb: :: $ make TEST=xxx DEBUG=gdb test-debug Detailed Documentation ---------------------- Current “make test-help” output: :: $ make test-help test - build and run (basic) functional tests test-debug - build and run (basic) functional tests (debug build) test-all - build and run functional and extended tests test-all-debug - build and run functional and extended tests (debug build) retest - run functional tests retest-debug - run functional tests (debug build) retest-all - run functional and extended tests retest-all-debug - run functional and extended tests (debug build) test-cov - generate code coverage report for test framework test-gcov - build and run functional tests (gcov build) test-wipe - wipe (temporary) files generated by unit tests test-wipe-cov - wipe code coverage report for test framework test-wipe-doc - wipe documentation for test framework test-wipe-papi - rebuild vpp_papi sources test-wipe-all - wipe (temporary) files generated by unit tests, docs, and coverage test-shell - enter shell with test environment test-shell-debug - enter shell with test environment (debug build) test-checkstyle - check PEP8 compliance for test framework test-refresh-deps - refresh the Python dependencies for the tests Arguments controlling test runs: V=[0|1|2] - set test verbosity level 0=ERROR, 1=INFO, 2=DEBUG TEST_JOBS=[|auto] - use at most parallel python processes for test execution, if auto, set to number of available cpus (default: 1) MAX_VPP_CPUS=[|auto]- use at most cpus for running vpp main and worker threads, if auto, set to number of available cpus (default: auto) CACHE_OUTPUT=[0|1] - cache VPP stdout/stderr and log as one block after test finishes (default: 1) FAILFAST=[0|1] - fail fast if 1, complete all tests if 0 TIMEOUT= - fail test suite if any single test takes longer than (in seconds) to finish (default: 600) RETRIES= - retry failed tests times DEBUG= - set VPP debugging kind DEBUG=core - detect coredump and load it in gdb on crash DEBUG=gdb - allow easy debugging by printing VPP PID and waiting for user input before running and tearing down a testcase DEBUG=gdbserver - run gdb inside a gdb server, otherwise same as above DEBUG=attach - attach test case to already running vpp in gdb (see test-start-vpp-in-gdb) STEP=[yes|no] - ease debugging by stepping through a testcase SANITY=[yes|no] - perform sanity import of vpp-api/sanity vpp run before running tests (default: yes) EXTENDED_TESTS=[1|y] - used by '[re]test-all' & '[re]test-all-debug' to run extended tests TEST= - filter the set of tests: by file-name - only run tests from specified file, e.g. TEST=test_bfd selects all tests from test_bfd.py by file-suffix - same as file-name, but 'test_' is omitted e.g. TEST=bfd selects all tests from test_bfd.py by wildcard - wildcard filter is .., each can be replaced by '*' e.g. TEST='test_bfd.*.*' is equivalent to above example of filter by file-name TEST='bfd.*.*' is equivalent to above example of filter by file-suffix TEST='bfd.BFDAPITestCase.*' selects all tests from test_bfd.py which are part of BFDAPITestCase class TEST='bfd.BFDAPITestCase.test_add_bfd' selects a single test named test_add_bfd from test_bfd.py/BFDAPITestCase TEST='*.*.test_add_bfd' selects all test functions named test_add_bfd from all files/classes VARIANT= - specify which march node variant to unit test e.g. VARIANT=skx test the skx march variants e.g. VARIANT=icl test the icl march variants COREDUMP_SIZE= - pass as unix { coredump-size } argument to vpp e.g. COREDUMP_SIZE=4g COREDUMP_SIZE=unlimited COREDUMP_COMPRESS=1 - compress core files if not debugging them EXTERN_TESTS= - path to out-of-tree test_.py files containing test cases EXTERN_PLUGINS= - path to out-of-tree plugins to be loaded by vpp under test EXTERN_COV_DIR= - path to out-of-tree prefix, where source, object and .gcda files can be found for coverage report PROFILE=1 - enable profiling of test framework via cProfile module PROFILE_SORT_BY=opt - sort profiling report by opt - consult cProfile documentation for possible values (default: cumtime) PROFILE_OUTPUT=file - output profiling info to file - use absolute path (default: stdout) TEST_DEBUG=1 - turn on debugging of the test framework itself (expert) SKIP_AARCH64=1 - skip tests that are failing on the ARM platorm in FD.io CI RND_SEED=seed - Seed RND with given seed Starting VPP in GDB for use with DEBUG=attach: test-start-vpp-in-gdb - start VPP in gdb (release) test-start-vpp-debug-in-gdb - start VPP in gdb (debug) Arguments controlling VPP in GDB runs: VPP_IN_GDB_TMP_DIR - specify directory to run VPP IN (default: /tmp/unittest-attach-gdb) VPP_IN_GDB_NO_RMDIR=0 - don't remove existing tmp dir but fail instead VPP_IN_GDB_CMDLINE=1 - add 'interactive' to VPP arguments to run with command line Creating test documentation test-doc - generate documentation for test framework test-wipe-doc - wipe documentation for test framework Creating test code coverage report test-cov - generate code coverage report for test framework test-wipe-cov - wipe code coverage report for test framework Verifying code-style test-checkstyle - check PEP8 compliance