VPP with VMware/Vmxnet3

This section describes a native Vmxnet3 driver that is included with VPP. This driver is written as a plugin and is found in src/plugin/vmxnet3.

Advantages

The native VPP native vmxnet3 driver provides the following features that are not provided with the standard dpdk vmxnet3 driver.

Interrupt mode
Adaptive mode
LRO/TSO mode

Does not support

This driver does yet support the following features.

VLAN filter

Prerequisites

This code is tested with vfio-pci driver installed with Ubuntu 18.04 which has kernel version 4.15.0-33-generic.
This code is tested with ESXi vSwitch version 6.5 and 6.7 for LRO/TSO support, VMware Workstation 15 Pro (no LRO/TSO), and VMware Fusion 11 Pro (no LRO/TSO).
Driver requires MSI-X interrupt support, which is not supported by uio_pci_generic driver, so vfio-pci needs to be used. On systems without IOMMU vfio driver can still be used with recent kernels which support no-iommu mode.

VMware Fusion for Mac

VMware fusion does not have a menu option to change the default driver (e1000) to the vmxnet3 driver. VPP supports the vmxnet3 driver.

These instructions describe how to change the e1000 driver for VMware fusion.

From the VMware Fusion menu bar select Window then Virtual Machine Library.
From the Virtual Machine menu right click on the Virtual Machine you are using and select Show in Finder
Find the name associated with the VM you are using, right click on it and select Show Package Contents
Find the .vmx file and edit it.
Find all the occurrences of e1000 and change them to vmxnet3

If you are concerned more with configuration not performance the vmxnet3 driver can be set to interrupt mode in VPP. This will save a great deal on battery usage. Do this with the following

VMware Workstation PRO 15 for Linux

VMware Workstation does not have a menu option to change the default driver (e1000) to the vmxnet3 driver. VPP supports the vmxnet3 driver.

These instructions describe how to change the e1000 driver for VMware Workstation PRO 15 Linux. You may need to be a superuser for performing these steps.

Shut down the VM you are about to change.
From the vmware folder where vmware creates and stores the VM's, change the directory to the specific VM which you want to modify, and use your favorite text editor to open the corresponding VM's .vmx file. By default, it is $HOME/vmware/<vm-name>/<vm-name>.vmx
Locate the line for the interface which you want to modify. For example, if it is ethernet1, then change the line ethernet1.virtualDev = "e1000" to ethernet1.virtualDev = "vmxnet3"
Save the file and power on the VM.

If you are concerned more with configuration not performance the vmxnet3 driver can be set to interrupt mode in VPP. This will save a great deal on battery usage. Do this with the following

$ sudo vppctl set interface rx-mode <interface> interrupt

System setup

To use the native VPP vmxnet3 driver use the following Steps

Load VFIO driver

$ sudo modprobe vfio-pci

Make sure the interface is down

$ sudo ifconfig <if-name> down

The next 2 steps are optional and may be accomplished by specifying the optional "bind" keyword when creating the vmxnet3 interface.

For systems without IOMMU only, enable unsafe NOIOMMU mode

$ echo Y | sudo tee /sys/module/vfio/parameters/enable_unsafe_noiommu_mode

To bind interface to vfio-pci first install get the dpdk_devbind.py script.

Bind the driver with the following commands:

$ sudo /usr/local/vpp/vpp-config/scripts/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller' if=ens160 drv=vmxnet3 unused=vfio-pci,uio_pci_generic
0000:0b:00.0 'VMXNET3 Ethernet Controller' drv=vfio-pci unused=vmxnet3,uio_pci_generic
0000:13:00.0 'VMXNET3 Ethernet Controller' drv=vfio-pci unused=vmxnet3,uio_pci_generic
.....

$ sudo /usr/local/vpp/vpp-config/scripts/dpdk-devbind.py --bind vfio-pci 0b:00.0

Interface Creation

Now create the interface dynamically with following. The bind option must be specified if pci is not already bound manually by above steps.

$ sudo vppctl create interface vmxnet3 0000:0b:00.0 bind
$ sudo vppctl set interface state vmxnet3-0/b/0/0 up

Interface Deletion

If the interface needs to be deleted:

$ sudo vppctl delete interface vmxnet3 <if-name>

Show vmxnet3

Interface and ring information can be obtained with the command show vmxnet3 [if-name] [desc]

For example:

$ sudo vppctl show vmxnet3
Interface: vmxnet3-0/b/0/0 (ifindex 1)
  Version: 1
  PCI Address: 0000:0b:00.0
  Mac Address: 00:50:56:88:63:be
  hw if index: 1
  Device instance: 0
  Number of interrupts: 2
  Queue 0 (RX)
    RX completion next index 786
    RX completion generation flag 0x80000000
    ring 0 size 4096 fill 4094 consume 785 produce 784
    ring 1 size 4096 fill 4096 consume 0 produce 0
  Queue 0 (TX)
    TX completion next index 216
    TX completion generation flag 0x0
    size 4096 consume 216 produce 245