Part 2: vSphere GPU Pass-through with XenDesktop 7. VMware: Virtual Dedicated Graphics Acceleration (vDGA). VMware vSphere vDGA Configuration. After configuring the VM, I pass through the PCI card and on startup of the VM, there is a high pitched noise (not a beep so much as a power draw) that cycles a few times. Its almost as if when the VM is starting it is attempting to fully start up the GPU. Once it does that it continues through and I attempt to install the drivers.
This blog is an update of Josh Simon’s previous blog “How to Enable Compute Accelerators on vSphere 6.5 for Machine Learning and Other HPC Workloads”, and explains how to enable Nvidia V100 GPU, which comes with a larger PCI BARs (Base Address Registers) than previous GPU models, in Passthrough mode on vSphere 6.0 p4 and beyond. In addition, performance results are presented to demonstrate that V100 GPU in passthrough mode can achieve very close performance to bare-metal.
Preliminaries
To enable Nvidia V100 GPU for passthrough your host BIOS must be configured correctly, and the virtual machine destined to run these accelerated workloads must meet specific requirements as well. This section describes all of these requirements.
Note that while VMware supports VM Direct Path I/O (passthrough) as an ESXi feature and a device listed here may work correctly in passthrough mode, you should contact your device/system vendor if you require a formal statement of support for a particular device model.
Which Devices?
This article is only relevant if your PCI device maps memory regions whose sizes total more than 16GB. Devices in this class include the Nvidia K40m, K80, P100, and V100; Intel Xeon Phi; and some FPGA cards. Devices for which these instructions should not be necessary include the Nvidia K40c, K2, K20m, and AMD FirePro W9100, S7150x2. For these cards, simply follow the published instructions to enable passthrough devices under vSphere. As a general rule, cards that require more than 16GB of memory mapping are high end cards. You should follow the instructions in this article to enable them for use in passthrough mode within a virtual machine.
Host BIOS
Your host BIOS must be configured to support the large memory regions needed by these high-end PCI devices. To enable this, find the host BIOS setting for “above 4G decoding” or “memory mapped I/O above 4GB” or “PCI 64 bit resource handing above 4G” and enable it. The exact wording of this option varies by system vendor, though the option is often found in the PCI section of the BIOS menu. Consult your system provider if necessary to enable this option.
Guest OS
To access these large memory mappings, your guest OS must boot with EFI. That is, you must enable EFI in the VM and then do an EFI installation of the guest OS.
Enabling High-end Devices
With the above requirements satisfied, two entries must be added to the VM’s VMX file, either by modifying the file directly or by using the vSphere client to add these capabilities. The first entry is:
Specifying the 2nd entry requires a simple calculation. Sum the GPU memory sizes of all GPU devices(*) you intend to pass into the VM and then round up to the next power of two. For example, to use passthrough with two 32 GB V100 devices, the value would be: 32 + 32 = 64, rounded up to the next power of two to yield 128. Use this value in the 2nd entry:
With these two changes to the VMX file, follow the standard vSphere instructions for enabling passthrough devices at the host level and for specifying which devices should be passed into your VM. The VM should now boot correctly with your device(s) in passthrough mode.
(*) Note that some products, like the Nvidia K80 GPU, have two PCI devices on each physical PCI card. Count the number of PCI devices you intend to pass into the VM and not the number of PCI cards. For example, if you intend to use both of the K80’s GPU devices within a single VM, then your device count is two, not one.
Performance evaluation
This section presents some preliminary performance evaluation by comparing GPU passthrough on vSphere 6.7 with bare-metal. The server used in this test is a Dell R740, which has 2 10-core sockets, 192 GB of memory, and 2 V100 GPUs. We used two popular HPC benchmark applications, i.e., LAMMPS and GROMACS, and measured the performance ratio which is defined as the bare-metal execution time divided by the virtual execution time (higher is better). For each application, 1 GPU and 2 GPUs were tested separately. Based on the figure below, it is clear to see that the performance gap is always below 3%. In the case of using 1 V100 GPU, LAMMPS can even achieve better performance in virtual than in bare-metal.
Figure 1: GPU performance comparison between passthrough mode in virtual and bare-metal. 1 and 2 GPUs are tested separated.
Troubleshooting
If you have followed the above instructions and your VM still does not boot correctly with the devices enabled, the material in this section may be helpful. If you have tried the suggestions below and are still having problems, please contact us and we’ll be happy to help you.
Mapping Issue
If you see an error similar to the following in the VM’s vmware.log file:
All three entries from the virtual machine’s log file above refer to the same PCI GPU device, the one located at device address 0000:09:00.0. Here, the GPU device has requested to map a total of just over 16GB – the sum of the three “size” entries shown before the word “flags” and measured in bytes.
GPU cards that map more than 16GB of memory (referred to as “high-end” cards) require the instructions in this section to be followed. Those GPU cards that map less than 16GB of memory do not require those instructions.
For a GPU device that maps more than 16GB of memory, proceed as follows.
In the vSphere Client again, choose the virtual machine and use the options
“Edit Settings -> VM Options ->Advanced -> Configuration Parameters -> Edit Configuration” to get to the list of PCI-related options shown in Figure 6.
Figure 6: The vSphere Client window for the “pciPassthru” parameters for a virtual machine
Here, you add the following two parameters, with the value of the second parameter, as shown, set to TRUE
pciPassthru.use64bitMMIO=”TRUE”
The value of the second parameter is adjusted to suit your specific GPU requirements:
pciPassthru.64bitMMIOSizeGB=<n>
We calculate the value of the second parameter using a straightforward approach. Count the number of high-end PCI GPU devices that you intend to pass into this VM. This can be one or more GPUs. Multiply that number by 16 and round it up to the next power of two.
For example, to use passthrough mode with two GPU devices in one VM, the value would be:
2 * 16 = 32, rounded up to the next power of two to give 64.
Figure 6 above shows an example of these settings in the vSphere Client for two such passthrough-mode high-end GPU devices being used in a VM.
The largest NVIDIA V100 device’s BAR is 32GB. So, for a single V100 device being used in passthrough mode in a VM, we would assign the value of the 64bitMMIOSizeGB parameter to be 64 (rounding up to the next power of two).
2.3 Installing the Guest Operating System in the VM
Install the Guest Operating System into the virtual machine. This should be an EFI or UEFI-capable operating system, if your GPU card has a requirement for large PCI MMIO regions.
The standard vendor GPU driver must also be installed within the guest operating system.
2.4 Assigning a GPU Device to a Virtual Machine
This section describes the assignment of the GPU device to the VM. Power off the virtual machine before assigning the GPU device to it.
To enable a virtual machine to have access to a PCI device, in the vSphere Client, highlight the virtual machine, use the “Edit Settings” option and scroll down to the PCI Device list. If your device is not already listed there, use the “Add New Device” button to add it to the list. Once added, your virtual machine settings should look similar to those shown in Figure 7. In this example, the relevant entry is “PCI Device 0”.
Figure 7: PCI devices that are enabled in a virtual machine using the “Edit Settings” option.
You can read more about this procedure in the knowledge base article here
2.5 Memory Reservation
Note that when the PCI device is assigned to a VM, the virtual machine must have a memory reservation for the full configured memory size for the VM. This is done in the vSphere client by choosing the VM, using “Edit Settings -> Virtual Hardware -> Memory” to access and change the value in the Reservation area.
Finally, power on the virtual machine. After powering on the virtual machine, login to the guest operating system and check that the GPU card is present using
- On a Linux virtual machine, use the command
“lspci | grep nvidia”
or
- On a Windows operating system, use the “Device Manager” from the Control Panel to check the available GPU devices
The GPU is now ready for application use in passthrough mode.
3. Multiple Virtual Machines using Passthrough Mode
To create multiple identical VMs, it is recommended that you clone a virtual machine without the PCI assignments mentioned above first and then assign specific device(s) to each virtual machine afterwards.
3.1 Multiple GPUs Assigned to One VM
You can use the process described in sections 1 and 2 above multiple times in order to dedicate more than one GPU to one virtual machine, if you wish to do that. This scenario is shown in figure 8 below.
Figure 8: One or more GPUs allocated fully to one virtual machine in passthrough mode
Using this DirectPath I/O method, one or more full GPUs is allocated to only one virtual machine. Multiple virtual machines on the same host server would require separate GPU devices to be fully allocated to them, if using passthrough mode.
References
For further detailed information on this subject consult these articles: