Xen Cloud Platform PCI and USB Passthrough

Hypervisors generally allow creation of virtual devices (sound, USB, network), which will work in most cases. However, if you need more control over the device or more options than the stock virtual device gives, you are going to find yourself stuck. To get around this, PCI passthrough is used to pass a raw device (such as a sound card, RAID card, video card, etc) directly into a virtual machine. This gives unfettered access and to the virtual machine, it is treated as if the hardware were directly connected. In the guide below, I explain how to pass through PCI and USB devices to Xen Cloud virtual machines.

A good use for PCI passthrough would be to pass through a RAID card, which has a set of disks attached to it. Without this option, you would be forced to create virtual disks on the RAID card’s array and pass that into the virtual machine. For shared storage, this is an ideal setup as one virtual machine does not have exclusive access to the resources. However, if this array was to store large amounts of data (e.g. file server), creating a 12+ TB virtual disk is cumbersome. Common tasks, such as managing the RAID card, expanding an array, or even monitoring the health of the disks isn’t possible because the hypervisor is managing the hardware. Giving the virtual machine direct access to the RAID card solves these problems, but usually requires a bit of setup.

For the hypervisors I have experience with — older VMWare server and VirtualBox — passing through devices is easy: simply select the device from the drop down box and off you go.

Unfortunately, in Xen Cloud Platform (XCP), it isn’t that easy. There are no GUI options (as of 1.6) to pass through devices, so it must be done through a command line. “Easy,” you say, “I use the command line often.” That experience certainly helps, but finding information online is difficult at best and head-bashingly frustrating at worst. I’ve written this guide to help save your forehead, desk, and keyboard from destruction and pain.

That being said, I will put a disclaimer up that these instructions work for me at the time of writing this guide, but Linux changes and new versions of software come out frequently. I can’t know of all the caveats or hardware setups, so you may hit some snags that are not covered here. However, I want to make sure this article is as correct as possible, so send me a message if you find something wrong, a step that you needed to do differently, or if something changes. Also, if you find an easier way to do any step of this guide, I’m certainly interested in hearing how you got it done! As with all guides that you find on the internet, pay attention to the commands given. Don’t blindly copy and paste them in. Understand what they do so your server doesn’t break.

Ultimately, what we are looking for is the PCI address of the device to pass through (e.g. 0000:00:1a.0 or 0000:06:00.0). For some devices, this is easy to get, such as actual PCI or PCIe devices. Getting this address for USB devices, however, is not quite as fun or direct. I’ll split this into two sections, one for PCI devices and another for USB.

Preparation

Before we get started, you may want to install some tools onto the server to make working in the command line easier. Programs, such as vim, are not included by default. This problem is compounded by the stock setting of having the repository “base” disabled. You could certainly enable it through the proper “.repo” file, but I would suggest simply using the YUM command “enablerepo” to get by it once. This command will give you access to “vim”, “lsusb”, and “lspci”, while bypassing the base repository base lockout:

[root@xcpserver ~]# yum install vim-common vim-enhanced usbutils pciutils --enablerepo=base

You can now create your virtual machine, but please note that we need to create it as a HVM (hardware-assisted virtual machine). To do this, simply select the template “Other install media” when creating the virtual machine. If you do not do this step, passing through will not work. XCP creates new instances as para-virtualized by default.

XenCenter showing the correct option selected to create a HVM.

XenCenter showing the correct option selected to create a HVM.

Additionally, your system needs to be capable of IOMMU and the feature needs to be enabled. This allows the system to create a virtual memory space for each device so it can safely be passed into your virtual machine.

PCI Passthrough

Luckily, PCI passthrough is fairly simple. As long as the device is in the system and the hypervisor recognizes it, we can look through all the devices with the “lspci” command. Below, I’ve removed a lot of items that will show when you type the command and I’ve highlighted the IBM M1015 RAID card that I will use in this example.

[root@xcpserver ~]# lspci
[Lots of items removed]
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 IDE interface: Intel Corporation 82801IB (ICH9) 2 port SATA Controller [IDE mode] (rev 02)
01:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
01:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
03:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2108 [Liberator] (rev 05)
05:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
05:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
06:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
07:00.0 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
07:00.1 Fibre Channel: Emulex Corporation Zephyr-X LightPulse Fibre Channel Host Adapter (rev 02)
08:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)

Note that it has “06:00.0” before the device needed. Your numbers will vary, but the idea is exactly the same. As stated previously, we need slightly longer version of that PCI address, which can be obtained by adding four zeroes at the beginning: “0000:06:00.0”. Make note of all the devices you want to pass through and then go to the section “Passing in PCI Devices” below. If you have USB devices to pass through, read that section as well.

USB Passthrough

This section is the main reason for this article. PCI passthrough is pretty well documented and straightforward, but passing in USB devices is not quite like that. Technically speaking, to pass in a USB device, you aren’t passing in the device itself, but rather the PCI address of the device it is plugged into (USB controller). The main problem with passing in a USB device is that it is not immediately obvious what that PCI address is, otherwise we could just skip to the end and I need not write this part.

Note: You are not able to pass in sub-devices for PCI — that is only allowed for PCIe. Because of this, if you have a PCI USB controller, the whole thing needs to be passed in. If you are going this route, please see the “PCI Passthrough” section of this guide, as this will not apply to you.

Make sure the device is plugged into the server and visible using “lsusb”. Make note of the ID tag of the device. In this example, I’m passing through a printer to my virtual machine to share out to the network (Brother HL-2140).

[root@xcpserver ~]# lsusb
Bus 005 Device 005: ID 0624:0248 Avocent Corp.
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 004: ID 04f9:0033 Brother Industries, Ltd
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

With this command, we now know the ID is “04f9:0033”. This is important because the next command does not give the pretty name, only the ID. To know what device it is plugged into, we use the “lsusb” command, but tell it to print in tree mode. Look for the ID of your USB device. I’ve highlighted the lines that we are interested in.

[root@xcpserver ~]# lspci -t
Bus#  6
`-Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  5
`-Dev#   1 Vendor 0x1d6b Product 0x0001
  `-Dev#   5 Vendor 0x0624 Product 0x0248
Bus#  4
`-Dev#   1 Vendor 0x1d6b Product 0x0001
Bus#  3
`-Dev#   1 Vendor 0x1d6b Product 0x0001
  `-Dev#   4 Vendor 0x04f9 Product 0x0033
Bus#  1
`-Dev#   1 Vendor 0x1d6b Product 0x0002
  `-Dev#   2 Vendor 0x0424 Product 0x2514

This tells us that the parent USB port is on Bus 03, which “lsusb” told us already, but we get the additional information that the parent is Dev 01. However, this still doesn’t give us the PCI address of the USB port, so we need to do some more digging. Using “lsusb” again, we will tell it to output much more information than normal. This will cover your terminal in a massive amount of text. In this pile of text, we are looking for the device with a Bus of 03 and a Dev of 01, with your results varying.

[root@xcpserver ~]# lspci -v
[Lots of items removed]
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Device Descriptor:
 bLength                18
 bDescriptorType         1
 bcdUSB               1.10
 bDeviceClass            9 Hub
 bDeviceSubClass         0 Unused
 bDeviceProtocol         0 Full speed (or root) hub
 bMaxPacketSize0        64
 idVendor           0x1d6b Linux Foundation
 idProduct          0x0001 1.1 root hub
 bcdDevice            2.06
 iManufacturer           3 Linux 2.6.32.43-0.4.1.xs1.6.10.734.170748xen uhci_hcd
 iProduct                2 UHCI Host Controller
 iSerial                 1 0000:00:1a.0

There we go, a PCI address! “0000:00:1a.0” is exactly what we need to continue to the last step. Repeat this for each device you need to pass through.

Note: In this example, you could probably safely assume that “Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub” is the parent, but it is always good to check and it takes little time to do so. There is nothing more frustrating working on a problem for hours on end to find you made a tiny mistake or assumption earlier.

Passing in PCI Devices

Now that we actually have the PCI address of the devices we want to pass through, we need to tell XCP which devices to assign and what virtual machine to assign them to. To start, we need to know where we are putting these devices. To get a list of the virtual machines, use the “xe vm-list” command on the server. Highlighted is the UUID of the virtual machine that I’m going to pass the devices into.

[root@xcpserver ~]# xe vm-list
uuid ( RO)           : a68d3b01-618c-47c1-ae87-4b3a00eedfea
     name-label ( RW): File Server
    power-state ( RO): running

We now have all the pieces needed to pass the device through to the virtual machine (finally!). The only special part left is how to write the PCI passthrough string. In my example, I’m passing through a RAID card and a printer, so I have two devices. Doing this is very simple. Note how each PCI address starts with a “#/”, with each address separated by a comma. Start the counter at 0 and increase by one for each device you add. Here is the string I will use:

0/0000:06:00.0,1/0000:00:1a.0

To put everything together into a command, you will tell the server what devices to include and what virtual machine to modify. We use the “other-config:pci” parameter to specify the string created above.

[root@xcpserver ~]# xe vm-param-set other-config:pci=0/0000:06:00.0,1/0000:00:1a.0 uuid=a68d3b01-618c-47c1-ae87-4b3a00eedfea

Note: You can modify this while the virtual machine is running without causing any problems. However, you need to shut down the virtual machine for the changes to take affect. Restarting will not work. Don’t waste an hour or two tracking down an issue that doesn’t exist (except maybe a PEBKAC error) like I did.

Once your virtual machine starts up, check to see if the device is listed. Check Device Manager in Windows or use “lspci” in Linux. Treat the device just as if it were directly attached to the virtual machine. That is quite a few steps to do such a simple operation, but I hope my guide helped clear up the steps needed to get it done.