Gigabyte 990FXA-UD3 with ESXi 5.5 and IOMMU
I have been able to set up an ESXi environment with the 990FXA-UD3 rev 4.0 motherboard. This motherboard is rather nice as it has four PCI Express x16 slots (two x16 speed, two x4), as well as two 1x and one conventional PCI. The downside is this board is a little pricey (around $180AUD), compared to the 3 slot ASRock 970-Extreme4 that retails for around $115AUD.
VMDirectPath (PCI Express passthrough) works when IOMMU has been enabled in the BIOS (which it isn't by default), and I have successfully passed through the following cards with it:
- Two Dell PERC 6i SAS/RAID Controllers (one in x16 slot 2, another in x16 slot 3)
- Intel Pro 1000/PT Quad network card
Unfortunately there are some side effects of using VMDirectPath:
- 100% of the guest memory allocation needs to be reserved
- A portion of host memory (~2GB) is reserved (possible IOMMU memory hole? Correct me if I am wrong..)
- ESXi then reserves a large amount of RAM for itself (System Reserved), it won't let me lower this down past 2GB, along with the above, leaving me with 12GB available for VMs (out of 16GB) :(
I have a low end Gigabyte 5450 installed as the video card (in the first x16 slot), but haven't tried to pass through that yet (will when I get my hands on a PCI video card)
And just a reminder, ESXi 5.5 removed (the unofficial) support for the Realtek 8111E, which appears as the onboard NIC on far too many motherboards, this one included. It can be slipstreamed in the ISO image used to install ESXi (see here for example).