Mathew McBride's website

Gigabyte 990FXA-UD3 with ESXi 5.5 and IOMMU

technologyvmwareesxiFri 27 Jun 2014 10:34:52No comments

I have been able to set up an ESXi environment with the 990FXA-UD3 rev 4.0 motherboard. This motherboard is rather nice as it has four PCI Express x16 slots (two x16 speed, two x4), as well as two 1x and one conventional PCI. The downside is this board is a little pricey (around $180AUD), compared to the 3 slot ASRock 970-Extreme4 that retails for around $115AUD.

VMDirectPath (PCI Express passthrough) works when IOMMU has been enabled in the BIOS (which it isn't by default), and I have successfully passed through the following cards with it:

  • Two Dell PERC 6i SAS/RAID Controllers (one in x16 slot 2, another in x16 slot 3)
  • Intel Pro 1000/PT Quad network card
The Intel Quad card cannot be enabled for passthrough by defualt, as its PCI-Express switch does not implement the required security extension (ACS). This check can be disabled in the ESXi advanced settings (see this page for an example). Once that is done, individual 'ports' on the card can be passed through (but enabling passthrough for one of a two port group requires the other port to be passed as well). I have a PERC 6i and two ports of the Intel card passed through two two seperate VMs each.

Unfortunately there are some side effects of using VMDirectPath:

  • 100% of the guest memory allocation needs to be reserved
  • A portion of host memory (~2GB) is reserved (possible IOMMU memory hole? Correct me if I am wrong..)
  • ESXi then reserves a large amount of RAM for itself (System Reserved), it won't let me lower this down past 2GB, along with the above, leaving me with 12GB available for VMs (out of 16GB) :(

I have a low end Gigabyte 5450 installed as the video card (in the first x16 slot), but haven't tried to pass through that yet (will when I get my hands on a PCI video card)

And just a reminder, ESXi 5.5 removed (the unofficial) support for the Realtek 8111E, which appears as the onboard NIC on far too many motherboards, this one included. It can be slipstreamed in the ISO image used to install ESXi (see here for example).

    Welcome to my site

    Mathew McBride, telecoms hardware access engineer, programmer, gamer and all round nerd

    Warning: contents of blog may not make any sense whatsoever.

    ipv6 ready

    You are accessing this page over IPv6!

    (C) Mathew McBride, 2006-2017
    Creative Commons License
    Unless specified, the content on this website is licensed under a Creative Commons Attribution-ShareAlike 3.0 Australia License.