mountainss Cloud and Datacenter Management Blog

Microsoft SystemCenter blogsite about virtualization on-premises and Cloud

Windows Server 2012 #HyperV – #SCVMM Design – Best Practices #WindowsAzure #Winserv


Funtional design Cloud

With Hybrid Cloud functionality you get the benefits of local private Cloud and public Cloud together like :

  • Active Directory Federation Services (ADSF) with Office365 and Windows Azure to provision Active Directory users from on-premises to the Cloud with
    Microsoft Forefront Identity Manager or Dirsync.
  • Making websites in the Cloud on WindowsAzure in a few minutes.
  • Using WindowsAzure BLOB Storage for Archiving
  • Manage and provision Virtual Machines with System Center 2012 SP1 from on-premises to the Cloud into Windows Azure
    (On  October 18th 2013, Windows Server 2012 R2 and System Center 2012 R2 will be released)
  • Using SQL databases for your on-premises applications or for your Cloud applications.
  • Managing applications for mobile devices.

The business benefits are :

  • You only pay for what you are using. (TCO Pricing)
  • Time to market, provisioning of Users, Virtual Machines, Websites is realy fast
  • Any time any place 24/7

Here you can find more information on Windows Azure :

To begin with private Cloud computing on-premises, you begin with architecture and design to match the business requirements.

Hyperv Cluster 2012 networking design

When the design of Microsoft Hyper-V Clustering, SQL, System Center 2012 Virtual Machine Manager, VPN Gateway to Windows Azure and ADFS is done, you don’t forget
disaster recovery with System Center 2012 Data Protection Manager and Windows Azure Backup for your Private Cloud solution.

Before you start installing Windows Server 2012 you should check this best practices:

Disclaimer: As with all Best Practices, not every recommendation can – or should – be applied. Best Practices are general guidelines, not hard, fast rules that must be followed. As such, you should carefully review each item to determine if it makes sense in your environment. If implementing one (or more) of these Best Practices seems sensible, great; if it doesn’t, simply ignore it. In other words, it’s up to you to decide if you should apply these in your setting.


⎕ Use Server Core, if possible, to reduce OS overhead, reduce potential attack surface, and to minimize reboots (due to fewer software updates).

⎕ Ensure hosts are up-to-date with recommended Microsoft updates, to ensure critical patches and updates – addressing security concerns or fixes to the core OS – are applied.

⎕ Ensure all applicable Hyper-V hotfixes and Cluster hotfixes (if applicable) have been applied. Review the following sites and compare it to your environment, since not all hotfixes will be applicable:

· Update List for Windows Server 2012 Hyper-V:

· List of Failover Cluster Hotfixes:

· Failover Cluster Management snap-in crashes after you install update 2750149 on a Windows Server 2012-based failover cluster:

⎕ Ensure hosts have the latest BIOS version, as well as other hardware devices (such as Synthetic Fibre Channel, NIC’s, etc.), to address any known issues/supportability

⎕ Host should be domain joined, unless security standards dictate otherwise. Doing so makes it possible to centralize the management of policies for identity, security, and auditing. Additionally, hosts must be domain joined before you can create a Hyper-V High-Availability Cluster.

· For more information:

⎕ RDP Printer Mapping should be disabled on hosts, to remove any chance of a printer driver causing instability issues on the host machine.

  • Preferred      method: Use Group Policy with host servers in their own      separate OU
    • Computer Configuration –> Policies –> Administrative Templates       –> Windows Components –> Remote Desktop Services –> Remote       Desktop Session Host –> Printer Redirection –> Do not allow client       printer redirection –> Set to “Enabled

⎕ Do not install any other Roles on a host besides the Hyper-V role and the Remote Desktop Services roles (if VDI will be used on the host).

  • When      the Hyper-V role is installed, the host OS becomes the “Parent      Partition” (a quasi-virtual machine), and the Hypervisor partition is      placed between the parent partition and the hardware. As a result, it is      not recommended to install additional (non-Hyper-V and/or VDI related)      roles.

⎕ The only Features that should be installed on the host are: Failover Cluster Manager (if host will become part of a cluster), Multipath I/O (if host will be connecting to an iSCSI SAN, Spaces and/or Fibre Channel), or Remote Desktop Services if VDI is being used. (See explanation above for reasons why installing additional features is not recommended.)

⎕ Anti-virus software should exclude Hyper-V specific files using the Hyper-V: Antivirus Exclusions for Hyper-V Hosts article, namely:

    • All folders containing VHD, VHDX, AVHD, VSV and ISO files
    • Default virtual machine configuration directory, if used (C:\ProgramData\Microsoft\Windows\Hyper-V)
    • Default snapshot files directory, if used       (%systemdrive%\ProgramData\Microsoft\Windows\Hyper-V\Snapshots)
    • Custom virtual machine configuration directories, if applicable
    • Default virtual hard disk drive directory
    • Custom virtual hard disk drive directories
    • Snapshot directories
    • Vmms.exe (Note: May need to be configured as process exclusions       within the antivirus software)
    • Vmwp.exe (Note: May need to be configured as process exclusions       within the antivirus software)
    • Additionally, when you use Cluster Shared Volumes, exclude the CSV       path “C:\ClusterStorage” and all its subdirectories.
  • For      more information:

⎕ Default path for Virtual Hard Disks (VHD/VHDX) should be set to a non-system drive, due to this can cause disk latency as well as create the potential for the host running out of disk space.

⎕ If you choose to save the VM state as the Automatic Stop Action, the default virtual machine path should be set to a non-system drive, due to the creation of a .bin file is created that matches the size of memory reserved for the virtual machine.  A .vsv file may also be created in the same location as the .bin file, adding to disk space used for each VM. (The default path is: C:\ProgramData\Microsoft\Windows\Hyper-V.)

⎕ If you are using iSCSI: In Windows Firewall with Advanced Security, enable iSCSI Service (TCP-In) for Inbound and iSCSI Service (TCP-Out) for outbound in Firewall settings on each host, to allow iSCSI traffic to pass to and from host and SAN device. Not enabling these rules will prevent iSCSI communication.

To set the iSCSI firewall rules via netsh, you can use the following command:

Netsh advfirewall firewall set rule group=”iSCSI Service” new enable=yes

⎕ Periodically run performance counters against the host, to ensure optimal performance.

  • Recommend      using the Hyper-V performance counter that can be extracted from the      (free) Codeplex PAL application:
  • Install      PAL on a workstation and open it, then click on the Threshold File      tab.
    • Select “Microsoft Windows Server 2012 Hyper-V” from the Threshold       file title, then choose Export to Perfmon template file. Save       the XML file to a location accessible to the Hyper-V host.
  • Next,      on the host, open Server Manager –> Tool –> Performance Monitor
  • In      Performance Monitor, click on Data Collector Sets –>      User Defined. Right click on User Defined and choose New –> Data      Collector Set. Name the collector set “Hyper-V Performance Counter      Set” and select Create from a template (Recommended) then      choose Next. On the next screen, select Browse and then locate the      XML file you exported from the PAL application. Once done, this will show      up in your User Defined Data Collector Sets.
  • Run      these counters in Performance Monitor for 30 minutes to 1 hour (during      high usage times) and look for disk latency, memory and CPU issues, etc.


⎕ Ensure you are running only supported guests in your environment. For a complete listing, refer to the following list:


⎕ Ensure NICs have the latest firmware, which often address known issues with hardware.

⎕ Ensure latest NIC drivers have been installed on the host, which resolve known issues and/or increase performance.

⎕ NICs should not use APIPA (Automatic Private IP Addressing). APIPA is non-routable and not registered in DNS.

⎕ VMQ should be enabled on VMQ-capable physical network adapters bound to an external virtual switch.

⎕ TCP Chimney Offload is not supported with Server 2012 software-based NIC teaming, due to TCP Chimney has the entire networking stack offloaded to the NIC. If software-based NIC teaming is not used, however, you can leave it enabled.

  • TO      SHOW STATUS:
    • From an elevated command-prompt, type the following:
      • netsh int tcp show global
        • (The output should show Chimney Offload State         disabled)
  • TO      DISABLE TCP Chimney Offload:
    • From an elevated command-prompt, type the following:
      • netsh int tcp set global chimney=disabled

⎕ Jumbo frames should be turned on and set for 9000 or 9014 (depending on your hardware) for CSV, iSCSI and Live Migration networks. This can significantly increase (6x increased throughput) throughput while also reducing CPU cycles.

  • End-to-End      configuration must take place – NIC, SAN, Switch must all support      Jumbo Frames.
  • You      can enable Jumbo frames when using crossover cables (for Live Migration      and/or Heartbeat), in a two node cluster.
  • To      verify Jumbo frames have been successfully configured, run the following      command from all your Hyper-V host(s) to your iSCSI SAN:
    • Ping –f –l 8000
      • This command will ping the SAN (e.g. with an 8K packet        from the host. If replies are received, Jumbo frames are properly        configured.

Command prompt Ip ping

NICs used for iSCSI communication should have all Networking protocols (on the Local Area Connection Properties) unchecked, with the exception of:

  • Manufacturers      protocol (if applicable)
  • Internet      Protocol Version 4
  • Internet      Protocol Version 6.
  • Unbinding      other protocols (not listed above) helps eliminate non-iSCSI      traffic/chatter on these NICs.

⎕ NIC Teaming should not be used on iSCSI NIC’s. MPIO is the best method. NIC teaming can be used on the Management, Production (VM traffic), CSV Heartbeat and Live Migration networks.

⎕ If you are using NIC teaming for Management, CSV Heartbeat and/or Live Migration, create the team(s) before you begin assigning Networks.

⎕ If using aggregate (switch-dependent) NIC teaming in a guest VM, only SR-IOV NICs should be used on guest.

⎕ If using NIC teaming inside a guest VM, follow this order:


  • Open      the settings of the Virtual Machine
    • Under Network Adapter, select Advanced Features.
    • In the right pane, under Network Teaming, tick the “Enable this       network adapter to be part of a team in the guest operating system.
  • Once      inside the VM, open Server Manager. In the All Servers view, enable NIC      Teaming from Server

Capture 2


  • Use      the following PowerShell command (Run as Administrator) on the Hyper-V      host where the VM currently resides:
    • Set-VMNetworkAdapter –VMName contoso-vm1 –AllowTeaming On
      • This PowerShell command turns on resiliency if one or more of the        teamed NICs goes offline.
    • Once inside the VM, open Server Manager. In the All Servers view,       enable NIC Teaming from Server

⎕ When creating virtual switches, it is best practice to uncheck Allow management operating system to share this network adapter, in order to create a dedicated network for your VM(s) to communicate with other computers on the physical network. (If the management adapter is shared, do not modify protocols on the NIC.)

Please note: we fully support and even recommend (in some cases) using the virtual switch to separate networks for Management, Live Migration, CSV/Heartbeat and even iSCSI.  For example two 10GB NIC’s that are split out using VLANs and QoS.

⎕ Recommended network configuration when clustering:

Min # of Networks on Host Host Management VM Network Access CSV/Heartbeat Live Migration iSCSI
5 “Management” “Production” “CSV/Heartbeat” “Live Migration” “iSCSI”

** CSV/Heartbeat & Live Migration Networks can be crossover cables connecting the nodes, but only if you are building a two (2) node cluster. Anything above two (2) nodes requires a switch. **

⎕ Turn off cluster communication on the iSCSI network.

  • In      Failover Cluster Manager, under Networks, the iSCSI network properties      should be set to “Do not allow cluster network communication on this      network.” This prevents internal cluster communications as well as CSV      traffic from flowing over the same network.

⎕ Redundant network paths are strongly encouraged (multiple switches) – especially for your Live Migration and iSCSI network – as it provides resiliency and quality of service (QoS).


⎕ If aggregate NIC Teaming is enabled for Management and/or Live Migration networks, the physical switch ports the host is connected to should be set to trunk (promiscuous) mode. The physical switch should pass all traffic to the host for filtering.

⎕ Turn off VLAN filters on teamed NICs. Let the teaming software or the Hyper-V switch (if present) do the filtering.


⎕ Legacy Network Adapters (a.k.a. Emulated NIC drivers) should only be used for PXE booting a VM or when installing non-Hyper-V aware Guest operating systems. Hyper-V’s synthetic NICs (the default NIC selection; a.k.a. Synthetic NIC drivers) are far more efficient, due to using a dedicated VMBus to communicate between the virtual NIC and the physical NIC; as a result, there are reduced CPU cycles, as well as much lower hypervisor/guest transitions per operation.

Example :

The first thing you want to do is create a team out of the two NICs and connect the team to a Hyper-V virtual switch. For instance with Powershell :

New-NetLbfoTeam Team1 –TeamMembers NIC1, NIC2 –TeamNicName TeamNIC1

New-VMSwitch TeamSwitch –NetAdapterName TeamNIC1 –MinimumBandwidthMode Weight –AllowManagementOS $false

Next, you want to create multiple vNICs on the parent partition, one for each kind of traffic (two for SMB). Here’s an example:

Add-VMNetworkAdapter –ManagementOS –Name SMB1 –SwitchName TeamSwitch

Add-VMNetworkAdapter –ManagementOS –Name SMB2 –SwitchName TeamSwitch

Add-VMNetworkAdapter –ManagementOS –Name Migration –SwitchName TeamSwitch

Add-VMNetworkAdapter –ManagementOS –Name Cluster –SwitchName TeamSwitch

Add-VMNetworkAdapter –ManagementOS –Name Management –SwitchName TeamSwitch

After this, you want to configure the NICs properly. This will include setting IP addresses, creating separate subnets for each kind of traffic. You can optionally put them each on a different VLAN.

Since you have lots of NICs now and you’re already in manual configuration territory anyway, you might want to help the SMB Multichannel by pointing it to the NICs that should be used by SMB. You can do this by configuring SMB Multichannel constraints instead of letting SMB try all different paths. For instance, assuming that your Scale-Out File Server name is SOFS, you could use:

New-SmbMultichannelConstraint -ServerName SOFS -InterfaceAlias SMB1, SMB2

Last but not least you might also want set QoS for each kind of traffic, using the facilities provided by the Hyper-V virtual switch. One way to do it is:

Set-VMNetworkAdapter –ManagementOS –Name SMB1 –MinimumBandwidthWeight 20

Set-VMNetworkAdapter –ManagementOS –Name SMB2 –MinimumBandwidthWeight 20

Set-VMNetworkAdapter –ManagementOS –Name Migration –MinimumBandwidthWeight 20

Set-VMNetworkAdapter –ManagementOS –Name Cluster –MinimumBandwidthWeight 5

Set-VMNetworkAdapter –ManagementOS –Name Management –MinimumBandwidthWeight 5

Set-VMNetworkAdapter –VMName * -MinimumBandwidthWeight 1

There is a great TechNet page with details on this and other network configurations at


⎕ New disks should use the VHDX format. Disks created in earlier Hyper-V iterations should be converted to VHDX, unless there is a need to move the VHD back to a 2008 Hyper-V host.

  • The      VHDX format supports virtual hard disk storage capacity of up to 64 TB,      improved protection against data corruption during power failures (by      logging updates to the VHDX metadata structures), and improved alignment      of the virtual hard disk format to work well on large sector disks.

⎕ Disks should be fixed in a production environment, to increase disk throughput. Differencing and Dynamic disks are not recommended for production, due to increased disk read/write latency times (differencing/dynamic disks).

⎕ Use caution when using snapshots. If not properly managed, snapshots can cause disk space issues, as well as additional physical I/O overhead. Additionally, if you are hosting 2008 R2 (or earlier) Domain Controllers, reverting to an earlier snapshot can cause USN rollbacks. Windows Server 2012 has been updated to help better protect Domain Controllers from USN rollbacks; however, you should still limit usage.

⎕ The recommended minimum free space on CSV volumes containing Hyper-V virtual machine VHD and/or VHDX files:

  • 15%      free space, if the partition size is less than 1TB
  • 10%      free space, if the partition size is between 1TB and 5TB
  • 5%      free space, if the partition size is greater than 5TB
  • To      enumerate current volume information, including the percentage free, you      can use the following PowerShell command:
    • Get-ClusterSharedVolume “Cluster Disk 1” | fc *
      • Review the “PercentageFree” output

⎕ It is not supported to create a storage pool using Fiber Channel or iSCSI LUNs.

⎕ Page file on Hyper-V Host should managed by the OS and not configured manually.


⎕ Use Dynamic Memory on all VMs (unless not supported).

⎕ Guest OS should be configured with (minimum) recommended memory

  • 2048MB      is recommended for Windows      Server 2012 (e.g. 2048 – 4096 Dynamic Memory). (The minimum      supported is 512 MB)
  • 2048MB      is recommended for Windows      Server 2008, including R2 (e.g. 2048 – 4096 Dynamic Memory).      (The minimum supported is 512 MB)
  • 1024MB      is recommended for Windows 7 (e.g. 1024 – 2048 Dynamic Memory). (The      minimum supported is 512 MB)
  • 1024MB      is recommended for Windows Vista (e.g. 1024 – 2048 Dynamic Memory). (The      minimum supported is 512 MB)
  • 512MB      is recommended for Windows Server 2003 R2 w/SP2 (e.g. 256 – 2048 Dynamic      Memory). (The minimum supported is 128 MB.
  • 512MB      is recommended for Windows Server 2003 w/SP2 (e.g. 256 – 2048 Dynamic      Memory). (The minimum supported is 128 MB).
  • 512MB      is recommended for Windows XP. Important: XP does not support Dynamic      Memory. (The minimum supported is 64 MB). Note:      Support for Windows XP Ends April 2014!


⎕ Set preferred network for CSV communication, to ensure the correct network is used for this traffic. (Note: This will only need to be run on one of your Hyper-V nodes.)

  • The      lowest metric in the output generated by the following PowerShell command      will be used for CSV traffic
    • Open a PowerShell command-prompt (using “Run as administrator”)
    • First, you’ll need to import the “FailoverClusters” module. Type the       following at the PS command-prompt:
      • Import-Module FailoverClusters
    • Next, we’ll request a listing of networks used by the host, as well       as the metric assigned. Type the following:
      • Get-ClusterNetwork | ft Name, Metric, AutoMetric,        Role

In order to change which network interface is used for CSV traffic,       use the following PowerShell command:

      • (Get-ClusterNetwork “CSV         Network”).Metric=900
        • This will set the network named “CSV          Network” to 900

Capture 3

*** Set preferred network for Live Migration, to ensure the correct network(s) are used for this traffic:

  • Open      Failover Cluster Manager, Expand the Cluster
  • Next,      right click on Networks and select Live Migration Settings
    • Use the Up / Down buttons to list the networks in order from most       preferred (at the top) to least preferred (at the bottom)
    • Uncheck any networks you do not want used for Live Migration traffic
    • Select Apply and then press OK
  • Once      you have made this change, it will be used for all VMs in the cluster

⎕ The Cluster Shutdown Time (ShutdownTimeoutInMinutes registry entry) should be set to an acceptable number

  • Default      is set using the following calculation (which can be too high, depending      on how much physical memory is installed)
    • (100 / 64) * physical RAM
      • For example, a 96GB system would have 150 minute timeout. (100/64)*96        = 150
  • Suggest      setting the timeout to 15, 20 or 30 minutes, depending on the number of      VMs in your environment.
    • Registry Key: HKLM\Cluster\ShutdownTimeoutInMinutes
      • Enter minutes in Decimal value.
      • Note: Requires a reboot to take effect

Capture 4

⎕ Run the Cluster Validation periodically to remediate any issues

  • NOTE:      If all LUNs are part of the cluster, the validation test will skip all      disk checks. It is recommended to set up a small test-only LUN and share      it on all nodes, so full validation testing can be completed.
  • If      you need to test a LUN running virtual machines, the LUN will need to be      taken offline.
  • For      more information:

⎕ Consider enabling CSV Cache if you have VMs that are used primarily for read requests, and are less write intensive. Scenarios such as Pooled VDI VMs; also can be leveraged for reducing VM boot storms.


⎕ If utilizing Hyper-V Replica, update inbound traffic on the firewall to allow TCP port ‘80’ and/or port ‘443’ traffic. (In Windows Firewall, enable “Hyper-V Replica HTTP Listener (TCP-In)” rule on each node of the cluster.

To enable HTTP (port 80) replica traffic, you can run the following from an elevated command-prompt:

netsh advfirewall firewall set rule group=”Hyper-V Replica HTTP” new enable=yes

To enable HTTPS (port 443) replica traffic, you can run the following from an elevated command-prompt:

netsh advfirewall firewall set rule group=”Hyper-V Replica HTTPS” new enable=yes

⎕ Compression is recommended for replication traffic, to reduce bandwidth requirements.

⎕ Configure guest operating systems for VSS-based backups to enable application-consistent snapshots for Hyper-V Replica.

⎕ Integration services must be installed before primary or Replica virtual machines can use an alternate IP address after a failover

⎕ Virtual hard disks with paging files should be excluded from replication, unless the page file is on the OS disk.

⎕ Test failovers should be performed monthly, at a minimum, to verify that failover will succeed and that virtual machine workloads will operate as expected after failover

⎕ Hyper-V Replica requires the Failover Clustering Hyper-V Replica Broker role be configured if either the primary or the replica server is part of a cluster.

⎕ Feature and performance optimization of Hyper-V Replica can be further tuned by using the registry keys mentioned in the article below:


⎕ Place all Cluster-Aware Updating (CAU) Run Profiles on a single File Share accessible to all potential CAU Update Coordinators. (Run Profiles are configuration settings that can be saved as an XML file called an Updating Run Profile and reused for later Updating Runs.


⎕ An Active Directory infrastructure is required, so you can grant permissions to the computer account of the Hyper-V hosts.

⎕ Loopback configurations (where the computer that is running Hyper-V is used as the file server for virtual machine storage) are not supported. Similarly, running the file share in VM’s that are hosted on compute nodes that will serve other VM’s is not supported.


⎕ Domain Controller VMs should have “Shut down the guest operating system” in the Automatic Stop Action setting applied (in the virtual machine settings on the Hyper-V Host)


⎕ Ensure Integration Services (IS) have been installed on all VMs. IC’s significantly improve interaction between the VM and the physical host.

⎕ Be certain you are running the latest version of integration services – the same version as the host(s) – in all guest operating systems, as some Microsoft updates make changes/improvements to the Integration Services software. (When a new Integration Services version is updated on the host(s) it does not automatically update the guest operating systems.)

  • Note:      If Integration Services are out of date, you will see 4010 events logged      in the event viewer.
  • You      can discover the version for each of your VMs on a host by running the      following PowerShell command:
    • Get-VM | ft Name, IntegrationServicesVersion
  • If      you’d like a PowerShell method to update Integration Services on VMs,      check out this blog:


⎕ If your SAN supports ODX (see this post for help; also check with your hardware vendor), you should strongly consider enabling ODX on your Hyper-V hosts, as well as any VMs that connect directly to SAN storage LUNs.

  • To      enable ODX, open PowerShell (using Run as Administrator) and type the      following:
    • Set-ItemProperty hklm:\system\currentcontrolset\control\filesystem       -Name “FilterSupportedFeaturesMode” –Value 0
    • Be sure to run this command on every Hyper-V host       that connects to the SAN, as well as any VM that connects directly to the       SAN.


Here you can find more best practices of Microsoft Hyper-v and SMB :


Here you can find more on Virtual networking services for Hybrid Cloud :


Here You can find the System Center 2012 SP1 Virtual Machine networking Architecture :

Here you can find an post on deploying Virtual Machines into Windows Azure via System Center 2012 SP1 App-Controller:

Here you can find the Technical documentation and Library of System Center 2012 SP1 Virtual Machine Manager :


            I Hope this blogpost will help you with the implementation of Cloud Computing
            If you need help see my website on Windows Azure

Author: James van den Berg

I'm Microsoft Architect and ICT Specialist and Microsoft MVP System Center Cloud and Datacenter Management

3 thoughts on “Windows Server 2012 #HyperV – #SCVMM Design – Best Practices #WindowsAzure #Winserv

  1. Pingback: Microsoft Most Valuable Professional (MVP) – Best Posts of the Week around Windows Server, Exchange, SystemCenter and more – #43 - Flo's Datacenter Report

  2. Pingback: Microsoft Most Valuable Professional (MVP) – Best Posts of the Week around Windows Server, Exchange, SystemCenter and more – #43 - TechCenter - Blog - TechCenter - Dell Community

  3. Pingback: Microsoft Most Valuable Professional (MVP) – Best Posts of the Week around Windows Server, Exchange, SystemCenter and more – #43 - Dell TechCenter - TechCenter - Dell Community

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s