Troubleshooting ISO file sharing in System Center Virtual Machine Manager 2012 SP1

So I was trying to use sharing of ISO files instead of copying when mounting on virtual machines. The really helpful dialog box in System Center Virtual Machine Manager (SCVMM) 2012 SP1 politely informs you that

image

OK, so what configuration?

I found these steps for VMM 2008: How to Enable Shared ISO Images for Hyper-V Virtual Machines in VMM

In short this is what you have to do:

  • Use a domain account as the VMM service account
  • Grant share and NTFS Read permissions to the VMM library shares that contain ISO files for both the VMM service account and the computer accounts of any Hyper-V hosts that need to access those ISO files
  • Configure constrained delegation for each Hyper-V host to allow delegation of credentials to the VMM library servers
  • Add the Hyper-V hosts to the Windows Autorization Access builtin group

If you use a group to grant access to you Hyper-V hosts to the library share, like I did you have to restart them to update their security tokens. Configuring constrained delegation also requires a restart.

But it wasn’t working! VMM just told med “Update failed” each time I tried to mount an ISO on a VM. This is the error I got:

Error (12700)
VMM cannot complete the host operation on the <Hyper-V host> server because of the error: ‘<VM>’ failed to add device ‘Virtual CD/DVD Disk’. (Virtual machine ID <GUID>)

‘<VM>’: User Account does not have sufficient privilege to open attachment ‘\<VMM Library server>MSSCVMMLibraryISOs<ISO file>’. Error: ‘General access denied error’ (0x80070005). (Virtual machine ID <GUID>)
Unknown error (0x8001)

Recommended Action
Resolve the host issue and then try the operation again.

The Internet didn’t know why it didn’t work either. I know because I asked it!

As far as I cloud see my permissions were correct so I had to figure out if there was another account involved. The best way I know to do that is by auditing object access. So first I enabled auditing of object access on my VMM Library server:

image

Next, to catch all security principals trying to access the files in the VMM Library, I added audit System Access Control List entries for both Authenticated Users and Everyone:

image

After setting this up I tried mounting the ISO again and then opened the Security Event Log. The first event of interest was a logon event (Event ID 4624):

image

The account listed here is the Run As account I use for Hyper-V host management. This was an Audit Success event since the account was allowed to log on to the VMM library server.

Next was an attempt to access a share (Event ID 5140):

image

Again it was the RunAs account this time trying to access the VMM Library share with read permissions (ReadData or ListDirectory). Again, this was an Audit Success event since this was just a request for access. This was followed by something called a detailed file share (Event ID 5145):

image

This was our first Audit Failure event. This event is the result of the File Share request in the previous event. As you can see it was not successful. For the user this will result in an access denied message; exactly what I was seeing in the VMM console.

So apparently the RunAs account is also involved in accessing the share. To test to see if this was the solution I added the RunAs account to the share and NTFS permissions with Read access. Immediately it was successful and the VM could access the ISO file directly from the VMM Library.

Happy that I made it work I was still interested in knowing why the RunAs account was used for this. The answer was to be found in the VMM Add Resource Wizard. This is the wizard used to add fabric resources to VMM, like Hyper-V hosts or clusters. On the Credentials page we see this:

image

So the RunAs account is stored and used to access the hosts discovered with the wizard. The universe was once again in balance.

Listing Windows Azure availability sets

Windows Azure guarantees a 99.95 % uptime SLA, but this is only for multiple instance roles. So, for example, one web server by itself will not be guaranteed 99.95 % uptime, but will rather have “best effort”. Whereas two, or more, in a multi instance role will. So how do you make something multi-instance?

You use something called an availability set. An availability set is spanned across multiple fault and update domains in the datacenter. A fault domain is a collection of resources that are expected to fail together, like a server rack. An update domain is a set of resources that are updated at the same time. An availability set ensures that your instances are spanned across multiple update and fault domains so that no single failure or update can impact all the instances. The point here is that all the instances must be the same and be able to offer the same service to your users. So going back to our example with web servers you have to have at least two identical ones in the same availability set for your service to have a 99.95 % uptime guarantee. It is not correct to say that any VM or instance is guaranteed 99.95 % since it is the service that the instance offers that is guaranteed. Furthermore only Internet facing services have the SLA, since you need to use the Windows Azure load balancer to publish your instances, but more on that another day. Back to availability sets.

You add VMs to availability sets either in the Windows Azure portal, with PowerShell or the REST API. In the portal or PowerShell there is no way to explicitly create an availability set, you always create one as a side effect of configuring a VM. In the portal it looks like this:

image

This is an availability set with only one instance. In the drop down list I can either remove the VM from the availability set or create a new availability set and add the VM to that set.

Note: when you either add or remove a VM from an availability set, it will get a new NIC. This will have the same IP address as before but you will see that its number has been incremented.

In PowerShell you use the Set-AzureAvailabilitySet or New-AzureVMConfig cmdlets. Set-AzureAvailabilitySet updates an existing VM while New-AzureVMConfig lets you specify an availability set when you create the VM.

Something I miss is the ability to list all my availability sets. Since there is no (yet) Get-AzureAvailabilitySet cmdlet you have to extract this information from each of your VMs. Here is a quick one-liner to do that:

https://gist.github.com/morgansimonsen/8039845

This will list the VM names, cloud service names and availability set names (if any) for all your VMs. Feel free to add any kind of formatting or sorting to the output.

Introducing BUSCON

I am often asked, usually on some form of IM media; “are you busy?”. I often find this question hard to answer since I regard busy-ness (is that a real English word?) as more of a state with varying degrees. This has led me to the definition of BUSCON or BUSy CONdition, based on the well-known DEFCON or defense readiness condition used by the United States Armed Forces. At the current time BUSCON is defined like this:

Busy condition Busy-ness Color
BUSCON 1 Metaphysically busy (AKA “Leave-me-the-fuck-alone” busy)* White
BUSCON 2 Extremely busy Red
BUSCON 3 Really busy Yellow
BUSCON 4 Above normal busy-ness Green
BUSCON 5 Normal busy-ness Blue

As BUSCONs creator I reserve the right to change its definition at any time. Check back here for updates if you find the BUSCON definition useful.

* So you take offense by my use of profanity in this post? Here are a couple of quotes that sum up my views on that area:

Stephen Fry: “It’s now very common to hear people say “I’m rather offended by that”, as if that gives them certain rights. It’s actually no more than a whine. “I find that offensive”. It has no meaning, it has no purpose, it has no reason to be respected as a phrase. “I’m offended by that”, well so fucking what.” **

Salman Rushdie: “What is freedom of expression? Without the freedom to offend, it ceases to exist.”

** See first asterisk