Working with differencing disks in VirtualBox

For some time now I have been trying to make persistent differencing disks work in VirtualBox. The idea is to have one base disk with all the software that all the VMs you want to deploy needs. Typically the OS, all patches up until the time you created the base disk, common apps like Flash etc. You then generalize this disk. In the Windows world this means running the System Preparation Tool (sysprep.exe) to remove all machine specific data from a VM. Now you can use this disk as the starting point for several new VMs and save a lot of disk space. Typically all virtualization software is able to use a differencing disk, or delta disk, for each VM. This disk only stores the changes made to the base disk in a new disk file, without making any changes to the base disk itself.

In VirtualBox the immutable disk type is what you use for the base disk. Once it is generalized you change its type from normal to immutable. Whenever you create a new VM and connect the immutable disk to it; VirtualBox will create a differencing disk with the VMs changes. This is where I have been having problems. In my scenario I want to keep the VM state in the differencing disk file between reboots, but VirtualBox, by default, will reset the differencing disk on every power cycle of the VM if the VM is running with an immutable base disk. According to the documentation you can modify this behaviour with the VBoxManage.exe tool. Here’s how:

VBoxManage.exe modifyhd <differencing disk file or disk uuid> –autoreset off

This is what I have been doing with my differencing disks. When I later inspected the disk with VBoxManage.exe, it correctly stated that autoreset was indeed off. But whenever the VM power cycled the disk was reset and the autoreset flag was set back to on.

Initially I thought that maybe this was a bug in VirtualBox regarding the VHD disk format, which is what I use for all my VMs, but the same problem exists with VirtualBox’s own virtual disk format VDI. I also read several posts from other people struggelig with this functionality, but their problems were related to the actual use of VBoxManage.exe, were for different host OSs (I use Windows 7/8), or something else. The reason I kept on trying, though, was that some people were actually able to make it work.

At some point I found another disk type in VirtualBox, the multiattach type. I really had not used it before since all the documentation on how use persistent differencing disks in VirtualBox used immutable disks. This is what the VirtualBox documentation online has to say about the multiattach disk type:

An image in multiattach mode can be attached to more than one virtual machine at the same time, even if these machines are running simultaneously. For each virtual machine to which such an image is attached, a differencing image is created. As a result, data that is written to such a virtual disk by one machine is not seen by the other machines to which the image is attached; each machine creates its own write history of the multiattach image.

Technically, a “multiattach” image behaves identically to an “immutable” image except the differencing image is not reset every time the machine starts.

So there it was! The multiattach disk type is the one to use! I changed by base disk from immutable to multiattach and now my VMs keep their differencing disks across power cycles. It seems that even though VBoxManage.exe can change the autoreset type of a differencing disk that is running off an immutable disk; the flag is not honored and is reset on every power cycle. The question now is why you can actually change that flag when it will not be honored? Would be very simple to just inform the user that immutable disks will always reset, and that multiattach should be used if you want to keep the VM state.

Anyway, VirtualBox is a great product, and also free!

Here is the link regarding the disk types in VirtualBox: http://www.virtualbox.org/manual/ch05.html#hdimagewrites

Until next time!

Poor-man’s Active Directory backups (export really)

Sometimes I need a “copy” of an Active Directory domain, partition or LDS instance. Usually this is when I remove decomissioned domains in a multi-domain forest and want to keep a record of what was left when I deleted it. You can do this With LDIFDE.EXE. Here is an example command to make a full export of a domain partition:

ldifde.exe -f <path and name of export file>.ldf -d “<DN of partition” -p subtree -x -l * -j <path to log folder>

I mean; you can never be too sure right?

VirtualBox 4 and physical NICs

I use Oracle VirtualBox for much of my testing and demos. Yesterday I figured I wanted to connect two machines in a way that enabled the VMs on one talk to the other, while at the same time keeping the traffic isolated. The easiest way I could think of was adding a second NIC in each machine (host) and connecting them with a crossover cable. One of the hosts already had a RealTek Gbps adapter and the adapter I added also used the same chip, and thus the same driver. Windows handled this fine, even though both NICs had the same device name (not connection name). Unfortunately VirtualBox did not. I couldn’t tell the two cards apart. Looking in the config files for VMs configured for bridged networking, I discovered that VirtualBox references physical NICs on the host by name, not GUID etc. So the only way I could think of for fixing this was to rename the second NIC.

Like I said, the connection name was different and easily changed, but the device name required some tweaking. Here’s how I did it:

  1. Look up the Device Instance Path for the NIC in Device Manager.
    This is most easily done by opening properties for the NIC from the Network Connections window, since you will not be able to tell the NICs apart if you use Device Manager (since they have the same device name).
    The Device Instance Path will look something like this:
    PCIVEN_10EC&DEV_8168&SUBSYS_816810EC&REV_014&98ECCA6&0&00E3
  2. Now open regedit as LOCAL SYSTEM
    This is most easily achieved by using PsExec.exe from Sysinternals. The command is:
    psexec.exe -i -s regedit.exe
  3. Navigate to the key HKEY_LOCAL_MACHINESYSTEMCurrentControlSetEnum.
  4. Under the Enum key you will find the “path” from the Device Instance Path.
  5. Under the last part of the path, 4&98ECCA6&0&00E3 in this case, you will find a value called DeviceDesc. It will probably have a value something like this:
    @oem49.inf,%rtl8168.devicedesc%;Realtek PCIe GBE Family Controller
    The value first makes a reference to the driver INF file and a variable within it. I do not know how this works in detail, but you can just change it to any sting value you want. The INF file will not contain the data you choose anyway. I changed mine to the name of the device with “#2” appended.
  6. Before the data in DeviceDesc can be changed you need to give the local Administrators group (or another security principal of your choice), access to the key. While the last part of Device Instance Path is selected, hit Edit and select Permissions. Add the security principal you want.
  7. Change the value of DeviceDesc.
  8. Exit regedit.
  9. Open or refresh the Network Connections window and check that your device has the new name.
    I did not have to reboot for the change to take effect, but that may be necessary.

After this procedure my “new” NIC instantly showed up in VirtualBox and was available for Bridged Networking. I filed a bug at the VirtualBox site so maybe they will fix it. I have no idea of the consequences for making this change yet. It may break Windows networking in some subtle way, but I doubt it. You may also have trouble if you need to upgrade the device driver for the NIC, but again, I doubt it. To make sure you do not run into trouble, save the previous value of the DeviceDesc value so you can change it back later. You could also change the permissions back to what they were before. Consider this my disclaimer.

Troubleshooting Forefront Endpoint Protection 2010 Installations

I had a hand in rolling out Forefront Endpoint Protection (FEP) for a customer recently. Some of our clients did not get FEP installed even though the SCCM client was installed and working correctly, and they had all prerequisites present and had successfully received the advertisement and downloaded the files from the distribution point (DP). It turned out that these clients were already running Microsoft Security Essentials (MSE), which FEP does not detect or uninstall. The solution was to manually uninstall MSE first and then wait for the next installation attempt from the SCCM client.

For future reference; these are the Anti-Malware products that FEP can detect and uninstall before it installs itself:

  • Symantec Endpoint Protection version 11
  • Symantec Corporate Edition version 10
  • McAfee VirusScan Enterprise version 8.5 and version 8.7
  • Trend Micro OfficeScan version 8.0 and version 10.0
  • Forefront Client Security version 1 including the Operations Manager agent

If you want to troubleshoot FEP deployments here are som interesting logfiles:

  • %WINDIR%%TEMP%FEP-ApplyPolicy-%COMPUTERNAME%.log
  • C:Documents and SettingsAll UsersProgramdataMicrosoftMicrosoft Security ClientSupportEppSetup.log
    (This folder also contains other interesting files regarding the FEP install.)

An overview of all SCCM 2007 logfiles is available here: http://technet.microsoft.com/en-us/library/bb892800.aspx

 

Authentication errors on NLB cluster

I configred a 2 node NLB cluster to load balance Remote Desktop Session Hosts with Windows Server 2008 R2. These were virtual servers running on VMWare so I selected to use multicast mode for the cluster. The cluster IP (.3) correctly resolved to the cluster multicast address with ARP. The cluster formed and converged successfully.

After a few minutes I could no longer access node 2 (.2) from any other computer. The error given by Windows was “The target account name is incorrect”. This is a Kerberos authentication error (KRB5KRB_AP_ERR_MODIFIED)., which means there is a mismatch between the name you entered in your request and the name of the server you are actually contacting. In other words, the name of the machine is different from the name you specified, yet you were sent to that machine by the networking stack. I also experienced another error when trying to remotely administering node 2; The RPC server is unavailable. How could this be?

Turns out the reason is IPv6 6to4. When both NLB nodes were given the same official IPv4 address as a secondary address on their adapters (the cluster address), they also both configured the corresponding IPv6 address for that IPv4 address on their 6to4 adapters. Both machines had the same IPv6 6to4 address. Once they both registered this in DNS the game was up. Whenever I typed in the name of node 2 in an UNC path it randomly resolved to the IPv6 address of node 1. When the connection was attempted on node 1 the actual name of node 1 was different from what was specified in the Kerberos ticket that the connecting machine got from the Domain Controller, thus producing the “mismatch” error, and in turn the RPC error.

To work around the error I disabled 6to4 on both NLB nodes:

netsh interface ipv6 6to4 set state disabled

This is not a perfect solution and I would much rather find a way to disable the registration of 6to4 addresses in DNS. I will have to look into this further…

Another approach would be to disable strict name checking on the hosts, in effect disabling the requirement that the name of the machine must match the name in the Kerberos ticket, but that is a major change with some serious security issues attached to it. Also the effect of having two machines with the same 6to4 address is unknown to me.

By the way, you might not see this error in your network or lab, since 6to4 addresses are only configured for machines with official IPv4 addresses. In other words, if you use any of the RFC1918 addresses (10/8, 172.16/12 or 192.168/16) you will not have 6to4 addresses.

The Case of The Strange Folder Redirection Error

I was enabling Folder Redirection for some Windows 7 Professional machines, or rather, for the users of some Windows 7 Professional machines. The users already had a server based home directory with a My Documents folder, which also had data. The purpose of the operation was to, firstly, enable Folder Redirectin, but also to merge the contents of the My Documents folder on the client machines with the My Documents folder on the network server. First, to see what kind of conflict resolution Folder Redirection had, I created a file with the same name (but different content) in both the local My Documents folder and the one on the server. After logging on the first time the Folder Redirection policy was active I found this error event in the Application log on the client machine:

Log Name:      Application
Source:        Microsoft-Windows-Folder Redirection
Event ID:      502
Level:         Error
User:          <domain><username>
Computer:      <computername>
Description: Failed to apply policy and redirect folder Documents to \<servername><share><username>Documents.
Redirection options=0x1001.
The following error occurred: Failed to copy files from C:Users<username>Documents to  \<servername><share><username>Documents.
Error details: This function is not supported on this system.

Originally I thought this was a problem with NTFS file permissions on the file server, but these we OK. After all, other clients were redirecting their folders without problem. Since the error details didn’t give me any clue I decided to try to remove my offending duplicate file. I deleted it from the client machine and on the next logon the My Documents folder redirected without problem.

The error in the log should have a better explanation of what is happening. Folder Redirection is definitely supported on Windows 7 Professional. The Folder Redirection specific event logs didn’t contain any more information either. The error text should have said something along the lines “File conflict; file X already exists”. Maybe in Windows 8.

Unfortunately I solved the problem before I had a chance to enable debug logging for the Folder Redirection Client Side Extension. Maybe that would have told me what the problem was. Should you want to enable debug logging for Folder Redirection you can do so with this command:

reg.exe add “HKLMSoftwareMicrosoftWindows NTCurrentVersionDiagnostics” /v FdeployDebugLevel /d 0x0f /t REG_DWORD

If you are on Windows XP/2003 or earlier this will give you a log file: %windir%debugusermodefdeploy.log. If you are running Windows Vista/2008 or newer you will simply get more events in the Windows event logs.

So, should you find yourself staring at this error in the middle of the night (or any other time), see if you have any duplicate files in the folders you are trying to redirect.

Happy redirecting!

UPDATE: The duplicate files I created were created by a different account than the user owning the client computer and home directory on the server. That means that the user actually owning the folders could not delete or move the duplicate file. That could also be the reason for this error. But the fact remains; it is still a very poor error message.

UPDATE 2: I have now had a chance to test the conflict resolution in Folder Redirection and from my tests it seems that the client wins if the same file (but with different content) exists on both the server where you are redirecting to and on the client. I performed the same experiment as outlined above; two files with the same name, one on the server and one on the client. This time they were both created by the user owning the client and the folder on the server. Upon the next logon the folder redirection policy took effect and the local files were copied to the server, merging them with the content that already existed there. But as I say, the copy of the identical file on the server was silently overwritten by the file on the client. So now you know.

Script to find outdated computer objects in Active Directory

Computers have accounts in Active Directory and log on just as user accounts do. The “user name” of a computer is its name with a dollar sign appended, e.g: MYPC1$. The password is set by the machine when it is joined to the domain and changed every 30 days by the machine. Just as with user accounts, computer accounts gets left in the domain when the machine is decomissioned or reinstalled under another name etc. This leaves our directory service full of outdated user and computer object. (Outdated in this context means “has not logged on to the domain for X days). This in turn makes it hard to know how many computers are actually active. So how to battle this problem?

On Windows 2000 every DC maintend an attribute on each computer object called lastLogon. It stored the timestamp of the last logon for a computer on that DC. The “on that DC” part is important, because the lastLogon attribute was not replicated beyond the local DC. So to figure out when computer CORP-PC1 last logged on, you would have to query the lastLogon attribute on all the DCs in the domain and find the most recent one. This could be done, but was tedious and time consuming. Enter Windows Server 2003.

In Windows Server 2003 a new attribute was introduced; lastLogonTimestamp. Just like lastLogon, lastLogonTimestamp stored the timestamp of the last logon to the domain for a computer account, but lastLogonTimestamp is a replicated attribute, which means that now every DC knows the most recent logon time for a computer. So to figure out when CORP-PC1 last logged on to any DC in the domain we can query any DC for the lastLogonTimestamp attribute of CORP-PC1. Very nice.

I often find myself in a situation where I need to “clean house” in customer’s directories so I created a script that uses lastLogonTimestamp to find all computers that have not logged on to the domain for X days. X varies greatly from customer to customer, but a good start would be 90 days. The sciprt is called FindOutdatedComputers.vbs and in this post I would like to share it with you.

FindOutdateComptuers.vbs has 3 user changeable variables: intTimeLimit, strDomain and strDC. These are defined at the very beginning of the code and should be the only variables you change.

intTimeLimit: the number of days since a computer logged on to the domain

strDomain: LDAP domain name; e.g. dc=mydomain,dc=com

strDC: FQDN of DC in domain; e.g. dc1.mydomain.com

The script must be run with cscript.exe and takes one command line argument; a 0 (zero) or a 1 (one). 0 mens just echo out the machines that have not logged on since the defined limit and 1 means echo out, but also move the comptuters to the Outdated Computer Objects OU (which the script creates). It will never delete any information from your AD, only move computer accounts to the Outdated Comptuer Objects OU. If you omit this argument, the default action is just to display the results and not move anything.

For FindOutdatedComptuers.vbs to work your domain functional level must be at least Windows Server 2003. The script checks for this and warns you if you are below the needed level.

Here is the code:

https://gist.github.com/morgansimonsen/8040007

Desktop.ini customizations do not take effect

You copy a desktop.ini file into a folder to customize and maybe localize it. You have correctly set the file’s attributes to Hidden, System and Read-Only, but still your customizations do not work. To make it work you need to set the Read-Only or System flags on the folder where the desktop.ini file resides. As I am sure you know, folders cannot be read-only, neither can they “remember” your settings so that all new files placed in the folder become read-only as well. So this flag is purely to tell Windows that this is a special folder and that it should look for a desktop.ini file inside it.

To change the attributes from the command line type:

attrib +R +S <your folder>

From PowerShell:

Set-ItemProperty <your folder> -Name Attributes -Value “ReadOnly,System”

It took me a while to figure this out so hopefully someone can make use of this info.

Here is a link to another issue where your desktop.ini customizations do not work:

If you won’t translate RDS profiles; I will!

Out of pure frustration with the fact that the Active Directory Migration Tool (ADMT) is unable (unwilling is my guess) to do security translation for users’ Remote Desktop Services (RDS) roaming profiles, I decided to take matters into my own hands and created the script below. It is not very refined just now, but I have a lot of ideas for future versions. In the meantime, if you can use it for something; great!

# TranslateRDSProfiles.ps1
# Morgan Simonsen
# http://morgansimonsen.wordpress.com
#
# Script to translate a user profile belonging to a migrated user.
# Primarily intended to solve the problem with ADMT not translating
# Remote Desktop Service roaming profiles.
#
# Version: 1.0 (13.02.2012)
#     Initial version.

#----------------------------------
# User changeable strings
# Only edit in this section!
#
# Root of folder or sharing storing the profiles
$RDSProfileRootDirectory = "Z:" # Include trailing backslash
# NetBIOS Name of source domain; the domain the user was migrated from
$SourceNBTDomainName = "DOMAIN_1" # Include trailing backslash
# NetBIOS Name of target domain; the domain the user was migrated to
$TargetNBTDomainName = "DDS" # Include trailing backslash
# DNS name of target domain
$TargetDNSDomainName = "dds.intern"
# FQDN of Domain Controller in target domain
$TargetDomainDC = "ddsdc1.dds.intern"
# Location of subinacl.exe
$SUBINACLLocation = "C:Program Files (x86)Windows Resource KitsToolssubinacl.exe"
# Location of echoArgs.exe (not actually used by the script)
$echoArgs = ($PSHome+"ModulesPscxAppsechoArgs.exe")
#----------------------------------

Get-ChildItem $RDSProfileRootDirectory | ForEach `
{
    Write-Host ("Processing folder: "+$_.name)
    If (Get-QADUser ($TargetNBTDomainName+$_.name) -service $TargetDomainDC)
    {
        $user = Get-QADUser ($TargetNBTDomainName+$_.name) -service $TargetDomainDC
        Write-Host (" Found matching user: "+$user.Userprincipalname)

        If ( (Test-Path ($_.FullName+"NTUSER.DAT")) -or (Test-Path ($_.FullName+"NTUSER.MAN")) )
        {
            Write-Host (" Found user registry hive: "+($_.FullName+"NTUSER.DAT"))
            Write-Host (" Updating permissions...")
            $entireprofile = ($_.FullName+"*.*")
            $completeusername = ($TargetNBTDomainName+$user.samaccountname)
            Write-Host (" SubInACL.exe File Output:")
            & $SUBINACLLocation /noverbose /subdirectories $entireprofile /grant="$completeusername=f" /setowner=$completeusername  > $null
            Write-Host (" SubInACL.exe Registry Output:")
            reg.exe load ("HKU"+$_.Name) ($_.FullName+"NTUSER.DAT") > $null
            $regkey = ("HKEY_USERS"+$_.Name)
            & $SUBINACLLocation /noverbose /subkeyreg $regkey /grant="$completeusername=f"  > $null
            reg.exe unload ("HKU"+$_.Name) > $null
        }
    }
    Else
    {
        Write-Host " Cannot find user in domain that matches folder name."
        Write-Host " Continuing with next user."
    }
}