Remember the HOSTS file!

After moving a virtual Exchange 2007 machine from Virtual Server to Hyper-V I started getting error messages both in the Event Logs and in the Exchange Management Console and Shell. During the move I had also changed the IP address of the server. Here are the error messages:

In the Application Log:

EventID: 4001
Source : MSExchange System Attendant Mailbox
EntryType : Error

Message: A transient failure has occurred. The problem may resolve itself in awhile. The service will retry in 56 seconds. Diagnostic information:

Cannot open mailbox /o=Lab Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=LAB-EXCH/cn=Microsoft System Attendant.

Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException: Cannot open mailbox /o=Lab Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=LAB-EXCH/cn=Microsoft System Attendant. —> Microsoft.Mapi.MapiExceptionNetworkError: MapiExceptionNetworkError: Unable to make co

nnection to the server. (hr=0x80004005, ec=2423)

Diagnostic context:

……
Lid: 12952dwParam: 0x6BAMsg: EEInfo: prm[0]: Long val: 135
Lid: 15000dwParam: 0x6BAMsg: EEInfo: prm[1]: Pointer val: 0x00000000
Lid: 15000dwParam: 0x6BAMsg: EEInfo: prm[2]: Pointer val: 0x00000000
Lid: 16280dwParam: 0x6BAMsg: EEInfo: ComputerName: n/a
Lid: 8600 dwParam: 0x6BAMsg: EEInfo: ProcessID: 2372
Lid: 12696dwParam: 0x6BAMsg: EEInfo: Generation Time: 2009-03-26 13:01:49:138
Lid: 10648dwParam: 0x6BAMsg: EEInfo: Generating component: 8
Lid: 14744dwParam: 0x6BAMsg: EEInfo: Status: 10060
Lid: 9624 dwParam: 0x6BAMsg: EEInfo: Detection location: 318
Lid: 13720dwParam: 0x6BAMsg: EEInfo: Flags: 0
Lid: 11672dwParam: 0x6BAMsg: EEInfo: NumberOfParameters: 0
Lid: 19778
Lid: 27970StoreEc: 0x977
Lid: 17730
Lid: 25922StoreEc: 0x977

at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Int32 ec, DiagnosticContext diagCtx)

at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Object objLastErrorInfo)

at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize)at Microsoft.Mapi.ConnectionCache.OpenMapiStore(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, ClientIdentityInfo clientIdentity, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)at Microsoft.Mapi.ConnectionCache.OpenMailbox(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, WindowsIdentity windowsIdentityAs, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

— End of inner exception stack trace —

at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags)at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString)at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsAdmin(ExchangePrincipal mailboxOwner, CultureInfo cultureInfo, String clientInfoString, Boolean useLocalRpc, Boolean ignoreHomeMdb)at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsAdmin(ExchangePrincipal mailboxOwner, CultureInfo cultureInfo, String clientInfoString)at Microsoft.Exchange.Servicelets.SystemAttendantMailbox.Servicelet.Work()

In Exchange Management Console (EMC) and Exchange Management Shell (EMS) when viewing the Mailbox node under Server Configuration:

——————————————————–

Microsoft Exchange Warning

——————————————————–

The following warning(s) were reported while loading topology information:

get-MailboxDatabase

Completed

Warning:

ERROR: Could not connect to the Microsoft Exchange Information Store service on server <server FQDN>. One of the following problems may be occurring: 1- The Microsoft Exchange Information Store service is not running. 2- There is no network connectivity to server lab-exch.lab.net. 3- You do not have sufficient permissions to perform this command. The following permissions are required to perform this command: Exchange View-Only Administrator and local administrators group for the target server. 4- Credentials have been cached for an unpriviledged user. Try removing the entry for this server from Stored User Names and Passwords.

This last error occurs when either the Management Console or Management Shell executes the Get-MailboxDatabase -Status cmdlet.

Outlook Web Access also did not work and displayed this error:

Outlook Web Access could not connect to Microsoft Exchange. If the problem continues, contact technical support for your organization.

Request

Url: https://<servername>:443/owa/default.aspx

User host address: <address of client>

Exception

Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException

Exception message: Cannot open mailbox /o=Lab Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=morgan.

Call stack

Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags)

Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString)

Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)

Microsoft.Exchange.Clients.Owa.Core.OwaWindowsIdentity.CreateMailboxSession(ExchangePrincipal exchangePrincipal, CultureInfo cultureInfo)

Microsoft.Exchange.Clients.Owa.Core.UserContext.CreateMailboxSession(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.UserContext.Load(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.CreateUserContext(OwaContext owaContext, UserContextKey userContextKey, UserContext& userContext)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.PrepareRequestWithoutSession(OwaContext owaContext, UserContextCookie userContextCookie)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.InternalDispatchRequest(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.DispatchRequest(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.OwaModule.OnPostAuthorizeRequest(Object sender, EventArgs e)

System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Inner Exception

Exception type: Microsoft.Mapi.MapiExceptionNetworkError

Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80004005, ec=2423) Diagnostic context: …… Lid: 12952 dwParam: 0x6BA Msg: EEInfo: prm[0]: Long val: 135 Lid: 15000 dwParam: 0x6BA Msg: EEInfo: prm[1]: Pointer val: 0x00000000 Lid: 15000 dwParam: 0x6BA Msg: EEInfo: prm[2]: Pointer val: 0x00000000 Lid: 16280 dwParam: 0x6BA Msg: EEInfo: ComputerName: n/a Lid: 8600 dwParam: 0x6BA Msg: EEInfo: ProcessID: 3372 Lid: 12696 dwParam: 0x6BA Msg: EEInfo: Generation Time: 2009-03-25 20:59:22:652 Lid: 10648 dwParam: 0x6BA Msg: EEInfo: Generating component: 8 Lid: 14744 dwParam: 0x6BA Msg: EEInfo: Status: 10060 Lid: 9624 dwParam: 0x6BA Msg: EEInfo: Detection location: 318 Lid: 13720 dwParam: 0x6BA Msg: EEInfo: Flags: 0 Lid: 11672 dwParam: 0x6BA Msg: EEInfo: NumberOfParameters: 0 Lid: 19778 Lid: 27970 StoreEc: 0x977 Lid: 17730 Lid: 25922 StoreEc: 0x977

Call stack

Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Int32 ec, DiagnosticContext diagCtx)

Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Object objLastErrorInfo)

Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize)

Microsoft.Mapi.ConnectionCache.OpenMapiStore(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, ClientIdentityInfo clientIdentity, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)

Microsoft.Mapi.ConnectionCache.OpenMailbox(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, WindowsIdentity windowsIdentityAs, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)

Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

The Exchange Best Practices Analyzer (BPA) also failed when run locally against the server. The error was: Exchange Server cannot be contacted.

Despite these errors all Exchange services were running and Outlook clients on the internal network could connect to Exchange and open their mailbox. Any access through CAS did not work, so Outlook Anywhere or OWA were unavailable.

After searching the web I did not find an exact match for these errors, at least none that had my particular symptoms. But from what I read it looked like an issue with the network configuration, i.e. the network card.

First I went into Device Manager and removed all missing hardware devices, this included the old Intel 21140 emulated NIC from Virtual Server. I had not removed the Virtual Server Virtual Machine Additions until after I had moved the server so its configuration was still in the Registry. For information on how to remove missing devices, have a look here. Finally I removed the Virtual Machine Bus Network Adapter, reinstalled it and set the correct IP address information. All this did not resolve the error either, but I discovered something interesting in the Registry regarding the Virtual Machine Bus Network Adapter, which I will cover in a later post.

Still working with the networking idea I ran the Exchange Best Practices Analyzer (BPA) from a different server, and this time I could connect and complete the scan. This meant that at least the network was working and that the Windows Firewall was not blocking the communication. The next step was to run Wireshark on the server while I ran the BPA against the local server, that is not over the network. This produced the first real indication on what the error was. Here is a screenshot from the capture:

The Exchange server’s new IP address on Hyper-V was 192.168.131.3. But while it was hosted on Virtual Server it’s IP address was 192.168.131.180. So judging from the ARP broadcasts, which are coming from the local server, the old IP address is still active somehow. To test this theory I changed the server’s IP address back to the old one 192.168.131.180. Now the BPA finished successfully and EMC/EMS were able to successfully run the Get-MailboxDatabase -Status cmdlet. I changed the IP address back to the new one, 192.168.131.3, hoping that this would somehow reset the erroneous configuration. It did not, and the errors immediately came back. So the old IP was still somewhere in the Registry. I did a search in the Registry for 192.168.131.180, and this is what I got:

reg query HKLM /s /f “192.168.131.180”
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkListNlaCacheIntranetAuth2.4.fe80::7de2:4df0:fa6e:2231%10 1.1.192.168.131.180
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkListNlaCacheIntranetAuth2.4.fe80::d0b2:3082:e67:d254%10 1.1.192.168.131.180
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkListNlaCacheIntranetAuth2.4.fe80::e1e0:a13d:50cc:f9b3%10 1.1.192.168.131.180
HKEY_LOCAL_MACHINESYSTEMControlSet001Services{3832C4EC-AC88-42FA-B266-51D68342079A}ParametersTcpip
IPAddress REG_MULTI_SZ 192.168.131.180
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServices{3832C4EC-AC88-42FA-B266-51D68342079A}ParametersTcpip
IPAddress REG_MULTI_SZ 192.168.131.180
End of search: 5 match(es) found.

I ignored everything under the NLA key which is part of the Network Location Awareness service. That left the GUID key ({3832C4EC-AC88-42FA-B266-51D68342079A}) under Services. Each GUID key under the Services key represents a Network Card. So which card was this?

Windows maintains a list of the network cards in the system under the Registry key HKLMSOFTWAREMicrosoftWindows NTCurrentVersionNetworkCards. This is the list from my system:

reg query “HKLMSOFTWAREMicrosoftWindows NTCurrentVersionNetworkCards” /s
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkCards8
ServiceName REG_SZ {3832C4EC-AC88-42FA-B266-51D68342079A}
Description REG_SZ Microsoft Virtual Machine Bus Network Adapter

So this was the Virtual Machine Bus Network Adapter. Time to examine the TCP/IP registry settings for this adapter more closely.

The main TCP/IP configuration location in the Registry is under HKLMSYSTEMCurrentControlSetServicesTcpip. The configuration of the network cards is in the subkey ParametersInterfaces:

reg query HKLMSYSTEMCurrentControlSetServicesTcpipParametersInterfaces /s
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersInterfaces{3832C4EC-AC88-42FA-B266-51D68342079A}
UseZeroBroadcast REG_DWORD 0x0
EnableDeadGWDetect REG_DWORD 0x1
EnableDHCP REG_DWORD 0x0
NameServer REG_SZ 192.168.131.2
Domain REG_SZ
RegistrationEnabled REG_DWORD 0x1
RegisterAdapterName REG_DWORD 0x0
DhcpServer REG_SZ 255.255.255.255
Lease REG_DWORD 0x1c20
LeaseObtainedTime REG_DWORD 0x49caa3c7
T1 REG_DWORD 0x49cab1d7
T2 REG_DWORD 0x49cabc63
LeaseTerminatesTime REG_DWORD 0x49cabfe7
AddressType REG_DWORD 0x0
IsServerNapAware REG_DWORD 0x0
DhcpConnForceBroadcastFlag REG_DWORD 0x1
IPAddress REG_MULTI_SZ 192.168.131.3
SubnetMask REG_MULTI_SZ 255.255.255.0
DefaultGateway REG_MULTI_SZ 192.168.131.1
DefaultGatewayMetric REG_MULTI_SZ 0

As we can see there is a configuration mismatch between the configuration under the GUID key ({3832C4EC-AC88-42FA-B266-51D68342079A}) under Services, and the configuration under the Tcpip key. Seemed like a buggy TCP/IP stack so I decided to reset it:

1. Uninstalled the Virtual Machine Bus Network card from Device Manager.
2. Reset WinSock:
   netsh winsock reset catalog
3. Reset TCP/IP:
   netsh int ip reset %temp%reset.log
4. Reboot

These steps would completely reinstall TCP/IP and reinstall the NIC.

But after the reboot the errors were still present and Wireshark saw the same ARP requests for the old address 192.168.131.180. The old address was now completely gone from the Registry and Active Directory does not store IP addresses (usually). So where was this information coming from? I looked in the forward and reverse lookup zones on the DNS server, but the correct IP was present in the forward zone. The old address was in the reverse zone, but that was not relevant here. I did a ping from the command line for the server name, and it returned the old address! But Nslookup did not! Finally I was on to something! The HOSTS file!

Here is the hosts file on the server:

# Copyright (c) 1993-2006 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a ‘#’ symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
127.0.0.1 localhost
#::1 localhost
192.168.131.180    lab-exch
192.168.131.180    lab-exch.lab.net

And there was the answer and the explanation for everything. I had previously entered the information for the local server into the hosts file to work around a bug in the DSProxy components on the CAS server where DSProxy would not listen on IPv6 addresses. More info on this issue here. This also explained why other machines could connect to the server and why EXBPA failed when run locally.

I updated the addresses and rebooted. The issue with DSProxy and IPv6 was fixed in Update Rollup 4 for Exchange 2007 with Service Pack 1, but I was not able to make it work in my setup without the entries in the hosts file. My guess is that this is because I have an ISA server between the Exchange server and the clients.

So there you have it. A very simple solution arrived at via very long, cumbersome and unnecessary process. So remember that old hosts file guys.