Finding free space in Exchange Server 2003 Databases

Finding out how much free space (white space) an Exchange database has is something I regularly do. I want to show you two ways you can get this information.

First of, you can use ESEUTIL /MS <database name> to dump the free space information, but this requires that the database is dismounted. That is not always practical, since users may want to read some email while you check for free space.

The second way is to view the Application log of the Exchange server. When Exchange finishes online defragmentation, which is scheduled to run each night by default, it logs an event with ID 1221 in the Application log. The source can either be MSExchangeIS Mailbox Store or MSExchangeIS Public Store, dependig on whether the store is a mailbox or a public folder store. A typical 1221 event looks like this:

Type:         Information
Event:        1221
Date Time:    11.07.2006 06:00:00
Source:       MSExchangeIS Mailbox Store
ComputerName: SERVER1
Category:     General
User:         N/A
Description:  The database “Storage Group Astore1” has 659 megabytes of free space after online defragmentation has terminated.

Another neat trick is to use the eventquery.vbs tool to dump all the recent 1221 events so you can easily see all your databases. A sample commmand to dumpt the events from the last 24 hrs would be:

cscript c:windowssystem32eventquery.vbs /L Application /V /FI “ID eq 1221” /FI “DATETIME eq 07/11/06,01:00:01AM-07/11/06,11:59:59PM” /FO LIST

Outlook tip

Just discovered a new hotkey in Outlook that I previously did not know about. When typing names in the To field you are often presented with the drop down list of previously resolved addresses from the Outlook local cache. This cache is stored in a file usually called outlook.nk2 in you userprofile directory tree. But if you have not previously used the address and want Outlook to try to resolve it you can press CTRL+K. This is the same as pressing the Check Names button on the toolbar.

Difference betwen groups in the Builtin container and Domain Local groups

Group sAMAccountType groupType systemFlags isCriticalSystemObject
Built-in 536870912 -2147483643 -1946157056 Yes
DL       536870912 -2147483644 <Not Set> <Not Set>The groups in the Builtin container may look like ordinary Domain Local groups, but they are not. In Windows Server 2003 Active Directory they are listed as Builtin Local.
These groups cannot be used on other machines in a domain when the domain is in native mode (as can other Domain Local groups in native mode).
The builtin groups are only valid on Domain Controllers.

<— From ADSI SDK : Start ————————————————————————>
The groupType attribute in Active Directory contains the group type definition:

The first three flags specify the group scope.
The ADS_GROUP_TYPE_SECURITY_ENABLED flag indicates the type of the group. If this flag is set, the group is a security group. If this flag is not set, the group is a distribution group.ADS_GROUP_TYPE_GLOBAL_GROUP        = 0x00000002
ADS_GROUP_TYPE_LOCAL_GROUP         = 0x00000004

Group that contains only accounts and other account groups from its own domain.
This group may be exported to a different domain.

Group that can contain accounts and universal groups from any domains. It may not be included in either access-control lists of resources in other domains or groups other than global groups in the same domain.

This flag is for the WinNT provider as the
ADS_GROUP_TYPE_DOMAIN_LOCAL_GROUP flag is for the LDAP provider.

Group that can contain accounts and account groups from any domains, but not domain local groups.

Group that is security enabled. This group can be used to apply an access-control list on an ADSI object or a file system.
<— From ADSI SDK : End ————————————————————————–>

In reality there exists a fifth group type:

These groups can only be security enabled, and when they do, they have a groupType attribute with the value -2147483643.

Exchange group relationships in a multi-domain forest

In a multi-domain forest with Exchange 2000/2003 installed there are some special group relationships.
Each domain for which DomainPrep has been run, has the following Exchange related groups:
  • Exchange Domain Servers (Global Group)
  • Exchange Enterprise Servers (Domain Local Group)

Exchange Enterprise Servers
Purpose: Group all Exchange servers in a specific Enterprise (organization/forest)
This group has the follwing members:
– The computer account of all Exchange servers in the organization
– The Exchange Domain Servers group from all domains where DomainPrep has been run

Exchange Domain Servers
Purpose: Group all Exchange Servers in a specified domain
This group has the following members:
– The computer account of all Exchange servers in the domain where the group exists

When adding a new Recipient Update Serveice to a domain in a multi-domain forest, that previously has not had Exchange, it is quite usual to get the following errors in the application log on the Exchange server (Exchange server is usually located in another domain):

Source:  MSExchangeAL
Category: LDAP Operations
Event ID: 8270
Description: LDAP returned the error [32] Insufficient Rights when importing the transaction
dn: <GUID=A907D19B-18F7-4098-95AB-A8E029C1634C>
changetype: Modify

In this event the dn: <GUID> is the GUID of the ‘Exchange Enterprise Servers’ group in the domain specified in the event. The member:add:<GUID> is the GUID of the ‘Exchange Domain servers group’ from another domain.
Probably a domain that was recently added to the forest. You will see this error for each of the other domains in the forest. The event will be repeated but with a different GUID in the member:add field.

You will also see this error:

Source:  MSExchangeAL
Category: LDAP Operations
Event ID: 8270
Description: LDAP returned the error [32] Insufficient Rights when importing the transaction
dn: <SID=0102000000000005200000002A020000>
changetype: Modify

In this event the <SID> is the SID of the ‘Pre-Windows 2000 Compatible Access’ group in the domain that is specified in the event (dc=xxx,dc=xxx),and the member:add:<GUID> is the GUID of the ‘Exchange Domain Servers’ group in one of the other domains in the forest. You will see this error for each of the other domains in the forest. The event will be repeated but with a different GUID in the member:add field.
These errors are most likely due to incorrect permissions in the target domain’s Active Directory.
The permissions are not correctly set or all information in not yet replicated.

Thus we can deduce the follwing member relationships:
Group Name:                          Memebership:
Exchange Domain Servers         All Exchange Servers in the group’s domain
Exchange Enterprise Servers      Exchange Domain Servers from each additional domain in the forest
Pre-Windows 2000 Compatible    Access Exchange Domain Servers from each additional domain in the forest

There is also a KB article that deals with this here:

Missing permissions cause the Recipient Update Service not to process accounts in Exchange 2000 Server and Exchange Server 2003

Logging on through Terminal Services on a Windows Server 2003 Domain Controller

I work extensively with multi-domain forests, usually in a configuration with an empty root domain and several child domains that host users and computers etc. The other day I was trying to log on to a newly added Domain Controller in a child domain. I was going to prepare the domain for Exchange so I was trying to log on with an account that was a member of the Enterprise Admins group in the root domain. This usually works very well, because this group is always a member of the local Administrators group in any child domain. This time however I could not log on and got this error message:
To log on to this remote computer, you must be granted the Allow log on through Terminal Services right. By default, members of the Remote Desktop Users group have this right. If you are not a member of the Remote Desktop Users group or another group that has this right, or if the Remote Desktop Users group does not have this right, you must be granted the right manually.

This, of course, led me to investigate. The first thing I discovered was that this Domain Controller had also been made a Terminal Server. I won’t go into how bad an idea that is here, suffice to say that I do not recommend it. From this fact it followed that someone also probably had changed the default settings of the Allow log on through Terminal Services right in some policy, and that was probably the reason I could not log on. Sure enough. The Default Domain Policy had been changed (again, not a good idea), granting the Allow log on through Terminal Services right to a global group in the domain only. Let’s call that group TSUsers. I also discovered that the same someone had also added the TSUsers group to the Remote Desktop Users group. Normally that should have been enough to allow log on through Terminal Services. Obviously it wasn’t. So I had two problems. First, why could I not log on as a member of the Administrators group when the Default Domain Policy had been changed, and second; why was it not enough to add the TSUsers group to the Remote Desktop Users group to allow them to log on through Terminal Service?
By default, the Allow log on through Terminal Services right is controlled through the Local Computer Policy, the one you can edit with gpedit.msc. The default setting for Windows Server 2003 is to grant this right to the Administrators and Remote Desktop Users local groups. If the server is promoted to a Domain Controller, the Remote Desktop Users group is removed from the Local Computer Policy, leaving only the Administrators group.  So on a Domain Controller it is not enough to be a member of the Remote Desktop Users group to log on through Terminal Services. You must be a member of the Administrators group in the domain. That is probably what confused the person who had set up the server. He had added his domain group (TSUsers) to the Remote Desktop Users group and been unable to log on, since the server was a DC. That answered my second question. To solve this problem he edited the Default Domain Policy and gave the right to his domain group. But in doing so he overrode the Local Computer Policy, which gives members of the Administrators group access. This was what made me unable to log on to the server, and the answer to my first question. Easy!
Interestingly enough the text in the Remote tab on a Domain Controller does not change, even if Remote Desktop Users no longer can log on through Terminal Services. It still says that members of the group has access.
To solve my immediate problem I added the Enterprise Admins group to the Default Domain Policy in the child domain and was able to log on and do my Exchange preparation. This setup is obviously not recommended. A DC should never be a Terminal Server and domain based policies should not be changed in such a way as to lock out administrators.
In researching this post I also found out another interesting thing about Terminal Services in Windows Server 2003. You no longer have to give a user or group both the Log on locally and Allow log on through Terminal Services rights to be able to log on via Terminal Services. This was needed in Windows 2000. In Windows Server 2003 it is handled this way:
  • Log on locally controls logon via the console (not RDP console, but keyboard attached to the server)
  • Allow log on through Terminal Services controls logons via Terminal Services.
You can read more about that in this KB article:
Difference in the user right “Deny log on locally” between Windows 2000 and Windows 2003

How to remove Event Logs from Event Viewer

The Event Viewer management console has several categories depending on the roles of a server. All systems running Windows have Application, Security and System logs, or categories. These logs are represented by .evt files on disk, typically located in the %SystemRoot%system32config directory.

When a system loses a role, eg. it is demoted from Domain Controller to member server, the logs associated with that role remain in the Event Viewer console on that system. This can be quite annoying, not to mention misleading. Not only is the category retained in Event Viewer, but all the events are there as well.

I will not show you how you can remove these logs. The problem is that the .evt files are locked and cannot be deteled. They are locked by the the Event Log Service which cannot be stopped. The solution is to use Mark Russinovich’s excellent utilities PendMoves and MoveFile. You can find them here:

Windows often needs to replace a file that is in use. This presents a problem when the process using the file cannot be stopped. To resolve this problem Windows has a spesial API that can tell Session Manager to delete that file, or replace it, on the next reboot. The MoveFile utility does just that. It tells Session Manager where to move, or delete, a file on the next reboot, before the system starts it’s services and applications. This info is stored in the registry key HKLMSystemCurrentControlSetControlSession ManagerPendingFileRenameOperations. You can write to this key using WMI or your own app, but I use Mark’s tool since it is already there.

To get rid of eg. the old File Replication Service Log from a server you would first go into Event Viewer and get the path to the .evt file by selecting properties on the log. Usually you would get C:WINDOWSSystem32configNtFrs.evt

Next, run the following command from the directory where you extracted PendMoves and MoveFile:

MoveFile.exe C:WINDOWSSystem32configNtFrs.evt “”

The “” indicates a NULL destination and is interpreted by Session Manager as a delete operation.

Now you can run PendMoves to get a list of any file move/delete operations scheduled for the next reboot.

But to get completely rid of the log we also will have to remove some setting in the registry, or else the Event Log Service would just recreate the file we deleted. The new file would be empty, of course, but the log would remain in Event Viewer.

Continuing the example with the File Replication Service Log, navigate to the key HKLMSYSTEMCurrentControlSerServicesEventlog
This is the main key for the Event Log Service and it has a subkey for each log that Event Viewer displays. Delete the entire key of the log you want to get rid of.

After the next reboot the logs should be gone from Event Viewer.

Sometimes they are not however. This usually happens because the service that uses the log is still set to Automatic startup. For example; when a DC is demoted to member server, the FRS service is not stopped and disabled. If this is the case the registry key you deleted will be recreated by FRS at startup and a new logfile created. So make sure to check all corresponding services before rebooting.

Meeting with Jesper Johansson from Microsoft

I just found out that I am invited to a lunch and later dinner date with Microsoft’s security guru Jesper Johansson. Jesper is a principal figure in Microsoft’s Trustworthy Computing program and is a frequent and popular speaker at the major Microsoft conferences such as TechEd and IT Forum. I have spoken with Jester at IT Forum before, but always very shortly and with thousands of people present. This time it will be only me and my colleagues from the Norwegian Microsoft TechNet Speaker Community. It will be a great opportunity to pick Jesper’s brain about security, a subject I feel strongly about, and anything else the conversation might turn to.

I am also invited to a seminar the next day where Jesper will be keynote speaker as well as hold some in-depth sessions. That also sounds very interesting.

I had the same oportunity earlier this year when I was invited to dinner with John Craddock and Sally Storey, but I had to cancel because I was going to Barcelona for a week 🙂

If you want to know more about Jesper Johansen and what he is up to (I recommend it), you can check out his blog:

ExMerge and Outlook 2003 PST Files

Recently I was trying to import some PST files to an Exchange 2003 server using ExMerge. The PST files had been collected from clients that had previously been running POP3 accounts using various versions of Outlook. The import was part of a migration process where we moved from these POP3 accounts to Exchange 2003. Earlier we upgraded all clients to Outlook 2003 and created profiles for Exchange. Although all the clients were running Outlook 2003 now, some PST files had been created with previous versions of Outlook and some with Outlook 2003. As it turned out, this was a problem.

Upon starting the import job I immedialey started to get failures in ExMerge. This was unusual since I had verified all the PST filenames and made sure they matched the mailbox name in the Exchange database. I exmained the ExMerge log and found these errors:

Merging data from file ‘E:EXMERGEDATAUSER.PST’ to mailbox ‘User’ (‘USER’) on server ‘SERVER1’.
Error configuring message service (MSPST MS)(MAPI_E_EXTENDED_ERROR)(CMapiSession::CreateEMSPSTProfile)
Errors encountered. Copy process aborted for mailbox ‘User’ (‘USER’).
I had never seen this particular error before so I started to investigate.
After some research I found out that Microsoft changed the format of the PST and OST files in Outlook 2003 to eliminate the problems with the 2GB limits on these files that the previous verions of Outlook were hampered by. The new Unicode PST/OST files can be up to 20 GB and are in Unicode by default. The older versions of Outlook user ANSI. If you want to know more about this feature I suggest the fillowing links:
Combining this information with the error from the ExMerge log (CreateEMSPSTProfile), I guessed that maybe ExMerge could not read this new Unicode format. This was indeed the case, as the Knowledge Base article 823176 proved:
“The ExMerge utility does not support Unicode .pst files. If you export data from Outlook 2003, the default .pst format is Unicode. To work around this issue, create an ANSI .pst file. This is a .pst file that is compatible with Outlook 97 and with Outlook 2003.”
Here is the link to the article if you want to read all of it:
The only soultion left to me in this case was to manually convert the new Unicode PST files to ANSI PST files. This was a very tedious process, that had to be done with Outlook 2003. First creating an ANSI PST file and the importing the Unicode file, and finally importing that file with ExMerge into the Exchange store. I could not find any utility to do this automatically or one to convert the file. Luckily for me there were only a few files out of the total that had to be converted.
The reason that I write this blog post is partly to share an interesting discovery with my peers, but also beacuse the error that ExMerge logs (CreateEMSPSTProfile) is not documented anywhere as being caused by an Outlook 2003 Unicode PST file. I think that it should be and so I blogged about it here. Hopefully Google will index this post soon and it will be available when someone has the same problem as I did.

Information wants to be free!