Exchange Me! » Exchange Tracking Center

DATABASE CORRUPTION IN EXCHANGE…PRT3

B00M3R — Thu, 17 Apr 2008 17:07:36 +0000

In this section we will look at exporting the damaged page file to a text file. Name the txt file in relation to the page file itself so as below this relates to page 3106. Therefore, you could use the following command to create a file called 3106.TXT.

This must be run from the \Program Files\EXCHSRVR\BIN directory

eseutil /m “d:\program files\exchsrvr\mdbdata\pri

1.edb” /p3106 >3106.txt

After running the command you will see the text file created.
Microsoft(R) Exchange Server(TM) Database Utilities
Version 6.0
Copyright (C) Microsoft Corporation 1991-2000. All Rights Reserved.
Initiating FILE DUMP mode…
Database: d:\program files\exchsrvr\mdbdata\priv1.edb
Page: 3106
pgnoThis <0×02360004, 4>: 3106 (0×00000064)
objidFDP <0×02360018, 4>: 11 (0x0000000b)
ulChecksumParity <0×02360000, 4>: 466674925 (0x1bd0e4ed)
dbtimeDirtied <0×02360008, 8>: 216893 (0x0000000000034f3d)
cbFree <0x0236001c, 2>: 4026 (0x0fba)
ibMicFree <0×02360020, 2>: 3265 (0x0cc1)
itagMicFree <0×02360022, 2>: 2 (0×0002)
cbUncommittedFree <0x0236001e, 2>: 0 (0×0000)
pgnoNext <0×02360014, 4>: 0 (0×00000000)
pgnoPrev <0×02360010, 4>: 0 (0×00000000)
fFlags <0×02360024, 4>: 3141 (0x00000c45)
Parent of leaf
Internal page
Root page
FDP page
Multiple Extent Space (ParentFDP: 97, pgnoOE: 340)
Index page (non-unique keys)
TAG   0    cb:   16    ib:    0    offset: 28 - 37    flags: 0×0000
TAG   1    cb:    6    ib: 3259    offset: ce3 – ce8    flags: 0×0000
Operation completed successfully in 1.91 seconds.

Things to Note:
When I generated my report, I picked page number 3106 at random. As you can see at the bottom of the text file, page 3106 is an index page. If you ever have to repair a 1018 error, you’ll usually lose all of the data on the page that you encountered the error on. Of course, that’s assuming that the error occurred on a leaf page. An index page links too many other pages. Therefore, if the page that I selected had actually been damaged, I might have lost the index page and all related leaf pages. This could possibly lead to a complete breakdown of the entire database. Fortunately, Exchange is really good about rebuilding structural components, such as index pages.

When you browse the file you can see the page number and previous page number and the next database pages. Also you should pick out the checksum parity bit number and you can use all this info when attempting to repair the database.

You also need to realize that you can get false 1018 errors. Now and then a faulty disk cache or a faulty hard drive will cause data to be read from a location other than the correct location specified by Exchange. When this happens the database is initially corrupt and you have a serious problem on your hands because the disks actions will soon cause the database to become totally corrupt.

At this stage you probably really need to understand about repairing a corrupt database which Ill talk about more in another post. For now understand how to read the txt file which will help you resolve and repair.

DATABASE CORRUPTION IN EXCHANGE…PRT2

B00M3R — Fri, 04 Apr 2008 07:00:15 +0000

In the first part of this corruption post I spoke of what happens when an Exchange database goes corrupt on you. I also mention it can happen at page level, database level or store level. I mentioned hardware as well which will cause problems. This post will concentrate on the 1018 error.

The common type as mentioned of database corruption is at the page level. You’ll get heaps of different error messages the can potentially relate to this type of corrupt database. The most common being 1018, now we have all seen the 1018 Jet_errReadVerifyFailure (havent we?) Most of the time this error indicates that the page has the incorrect checksum value or the page number is incorrect. Microsoft Exchange will begin reading the database page by processing the page number; this is so it knows its reading the correct data. If for some reason the number is different from the expected number, then yes you guessed it exchange generates the 1018 error.

Otherwise Microsoft Exchange will calculate a checksum value for the page then verify the calculated value and make sure it matches the stored checksum value. If it does match then the page is assumed to correct and valid otherwise, yes again the 1018 gets generated.

Should I take the 1018 seriously?

The 1018 is often a warning that bad things are on their way. Exchange is trying to tell you that the database has failed once and may go again. Normally they will be non fatal failures; however future failures could be worst and corrupt the store.

So if you haven’t worked it out already the 1018 is associated with an individual page within the database rather than the whole database. The 1018 is normally not fatal and some may take it as of no interest at all because Exchange has been known to generate this error with infrequently used data such as in a deleted items folder or just a plain old blank database.

What do I do with the 1081?

Try and find more about the error before trying to repair, first instinct is to fix but learn more about it first. This way you can try and determine exactly what data was affected and how fatal the failure is, also what’s the likelihood of it happening again.

In the next section we will look further at the 1018 but in the meantime check out some of the links on the 1018 error.

Understanding and analyzing -1018, -1019, and -1022 Exchange database errors

New error correcting code is included in Exchange Server 2003 SP1

Microsoft engineer Mike Lee recorded a great support webcast last year that is also helpful

Recipient Policies What Are They?

B00M3R — Sat, 12 Jan 2008 12:11:04 +0000

Recipient Policies What Are They?

Recipient policies are organisation wide objects held in the “Recipient Policies” container – a sub container of Recipients. When you install Exchange the program creates a default policy and you can then create as many policies as you want to after. Although you must keep the default policy and you cannot delete it.

What do they do?

Set a default value for the domain used by Exchange to reference files via IFS. (IFS provides access to the Exchange information store by using Win32 file system APIs)Generates email proxy address. RUS generates and sets email addresses on new mail enabled objects but you define the format for the addresses and the type of proxy address that RUS generates through policy.Controls how the mailbox manager processes mailboxes.

Enables SMTP virtual servers to accept incoming mail. When you make an installation of Exchange the virtual servers will accept email from the domain defined in the default policy, but you add policies to cover additional domains if you require.

The Recipient Update Service (RUS) is responsible for creating and maintaining E-Mail Addresses in your Exchange Organization. The Recipient Update Service creates an Entry (Recipient Update Service (Enterprise Configuration)) for the entire Exchange Organization for modifying objects in the Configuration Container Partition in Active Directory and one RUS for every Exchange enabled Domain in this Forest.

Exchange Install.After install there is one default policy created called “Default Policy” This policy contains Proxy address for the default SMTP domain and one for X.400 – You can add additional Proxy addresses to the default policy or have different Recipient Policies for different users.Note:If you want to remove old or unwanted E-Mail addresses, you must either remove the addresses manually in Active Directory Users and Computers or use an automated process. Or you can use LDIFDE.

Build numbers and release dates for Exchange Server

B00M3R — Thu, 20 Dec 2007 09:40:46 +0000

How to tell what Exchange version you are using? Many of us know of course but Microsoft have build numbers to indentify exact builds.

To find your build: The easiest way is to open ESM>Administrative Groups>Domain Name>click on the servers folder and to the right you will see all your servers and under Server Version is the version type. Make sure you are running the latest service pack for the version you are using.

Version                                             Build number              Release date
———————————————————————————————-
Microsoft Exchange Server 4.0                      4.0.837                   April 1996
Microsoft Exchange Server 4.0 (a)                4.0.993                   August 1996
Microsoft Exchange Server 4.0 SP1               4.0.838                   May 1996
Microsoft Exchange Server 4.0 SP2               4.0.993                   August 1996
Microsoft Exchange Server 4.0 SP3               4.0.994                   November 1996
Microsoft Exchange Server 4.0 SP4               4.0.995                   April 1997
Microsoft Exchange Server 4.0 SP5               4.0.996                   May 1998

Microsoft Exchange Server 5.0                       5.0.1457                  March 1997
Microsoft Exchange Server 5.0 SP1                5.0.1458                  June 1997
Microsoft Exchange Server 5.0 SP2                5.0.1460                  February 1998

Microsoft Exchange Server 5.5                        5.5.1960                  November 1997
Microsoft Exchange Server 5.5 SP1                 5.5.2232                  July 1998
Microsoft Exchange Server 5.5 SP2                 5.5.2448                  December 1998
Microsoft Exchange Server 5.5 SP3                 5.5.2650                  September 1999
Microsoft Exchange Server 5.5 SP4                 5.5.2653                  November 2000

Microsoft Exchange 2000 Server                      6.0.4417                  October 2000
Microsoft Exchange 2000 Server (a)                 6.0.4417                  January 2001
Microsoft Exchange 2000 Server SP1                6.0.4712                  July 2001
Microsoft Exchange 2000 Server SP2                6.0.5762                  December 2001
Microsoft Exchange 2000 Server SP3                6.0.6249                  August 2002
Microsoft Exchange 2000 Server post-SP3       6.0.6487                  September 2003
Microsoft Exchange 2000 Server post-SP3       6.0.6556                  April 2004
Microsoft Exchange 2000 Server post-SP3       6.0.6603                  August 2004

Microsoft Exchange Server 2003                        6.5.6944                  October 2003
Microsoft Exchange Server 2003 SP1                6.5.7226                  May 2004
Microsoft Exchange Server 2003 SP2                6.5.7638                  October 2005

Microsoft Exchange Server 2007 8.0.685.24 or 8.0.685.25 December 2006
Microsoft Exchange Server 2007 SP1 8.1.0240.006 November 2007

Message tracking event IDs in Exchange Server 2003

B00M3R — Sun, 16 Dec 2007 10:16:20 +0000

Here are some of the event IDs that are logged to message tracking log files. You can enable message tracking logs to track or to troubleshoot the flow or status of a message in Exchange Server 2003 as shown in previous blog. You can record information about the sender, the message, and the recipient. If you want to log more detailed information, you can also record the subject line of messages.

By default, the tracking logs are located in the C:\Program Files\Exchsrvr\YourServerName.log folder. Each daily log is named in the yyyymmdd.log format according to the date that the log was created. The file name date is in Coordinated Universal Time (UTC). Here is a list of event ID’s and there meaning. You can import this log file into Excel which makes it easier to read as opening the text file is too busy.

A few FAQ’s
Q1: When a message is generated in the system for the first time, what event is associated with that message in the tracking log?
A1: There are different events for different message submission paths to Exchange Server 2003. For example, for messages that are submitted through the SMTP component, the first event ID in the tracking log is 1019. For messages that are submitted through the Store component, the first event ID in the tracking log is 1027.
Q2: Is there one event ID that covers the creation of all messages and that only appears one time per message?
A2: There is no one event that covers the creation of all messages because messages can be created in various ways by various clients, remote servers, and pickup directory. It would make no sense to use the same event for all these code paths. Or, it would be impossible to use the same event for all these code paths. However, event 1019 is logged when any message enters Inetinfo-side transport processing. The tracking log may frequently contain multiple 1019 events that have the same message ID. For example, this may occur if the server is restarted multiple times during a period when the remote destination for the particular message is down. On each restart, the message is resubmitted, and event 1019 is logged. This is expected behavior.
Q3: Why are there multiple 1020 and 1031 events that are logged for the same message ID?
A3: This is expected behavior. The same message ID can be transferred out multiple times. When the same message ID is transferred out multiple times, events 1020 and 1031 are generated.

Event Number	Event Type	Description
0	Message transfer in	The message was received from a server, a connector, or a gateway.
1	Probe transfer in	An X.400 probe was received from a gateway, a link, or a message transfer agent (MTA).
2	Report transfer in	A delivery receipt or a non-delivery report (NDR) was received from a server, a connector, or a gateway.
4	Message submission	The message was sent by the client.
5	Probe submission	An X.400 probe was received from a user.
6	Probe transfer out	An X.400 probe was sent to a gateway, a link, or an MTA.
7	Message transfer out	The message was sent to a server, a connector, or a gateway.
8	Report transfer out	A delivery receipt or an NDR was sent to a server, a connector, or a gateway.
9	Message delivered	The message was delivered to a mailbox or a public folder.
10	Report delivered	A delivery receipt or an NDR was delivered to a mailbox.
18	StartAssocByMTSUser
23	ReleaseAssocByMTSUse
28	Message redirected	The message was sent to mailboxes other than the mailboxes of the recipients.
29	Message rerouted	The message was routed to an alternative path.
31	Downgrading	An X.400 message was downgraded to 1984 format before relay.
33	Report absorption	The number of delivery receipts or of NDRs exceeded a threshold and the reports were deleted.
34	Report generation	A delivery receipt or an NDR was created.
43	Unroutable report discarded	A delivery receipt or an NDR could not be routed and was deleted from the queue.
50	Gateway deleted message	The administrator deleted an X.400 message that was queued for a gateway.
51	Gateway deleted probe	The administrator deleted an X.400 probe that was queued for a gateway.
52	Gateway deleted report	The administrator deleted an X.400 report that was queued for a gateway.
1000	Local delivery	The sender and the recipient are on the same server.
1001	Backbone transfer in	Mail was received from another MAPI system across a connector or across a gateway.
1002	Backbone transfer out	Mail was sent to another MAPI system across a connector or across a gateway.
1003	Gateway transfer out	The message was sent through a gateway.
1004	Gateway transfer in	The message was received from a gateway.
1005	Gateway report transfer in	A delivery receipt or an NDR was received from a gateway.
1006	Gateway report transfer out	A delivery receipt or an NDR was sent through a gateway.
1007	Gateway report generation	A gateway generated an NDR for a message.
1010	SMTP queued outbound	Outgoing mail was queued for delivery by the Internet Mail Service.
1011	SMTP transferred outbound	Outgoing mail was transferred to an Internet recipient.
1012	SMTP received inbound	Incoming mail was received from by the Internet Mail Service.
1013	SMTP transferred	Incoming mail that was received by the Internet Mail Service was transferred to the information store.
1014	SMTP message rerouted	An Internet message is being rerouted or forwarded to the correct location.
1015	SMTP report transferred In	A delivery receipt or an NDR was received by the Internet Mail Service
1016	SMTP report transferred out	A delivery receipt or an NDR was sent to the Internet Mail Service.
1017	SMTP report generated	A delivery receipt or an NDR was created.
1018	SMTP report absorbed	The receipt or the NDR could not be delivered and was absorbed. (You cannot send an NDR for an NDR.)
1019	SMTP submit message to AQ	A new message is submitted to Advanced Queuing.
1020	SMTP begin outbound transfer	A message is about to be sent over the wire by SMTP.
1021	SMTP bad mail	The message was transferred to the Badmail folder.
1022	SMTP AQ failure	A fatal Advanced Queuing error occurred. Information about the failure was written to the Event Manager.
1023	SMTP local delivery	A message was successfully delivered by a store drive (logged by Advanced Queue).
1024	SMTP submit message to cat	Advanced Queuing submitted a message to the categorizer.
1025	SMTP begin submit message	A new message was submitted to Advanced Queuing.
1026	SMTP AQ failed message	Advanced Queuing could not process the message. The message caused an NDR to be sent, or the message was put in the Badmail folder.
1027	SMTP submit message to SD	A message was submitted to the store driver by the MTA.
1028	SMTP SD local delivery	The store driver successfully delivered a message (logged by store driver).
1029	SMTP SD gateway delivery	The store driver transferred the message to the MTA.
1030	SMTP NDR all	All recipients were sent an NDR.
1031	SMTP end outbound transfer	The outgoing message was successfully transferred.
1032	SMTP message scheduled to retry categorization
1033	SMTP message categorized and queued for routing
1034	SMTP message routed and queued for remote delivery
1035	SMTP message scheduled to retry routing
1036	SMTP message queued for local delivery
1037	SMTP message scheduled to retry local delivery
1038	SMTP message routed and queued for gateway delivery
1039	SMTP message deleted by Intelligent Message Filtering
1040	SMTP message rejected by Intelligent Message Filtering
1041	SMTP message archived by Intelligent Message Filtering
1042	Message redirected to the alternate recipient

Exchange Message Tracking – A Great Tool!!

B00M3R — Fri, 14 Dec 2007 07:24:37 +0000

Exchange has a great feature called message tracking that enables you to track messages. It works for both directions inbound/outbound – it also does internal messages. This function has a low overhead so I leave it enabled so I can get my hands on the info when I want, although I do have a large amount of emails that pass through my organization on a daily basis so I set log removal to be low.

—————————————————————————————————————————————————————–

Here is the scenario. Your Boss calls at the wrong moment as per usual raving about a SUPER important email message that never got delivered. So what do you do? This is when you need to know how to use Message Tracking so let’s have a look at how.

How to Enable

1.       Open ESM go to servers
2.       Right click on the server and choose properties
3.       Select these options “enable subject logging and display” “enable message tracking”
4.       “Remove log files” This option set to 30 days which is long enough. If you have massive traffic consider lower times say 7-10 days.
5.       Also check out the location of the log files. Keep them away from the main store on a separate drive if possible.

Now mine looks slightly different because I do mine through a server policy as I have multiple Exchange servers. Although greyed out you can see the ticks and where I store them.

Now let’s look at Tracking Messages.

Once tracking has been running for a while you will have collected some information, then we can track messages. Let’s look at how

1.       Open ESM and then go to tools
2.       Scroll down to Message Tracking Center
3.       Choose the server you want to track the message from. This of course will be the server that the user has his or her mailbox on, depending on whether you want to track inbound or outbound messages.

At this point we can search even though nothing else is configured. But this will result in heaps of results up to a max of 1000 every message since midnight will be processed. Best case – use the other fields to narrow the search results. Once the system finds the message you can double click it which will show what exchange did with the message.

Tracking log files will be stored (by default) in a folder located at x:\Program Files\Exchsrvr\servername.log, where x is the volume you have installed Exchange Server onto. Inside this folder you will find a text file for each day that logs are being retained for. You can open these files and work from them if you want, but I would recommend doing it in Excel as the files are tab-delimited and very hard to sort through otherwise.

Ok so we have a great way of searching and finding out what has happened with an email. Now that’s it but we can advance things a bit by utilizing third party tools and REALLY bringing Message Tracking ALIVE.

Check out these links for advanced use of Message Tracking. If you search the web you will find various software, some users have created scripts to work with these logs – Just make sure you test them and not in your live enviroment

Exchange Log Analyzer Promodag Now This is great software