Transaction logs are extremely important in the running of Exchange and crucial to the proper operation. Many Admin users know about them but rarely mix it with the logs until its too late. Im going to give you an overview of the logs so you will have a better understanding for when the brown stuff hits the fan…
Exchange was built to write all transactions to these logs and then commit the changes to the database when the system allows. This way messages can be sent and received without direct contact with the database all because of this write-ahead method of logging.
They work like this…
When a message is sent the transaction is recorded in the transaction logs. Not until this transaction is commited to the database does it move from system memory so in the event of a crash, you lose the contents from memory but gain the record in the log. These transaction logs are CRITICAL to the recovery of a crashed Exchange server. The same applies for other transactions such as received messages, moved messages and deleted messages. Thats why Microsoft recommend storing the logs on redundant storage like RAID. By the way, losing a set of tran logs wont prevent you from restoring from tape, but you will lose all the messages and changes since that last full back up.
Maintain Your Logs
- Make sure you perform regular backups to commit the transactions and flush the logs.
- Make sure you place them on a dedicated drive thats set up to support heavy write loads.
- Place them on redundant arrays. RAID 1 great and RAID 5 good but not as write friendly.
- Also make sure there is plently of room on the drive, if the drive runs out of space Exchange will stop.
- Dont use circular logging unless you have very few and I mean VERY few mail boxes.
Question: What Do I Do When An Exchange Store Won’t Mount?
When a store from your Microsoft Exchange server wont mount users cannot access data and mailflow is interrupted, the phones get hot and often panic can creep in. Having a systematic approach to troubleshooting this issue is very important. Here i will try and show you how to address this issue, but as with any trouble shooting situation begin with eliminating as many variables as possible and check the event log for clues to the failure.
Answer: Information Store - Is the Microsoft Information Store (MSExchangeIS service running? Can you successfully start the store service if it is stopped? If you cannot start the service is the Microsoft Exchange System Attendant (MSExchangeSA) service started?
System Attendant - Is the System Attendant service started? Can you start the service if its stopped? If you successfully start the system attendant service make sure to start the Information Store service as well.
Reboot The Server - I’ve often spent time troubleshooting this issue and forgot to try this one. Remember if in doubt reboot. The users will be offline for a while but they are offline anyway and if the store comes back up bingo back in business.
Anti-Virus Software - Generally not an issue if your Exchange Server has been already up and running for a while and working. But if you have updated the engine or you have a new install check this out. Exchange Av comes in two flavours, generic and Exchange aware. If you are using generic make sure you are not scanning your Exchange data directory. Try disabling your AV solution and attempt a mount then you know the issue. If it mounts after disabling check with the AV software provider and make sure there is no known issues with Exchange.
Mount a Blank Database (Enterprise Edition Only) – If you are trying to isolate the problem and are not sure if the problem is with a particular database, you can try to mount a blank database. If you can successfully mount a blank database, you know that your existing database has a problem. When mounting a blank database, do not move your current database files, simply create a new database.
Database Size - Are you running Exchange Server Standard Edition? If so, has the server reached the 16-GB store limit? If you are running Exchange Standard Edition, your mailbox stores will dismount when you reach the 16-GB size limit.
Problems With The Database – Is there a -1018 or -1022 error in the application log indicating a problem with your physical storage? If so search the Microsoft knowledge base re these errors
Back Up – Was a backup of the database restored? If yes, make sure it was restored properly in accordance with the Microsoft restore guide.
Shutdown Dirty - Verify that the database was shut down in a clean (consistent) state. Use ESEUTIL /MH. If the database is shutdown in a dirty (inconsistent) state, you cannot mount it and you must replay at least one transaction log file to bring it to a clean state.
Verify - Verify the integrity of the database. Use ESEUTIL /G, which will check the low-level integrity of the database. Were any utilities, such as Exchange Server Database Utilities (Eseutil.exe), run against the database? If so, what was done? Was it done correctly? Check against the knowledge base for confirmation.
Error ID’s - Error IDs can be confusing. The initial error you may see often has a generic message. If the error you receive does not tell you exactly what is wrong, the next step is to check the application event log. Once you find the errors in the event log, be sure to look in the error text itself for the actual error number. It is better to search Microsoft.com for information about the error number in the text, than to search for the event ID.
An SMTP session is nothing more than a telnet session over TCP port 25 instead of port 22. Because of this you can instantly test your own SMTP sessions and send an e-mail without even using a mail client.
The steps that we go thorugh in test are as follows:
- Identify Sender
- Identify recipient
- Provide message body
- End session
There are some other optional requests (Message priority, subject line, etc) To connect just use your command prompt or some software like putty which can be downloaded here. Just be sure to specify the SMTP port number as below.
- Lets Connect…..
- Telnet mailserver.com 25
The handshake is where you specify your host address. You can really put what you want here but if the mail server you are trying to connect to has spam filtering set up it will reject data that is invalid or does not resolve.
- Lets Handshake…..
- EHLO mail.otherservername.com
Ok now when you specify the FROM address if the receiving mail server is filtering for spam it will expect that the domain listed in your FROM address resolves to the same IP as the domain listed when you provided your mail server name at the handshake.
- Lets Identify Sender…..
- MAIL FROM:email@example.com
Now we need to identify recipient. Most mail servers will accept a message to anyone residing on the mail server. If the mail server accepts a message from anyone this is known as open relay, great news for spammers.
- Lets Identify Recipient…..
- RCPT TO:firstname.lastname@example.org
OK now for the data. Provide your message body, start by typing the word data and then press enter. The mail server then will instruct you to begin typing the message body and to end, press enter followed by a period and then enter again.
- Lets provide Message Body…..
- DATA This is a test message please do not respond. (Make sure you remember the enter/period/enter otherwise no work)
Now lets end the session. To end the session just type quit and game over. Below is a test session i did connecting to a server in Australia.
As i mentioned in my last post SMTP commands are answered with numeric responses. Here I have included some common SMTP server response codes and what they mean. If at any stage in an SMTP transaction there is an error, it will have a unique code assigned to it so the user can hopefully determine why it has happened. However the response codes are not always errors, SMTP will also output a success code.
The first digit of the response denotes whether it is good, incomplete or simply bad.
2 = success – 4 = temporary failure 5 = permanent failure
- 211 – System status or system help reply
- 214 – Help message
- 220 – The SMTP service is ready
- 221 – SMTP is closing the transmision channel
- 250 – The command has been completed
- 354 – OK to transmit message
- 450 – Command can not be completed because mailbox is busy.
- 451 – Command has been aborted because of an error
- 452 – Command has been aborted because the receiving server is out of disk space
- 500 – Syntax Error
- 550 – Mailbox is not available or doesnt exist
- 552 – Command was aborted because the recipient exceeded their storage quota
- 554 – The transaction has failed.
- 556 – Transaction failed due to bad weather! (only joking)
SMTP is an asymmetric response protocol which means it sends a command and then waits for a response before transmitting another command. The commands that SMTP use are words but the responses are numeric values/codes. I have included some common SMTP commands.
HELO - The hello command is used to start an SMTP session. When a host needs to make a connection (SMTP session) with another server, it sends the HELO command to the other server. The server wanting to make the connection cannot send anything until it has received a response from the initial command. To ensure the host that is receiving the request knows where to send its response, the sending server uses it’s FQDN as an argument to the HELO command.
MAIL FROM: – This command is primarily used to send email addresses. The HELO command provides the receiving host with the FQDN of the server sending the message but the receiving server doesnt know which email address within that organsiation (domain) actually sent the message. Of course, there is no guarantee that the sending server belongs to the same domain as the server that transmitted the orginal HELO command because that server could have been relaying. So, its very important for the sending server to send the receiving server the originating email address so that the recipient can reply to the message if neccessary.
RCPT TO: - This command tells the receiving server the email address of the message sender. Its very common for mail messages to be sent to multiple users often in different domains. If the users are in a common domain SMTP sends the RCPT TO: command manytimes for each user. If the users are in different domains then the sending SMTP server performs a DNS lookup against all of the users to obtain the MX record that is associated with the users domain. And dont forget, this happens everytime a SMTP message is sent regardless of the number of recipients. This query returns the IP address of the recipients mail server, if the message is going to many servers which is often the case when recipients are in different domains then SMTP has to establish a session with each domain.
DATA - There is one thing all the above commands have in common, they are all used in conjuction with a parameter such as the email address. Well, the data command works differently. When the sending server transmits the data command it tells the receiving server that a stream of data will follow which of course is the message body. And the things is, because an email message can be long the receiving server needs to know when the message is complete. To do so, SMTP appends a CRLF and a period to the end of the message body. This period on a line by itself allows the receiving host to work out when the complete message has been received.
QUIT – This command is used to end the SMTP session.
RSET - This command performs an SMTP reset and aborts the current message being sent.
Again another brief post that was born from a question from one of my site administrators. He ask could I explain the meaning of the queue icons. I’m going to talk about queues in detail soon but here’s the run down on the icons and what they mean. Depending on their state each queue will be represented by a different folder icon as shown below.
- Indicates an active state.
- Indicates a ready state.
- Indicates a retry state.
- Indicates a warning state such as not available or error
Ok let’s get started with WinRoute. You have some background info and hopefully now you have the tool.
To run – open the command prompt, CD to the bin folder and run winroute.
Once open we need to specify an Exchange server from which you want to load the routing table. In my demo below I have chosen a local Exchange master. You can also load a saved routing table (winroute file) if you have one. The idea apparently from Microsoft of saving a winroute file is you can post to them if you want them to diagnose a routing issue.
You can see once WinRoute is operational you have two choices. New Server Query or Open WinRoute File. Let’s do a new server query.
The new server option asks for the server you want to use. Type in your server and click OK.
Finally we are in WinRoute.
You will see three window panes. In the tree view section you can see graphical information about your Exchange organisation. One below into the address space area. This is all the address spaces known by this server and the costs, restrictions if any, source routing groups and administrative groups.
The raw data you see at the very bottom contains routing information that Exchange uses, it’s really only for informational purposes and administrators who understand the real guts of Exchange.
From here browse around and check out what’s where and going on. You can see status information that’s interesting and will help you get your head around your Exchange set up.
If you want to know the number of routing groups, servers, connectors and much more, click actions – Get statistics.
I hope this brief intro has been useful and interesting for you. For more information on WinRoute check out this Microsoft link.
Issue E-mail messages are misrouted.
Possible solutions Look at the final connector in the expected path. Does it have any restrictions?
Look at the address space pane. Is there a more specific connector whose address space matches the recipient e-mail domain?
Issue Changes to objects in a routing group are not acknowledged by the master.
Possible solutions Verify who the master is in the Exchange System Manager. Query that server with WinRoute. Look at the member’s information list. Is the master or any other server disconnected?
Does the major version increment after adding or changing a connector?
Issue Routing groups are not found in the Directory Service.
Possible solutions For information, see Microsoft Knowledge Base article 330279, “Deleted routing groups are listed in the WinRoute tool; fix requires Exchange 2000 SP3.”
This issue can be caused by bad objects that are stuck in the routing memory of all Exchange servers in the organization. This can be caused by deleted a routing group or routing group connector. The only way to flush is by restarting all servers.
Install the Exchange 2000 Server post-Service Pack 3 rollup described in Knowledge Base article 813840, “XGEN: March 2003 Exchange 2000 Server Post-Service Pack 3 Rollup.” After applying the post-Service Pack 3 rollup, shut down all Exchange Server services simultaneously across the organization to remove the routes.
Run Regsvr32 /u xlsasink.dll in your exchsrv\bin directory for every inbound and outbound bridgehead, and stop the MTA on every inbound and outbound X.400 bridgehead. If the size of the routing groups is small, from two to four computers, and there are multiple bridgeheads, the easiest method is to run Regsvr32 /u xlsasink.dll on each computer.
A Little knowledge
As you know Exchange server uses SMTP to transfer user and system messages through your Exchange organisation. To keep network traffic low and allow flow control, Exchange uses routing groups. In every group as mentioned there is a master, this master is responsible for updating and holding the link state tables, it also propagates information back to other Exchange servers should you have any.
LSA (Link State Algorithms) are very similar to OSPF (Open Shortest Path First) More on OSPF here
Routers choose and calculate the shortest path to send network messages using OSPF we’ll Exchange uses a similar concept with LSA with a few differences. In Exchange routing groups and connections are called connectors or links. The Link State Algorithm creates a link state table which contains all the information about the routing groups, connectors and more.
Any changes made on your other Exchange servers will be sent across to the routing group master. The routing group master updates it’s LST (Link State Table) and sends the information back out to update all the other Exchange servers.
The LST info is always in RAM and is not written to hard disk. New information will be sent to other routing group masters each time the routing group master restarts etc.
What does LST contain = Information on connectors, servers, routing groups, your organization, address spaces, the link state, costs and the version.
Let’s talk WinRoute. WinRoute will determine the routing status of your Exchange organization. WinRoute is a tool from Microsoft specifically for Exchange 2000/2003 and used correctly you can determine the link state routing information that is known to your routing group master. This tool can be used for troubleshooting and is an excellent tool to have to hand.
Where to get WinRoute. You can download from Version 6.5.7226 from here
Once you have the tool extract the files and copy to a folder of your choice. I like to place the files in the \exchsrvr\bin directory.
Next we need to load the routing table from the routing group master. Now if you only have one routing group then you’ll only have one master but if you have several routing groups you’ll have many masters. So choose the master. See below for determining your master. In my scenario I have many routing groups and have decided on NDOEX02 master. Go to ESM open your routing groups/group and check the name of your master.
Now we know the name of the master we can run WinRoute and start to investigate. Next…
SMTPDiag is a great tool. I mentioned it briefly in my last post so I thought I’d give you the run down on how to use it. If someone says “I can’t seem to send an email to this address “and you want to check out resolution information check out SMTPDIAG.SMTPDiag issues DNS queries using both UDP and TCP as mentioned. The first thing it does after of course checking the syntax is to check Start of Authority (SOA) for the remote address domain.
An SOA(State of Authority) Record is the most essential part of a Zone file. The SOA record is a way for the Domain Administrator to give out simple information about the domain like, how often it is updated, when it was last updated, when to check back for more info, what is the admins email address and so on. A Zone file can contain only one SOA Record.
OK the next step is to validate that the local domain MX/A records can be resolved. The test is used to verify that the sender domain is valid and any bounces can be returned to the original server. This test could fail if the domain name is not resolvable from inside the firewall. The remote MX/A records are then also checked. NOTE: If this step fails, mail will not route because of DNS issues, then you know you must check out your DNS infrastructure.
What’s an MX record?
MX stands for Mail Exchange Records. MX records are used in DNS records(or Zone files) to specify how email should be routed. More on MX records here
When all the DNS checks are complete and successfully checked, the tool will try and establish a connection to all the MX records that were published from the remote domain on port 25 and try to EHLO them all, mail from, rcpt to and then the quit command.If you use the verbose (/v) option when you run the tool, more information will be provided about what each test is doing, as well as detailed results of each test step. See my example below.
This is The Usage Syntax: SMTPDIAG “sender address” “recipient address” [-d target DNS] [/v]You will also notice from the results that there is colour coding in the syntax. This means the following:
White text indicates action being taken.
Gray indicates informational results
Green indicates a successful test result.
Red indicates a failed test result.
|sender address||Required. Address of a local mailbox. Used to verify SMTP submission and check inbound DNS.|
|recipient address||Required. E-mail address of remote mailbox you are trying to send mail to. Used to verify DNS, and remote mailbox availability.|
|-d target DNS||Optional. IP address of target DNS server to use to look up remote MX (mail exchange) records for testing. This is often configured as an external DNS server in Exchange. The external DNS setting is not available for Internet Information Services (IIS) SMTP.|
|/v||Optional. Displays additional information about each test.|
I’ve deleted my domain name in this example: Hope this helps!
So download the tool and get testing its the only way to learn…..