DNS Issue in Windows 2003 Enviornment
Enviornment Variables:
Company Network Running Windows 2003 on 2 Servers for DNS.
Using Private IP's Internally, NAT through PIX 515E Firewall to Cisco 2610 Router with 2 T1's load balancing to different networks. clients running DHCP.
The configuration was working. Something has affected DNS resolution to certain domains and hosts. The only factors that have knowingly changed is the addition of a second (different homed) T1 line, although tried to eliminate this by shutting down the interface on the router to the second T1, giving us the network config of the working config. Most sites work fine.
The problem manifested itself in users being unable to access www.yahoo.com or www.bbcnews.com (can access websites via IP's).
Forwarders are set up in MS DNS for All Domains (unresolveable). NSLookup can get a record for the domain (yahoo.com) but not for www.yahoo.com (this request times-out, without ever going to the next DNS server). There are no entries/zones for yahoo anywhere in our DNS infrastructure. I have verified and bypassed by hardcoding one of our ISP's DNS servers on an internal computer... it works flawlessly. So somewhere the server(s) is not forwarding the request. I even attempted setting up a specific forwarder in DNS for the domain yahoo.com with its DNS servers listed... but to no avail (although in NSLookup, it did attempt contact through yahoo.com's DNS servers before reverting back to our internal server at which point it timed-out.)
Any ideas? Any addition information required?
Thanks in advance.
-Matt Schipper
Software/Hardware used:
Software/Hardware used:
ASKED:
December 6, 2004 1:39 PM
UPDATED:
December 8, 2004 10:36 AM
Answer Wiki:
have you tried flushing the DNS? start>cmd ipconfig /flushdns
To see all answers submitted to the Answer Wiki: View Answer History.
It’s a bit of a reach, but some viruses and spyware plant bogus entries in the hosts file (c:winntsystem32driversetchosts).
To troubleshoot, I’d start with a command prompt, and try to ping http://www.yahoo.com as well as yahoo.com and see (when it times out or not) what IP address it was trying to go to. Try this on several systems, including the DNS servers themselves.
Another step is to go to the command prompt and run nslookup from the command line, set the server equal to a number of different servers – starting with the default, and then stepping through all your forwarders.
Please let me know if these steps work for you or not. I’ve got other tricks up my sleeve, but these two usually provide enough pointers so that you figure the rest out on your own.
Bob
Minor followup based on seeing my own reply. Apparently, whatever software parses these replies has either filtered out the backslash, or is using it as a metacharacter.
The path to the hosts file SHOULD look like this (using double-backslashes to see if I can force them).
C:winntsystem32driversetchosts
Bob
Bob,
Thanks for the reply.
One of the things you mentioned struck a bell and led me down a path which I believe has now resolved the issue.
It seems that something changed in external DNS somewhere, and my systems admin had our default listing for the domain set with a wildcard. Possible causing a looping effect. Especially since this record is default to our webhost, and not the company network. I am not exactly sure, but upon verifying through one of the forwarders I recognized the IP being returned… and went from there.
Again, thank you.
Sincerely,
Matthew A. Schipper
Spoke too soon. Forgot to /FlushDNS .
Still not working. It appears as though it is not passing the request to the forwarders.
Spoke too soon. It didn’t work.
the IP was still cached locally on the Client when testing.
There are no eronous entries in the HOSTS file.
It appears that the request is not getting forwarded. The internal DNS server just sits on the request until it times out. The request never gets to the forwarders
However… the forwarders can access it just fine.
Any ideas why the servers stopped working on only a few domains/sites?
Ok Matt, try this, and variations on it.
go into nslookup at the command prompt, and make sure that it’s pointing at one of the servers in question. You’ll see that IP Address/name as soon as you’re in nslookup (dig is a more recent tool that does that and more, but nslookup is installed on most windows systems, so we’ll go with what’s known to be there).
If one of the key servers is not currently the default server, then make it so by entering the command “server 1.2.3.4″ or whatever the IP address is.
Then try “ls -d yahoo.com”
The name server should respond with “Can’t list…” indicating that it has no knowledge of it. If you get query refused, there’s something fishy going on. The purpose of this test is to make sure that no one or no-thing has made the DNS server authoritative for any domains that you didn’t have in mind. If you were running BIND, or Windows DNS with EXTERNAL files (as opposed to using the registry), I’d have you look at the BOOT file (C:winntsystem32dnsboot), but if all your values are in the registry, then you’ll have to look at the DNS GUI to check for all zones for which the server is set.
Next, at the nslookup prompt, do a “set debug”
and lookup yahoo.com and http://www.yahoo.com and see if there’s any difference. You’re going to get a BUNCH of stuff, but just wade through it carefully. If you want to, you can do a “set d2″ which provides even MORE information. The purpose in this is to look carefully at the request that is leaving the nslookup client (the resolver), and to see exactly what information is coming back from the name server.
As another test, try looking up “www.yahoo.com.” with a trailing period. This will force DNS to recognize that there is no other suffix.
Oh, and look for any instance of “yahoo” or “bbc” in your A record listing on the DNS server too.
You didn’t specifically say so, but have you been running a sniffer on the DNS servers themselves to see what (if any) traffic is passing as a result of your testing? That might also be useful.
If that doesn’t help, then consider calling me – but at that point, I’m likely to charge you money – fair warning.
Bob
Resolution Found:
The two combined elements that caused the issue were the
Cisco PIX (v6.3(1))
Windows Server 2003 Running DNS
The problem is caused by Server 2003′s utilization of EDNS0 packets, occassionaly larger than 512 bytes via UDP to servers that accept EDNS0.
(This is why it was so rare, but constant to certain sites)
The Cisco PIX before software v6.3(2) drops packets larger than 512 bytes to UDP port 53.
In version 6.3(2) and above you can alter the
- (No) Fixup Protocol DNS
command (to either increase the drop size, or remove this parameter altogether)
Alternatly you can modify Windows Server 2003 DNS not to use EDNS0 Packets.
The solution I chose, was to Update PIX IOS to 6.3(4) and remove the Fixup Protocol DNS command.
On a side note, make sure to Update your PDM image as well so you can modify this setting from the PDM if desired.
Attached below are two articles that might help others on this issue.
Microsoft Article on Subject
http://support.microsoft.com/default.aspx?scid=kb;en-us;828731
Corrected Cisco Link for PIX IOS Release Notes:
http://www.cisco.com/en/US/products/sw/secursw/ps2120/prod_release_note09186a00801a6d21.html
Sincerely,
Matthew A. Schipper