Posted by: Eric Hansen
Caching, DNS, ISP
Over the past few days, I’ve been messing with CDN and Object Storage from Soft Layer. Overall the experience has been pretty smooth sailing. The web UI is very easy and fluid, their offering is pretty impressive, and they even have bindings to their API via PHP which helps with my business.
One thing though that I experienced was that while some places (i.e.: Facebook when sharing a photo from the CDN) would display the picture, but I could not view it. Even after waiting 2 days for any possible DNS propagation to finish, I had this issue.
Basically, on the second day of having the CDN set up, I tested connectivity to it from a VPS I have that is in a different state. I could ping, wget the image, dig & perform an nslookup on it without issue. However, from my desktop PC it would always say that it can’t find the host. After many hours of dealing with tech support, someone mentioned that it could be poor ISP DNS issues. While I found this impossible due to the fact my computer uses Google’s DNS servers, I figured I would investigate this more.
I spent time on the phone with my ISP’s tech support to see what can be done. Initially when I talked to them, nslookup and dig would report that the domain couldn’t be found (NXDOMAIN in nslookup). After reconfiguring my network adapter in Ubuntu to use Google’s DNS instead of the system’s defaults (192.168.1.1 which pointed to Google’s DNS), and restarted it, I could finally pull up more information using nslookup, but it still gave NXDOMAIN (I could pull the SOA and such, just no A record). This was a step in the right direction, however. A couple hours later I tried again and was finally able to pull A records for the subdomain I needed.
How does this tie into the ISP and not just some home network issue? Well, even though you use Google’s DNS records, you still have to pass through your ISP’s DNS servers as well. While not all ISPs do this, mine (Wide Open West) does cache their entries. From the time I created the CDN until it started working, I kept checking it every so often to see if it was fixed, and this just furthered the retention of DNS information pertaining to my CDN. So basically even though Google would see the A record, my ISP would override it with their cached result of there being no A record present.