This can turn into quite a long topic and I anticipate that my response will be further groomed by other members of the community. To get the discussion started, however, here are a few things to consider
First and foremost, physical location of the data center; it should be as discrete possible. That means no signs with “data center” on it, etc. You may also want to consider going with no windows or at a minimum, tinted or one way mirror windows.
There definitely needs to be controlled access to the building(s), floor(s) and computer room(s). There are the typical ID cards that can be swiped to enter and exit the facility for security logging purposes; as well as a sign in mechanism — preferably electronic which can also be linked into the change management system.
Of course, no matter how strong the door lock, don’t circumvent it with slip ups in construction, such as using lowered tiles for the ceiling, where someone can simply climb over the wall from an insecure area into the data center. Also, no external door hinges!
To further preserve security, entrance to the facility can be granted only on a two person basis, so that no one is alone in the data center. In fact, physical access to the data center should be permitted only when necessary, such as to perform hardware installation and maintenance. Server administrators may use remote access for day-to-day activities. There are several mechanisms by which to accomplish this, such as terminal services, other remote clients, remote access boards, IP-based KVM and so on.
There are yet other considerations such as fire suppression systems, redundant power and other utilities. Hopefully, these help to get the thought motors going on the subject.