Instead of just doing an isolated test on your second site systems, consider actually failing over real production processing to the second site, let it run for a while there, then transfer processing back home.
Things to watch out for. It may take an outage to transfer processing. You may have capacity issues due to divergent equipment or telecomm. You will need to have well planned out “return home” procedures, which also may take an outage to execute.
The advantages are huge though. Like you say, how else will you know? Isolated testing can only prove viability to an extent. Can you ever have all users available to participate in an isolated test so you can prove capacity viability? How else will you prove your return home procedure work? Better question is have you even considered needing return home procedures?
As much as it makes sense, my experience is business units rarely want to accept the inherent risk in doing an actual production failover when there is no disaster. This is a level of maturity usually only found in very large corporations in the financial industries who build hot, remote clustered, HA systems. The rest of us tend to come up short of this level of testing maturity and accept the risk of not doing the real failover tests.