It is difficult to completely isolate a voice network or VoIP server. By design they are intended to initiate and receive communications from the outside world. Attackers know this as well which is why VoIP servers represent an attractive target.
There is a variety of information that can be gathered by an attacker just by sniffing network traffic and placing calls to your network. There are also tools available to enable attackers to conduct vulnerability assessments and penetration tests against your voice network to find the weaknesses.
Read Locking Down VoIP for more about the security issues facing VoIP networks and the steps you should take to proactively identify and secure any holes in your voice infrastructure.