It depends on the risk you are willing to take and how layered your other network protection is. Your network is as strong as your weakest system (or users where the issue typically lies). Antivirus is always recommended or something like intrusion prevention or network admission control.
A properly configured firewall will suffice, it will have all of the above features. Just keep your VOIP server up to date and ensure no zerodays are out by reading the whitepapers, etc. By the way, if you shoot for a hardwall, get something with SPI/DPI (DPI solves the need of AV) and ensure you have at least 3 layers protected on the OSI Model. When it comes to VoIP server penetration, on a commercial level you’re fighting off stack overflows, and zero day remote exploits, all day to say the least.