Linux & Windows both are okay and capable to install mail server. It depends on permit of ur pocket and how much cost u may carry, what kind of OS u want to install…. Just I leave here some discussions:
If u want to go for free softwares and inbuilt mechanism as well as cost effectiveness, u may proceed for Linux, there r no of linux flavours and some out of them r very intregated and consistent performance. viz. Red Hat, SuSE, Debian, Ubuntu (by cost) and so on. Another point is if u r using linux, u don’t worry for virus. cause maximum virus related progs r written to aim windows. So it is not necessary to install Anti-virus sw with extra cost, because linux has in-built virus scanner (Clamv etc) though 3rd party sw r also available.
If u want to use windows, then u may purchase maximum licensed softwares related to mail server to complete the jobs. And also u have a valid licensed Anti virus sw. But it is easy to install rather than linux system and u have more expert person for linux system (I guess already u have) So comparing both of them, they r equal and give numerous advantages and smoothness. Just cost and expertisation are differ.
Hope it may help u !