You do not tell us about how do you publish your Exchange to the outside world.
Considering your security concerns which I subscribe entirely, I assume that you are using ISA server 2006 or TMG to publish your Exchange to the *World*.
These are *the best* way to secure your Exchange. (not only, bu also)
If this is correct, You can easily configure a publishing rule, to publish IMAP access, in a way that only internal or VPN.users can access it.
If you’re not using either of the above, we need to know how you are publishing you exchange installation or what are you using at the edge of your network.
As a last resort, you can always enable a firewall rule in your Exchange host to prevent access to IMAP from other that you LAN and/or VPN users.