It’s hard to say without knowing more.
I would check the eventlog on both servers looking for clues. It sounds like the SMTP service is hanging or a hardware (or software) firewall is getting in the mix somehow and botching things up.
The eventlog will give you some good clues… plus it will give you some raw error data to either post back here or google.
I tried the maximising the logs for transport and no errors there as well.
which logs genearate for this type of problems so that I increase the logging level for that?