1. Are there differences in interface speeds between the switches(ie 100Mbps vs 1Gbps)?
2. Does putting a client on the same switch the server uses make a difference?
If you aren’t seeing any errors on the switch ports or the NIC’s themselves, I would run a packet capture on either of the machines and see what shows up. If it is truly intermittent, you may have to capture for a while before you get anything usable. I would be interested to see what happens when you encounter problems. If the traffic flows are TCP based, do you get any resets or requests to retransmit specific segments? Packet captures will usually tell you where the problem lies. You might have to capture from both sides to really get a good idea of where the problem lies.