Sounds to me like spanning tree issues. I think the 3Com is probably a single spanning tree for all vlans, and the Cisco is per vlan spanning tree. The Cisco ‘sees’ it’s own bpdus, or those from another Cisco switch you have on the network which is running VLANs, and shuts the port down.
Provided the Catalyst 500 is only used for end devices, and not on linking to other switches, you can safely turn off spanning tree on the switch. Try it and see if the link stays up.
Just a guess, but I have seen sometihng similar on other switches where Cisco was connected to 3Com. It’s not really a compatibility issue, just a different way of doing the same thing. Please post back with any results.