I assume from your question, that you’re not using Java within Oracle. You’re “just” using JDBC to access Oracle.
In case of performance problems, it important that you identify the wait causes. Is it the DB that’s busy, or your frontend application that’s slowing things down – or a combination. Reasons for performance problems are plentitude – without knowing excatly where to look, you just left to guess work and probably will never find the right solution.
In general, if an application gets slow under load, it’s archtectural. Table design wrong, or your SQLs aren’t build right. If you’re faced with a climbing memory consumption that won’t go down, it looks like you’re not freeing data from the heap. Remember to close connections, cursors etc. to free up used resources.
Your Java VM is controlled via parameters to the “java” executable on your platform. -mX and -mS can help. However, I’ve never seen them cause performance problems – I’ve raised them from their defaults when our applications run out of space and we get memory errors specifically.
Having a garbage collector doesn’t mean you can ignore cleanup in your code. Depending on your JVM version there are options to control the garbage collector.
However, before you fully target that – make sure the garbage collector is the reason your application is working slow. Maybe you’re selecting 1000s and 1000s of records back to your client only to show 10 rows, and then dismissing the rest? That wourld certainly keep the garbage collector busy cleaning up all the time. It would be better only to transfer the 10 needed rows – it’s faster and doesn’t waste memory. In other words, if the garbage collector is spending a lot of time, find out why.