Running two instances of SQL server on the same machine will cause them to contend for memory and slow down.
I would definitely increase memory on the machine.
Are you using standard or enterprise versions for windows and sql
You will want to add as much memory to the server as you can afford to (with SQL this is just a best practice anyway).
When running multiple instances (no matter which version) you’ll want to limit the maximum amount of memory that each instance can use so that they aren’t fighting for memory.