You have to analyze more regarding the various processes and taking how much memory. Check on the virtual memory also. if your ram size is 4GB, VRAM can be fixed max upto 16GB. This will definitely improve the performance.
Last of all if it does not work, then you have to check the server configuration (RAM, processor, hard disk size and their usage), and increase few of the resources based on the available versus the peak time usage ratio..