While it would require lot more inputs to correctly identify your bottleneck, here are some suggestions:
1. Apart from sar try iostat to identify disk IO bottlenecks.
2. Use prstat to list all the processes running and how much resource they are consuming. Check the Mem and Paging utilization particularly.
3. Use vmstat and swap -a to check swap space utilization. I’ve come across several poorly tuned servers running Oracle RAC with similar problem that you are reporting.
4. Check how your http server is servicing traffic.
5. You mentioned slow opening of application upon entering username/password. Could this be an authentication issue? Check LDAP, NIS or whatever authentication mechanism you are using.
Hope this helps!