Lots of I/O for multiple concurrent queries or transactions is probable when the disk queue length climbs. Another possibility is that logging activity due to a heavy update load requires a different storage system configuration to perform well at peak times.
To reduce read I/O:
1) Size the database shared memory (SGA) correctly. There is lots of information about this topic in the Oracle doc and online. Quick summary: if Oracle isn’t configured to use the RAM then it will keep reading from disk even if caching data in RAM would improve performance (unlike SQL Server). This get more automagic in version 10g and above, but Oracle will never take over RAM without being given permission to do so.
2) If the SGA is sized correctly and you still have the high I/O rates, then you either need better execution plans or additional indexes (or both).
a) To improve execution plans, the first thing to do as a DBA is to make sure that the optimizer statistics have been gathered properly and are reasonable up to date. With 9i you should never use the ANALYZE command, always use one of the DBMS_STATS.GATHER_xxx_STATS methods. The Oracle performance tuning doc and online resources will get you going.
b) With the SGA sized right and good stats, next you must actually look at the workload. If you already know the problem queries, use EXPLAIN PLAN on them, see if there is an index or some indexes that can be created to reduce the I/O’s. If you need to identify the long-running queries, you need a tool or tools.
If you don’t have any tools for this, “STATSPACK” is built-in so start with it. Since 9i is not a current version of Oracle, you can get a copy of Don Burleson’s book for tuning 9i with STATSPACK for not much money. This will help you identify the SQL causing all the I/O, so you can either add indexes or suggest some changes to the application developer.
Heavy updates and logging
With RAID 1+0, the storage configuration and logging shouldn’t be the problem, but incorrectly sized redo log files could be a contributor. Check the log switches; if switching to new redo in less than 5 minutes during peak times, increase the redo file sizes.
Oracle’s SAME (stripe and mirror everything) recommended storage configuration is predicated on having a very high bandwidth connection (or connections). If you have the database, redo logs, and archive logs all in one RAID array attached to one SCSI channel, you may need to move the redos to another storage system. Measure your redo log write rate at peak time (just divide file size by minutes between log switches). If you’re writing more than about 50MB per minute via the same SCSI channel as your tablespaces, I would recommend separate storage for redo’s (and for arcive logs if you can get it).