This leads to a problem in tuning, since you might fix one problem and not see an improvement in system performance, so it seems like your fix didn’t work. Often the system is really overloaded/choked – if it is trying to run with 130% load, and you take out a 10% problem, the system is still trying to run at a 120% load, and everything still runs slow.

To address these, I use that query I posted earlier and start working down through the “worst of the worst” – looking at each query to determine what is causing it to behave badly. In one large system, I had to fix over 30 queries (mostly by adding/changing indexes) before the system dropped out of saturation mode and started being responsive to further improvements.

The Disk I/O and Buffer Get numbers suggest that the system is doing some serious table (or index) scanning, and that the data of interest isn’t cached, so it needs to hit the disk. This implies that the indexes aren’t efficient, and may explain why the optimizer doesn’t choose to use some of the indexes.

Are you using “covering” indexes? (Indexes where the index itself contains the column(s) being retrieved based on the key values.) These can seriously improve system performance, as the system can often avoid fetching the actual data row if the desired retrieval value is in the index. Sometimes a modest increase in the size of an index can cut performance by 10x or more.

I would start looking at the overall systems performance issues before focusing on this particular query. 40 minutes to retrieve 900 rows seems excessive by itself – that is the kind of performance time I would expect for a monthly detail financial report for a large corporation.

Hmm, any changes to the database server recently such as a migration request to some schemas or some parameter changes or even hardware changes ? I mean things don’t happen out of a sudden if this server has been going on fine for the last few weeks/months. Are there any batch jobs that run over the night doing lots of DML operations such as inserts, deletes and updates?

Any reasons why this query in particular ? Is it because it appeared in the top 10 SQL statements in your Statspack reports ? Btw, I think the buffer gets are really off the charts. Thats why the query is so slow.

I thought I might have made a mistake, so I started again (killed sessions, closed sql+),

Personally, I think if the problem resides purely in your database server, there is no need to test the queries on the database server (locally) and on another machine a hop away. (Thanks Carlosdl for pointing that out)
Just do all the sqlplus sessions locally on the database server. Is this machine a critical production database ? If not, you might want to do the following to get a “cleaner” reading for explain plan instead of just kill sessions.

SQL> alter system flush shared_pool;
SQL> alter system checkpoint;
SQL> alter session set events = ‘immediate trace name flush_cache’;

I don’t really fancy SQL tuning since you said that it’s poor performance throughout the database but having said that I think if this query is important, perhaps tuning it as an interim solution is better than no solution at all.

The execution plan has only slightly changed though, with 1 index from a table being used in place of another

I’m of the opinion you need to take a look at the clustering factor of the indexes in question. On top of that, another back to basics review of the indexes in question should be based on the following questions :

1) How many distinct values in that particular column of the table that was indexed are we talking about? If the number of distinct values is very low and the index used is a normal B-Tree index, it would make more sense to use a bit-map index instead on columns such as Gender i.e. Male/Female.

2) Are there lots of NULL values in that particuar column that was indexed ? If there are, chances are that index wouldn’t be utilized no matter how hard you try as the index is invalidated the minute it encounters NULL. Try using a function based index on a nvl(<NULL columnn>, <some value>) function to convert the NULL value to something instead. That is the silver bullet that burleson always preach.

3) Are your indexes composite indexes i.e. indexes placed on several columns ? If they are, the order of the columns by which your composite index is placed play a part in the attractiveness of the index to the optimizer.

The only other thing I can think of is to rebuild all the indexes. That would certainly help but given that the data in the tables are so huge, the rebuild might take a lot of time.

3) Is the amount of data in the tables queried identical ?

Paladine Solamnia: The same query is being run against the same database (on the same database server), but from different servers.

1) Are the servers hardware in terms of CPU and memory allocation for your various SGA, DB buffer cache, shared pool, etc identical ? Are you using Automatic Memory Management, btw ?
We have the pga_aggregate_target set to 3758096384 and the workarea_size_policy set to AUTO.

1) Are the servers hardware in terms of CPU and memory allocation for your various SGA, DB buffer cache, shared pool, etc identical ? Are you using Automatic Memory Management, btw ?
I’m not sure, I will need to check.

2) You said the explain plans are the same except the costs. Which component in the explain plan is different with respect to the cost? Is it the index, the join, the full table scans or the index scans ?
E.g.
Adj100 – Table access by local index row id – cost 226,644; cpu_cost 9,219,635,858; io_cost 225,960
– Index Range Scan – cost 1,517, cpu_cost 216,554,762; io_cost 1,500

Adj10 – Table access by local index row id – cost 78,757; cpu_cost 921,963,586; io_cost 78,689
– Index Range Scan – cost 1,517, cpu_cost 216,554,762; io_cost 1,500

3) Is the amount of data in the tables queried identical ? This has a big impact on how the optimizer plans its execution. For instance, querying against a small table would yield better performance if a full table scan is used while an index scan would be ideal if only 5% of the table rows are to be returned.
Yes, they are all fairly large tables (375 mil, 113 mil, 84 mil, 3 mil). I noticed the change in performance yesterday when a different index was chosen on the 375 mil table (there are only 2 indexes on the table). The other indexes are direct table->table joins, so no full table scans are done in the query.

4) I cannot emphasise this more often than I would but it is imperative that the statistics for the tables in the query along with any related indexes are accrued correctly in the form of high-quality statistics. I’ve seen the performance of queries degrade badly ‘cos the statistics are not updated and skewed badly. At the very least, try querying against dba_tables and dba_segments using this query I wrote to ascertain when the statistics are last updated:
Our stats are updated every weekend, and I have checked that they were throughout the process.

Note:- We are experiencing poor performance throughout the database, we are just using this query as a test.

Could it be that we have a problem with our optimizer?

“Setting the optimizer_index_cost_adj value to 10 effectively forces the optimizer to try to use indexes even when a full table scan may be a lot more efficient.”

Yes, we were trying to force the optimizer to use the indexes, as we noticed that our indexes weren’t being used when they should be (that is a another problem, although related). When we used hints to use the indexes, the queries ran far quicker than the ones that did full table scans, which is why we set the index cost adjustment.

” I have used the following to look at the query activity”

As this is a reporting database, both the disk and buffer fetches are high (running the query returns almost 900 rows). If I run it for just the query in question, with the queries started 1.5 mins apart (both with same explain_plan), I get the following:

EXES DISK DISK/EXE GETS GETS/EXE
1) 2 115198 57599 7468269 3734134
2) 4 165601 41400 8963173 2240793

I think we need a new database…….

I think the easiest way to troubleshoot is to find out what is the main difference between both your servers S1 and S2. As what kccrosser suggested, try setting your optimizer_index_cost_adj back to the default value of 100. Ceteris Paribus, you should probably ask yourself the following:

1) Are the servers hardware in terms of CPU and memory allocation for your various SGA, DB buffer cache, shared pool, etc identical ? Are you using Automatic Memory Management, btw ?

2) You said the explain plans are the same except the costs. Which component in the explain plan is different with respect to the cost? Is it the index, the join, the full table scans or the index scans ?

3) Is the amount of data in the tables queried identical ? This has a big impact on how the optimizer plans its execution. For instance, querying against a small table would yield better performance if a full table scan is used while an index scan would be ideal if only 5% of the table rows are to be returned.

4) I cannot emphasise this more often than I would but it is imperative that the statistics for the tables in the query along with any related indexes are accrued correctly in the form of high-quality statistics. I’ve seen the performance of queries degrade badly ‘cos the statistics are not updated and skewed badly. At the very least, try querying against dba_tables and dba_segments using this query I wrote to ascertain when the statistics are last updated:

set pagesize 1000
set linesize 300

spool ana_progress.log

alter session set NLS_DATE_FORMAT=’DD-MON-YYYY HH24:MI:SS’;

select a.owner, a.table_name, b.partition_name, a.last_analyzed
from dba_tables a, dba_segments b
where a.table_name = b.segment_name
and a.owner = b.owner
and a.owner in(‘<owner>’)
order by 1,2;

spool off

Similarly, query against the dba_indexes and their last_analyzed to see any differences.
If there are any discrepancies, run the following against the tables in your query:

execute dbms_stats.gather_table_stats(tabname=>’<table_name>’,ownname=>’<owner>’,cascade=>TRUE);

I hope this helps.