Drop those Indexes and see the Magical Unicorn of Exadata do it’s Magic !

Now that I have got your attention with my sales pitch. Let me quote a sales pitch we have heard quite often from Exadata Sales rep “Drop the indexes on Exadata ; It is a magical unicorn and it will guarantee performance even without them”  Well, 50 years of computer science begs to differ. This Urban legend has to die for once and all. Indexes are computer science constructs and one of the basics of search optimization.

Hardware like Exadata cannot always be a solution to a Software problem. Exadata though a very well engineered system design to run Oracle database it cannot always compensate for wrong SQL logic. Recently we encountered a issue where a query running in a constant loop was doing lot of user I/O and hence consuming way too much CPU. We decided to dig a little further by getting the execution plan for the query.

Query

SELECT  t.*                          ,
        e.RETURNCODEMSG              ,
        r.RETURNCODE            AS R_RC  ,
        R.RETURNCODEDESC        AS R_RCD ,
        R.RETURNCODEPOVIDER     AS R_RCP ,
        R.RETURNCODEPOVIDERDESC AS R_RCPD
FROM
        (
                SELECT  *
                FROM
                        (
                                SELECT  ROWNUM rown ,
                                        t1.*
                                FROM
                                        (
                                                SELECT  TRAN.*
                                                FROM    transactions TRAN,
                                                        TRANSREVERSED RTRAN
                                                WHERE   RTRAN.LOGREFID(+)    =TRAN.ID
                                                        AND TRAN.UTIMESTAMP >= to_date('26/10/2017', 'dd/MM/yyyy')
                                                        AND TRAN.UTIMESTAMP <= to_date('26/10/2017 23:59:59','dd/MM/yyyy HH24:MI:ss')                                                          AND 1                =1                                                          AND TRAN.SERVICENAME ='STRINGTOSEARCH'                                                          AND TRAN.returncode !='000004'                                                  ORDER BY TRAN.UTIMESTAMP DESC                                          )                                          t1                          )                          q1                  WHERE   q1.rown    >=1
                        AND q1.rown
        )
        t               ,
        TRANSREVERSED r ,
        esbreturncode e
WHERE   r.LOGREFID(+)      =t.ID
        AND e.returncode(+)=t.returncode
ORDER BY t.UTIMESTAMP DESC

Now If you analyze the Query, it’s a complex one with Joins on two tables selecting data from table TRANSACTIONS and then doing an inner join on a column in table TRANSREVERSED. The predicate LOGREFID is used for this inner join. In the Explain plan we saw a full table scan being done on the TRANSREVERSED table. Even though the lookup table is small the result set via which it has to be compared is quite huge.

Execution Plan Pre-Index Ethos


<hr />
<p class="x_MsoNormal">| Id  | Operation                          | Name          | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |</p>
<p class="x_MsoNormal">--------------------------------------------------------------------------------------------------------------------</p>
<p class="x_MsoNormal">|   0 | SELECT STATEMENT                   |               |    39 | 84786 | 17007   (1)| 00:03:25 |       |       |</p>
<p class="x_MsoNormal">|   1 |  SORT ORDER BY                     |               |    39 | 84786 | 17007   (1)| 00:03:25 |       |       |</p>
<p class="x_MsoNormal">|*  2 |   HASH JOIN OUTER                  |               |    39 | 84786 | 17006   (1)| 00:03:25 |       |       |</p>
<p class="x_MsoNormal">|*  3 |    HASH JOIN RIGHT OUTER           |               |    39 | 80106 |  9053   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|   4 |     TABLE ACCESS STORAGE FULL      | ESBRETURNCODE |    76 |  2964 |     4   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">|*  5 |     VIEW                           |               |    39 | 78585 |  9049   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|   6 |      COUNT                         |               |       |       |            |          |       |       |</p>
<p class="x_MsoNormal">|   7 |       VIEW                         |               |    39 | 78078 |  9049   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|   8 |        SORT ORDER BY               |               |    39 | 12558 |  9049   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|*  9 |         HASH JOIN OUTER            |               |    39 | 12558 |  9048   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|  10 |          PARTITION RANGE SINGLE    |               |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|* 11 |           TABLE ACCESS STORAGE FULL| TRANSACTIONS  |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|  12 |          PARTITION RANGE ALL       |               | 16278 |   429K|  7953   (1)| 00:01:36 |     1 |1048575|</p>
<p class="x_MsoNormal">|  13 |           TABLE ACCESS STORAGE FULL| TRANSREVERSED | 16278 |   429K|  7953   (1)| 00:01:36 |     1 |1048575|</p>
<p class="x_MsoNormal">|  14 |    PARTITION RANGE ALL             |               | 16278 |  1907K|  7953   (1)| 00:01:36 |     1 |1048575|</p>
<p class="x_MsoNormal">|  15 |     TABLE ACCESS STORAGE FULL      | TRANSREVERSED | 16278 |  1907K|  7953   (1)| 00:01:36 |     1 |1048575|</p>

 

After trying a couple of silly scenarios for tuning it like moving the Table to the flash cache permanently with FLASH_CACHE=KEEP option and trying to ask the developer to reduce the time span for which it searches the first table via field UTIMESTAMP (developers will never admit their code is crappy). Then we went back to the hammer and nails approach. “Why not create an index on the field instead of trying to put a nail through with a Jedi lightsaber !!”

So after a few hours of excersizing our grey cells we created a non-unique b*tree index on the field LOGREFID and rebuild the table stats. The performance of the query immediately saw improvement from consuming a helluva lot user I/O to almost nothing.  In case of heavy reads a missing index is almost always the silver bullet.

Execution Plan Post-Index Ethos

<p class="x_MsoNormal">----------------------------------------------------------------------------------------------------------------------</p>
<p class="x_MsoNormal">| Id  | Operation                            | Name          | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |</p>
<p class="x_MsoNormal">----------------------------------------------------------------------------------------------------------------------</p>
<p class="x_MsoNormal">|   0 | SELECT STATEMENT                     |               |    39 | 84786 |  1174   (1)| 00:00:15 |       |       |</p>
<p class="x_MsoNormal">|   1 |  SORT ORDER BY                       |               |    39 | 84786 |  1174   (1)| 00:00:15 |       |       |</p>
<p class="x_MsoNormal">|   2 |   NESTED LOOPS OUTER                 |               |    39 | 84786 |  1173   (1)| 00:00:15 |       |       |</p>
<p class="x_MsoNormal">|*  3 |    HASH JOIN RIGHT OUTER             |               |    39 | 80106 |  1127   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|   4 |     TABLE ACCESS STORAGE FULL        | ESBRETURNCODE |    76 |  2964 |     4   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">|*  5 |     VIEW                             |               |    39 | 78585 |  1123   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|   6 |      COUNT                           |               |       |       |            |          |       |       |</p>
<p class="x_MsoNormal">|   7 |       VIEW                           |               |    39 | 78078 |  1123   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|   8 |        SORT ORDER BY                 |               |    39 | 12558 |  1123   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|*  9 |         HASH JOIN OUTER              |               |    39 | 12558 |  1122   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|  10 |          PARTITION RANGE SINGLE      |               |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|* 11 |           TABLE ACCESS STORAGE FULL  | TRANSACTIONS  |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|  12 |          INDEX STORAGE FAST FULL SCAN| LOGREFID_IDX  | 16278 |   429K|    27   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">|  13 |    TABLE ACCESS BY GLOBAL INDEX ROWID| TRANSREVERSED |     1 |   120 |     2   (0)| 00:00:01 | ROWID | ROWID |</p>
<p class="x_MsoNormal">|* 14 |     INDEX RANGE SCAN                 | LOGREFID_IDX  |     1 |       |     1   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">

Lets look at more graphical representation  from OEM ;  A picture speaks louder than a thousand words.

image001-2

/* The left-side mountain spikes are pre-index ethos and the right side hills are post. Self explanatory */

Heavy reads with high cost screams Index missing. Now this does not mean you go and start creating indexes in every SQL query you see a bit of reads. The Crux of the matter is that before you go ahead and blame your storage admin or disk subsystem or get new hardware. Doing a bit of the basics like looking at AWR reports, generating plans and applying good ol’ DBA intuition, it can do more good than throwing hardware at it. Hardware will never guarantee solution to a software problem.

P.S: We had the above issue on a densely packed Exadata X4-2 Quarter rack sharing 20+ instances. Disk I/O is the number one performance killer on a consolidated platform. Waiting on Disk I/O will always lead to higher CPU consumption.

 

 

Advertisements

Rebuild All Indexes on a Schema in Oracle

spool index_rebuild.sql

select ‘alter index ‘||owner||’.’||index_name ||’ rebuild online nologging;’
from dba_indexes
where owner=upper(‘SAMPLE’);
spool off

How to drop all indexes in a SCHEMA : Oracle 10g

How to drop all indexes in a SCHEMA

–Login as the schema user whose indexes you want to drop

SQL>set pages 10000
SQL>set wrap off
SQL>set heading off
SQL>spool /u2/scripts/idxswx.txt

SQL> select ‘ drop index ‘ || index_name ||’;’ from user_indexes;
drop index IND1;
drop index IND2;
drop index IND3;

SQL> spool off;

— This will spool the file to the set location and now run the Script as the schema user
SQL> conn scott/tiger

SQL>@indx.sql

–in the particular schema in which you want to drop indexes.

A similar script can be used for synonyms especially when you import a schema into a different name than when it was exported.

Myths about Index Rebuilding

Index Re-building is a debatable concept amongst DBA’s. Experts like Ask Tom think  rebuilding indexes doesn’t  create any considerable performance improvement.

http://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:2913600659112
Testimony to this is a small exercise I did in my production environment. We have a table with over 10 million rows. A full table scan on this takes approximately 15 minutues. the table has 6 b*tress indexes. I rebuild all the indexes online, the result was same. Full table scan took a few seconds less. The result was neglible. For performance tuning you have to carefully consider all symptoms and not blinly rebuild all indexes. The Cost Based Optimizer(CBO) in Oracle takes care of optimizing your queries. Below are the general myths associated with index rebuilding.

1. The vast majority of indexes do not require rebuilding
2. Oracle B-tree indexes can become “unbalanced” and need to
be rebuilt is a myth
3. deleted space in an index is “deadwood” and over time
requires it to be rebuilt is a myth
4. If an index has a height of “X” levels, it’s inefficient
and must be rebuilt is a myth
5. If an index has a poor CF, rebuilding it is a myth
6. To improve performance indexes must be rebuilt on a regular basis is a myth