Oracle DBA – Tips and Techniques

Knowledge is Power – so share it ….

11g Flashback Data Archive – Part One

Posted by Gavin Soorma on November 6, 2009

Businesses are increasingly realizing the immense value that historical data can provide to help them understand market trends and customer behaviour as well. Further, many regulatory and compliance policies
like SOX and BASEL-2 mandate retention of historical data.

Until now, this unfortunately involved application rewrites,administration overheads or costly third-party software solutions. Total Recall or the new Oracle 11g Flashback Data Archive feature has greatly enhanced the flashback technology which was introduced in Oracle 9i to make it far more than just a tool to recover from logical corruptions and human error. It is now an out-of-the box Information Life Cycle Management tool and provides ease of management of historical information and long term secure data tracking with minimal performance overheads.

11g Flashback Data Archive provides the automated ability to track and store all transactional changes to a table over its lifetime without having to build this intelligence into your application. Prior to 11g, Flashback technology to a large part was based on the availability of undo data or flashback logs and both the undo data as well as flashback logs are subject to recycling when out of space pressure exists. The UNDO tablespace in Oracle was primarily meant for transaction consistency and not data archival.

Flashback Data Archive or Total Recall Features

  • Easy to configure – apply to all tables, one or a group of tables with simple “enable archive” command
  • Secure – complete protection from accidental or malicious updates and deletes
  • Efficiency of performance and storage – capture process is asynchronous background process and data in history tables is partitioned as well as compressed automatically
  • Easy to access historical data using standard SQL “AS OF” constructs
  • Automated data management – historical data is automatically purged without any human intervention
  • Retention policies customised to suit business needs

  • Flashback Data Archive uses a background process fbda to capture data asynchronously. It runs every 5 minutes which is the default as well as at more frequent intervals depending on system activity. The Primary source for historical data is the Undo data, but this undo data is not reused until the historical data is written first.

    Getting Started

  • System Privilege – FLASHBACK ARCHIVE ADMINISTER to create and administer a flashback data archive
  • Connect as SYSDBA
  • Object Privilege – FLASHBACK ARCHIVE privilege on the specific flashback data archive to enable historical data tracking
  • Quota on the tablespace where the flashback data archive has been created
  • Creating a Flashback Data Archive (lets call it FBDA)

  • Create a new tablespace or use existing tablespace – tablespace needs to be ASSM
  • Specify the FBDA as the default (optional)
  • Assign a quota for the FBDA (optional)
  • Assign a retention period for the FBDA
  • Retention period integer denoting days,months or years
  • Enable flashback archive for a specific table via the CREATE TABLE or ALTER TABLE clause. By default it is turned off.
  • SQL> CREATE TABLESPACE his_data_1
      2  DATAFILE '+data' SIZE 500M;
    
    Tablespace created.
    
    SQL> CREATE FLASHBACK ARCHIVE DEFAULT fba1
      2  TABLESPACE his_data_1
      3  RETENTION 2 DAY;
    
    Flashback archive created.
    
    SQL> SELECT FLASHBACK_ARCHIVE_NAME, to_char(CREATE_TIME,'dd-mon-yyyy') Created,
      2  RETENTION_IN_DAYS,STATUS FROM DBA_FLASHBACK_ARCHIVE;
    
    FLASHBACK_ARCHIVE_NA CREATED     RETENTION_IN_DAYS STATUS
    -------------------- ----------- ----------------- -------
    FBA1                 02-nov-2009                 2 DEFAULT
    
    SQL> GRANT FLASHBACK ARCHIVE ON fba1 TO scott;
    
    Grant succeeded.
    
    SQL> ALTER TABLE mysales FLASHBACK ARCHIVE fba1;
    
    Table altered.
    
    SQL> CREATE TABLE
      2  EMPSAL_HIS
      3  (EMPNO number,
      4  ENAME VARCHAR2(10),
      5  SAL NUMBER,
      6  FLASHBACK ARCHIVE;
    
    Table created.
    
    SQL> select TABLE_NAME,FLASHBACK_ARCHIVE_NAME,ARCHIVE_TABLE_NAME,STATUS
      2  from USER_FLASHBACK_ARCHIVE_tables;
    
    TABLE_NAME            FLASHBACK_ARCHIVE_NA  ARCHIVE_TABLE_NAME               STATUS
    -----------	  --------------------  --------------------	      ------------
    MYSALES               FBA1                 SYS_FBA_HIST_77429               ENABLED
    EMPSAL_HIS            FBA1                 SYS_FBA_HIST_77419               ENABLED
    

    Note: for every table where Flashback Archive is enabled, corresponding internal history tables are created in the flashback archive tablespace as shown below. These tables are automatically partitioned and compressed as well

    SQL> select object_id from dba_objects where object_name=‘DEPT';
    
     OBJECT_ID
    ----------
         73201
    
    SQL> select table_name,tablespace_name from user_tables;
    
    TABLE_NAME                     TABLESPACE_NAME
    ------------------------------ ------------------------------
    DEPT                           USERS
    SYS_FBA_DDL_COLMAP_73201       HIS_DATA_1
    SYS_FBA_TCRV_73201             HIS_DATA_1
    SYS_FBA_HIST_73201
    
    SQL> desc SYS_FBA_HIST_73201
     Name                                      Null?    Type
     ----------------------------------------- -------- ----------------------------
     RID                                                VARCHAR2(4000)
     STARTSCN                                           NUMBER
     ENDSCN                                             NUMBER
     XID                                                RAW(8)
     OPERATION                                          VARCHAR2(1)
    EMPNO                                              NUMBER(4)
     ENAME                                              VARCHAR2(10)
     MGR                                                NUMBER(10)
    DEPTNO                                             NUMBER(2)
     JOB                                                VARCHAR2(20)
    
    SQL> SELECT TABLE_NAME,PARTITION_NAME,COMPRESSION from USER_TAB_PARTITIONS;
    
    TABLE_NAME                     PARTITION_NAME                 COMPRESS
    ------------------------------ ------------------------------ --------
    SYS_FBA_HIST_77429            HIGH_PART                      ENABLED
    SYS_FBA_HIST_77419            HIGH_PART                      ENABLED
    

    Posted in Administration, Oracle 11g | Tagged: , , , | Leave a Comment »

    Oracle Restart – new in 11g R2

    Posted by Gavin Soorma on October 29, 2009

    Similar to the clusterware processes in a RAC environment, in 11g R2 even for a standalone instance, functionality is now available to automatically restart components like the database, listener, ASM diskgroup, service etc in the event of their failure.

    This feature is called Oracle Restart and it runs out of the Grid Infrastructure home which is separate from the database home.

    Oracle Restart is managed by CRSCTL utility and it uses Oracle High Availability Services to start and stop the components managed by Oracle Restart.

    Using Oracle Restart, we can stop or start all the components running out of a single Oracle Home with a single srvctl command as well as Oracle Restart will automatically start all components on machine reboot in the proper order taking into account dependencies like ASM instance and the managed database instances. So no more do we have to use the dbstart and dbstop scripts to manage this.

    Oracle Restart can be extended to Data Guard configurations as well by integrating with Data Guard Broker to manage the start and stop of database services following a data guard role transition.

    Let us examine some of the srvctl commands which can be executed in a single instance environment as well as see the use of the crsctl command – normally previously only used for RAC environments.

    -bash-3.2$ crs_stat -t
    Name           Type           Target    State     Host
    ------------------------------------------------------------
    ora.DATA1.dg   ora....up.type ONLINE    ONLINE    redhat64
    ora....ER.lsnr ora....er.type ONLINE    ONLINE    redhat64
    ora....WARE.dg ora....up.type ONLINE    ONLINE    redhat64
    ora.asm        ora.asm.type   ONLINE    ONLINE    redhat64
    ora.cssd       ora.cssd.type  ONLINE    ONLINE    redhat64
    ora.diskmon    ora....on.type ONLINE    ONLINE    redhat64
    ora.eons       ora.eons.type  ONLINE    ONLINE    redhat64
    ora.ons        ora.ons.type   ONLINE    ONLINE    redhat64
    
    bash-3.2$ ./crsctl check has
    CRS-4638: Oracle High Availability Services is online
    
    -bash-3.2$ ./crsctl check css
    CRS-4529: Cluster Synchronization Services is online
    
    -bash-3.2$ ./srvctl start listener
    
    -bash-3.2$ ps -ef |grep tns
    oracle   14899     1  0 14:08 ?        00:00:00 /u02/app/oracle/product/11.2.0/grid/bin/tnslsnr LISTENER -inherit
    
    -bash-3.2$ ./srvctl status listener
    Listener LISTENER is enabled
    Listener LISTENER is running on node(s): redhat64
    
    -bash-3.2$ ./srvctl status asm
    ASM is running on redhat64
    
    -bash-3.2$ ./srvctl status diskgroup -g DATA1
    Disk Group DATA1 is running on redhat64
    
    -bash-3.2$ ./srvctl config asm
    ASM home: /u02/app/oracle/product/11.2.0/grid
    ASM listener: LISTENER
    Spfile: +DATA1/asm/asmparameterfile/registry.253.700932479
    ASM diskgroup discovery string: /dev/raw/raw*
    

    In case a process dies unexpectedly, Oracle Restart will automatically start the process. In the example below, we kill the PMON process of the ora11gr2 instance and find that in less than 5 seconds the instance has been restarted by the Oracle High Availablity Service running in the background constantly monitoring the health of the managed components.

    -bash-3.2$ ps -ef |grep pmon
    oracle   12710     1  0 Oct22 ?        00:00:01 asm_pmon_+ASM
    oracle   15752     1  0 14:14 ?        00:00:00 ora_pmon_ora11gr2
    oracle   15975 18271  0 14:15 pts/2    00:00:00 grep pmon
    
    -bash-3.2$ kill -9  15752
    
    -bash-3.2$ ps -ef |grep pmon
    oracle   12710     1  0 Oct22 ?        00:00:01 asm_pmon_+ASM
    oracle   16679     1  0 14:22 ?        00:00:00 ora_pmon_ora11gr2
    oracle   16775 18271  0 14:22 pts/2    00:00:00 grep pmon
    

    In this example we stop all the components running out of a particular home by executing the srvctl stop home command. After stopping the database home, we find that only the ASM instance is running as it is managed by a seperate home which is the Grid Infrastructure home. After the database home is started, the database instance ora11gr2 also is automatically started.

    [oracle@redhat346 ~]$ srvctl stop home -o /u01/app/oracle/product/11.2.0/dbhome_1 -s /tmp/state
    
    [oracle@redhat346 ~]$ ps -ef |grep pmon
    oracle   19841     1  0 Sep18 ?        00:01:24 asm_pmon_+ASM
    oracle   31843 27855  0 15:19 pts/1    00:00:00 grep pmon
    
    [oracle@redhat346 ~]$ srvctl start home -o /u01/app/oracle/product/11.2.0/dbhome_1 -s /tmp/state
    
    [oracle@redhat346 ~]$ ps -ef |grep pmon
    oracle   19841     1  0 Sep18 ?        00:01:24 asm_pmon_+ASM
    oracle   26690     1  0 Oct08 ?        00:00:41 ora_pmon_emrep
    oracle   32054     1  0 15:20 ?        00:00:00 ora_pmon_testdb
    oracle   32344     1  0 15:20 ?        00:00:00 ora_pmon_ora11gr2
    oracle   32620 27855  0 15:22 pts/1    00:00:00 grep pmon
    

    Posted in Administration, High Availability, Oracle 11g release 2 | Tagged: , , , | Leave a Comment »

    11g Release 2 Grid Infrastructure Installation

    Posted by Gavin Soorma on October 23, 2009

    Let us have a look at the installation screenshots of 11g Release 2 on a Red Hat Linux platform. Currently the 11g R2 software has only been released for the Linux platform with release dates for other platforms like Solaris and AIX set for sometime end October to mid November.

    There are quite a few new installation options available as can be seen from the screenshots below. What we are installing below is “Grid Infrastructure for a stand alone server”.

    In 11g R2, ASM is now part of what is called the Grid Infrastructure (nothing to do with Grid Control). It is no longer an option available in dbca and there is a command line option asmca which is launched from the Grid Infrastructure home. The ASM instance will be running from the Grid Home and not the database Oracle Home.

    Further, the Grid Infrastructure can be installed for a stand alone server or for a cluster deployment. The ASM and Clusterware are instaled in the same home directory and it should be noted that now in 11g R2, the clusterware files like the OCR and the Voting Disk can be located on ASM storage and raw devices is no longer supported.

    Also, now there is separation between SYSASM and SYSDBA and if you want to connect to the ASM instance you need to do it as SYSASM. It is recommended to also create a separate group in addition to the DBA group specific for ASM administration and make SYSASM grantees members of this group. In one of the screenshots below we will see an alert being displayed when we have selected the dba group for the ASM installation.

    Posted in Administration, Oracle 11g, Oracle 11g release 2 | Tagged: , , | Leave a Comment »

    11g Release 2 ACFS (Automatic Storage Management Cluster File System)

    Posted by Gavin Soorma on October 16, 2009

    The following are some of the new ASM related features introduced in 11g R2:

  • In Oracle 11g Release 2, ASM and the Oracle Clusterware are now installed in a common home called the Grid Infrastructure Home.
  • The clusterware files like the Voting Disk and the Cluster Registry can no longer be installed on raw devices. They can now be installed on ASM Disk Groups
  • Need to connect as SYSASM to perform any administrative operations on the ASM instance or we will get an ORA-01031 error as shown below
  • [oracle@redhat346 stage]$ sqlplus sys/xxx as sysdba
    
    SQL*Plus: Release 11.2.0.1.0 Production on Fri Oct 16 09:10:55 2009
    
    Copyright (c) 1982, 2009, Oracle.  All rights reserved.
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
    With the Automatic Storage Management option
    
    SQL> shutdown immediate;
    ORA-01031: insufficient privileges
    
  • ACFS or Automatic Storage Management Cluster File System is an extension of the ASM technology to support storage of files like Oracle binaries and other application files related to text,video, audio etc
  • The example below will show how we can configure and mount a ACFS on Red HatLinux with 11g R2.

    Note: ASM operations like creating and adding disk groups etc which were earlier performed via DBCA will now be performed via the GUI ASM Configuration Assistant which needs to be launched via the ‘asmca‘ command from the Grid Infrastructure Home.

    We first create a disk group called ASMCFS and then create a volume called ASMCFS_VOL1 of 25 GB using the disk group that we just created.

    We will then use a mount point /u03 to mount this file system. A configuration script is created which includes the commands to be run as root to mount this file system. After the script is run we will see the ACFS file system mounted on /u03.

    /bin/mount -t acfs /dev/asm/asmcfs_vol1-44 /u03
    
    chown oracle:dba /u03
    
    chmod 775 /u03
    
    [oracle@redhat346 stage]$ df -k
    
    Filesystem           1K-blocks      Used Available Use% Mounted on
    .....
    ......
    /dev/asm/asmcfs_vol1-44
                          26214400     89112  26125288   1% /u03
    

    asmca_11gr2

    asmca_2

    asmca_3

    asmca_4

    asmca_5

    asmca_6

    asmca_7

    asmca_8

    asmca_9

    Posted in Administration, Oracle 11g, Oracle 11g release 2 | Tagged: , , , | Leave a Comment »

    Cross Platform Transportable Tablespace using ASM with Oracle XML DB

    Posted by Gavin Soorma on October 14, 2009

    In one of my earlier posts How To Transport a Tablespace from 10g ASM to 11g ASM on AIX we discussed how to transport a tablespace stored on ASM storage from Windows to AIX using the DBMS_FILE_TRANSFER package.

    We will now look at how the same thing can be done using FTP with Oracle XML DB Repository and its virtual folders.

    All we need to do is to run the following script and provide the port numbers which will be used for the FTP and HTTP protocols for accessing the repository.

    @?/rdbms/admin/catxdbdbca 8080 7787

    In this case we have assigned the port 8080 for the FTP protocol and the port 7787 is for HTTP access.

    The ASM related directories on the Windows Oracle database are accessed via the sys/asm virtual folder and then we use FTP to copy the ASM files over the network and then use RMAN to convert it to AIX 64 bit format

    The example below shows us how we are connecting to the Windows server via port 8080 using the FTP protocol. Note we are then connecting with a database user – SYSTEM in this case.

    testdb:/u01/oracle > ftp cbdorca261 8080
    Connected to cbdorca261.bankwest.com.
    220- cbdorca261
    Unauthorised use of this FTP server is prohibited and may be subject to civil and criminal prosecution.
    220 cbdorca261 FTP Server (Oracle XML DB/Oracle Database) ready.
    Name (cbdorca261:oracle): system
    331 pass required for SYSTEM
    Password:
    230 SYSTEM logged in
    ftp> i
    200  Type set to I.
    ftp> cd sys/asm
    250 CWD Command successful
    ftp> cd DATA/genua1/datafile
    250 CWD Command successful
    ftp> get test_tts.376.699110247
    200 PORT Command successful
    150 BIN Data Connection
    226 BIN Transfer Complete
    104865792 bytes received in 2.25 seconds (4.551e+04 Kbytes/s)
    local: test_tts.376.699110247 remote: test_tts.376.699110247
    ftp> quit
    221 QUIT Goodbye.
    
     testdb:/u01/oracle > rman target /
    
    Recovery Manager: Release 11.1.0.7.0 - Production on Thu Oct 1 13:50:02 2009
    
    Copyright (c) 1982, 2007, Oracle.  All rights reserved.
    
    connected to target database: TESTDB (DBID=2482257163)
    
    RMAN>  convert datafile2>
     '/u01/oracle/test_tts.376.699110247'
    3> FORMAT '+DATA'
    4> from platform 'Microsoft Windows IA (32-bit)';
    
    Starting conversion at target at 01-OCT-09
    using target database control file instead of recovery catalog
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: SID=140 device type=DISK
    channel ORA_DISK_1: starting datafile conversion
    input file name=/u01/oracle/test_tts.376.699110247
    converted datafile=+DATA/testdb/datafile/test_tts.267.699112231
    channel ORA_DISK_1: datafile conversion complete, elapsed time: 00:00:03
    Finished conversion at target at 01-OCT-09
    
    imp file=tts.dmp transport_tablespace=y tablespaces=TEST_TTS datafiles=+DATA/testdb/datafile/test_tts.267.699112231
    
    Import: Release 11.1.0.7.0 - Production on Thu Oct 1 13:57:33 2009
    
    Copyright (c) 1982, 2007, Oracle.  All rights reserved.
    
    Username: sys as sysdba
    Password:
    
    Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production
    With the Partitioning and Real Application Testing options
    
    Export file created by EXPORT:V10.02.01 via conventional path
    About to import transportable tablespace(s) metadata...
    import done in US7ASCII character set and AL16UTF16 NCHAR character set
    import server uses WE8MSWIN1252 character set (possible charset conversion)
    export client uses WE8MSWIN1252 character set (possible charset conversion)
    . importing SYS's objects into SYS
    . importing SYS's objects into SYS
    . importing SYSTEM's objects into SYSTEM
    . . importing table                    "MYOBJECTS"
    . importing SYS's objects into SYS
    Import terminated successfully without warnings.
    
    SQL> conn system/oracle
    Connected.
    
    SQL> select file_name from dba_data_files
      2  where tablespace_name='TEST_TTS';
    
    FILE_NAME
    --------------------------------------------------------------------------------
    +DATA/testdb/datafile/test_tts.267.699112231
    
    SQL> select tablespace_name from user_tables
      2  where table_name='MYOBJECTS';
    
    TABLESPACE_NAME
    ------------------------------
    TEST_TTS
    
    SQL> select count (*) from myobjects;
    
      COUNT(*)
    ----------
         56454
    

    Posted in Administration | Tagged: | Leave a Comment »

    ASMCMD examples

    Posted by Gavin Soorma on September 30, 2009

    asmcmd is a command line tool that we can use to manage Oracle ASM instances, ASM disk groups, files and dierctories, templates very much as in the same way we would do while working with say a UNIX file system.

    ASMCMD can be launched in interactive or non-interactive modes and we need to first ensure that our environment points to the “Grid Infrastructure Home” as in 11g R2, ASM is not part of the standard database software installation. Also chec that the ORACLE_SID points to the ASM instance which should be running.

    Let us have a quick look at some of the useful command options which are now available with 11g Release 2.

    lsct: Lists information about current Oracle ASM clients from the V$ASM_CLIENT view.

    ASMCMD> lsct
    DB_Name  Status     Software_Version  Compatible_version  Instance_Name  Disk_Group
    +ASM     CONNECTED        11.2.0.1.0          11.2.0.1.0  +ASM           DATA
    testdb   CONNECTED        11.2.0.1.0          11.2.0.0.0  testdb         DATA
    

    cp: We can copy files from ASM to OS as wel as from OS to ASM disk groups

    ASMCMD> cp EXAMPLE.265.697907183 /tmp/example01.bak
    copying +DATA/TESTDB/DATAFILE/EXAMPLE.265.697907183 -> /tmp/example01.bak
    
    ASMCMD> cp /tmp/example01.bak +DATA/TESTDB/DATAFILE/example01.bak
    copying /tmp/example01.bak -> +DATA/TESTDB/DATAFILE/example01.bak
    
    ASMCMD> ls
    EXAMPLE.265.697907183
    SYSAUX.257.697907095
    SYSTEM.256.697907095
    UNDOTBS1.258.697907095
    USERS.259.697907095
    example01.bak
    

    du: Total space in MB used by files – particular directory can also be specified

    ASMCMD> pwd
    +DATA/TESTDB/DATAFILE
    
    ASMCMD> du
    Used_MB      Mirror_used_MB
       1574                1574
    
    ASMCMD> du +DATA/TESTDB/ONLINELOG
    Used_MB      Mirror_used_MB
        153                 153
    

    find: we can use the wildcard or can specify a particular file type by using the ‘–type’ clause

    ASMCMD> find --type ONLINELOG +DATA *
    +DATA/TESTDB/ONLINELOG/group_1.261.697907177
    +DATA/TESTDB/ONLINELOG/group_2.262.697907179
    +DATA/TESTDB/ONLINELOG/group_3.263.697907179
    
    ASMCMD> find +DATA example*
    +DATA/ASM/DATAFILE/example01.bak.267.698929915
    +DATA/TESTDB/DATAFILE/EXAMPLE.265.697907183
    +DATA/TESTDB/DATAFILE/example01.bak
    

    ls: list the contents of an ASM Disk Group directory as well as attributes of files located in the directory

    ASMCMD> ls -s
    Block_Size  Blocks      Bytes      Space  Name
          8192   12801  104865792  106954752  EXAMPLE.265.697907183
          8192   89601  734011392  736100352  SYSAUX.257.697907095
          8192   89601  734011392  736100352  SYSTEM.256.697907095
          8192    7681   62922752   65011712  UNDOTBS1.258.697907095
          8192     641    5251072    6291456  USERS.259.697907095
                                              example01.bak => +DATA/ASM/DATAFILE/example01.bak.267.698929915
    
    ASMCMD> ls -l
    Type      Redund  Striped  Time             Sys  Name
    DATAFILE  UNPROT  COARSE   SEP 30 11:00:00  Y    EXAMPLE.265.697907183
    DATAFILE  UNPROT  COARSE   SEP 28 23:00:00  Y    SYSAUX.257.697907095
    DATAFILE  UNPROT  COARSE   SEP 29 22:00:00  Y    SYSTEM.256.697907095
    DATAFILE  UNPROT  COARSE   SEP 18 22:00:00  Y    UNDOTBS1.258.697907095
    DATAFILE  UNPROT  COARSE   SEP 18 22:00:00  Y    USERS.259.697907095
                                                N    example01.bak => +DATA/ASM/DATAFILE/example01.bak.267.698929915
    

    iostat: Uses the V$ASM_DISK_IOSTAT view to display I/O statistics of disks in mounted ASM disk groups

    ASMCMD> iostat -G DATA
    Group_Name  Dsk_Name   Reads        Writes
    DATA        DATA_0000  25448671744  19818926592
    
    ASMCMD> iostat -t
    Group_Name  Dsk_Name   Reads        Writes       Read_Time    Write_Time
    DATA        DATA_0000  25450195456  19819686912  6491.434444  8042.604156
    

    lsdg: Uses V$ASM_DISKGROUP_STAT view to list information about a particular disk group

    ASMCMD> lsdg DATA
    State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB    Name
    MOUNTED  EXTERN  N         512   4096  1048576     51200    49375                0           49375    DATA/
    
    ASMCMD> lsdsk -t -G DATA
    Create_Date  Mount_Date  Repair_Timer  Path
    18-SEP-09    18-SEP-09   0             /dev/raw/raw1
    

    lsattr: List attributes of a disk group

    ASMCMD> lsattr -l -G DATA
    Name                     Value
    access_control.enabled   FALSE
    access_control.umask     066
    au_size                  1048576
    cell.smart_scan_capable  FALSE
    compatible.asm           11.2.0.0.0
    compatible.rdbms         10.1.0.0.0
    disk_repair_time         3.6h
    sector_size              512
    

    Use the setattr command to change an attribute

    ASMCMD> setattr -G data compatible.rdbms 11.2.0.0.0
    ASMCMD> lsattr -l -G DATA
    Name                     Value
    access_control.enabled   FALSE
    access_control.umask     066
    au_size                  1048576
    cell.smart_scan_capable  FALSE
    compatible.asm           11.2.0.0.0
    compatible.rdbms         11.2.0.0.0
    disk_repair_time         3.6h
    sector_size              512
    

    Posted in Administration, Oracle 11g | Tagged: | Leave a Comment »

    10g EM Grid Control Notification Rules and Corrective Action Jobs

    Posted by Gavin Soorma on September 29, 2009

    In 10g Enterprise Manager Grid Control, we can create corrective action or ‘fixit’ jobs which can automatically run if say a metric threshold is crossed or a particular event occurs. For example, we may have situations where we monitor the disk space used by the archive log files and as soon as the disk space free or available comes below a threshold, we would like another action to automatically occur which in this case could be to backup the archive log files to tape and then delete them to free up the disk space.

    Read more about this feature in this worked example …….

    Posted in EM Grid Control | Tagged: , , | 3 Comments »

    11g RMAN Substitution Variables

    Posted by Gavin Soorma on September 22, 2009

    We can now use SQL*PLUS type substitution variables in RMAN like &1, &2 and so on and the same can be linked to shell scripts as shown below with the new clause USING where we are passing the Incremental Level and the backup tag to the actual RMAN command.

    Also note that in 11g we can run RMAN scripts using the ‘@’ like a SQL script

    testbkp.rcv

    connect target sys/oracle
    run{
    backup incremental level &1
    database
    tag &2;
    }
    exit;

    testbkp.sh

    #! /bin/ksh
    export level=$1
    export tag=$2_`date +%d%b%y`
    rman @testbkp.rcv using $level $tag

    Note the actual RMAN command that is run …….. it adds a date along with the tag name (parameter 2) and runs a level 2 backup as the first parameter was ‘2′

    apex:/u01/oracle> ./testbkp.sh 2 dly_inc_bkp

    Recovery Manager: Release 11.1.0.6.0 – Production on Tue Sep 22 11:28:05 2009

    Copyright (c) 1982, 2007, Oracle. All rights reserved.

    RMAN> connect target *
    2> run{
    3> backup incremental level 2
    4> database
    5> tag dly_inc_bkp_22Sep09;
    6> }
    7> exit;
    connected to target database: APEX (DBID=1312143933)

    Starting backup at 22-SEP-09
    using target database control file instead of recovery catalog
    allocated channel: ORA_SBT_TAPE_1
    channel ORA_SBT_TAPE_1: SID=349 device type=SBT_TAPE
    channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.5.1.0
    channel ORA_SBT_TAPE_1: starting incremental level 2 datafile backup set
    channel ORA_SBT_TAPE_1: specifying datafile(s) in backup set
    input datafile file number=00003 name=/u02/oradata/apex/undotbs01.dbf
    input datafile file number=00002 name=/u02/oradata/apex/sysaux01.dbf
    input datafile file number=00001 name=/u02/oradata/apex/system01.dbf
    input datafile file number=00005 name=/u02/oradata/apex/monitor_data01.dbf
    input datafile file number=00007 name=/u02/oradata/apex/encrypt_data01.dbf
    input datafile file number=00008 name=/u02/oradata/apex/disc10_data01.dbf
    channel ORA_SBT_TAPE_1: starting piece 1 at 22-SEP-09
    channel ORA_SBT_TAPE_1: finished piece 1 at 22-SEP-09
    piece handle=a5kpsim9_1_1 tag=DLY_INC_BKP_22SEP09 comment=API Version 2.0,MMS Version 5.5.1.0
    channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 00:00:25
    Finished backup at 22-SEP-09

    Starting Control File and SPFILE Autobackup at 22-SEP-09
    piece handle=c-1312143933-20090922-05 comment=API Version 2.0,MMS Version 5.5.1.0
    Finished Control File and SPFILE Autobackup at 22-SEP-09

    Recovery Manager complete.

    Posted in Backup and Recovery, Oracle 11g | Tagged: , , | Leave a Comment »

    11g Release 2 Edition-based Redefinition

    Posted by Gavin Soorma on September 21, 2009

    11g Release 2 Edition-based redefinition enables you to upgrade or change the database component of an application while it is in use, thereby minimising or eliminating down time which was earlier required when an object like a procedure required to be modified, but we could not do it without any outage as the application was accessing the procedure or package which needed to be modified.

    An edition is like a workspace or private environment where database objects are redefined. When we are satisfied with the change that we have made, those changes in the edition can be then rolled out to all the application users.

    Let us see an example of this where we create a procedure, make some changes to the procedure in a new edition and then make those changes visible to other database users.

    SQL>  GRANT CREATE ANY EDITION, DROP ANY EDITION to sh;
    
    Grant succeeded.
    
    SQL> conn sh/sh
    Connected.
    
    SQL> CREATE OR REPLACE PROCEDURE hello IS
      BEGIN
        DBMS_OUTPUT.PUT_LINE('Hello, edition 1.');
      END hello;
    /
      2    3    4    5
    Procedure created.
    
    SQL> set serverout on
    
    SQL> BEGIN hello(); END;
    /
      2  Hello, edition 1.
    
    PL/SQL procedure successfully completed.
    
    SQL> CREATE EDITION e2;
    
    Edition created.
    
    SQL> ALTER SESSION SET EDITION = e2;
    
    Session altered.
    
    SQL> CREATE OR REPLACE PROCEDURE hello IS
      BEGIN
        DBMS_OUTPUT.PUT_LINE('Hello, edition 2.');
      END hello;
    /
      2    3    4    5
    Procedure created.
    
    SQL> BEGIN hello(); END;
      2  /
    Hello, edition 2.
    
    PL/SQL procedure successfully completed.
    

    Note, now if we change the edition to the default edition ‘ORA$BASE’, we will now see the original procedure and not the updated one

    SQL> ALTER SESSION SET EDITION = ora$base;
    
    Session altered.
    
    SQL>  BEGIN hello(); END;
      2  /
    Hello, edition 1.
    

    Let us now see how this can be rolled out to database user HR

    SQL> grant execute on hello to hr;
    
    Grant succeeded.
    
    SQL> GRANT USE ON EDITION e2 to public;
    
    Grant succeeded.
    

    We now make the default edition of the database, E2 so that all changes in the new edition E2 will be visible to all database users

    SQL> conn sys as sysdba
    Enter password:
    Connected.
    
    SQL> ALTER DATABASE DEFAULT EDITION=e2;
    
    Database altered.
    
    SQL> conn hr/hr
    Connected.
    
    SQL> set serverout on
    
    SQL>  exec sh.hello;
    Hello, edition 2.
    

    Posted in Administration, Oracle 11g, Oracle 11g release 2 | Tagged: , , , | Leave a Comment »

    11g Pending and Published Statistics

    Posted by Gavin Soorma on September 18, 2009

    Prior to Oracle 11g, the default behaviour was to make statistics available for the optimizer to use as soon as they were gathered.

    While statistics are required to enable the optimizer to generate optimal execution plans, sometimes just by gathering fresh statistics tried and trusted execution plans can abruptly change and thereby adversely affect application performance.

    In Oracle 11g we can now ‘defer’ the publication of statistics until they have been tried and tested and once we have confirmed that the execution plans are correct and optimal. Statistics remain in the pending state until they are published and the parameter optimizer_use_pending_statistics (default value FALSE) which when set at the session level will enables us to test the pending statistics independently of other database sessions.

    Let us look at a test case using the SALES table in the SH schema.

    If we see the query below, it appears that the data for the column CHANNEL_ID is skewed where majority of the rows have the value 3 and a very small majority have the value 9.

    SQL> select channel_id,count(*) from sales
    2 group by channel_id order by 2;

    CHANNEL_ID COUNT(*)
    ———- ———-
    9 2074
    4 118416
    2 258025
    3 540328

    But both the queries below are performing a full table scan of the SALES table when ideally it should be performing an index scan when the value 9 is used in the WHERE clause considering it accounts for a very small proportion of the rows in the SALES table.

    SELECT
    S.cust_id
    ,S.prod_id
    ,SUM(S.amount_sold)
    FROM sh.sales S
    WHERE
    channel_id=9
    GROUP BY S.cust_id, S.prod_id
    ORDER BY S.cust_id, S.prod_id
    /

    SELECT
    S.cust_id
    ,S.prod_id
    ,SUM(S.amount_sold)
    FROM sh.sales S
    WHERE
    channel_id=3
    GROUP BY S.cust_id, S.prod_id
    ORDER BY S.cust_id, S.prod_id
    /

    As earlier mentioned, statistics for a table are published immediately by default, so we use the DBMS_STATS package to change this default behaviour.

    SQL> exec dbms_stats.set_table_prefs('SH','SALES','PUBLISH','FALSE');
    
    PL/SQL procedure successfully completed.
    
    SQL> select dbms_stats.get_prefs('PUBLISH', 'SH', 'SALES' ) FROM DUAL;
    
    DBMS_STATS.GET_PREFS('PUBLISH','SH','SALES')
    ----------------------------------------------------------------------------------------------
    FALSE
    

    Since we observed that the data in the table is skewed, we will gather histograms which may help the optimizer make more informed decisions when generating an execution plan.

    Note that since we have disabled the publishing of statistics for the table SALES, even though we have analyzed the table, the LAST_ANALYZED column shows that the table has not been recently analyzed.

    SQL> EXEC DBMS_STATS.GATHER_TABLE_STATS('SH','SALES',METHOD_OPT => 'FOR ALL COLUMNS SIZE AUTO');
    
    PL/SQL procedure successfully completed.
    
    SQL> SELECT LAST_ANALYZED FROM DBA_TABLES WHERE TABLE_NAME='SALES';
    
    LAST_ANAL
    ---------
    18-AUG-2009
    

    However, we we query the DBA_TAB_PENDING_STATS view, it shows that the SALES table which is partitioned has been analyzed today.

    SQL> SELECT TABLE_NAME,PARTITION_NAME ,LAST_ANALYZED  FROM DBA_TAB_PENDING_STATS;
    
    TABLE_NAME                     PARTITION_NAME                 LAST_ANAL
    ------------------------------ ------------------------------ ---------
    SALES                                                         17-SEP-09
    SALES                          SALES_Q3_1998                  17-SEP-09
    SALES                          SALES_Q3_1999                  17-SEP-09
    SALES                          SALES_Q3_2000                  17-SEP-09
    SALES                          SALES_Q3_2001                  17-SEP-09
    SALES                          SALES_Q3_2002                  17-SEP-09
    SALES                          SALES_Q3_2003                  17-SEP-09
    SALES                          SALES_Q4_1998                  17-SEP-09
    SALES                          SALES_Q4_1999                  17-SEP-09
    SALES                          SALES_Q4_2000                  17-SEP-09
    SALES                          SALES_Q4_2001                  17-SEP-09
    SALES                          SALES_Q4_2002                  17-SEP-09
    SALES                          SALES_Q4_2003                  17-SEP-09
    SALES                          SALES_1995                     17-SEP-09
    SALES                          SALES_1996                     17-SEP-09
    SALES                          SALES_H1_1997                  17-SEP-09
    SALES                          SALES_H2_1997                  17-SEP-09
    SALES                          SALES_Q1_1998                  17-SEP-09
    SALES                          SALES_Q1_1999                  17-SEP-09
    SALES                          SALES_Q1_2000                  17-SEP-09
    SALES                          SALES_Q1_2001                  17-SEP-09
    SALES                          SALES_Q1_2002                  17-SEP-09
    SALES                          SALES_Q1_2003                  17-SEP-09
    SALES                          SALES_Q2_1998                  17-SEP-09
    SALES                          SALES_Q2_1999                  17-SEP-09
    SALES                          SALES_Q2_2000                  17-SEP-09
    SALES                          SALES_Q2_2001                  17-SEP-09
    SALES                          SALES_Q2_2002                  17-SEP-09
    SALES                          SALES_Q2_2003                  17-SEP-09
    

    We would like to see if the query execution plan has changed or improved after the recent gathering of statistics. We can use the parameter optimizer_use_pending_statistics for this by setting the value to TRUE (default is FALSE)

    SQL> alter session set optimizer_use_pending_statistics=TRUE;

    Session altered.

    We now find that the optimizer is performing an index scan when the value 9 is used as a predicate value for CHANNEL_ID while it performs a full table scan when the value 3 is used which is a much more optimal plan than the original one before gathering fresh statistics.

    explain plan
    for
    SELECT
         S.cust_id
        ,S.prod_id
        ,SUM(S.amount_sold)
      FROM sh.sales S
     WHERE
    channel_id=3
     GROUP BY S.cust_id, S.prod_id
     ORDER BY S.cust_id, S.prod_id
    /  2    3    4    5    6    7    8    9   10   11   12
    
    Explained.
    
    SQL> select * from table(dbms_xplan.display);
    
    PLAN_TABLE_OUTPUT
    ----------------------------------------------------------------------------------------------
    Plan hash value: 3701591983
    
    ----------------------------------------------------------------------------------------------
    | Id  | Operation            | Name  | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
    ----------------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT     |       |  4992 | 79872 |  2649   (4)| 00:00:32 |       |       |
    |   1 |  SORT GROUP BY       |       |  4992 | 79872 |  2649   (4)| 00:00:32 |       |       |
    |   2 |   PARTITION RANGE ALL|       |   541K|  8461K|  2601   (2)| 00:00:32 |     1 |    28 |
    |*  3 |    TABLE ACCESS FULL | SALES |   541K|  8461K|  2601   (2)| 00:00:32 |     1 |    28 |
    ----------------------------------------------------------------------------------------------
    
    SQL> explain plan
    for
    SELECT
         S.cust_id
        ,S.prod_id
        ,SUM(S.amount_sold)
      FROM sh.sales S
     WHERE
    channel_id=9
     GROUP BY S.cust_id, S.prod_id
     ORDER BY S.cust_id, S.prod_id
    /
      2    3    4    5    6    7    8    9   10   11   12
    Explained.
    
    SQL> select * from table(dbms_xplan.display);
    
    PLAN_TABLE_OUTPUT
    --------------------------------------------------------------------------------------------------------
    Plan hash value: 3360497850
    
    --------------------------------------------------------------------------------------------------------
    -
    
    | Id  | Operation                           | Name              | Rows  | Bytes | Cost (%CPU)| Time
    |
    
    -------------------------------------------------------------------------------------------------------
    -
    
    |   0 | SELECT STATEMENT                    |                   |   913 | 14608 |   315   (1)| 00:00:04
    |
    
    |   1 |  SORT GROUP BY                      |                   |   913 | 14608 |   315   (1)| 00:00:04
    |
    
    |   2 |   PARTITION RANGE ALL               |                   |  1423 | 22768 |   314   (0)| 00:00:04
    |
    
    |   3 |    TABLE ACCESS BY LOCAL INDEX ROWID| SALES             |  1423 | 22768 |   314   (0)| 00:00:04
    |
    
    |   4 |     BITMAP CONVERSION TO ROWIDS     |                   |       |       |            |
    |
    
    |*  5 |      BITMAP INDEX SINGLE VALUE      | SALES_CHANNEL_BIX |       |       |            |
    |
    
    --------------------------------------------------------------------------------------------------------
    

    Since we have now confirmed that the plan is acceptable and the new statistics can be used by the optimizer, we PUBLISH the statistics now for the SALES table. Note that the LAST_ANALYZED column is also updated and the DBA_TAB_PENDING_STATS shows that there are no more pending statistics which need to be published.

    SQL> EXEC DBMS_STATS.PUBLISH_PENDING_STATS ('SH','SALES');
    
    PL/SQL procedure successfully completed.
    
    SQL> SELECT LAST_ANALYZED FROM DBA_TABLES WHERE TABLE_NAME='SALES';
    
    LAST_ANAL
    ---------
    17-SEP-09
    
    SELECT COUNT(*) FROM DBA_TAB_PENDING_STATS;
    
      COUNT(*)
    ----------
             0
    
    SQL>  alter session set optimizer_use_pending_statistics=FALSE;
    
    Session altered.
    

    Posted in Administration, Oracle 11g | Tagged: , , | Leave a Comment »

    11g Interval Partitioning

    Posted by Gavin Soorma on September 15, 2009

    In versions prior to Oracle 11g, we were very likely to have faced the error shown below especially if we used range partitioning and the partition column was a date field.

    We needed to ensure that we precreated all the partitions before hand based on the expected values of data that would be inserted (or updated) in a table.

    ERROR at line 1:
    ORA-14400: inserted partition key does not map to any partition

    While we could use the MAXVALUE clause to create a ‘catch-all’ partition, this would not help us when we would like to perform any partition maintenance operations based on a date range or if wanted to use the partitioning feature to perform some kind of data archiving at the partition level.

    11g Interval Partitioning

    In Oracle 11g, the creation of partitions (for range) is automated and partitions are created as and when needed and takes the task of managing the creation of new partitions from the DBA. All that is required is to define the interval criteria and create the first partition. Subsequent partitions are created automatically based on the interval criteria.

    create table mypart
       (ename varchar2(20), doj date)
       partition by range (doj)
       INTERVAL (NUMTOYMINTERVAL(1,'YEAR'))
       STORE IN (tbs1,tbs2)
      (partition p_2009 values less than (to_date('01-JAN-2010','DD-MON-YYYY'))
      )
    ;
    

    Note the NUMTOYMINTERVAL is an SQL Funtion used to convert a number to an INTERVAL YEAR TO MONTH literal. The accepted values are ‘YEAR’ and ‘MONTH’.

    The STORE IN clause will create in the partitions in a round robin manner in tablespaces tbs1 and tbs2 as we will see below.

    Let us now insert some values into the table.

    SQL> insert into mypart
      2   values
      3   ('Tom','21-SEP-2009');
    
    1 row created.
    
    SQL> insert into mypart
      2  values
      3  ('Joe','02-JAN-2010');
    
    1 row created.
    

    What has happened after the second insert? – a new partition ‘SYS_P42′ has been created for the year 2010 with a high value of ‘01-JAN-2011′.

    The first partition that we had precreated has been created in the default tablespace USERS since no tablespace name has been prescribed and the new partition has been created in the tablespace TBS2. The next partition that comes along will be created in tablespace TBS1 and so on.

    SQL> select partition_name,high_value from user_tab_partitions
      2  where table_name='MYPART';
    
    PARTITION_NAME                 HIGH_VALUE
    ------------------------------ --------------------------------------------------------------------------------
    P_2009                         TO_DATE(' 2010-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA
    SYS_P42                        TO_DATE(' 2011-01-01 00:00:00', 'SYYYY-MM-DD HH24:MI:SS', 'NLS_CALENDAR=GREGORIA
    
    SQL> select partition_name,tablespace_name from user_tab_partitions
      2  where table_name='MYPART';
    
    PARTITION_NAME                 TABLESPACE_NAME
    ------------------------------ ------------------------------
    P_2009                         USERS
    SYS_P43                        TBS2
    

    We can use the ALTER TABLE SET INTERVAL command to convert a range partitioned table to an interval partitioned table as shown below.

    SQL> create table mypart2
       (ename varchar2(20), doj date)
       partition by range (doj)
      (partition p_2009 values less than (to_date('01-JAN-2010','DD-MON-YYYY'))
      )
    ;    2    3    4    5    6
    
    Table created.
    
    SQL> alter table mypart2
      2  SET INTERVAL (NUMTOYMINTERVAL(1,'YEAR');
    
    Table altered.
    

    We need to keep the following points in mind when using Interval Partitioning:

  • The partitioning column can be only one and it must be of type NUMBER or DATE
  • We cannot use the MAXVALUE clause
  • We cannot use this with Index Organised Tables
  • Posted in Administration, Oracle 11g | Tagged: , , | Leave a Comment »

    11g Real Application Testing – Part 2

    Posted by Gavin Soorma on September 14, 2009

    Analyze the impact of change on SQL statements using the SQL Performance Analyzer

    In the earlier post, we had discussed the Database Capture and Replay component of 11g Real Application Testing – (http://gavinsoorma.wordpress.com/2009/09/11/11g-real-application-testing-making-changes-with-confidence/)

    We can also analyse the impact of upgrading from 10.2.0.4 to 11.1.0.6 at the SQL statement level by using the other component of 11g Real Application Testing which is called SQL Performance Analyzer or SPA.

    We will use the following two SQL statements to create a SQL Tuning Set (STS) in the 10.2.0.4 environment. This STS is then transported to the 11g target environment and then we will use the GUI Database Control to replay the statements in the STS and generate a Performance Analysis report which will help us identify if any statements have improved or digressed in the changed environment.

    SELECT /*+TEST_SPA*/ p.prod_name, s.time_id, t.week_ending_day, SUM(s.amount_sold)
    FROM sales s, products p, times t
    WHERE s.time_id=t.time_id
    AND s.prod_id = p.prod_id
    GROUP BY p.prod_name, s.time_id, t.week_ending_day;

    SELECT /*+TEST_SPA*/ p.prod_category, t.week_ending_day, s.cust_id, SUM(s.amount_sold)
    FROM sales s, products p, times t
    WHERE s.time_id = t.time_id
    AND s.prod_id = p.prod_id
    AND p.prod_category = ‘Peripherals and Accessories’
    GROUP BY p.prod_category, t.week_ending_day, s.cust_id;

    We are using a ‘tag’ TEST_SPA to help us easily identify these SQL statements in the shared pool.

    Source 10.2.0.4 environment – Create and Export SQL Tuning Set

    SQL> exec dbms_sqltune.drop_sqlset(‘SH_10204′);

    PL/SQL procedure successfully completed.

    SQL> exec dbms_sqltune.create_sqlset(‘SH_10204′,’Demo Workload to test SPA’);

    PL/SQL procedure successfully completed.

    SQL> DECLARE
    2 stscur dbms_sqltune.sqlset_cursor;
    3 BEGIN
    4 OPEN stscur FOR
    5 SELECT VALUE(P)
    6 FROM TABLE(dbms_sqltune.select_cursor_cache(
    7 ’sql_text like ”SELECT /*+TEST_SPA%”’,
    8 null, null, null, null, null, null, ‘ALL’)) P;
    9 — populate the sqlset
    10 dbms_sqltune.load_sqlset(:sts_name, stscur);
    11 end;
    12 /

    PL/SQL procedure successfully completed.

    SQL> select sql_id,plan_hash_value, substr(sql_text,1,40) text, executions
    2 from dba_sqlset_statements
    3 where sqlset_name =’SH_10204′
    4 order by sql_id, plan_hash_value;

    SQL_ID PLAN_HASH_VALUE TEXT EXECUTIONS
    ————- ————— ————————————————– ———-
    5930m31ryy4b1 3901800458 SELECT /*+TEST_SPA*/ p.prod_category, t. 1
    5tbr81fmrnyc0 1679814994 SELECT /*+TEST_SPA*/ p.prod_name, s.time 1

    In order to transport the SQL tuning set that we have just created, we need to create a staging table to hold the contents of the STS and then we will use the export utility to generate a dump file which will then be copied to the target 11g environment.

    SQL> execute DBMS_SQLTUNE.CREATE_STGTAB_SQLSET(table_name =>’SH_STS’);

    PL/SQL procedure successfully completed.

    SQL> execute DBMS_SQLTUNE.PACK_STGTAB_SQLSET(sqlset_name => ‘SH_10204′,staging_table_name => ‘SH_STS’);

    PL/SQL procedure successfully completed.

    SQL> select count (*) from sh_sts;

    COUNT(*)
    ———-
    2

    Read More …..

    Posted in Administration, Oracle 11g | Tagged: , , , , | Leave a Comment »

    11g Real Application Testing – making changes with confidence

    Posted by Gavin Soorma on September 11, 2009

    CASE STUDY:

    Using 11g Real Application Testing to analyse the impact of making the following changes:

    1) Upgrade the database from 10g to 11g
    2) Convert from single instance database to a two node RAC database
    3) Convert from file system based storage to 11g ASM

    Overview of Real Application Testing

    One of the best new features in Oracle 11g is the Real Application Testing option which enables us to test the result of a change without actually implementing the same in a production environment. Changes such as database version upgrades, optimizer changes, hardware and storage changes all carry with them an element of risk and uncertainty. Using this feature much of that uncertainty and guesswork can be avoided.

    Real Application Testing comprises of two components:

    a) Database Replay
    b) SQL Performance Analyzer (SPA)

    SPA enables us to test the impact of a change at the level of a particular SQL statement to see if any improvements or regressions occurred in the execution of the SQL statement because of the change.

    Database Replay enables us to capture workload real time in a production environment and then replay the same in a clone or copy of the changed production environment with the exact timings, concurrency and transaction characteristics found in the original workload. The AWR comparison and Database Replay reports can easily help us pinpoint any potential problems and errors as well any divergence in performance.

    This feature will afford businesses increased confidence and surety when faced with a system change as well as significantly reducing the costs as well as resources associated with the testing phase.

    Database Capture is now supported in versions 9.2.0.8 as well as 10.2.0.3 and 10.2.0.4 enabling us to test the impact of upgrades from both 9i as well as 10g to 11g. Note, however, that replay can only be performed in a Oracle 11g database.

    At a very high level, the database capture and replay will involve the following steps:

    1) Prepare the source database for capture
    2) Capture the workload
    3) Copy the workload capture files to the target machine
    4) Prepare the target database for replay and process captured workload files
    5) Start the workload replay clients
    6) Once the workload replay has finished, examine the generated reports and make the analysis

    Read More ….

    Posted in Administration, Oracle 11g | Tagged: , , , , | Leave a Comment »

    Oracle 11g Advanced Compression

    Posted by Gavin Soorma on September 3, 2009

    Although table compression was introduced in Oracle 9i and 10g, it was aimed more at bulk load operations for data warehouse environments. The overhead associated with compression and uncompression made it unsuitable for OLTP type shops.

    In Oracle 11g, Advanced Compression (which is an additional licensed feature) includes OLTP compression capabilities as well as compression of unstructured data like images and text with Secure Files and also direct compression of the export dump file generated by Data Pump which is also directly read without any uncompress while doing an import.

    The compression feature has been greatly enhanced so as to remove any overhead associated with dealing with compressed data while performing any DML activities.The COMPRESS FOR ALL OPERATIONS keyword needs to be used for enabling OLTP data compression.

    While Oracle does claim compression ratios of 3:1, we were able to see clearly 2:1 ratios in terms of storage reductions and no real performance overhead while performing deletes and inserts on compressed data as shown in the example below.

    So this feature could provide us both Storage as well as Performance gains due to the reduced number of blocks which will need to be read while performing I/O.

    UNCOMPRESSED DATA

    SQL> create table myobjects
    2 as select * from all_objects;

    Table created.

    SQL> select sum(bytes)/1048576 from user_segments where segment_name=’MYOBJECTS’;

    SUM(BYTES)/1048576
    ——————
    21

    SQL> insert into myobjects
    2 select * from all_objects;

    56261 rows created.

    Elapsed: 00:00:09.39

    SQL> select count(*) from myobjects where object_type=’TABLE’;

    COUNT(*)
    ———-
    444

    Elapsed: 00:00:00.66

    SQL> delete myobjects;

    168780 rows deleted.

    Elapsed: 00:00:12.22

    COMPRESSED DATA

    SQL> create table myobjects_compress
    2 compress for all operations
    3 as select * from all_objects;

    SQL> select sum(bytes)/1048576 from user_segments where segment_name=’MYOBJECTS_COMPRESS’;

    SUM(BYTES)/1048576
    ——————
    9

    SQL> insert into myobjects_compress
    2 select * from all_objects;

    56261 rows created.

    Elapsed: 00:00:09.08

    SQL> select count(*) from myobjects_compress where object_type=’TABLE’;

    COUNT(*)
    ———-
    444

    Elapsed: 00:00:00.21

    SQL> delete myobjects_compress;

    168783 rows deleted.

    Elapsed: 00:00:10.96

    Compression of Data Pump Exports

    While doing an export, we can use the parameter COMPRESSION=ALL (other options are DATA_ONLY or METADATA) and we see that the size of the dumpfile has reduced very significantly while using this compression feature.

    The uncompressed dumpfile was 113 MB while the compressed dumpfile was just 13 MB. This feature can be very useful when we need to take a large export on disk but are limited by disk space. The earlier methods required us to use a pipe while performing an export, but we had to then uncompress the file while performing the import as well which is a very costly operation in terms of time and resources.

    $ expdp directory=data_file_dir dumpfile=uncompress.dmp schemas=SH compression=ALL

    middba1:/u01/oracle/product/11.1.0/db/demo/schema/sales_history> ls -l *.dmp
    -rw-rw—- 1 oracle dba 113610752 Sep 3 12:25 uncompress.dmp
    -rw-rw—- 1 oracle dba 13426688 Sep 3 12:21 compress.dmp

    Posted in Administration, Oracle 11g | Tagged: , , | Leave a Comment »

    DBA Monitoring Menu

    Posted by Gavin Soorma on September 3, 2009

    I have created a simple but very useful Unix shell script driven menu to enable easy monitoring of any number of Oracle databases from a single central location. This covers most of the normal DBA day to day standard monitoring requirements as well as a good interface for aiding in performance tuning.

    Set up is pretty simple requiring a single user called MONITOR in each database with SELECT privilege on the data dictionary tables.

    Drop me an email at gavin.soorma@bankwest.com.au so that I can send you the zip file along with the installation instructions.

    A few screen shots of the various menu options are shown below

           #######################################################
                            [DBA Menu - PRDU010]
    
                            Connected to Database- cpsprd
            #######################################################
    
            (1) Database Overview
            (2) Database Storage
            (3) Database Objects
            (4) Database Security
            (5) Database Monitor
            (6) Database Performance
    
            Enter your choice (q to quit):
    
            ########################################################
    
            #######################################################
                            Database Monitoring Menu - cpsprd
            #######################################################
    
            (1)  User Sessions
            (2)  Redo Log File Switches
            (3)  Current SQL (by Unix PID)
            (4)  Current SQL (by SID)
            (5)  Current SQL (by Oracle Username)
            (6)  Rollback Segment Usage
            (7)  Locked Sessions
            (8)  Locked Objects
            (9)  Datafile I/O
            (10) DBMS Jobs Status
            (11) View Long Operations
            (12) Active Transactions
            (13) Idle Sessions
            (14) Flashback Logging
            (15) Sessions With High I/O
            (16) Data Guard Log Shipping
            (17) Redo Generation By Day
    
            Enter your choice (q to quit):
    
            #######################################################
    
                 #######################################################
                            Database Performance Menu - cpsprd
            	  #######################################################
    
            (1)  Waits - Instance Level (by time waited)
            (2)  Waits - By Class
            (3)  Waits - Waiting Session's SQL
            (4)  Waits - What are users currently waiting on?
            (5)  Waits - Datafile Level
            (6)  Waits - Block & Object Level
            (7)  Locked Sessions
            (8)  Large Objects In Shared Pool
            (9)  Hit Ratios - Quick Check
            (10) Top 5 Problem Queries (Disk Reads)
            (11) Top 5 Problem Queries (Buffer Gets)
            (12) Top 5 Problem Queries (Disk Reads/Executions)
            (13) Top 5 Queries (Executions)
            (14) Top 5 Latch Waits (by sleeps)
            (15) Top 5 Latch Waits (by misses)
            (16) Current Waits On Latches
            (17) Enqueue Waits - By Session
            (18) Enqueue Waits - By Database
            (19) Enqueue Waits - By Object
            (20) Key CBO init.ora Parameters
            (21) Tables ANALYZE Status
            (22) Indexes ANALYZE Status
            (23) Top Segments with most I/O
    
            Enter your choice (q to quit):
    
            #######################################################
    

    Posted in Administration | Tagged: , | 2 Comments »

    11g Snapshot Standby for real time testing

    Posted by Gavin Soorma on September 2, 2009

    One of the very good new features in Oracle 11g is the Snapshot Standby database where we can basically convert the physical standby database into a read-write real time clone of the production database and we can then temporarily use this environment for carrying out any kind of development testing, QA type work or to test the impact of a proposed production change on an application.

    Basically, the Snapshot Standby database in turn uses the Flashback Database technology to create a guaranteed restore point to which the database can be later flashed back to.

    So we need to keep in mind that whatever limitations are present in the Flashback database technology, the same will also be inherent in the Snapshot database fetaure.

    Let us illustrate this feature by using an example where we would like to test the impact of the change of some important optimizer related parameters on a key application query using the actual production data.

    In production, the SQL query below is shown to be peforming a full table/partition scan and using hash joins. We would like to see if a nested loop join and index usage will prove beneficial to this SQL query that is used frequently by the application.

    explain plan for
    SELECT c.cust_last_name, sum(s.amount_sold) AS dollars,sum(s.quantity_sold) as quantity
    FROM sales s , customers c, products p
    WHERE c.cust_id = s.cust_id
    AND s.prod_id = p.prod_id
    AND c.cust_state_province IN ('Dublin','Galway')GROUP BY c.cust_last_name
    /
    
    SQL> select * from table(dbms_xplan.display);
    
    PLAN_TABLE_OUTPUT
    ------------------------------------------------------------------------------------------------------
    Plan hash value: 1248656060
    
    ------------------------------------------------------------------------------------------------------
    | Id  | Operation              | Name        | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
    ------------------------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT       |             |   519 | 23355 |   935   (6)| 00:00:12 |       |       |
    |   1 |  HASH GROUP BY         |             |   519 | 23355 |   935   (6)| 00:00:12 |       |       |
    |*  2 |   HASH JOIN            |             | 99644 |  4378K|   927   (5)| 00:00:12 |       |       |
    |   3 |    INDEX FULL SCAN     | PRODUCTS_PK |    72 |   288 |     1   (0)| 00:00:01 |       |       |
    |*  4 |    HASH JOIN           |             | 99644 |  3989K|   924   (5)| 00:00:12 |       |       |
    |*  5 |     TABLE ACCESS FULL  | CUSTOMERS   |   766 | 18384 |   408   (1)| 00:00:05 |       |       |
    
    PLAN_TABLE_OUTPUT
    ------------------------------------------------------------------------------------------------------
    |   6 |     PARTITION RANGE ALL|             |   918K|    14M|   507   (6)| 00:00:07 |     1 |    28 |
    |   7 |      TABLE ACCESS FULL | SALES       |   918K|    14M|   507   (6)| 00:00:07 |     1 |    28 |
    ------------------------------------------------------------------------------------------------------
    

    We will now convert the physical standby database to a Snapshot Standby database and note that after the conversion takes place, internally a guaranteed restore point has been created and the database has been opened in read-write mode. Also note the DATABASE_ROLE column.

    SQL>  ALTER DATABASE CONVERT TO SNAPSHOT STANDBY;
    
    Database altered.
    
    SQL> shutdown immediate;
    ORA-01507: database not mounted
    
    ORACLE instance shut down.
    SQL> startup;
    ORACLE instance started.
    
    Total System Global Area  521936896 bytes
    Fixed Size                  2139784 bytes
    Variable Size             415238520 bytes
    Database Buffers           96468992 bytes
    Redo Buffers                8089600 bytes
    Database mounted.
    Database opened.
    
    SQL> select NAME,GUARANTEE_FLASHBACK_DATABASE from  v$restore_point;
    
    NAME                                                         GUA
    ------------------------------------------------------------ ---
    SNAPSHOT_STANDBY_REQUIRED_09/02/2009 10:12:21                YES
    
    SQL> select open_mode,database_role from v$database;
    
    OPEN_MODE  DATABASE_ROLE
    ---------- ----------------
    READ WRITE SNAPSHOT STANDBY
    

    We will now make some changes to some optimizer related parameters:

    SQL> alter system set optimizer_index_cost_adj=20;
    
    System altered.
    
    SQL> alter system set optimizer_index_caching=10;
    
    System altered.
    
    SQL> alter system set optimizer_mode=FIRST_ROWS;
    
    System altered.
    

    We run the same query again and we can see that the plan has changed and the optimizer is now using the index scan with nested loop joins.

    --------------------------------------------------------------------------------------------------------------
    | Id  | Operation                             | Name                 | Rows  | Bytes | Cost (%CPU)| Time     |
    --------------------------------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT                      |                      |   519 | 23355 |   588  (11)| 00:00:08 |
    |   1 |  HASH GROUP BY                        |                      |   519 | 23355 |   588  (11)| 00:00:08 |
    |   2 |   NESTED LOOPS                        |                      | 99644 |  4378K|   580  (10)| 00:00:07 |
    |*  3 |    HASH JOIN                          |                      | 99644 |  3989K|   577  (10)| 00:00:07 |
    |*  4 |     TABLE ACCESS BY INDEX ROWID       | CUSTOMERS            |   766 | 18384 |   241   (2)| 00:00:03 |
    |   5 |      BITMAP CONVERSION TO ROWIDS      |                      |       |       |            |          |
    |   6 |       BITMAP INDEX FULL SCAN          | CUSTOMERS_GENDER_BIX |       |       |            |          |
    |   7 |     PARTITION RANGE ALL               |                      |   918K|    14M|   326  (12)| 00:00:04 |
    |   8 |      TABLE ACCESS BY LOCAL INDEX ROWID| SALES                |   918K|    14M|   326  (12)| 00:00:04 |
    |   9 |       BITMAP CONVERSION TO ROWIDS     |                      |       |       |            |          |
    |  10 |        BITMAP INDEX FULL SCAN         | SALES_PROMO_BIX      |       |       |            |          |
    |* 11 |    INDEX UNIQUE SCAN                  | PRODUCTS_PK          |     1 |     4 |     1   (0)| 00:00:01 |
    --------------------------------------------------------------------------------------------------------------

    While the original physical standby database has been now converted to snapshot database, some changes are happening on the Primary database and those changes are shipped to the standby site but not yet applied. They will accumulate on the standby site and will be applied after the snapshot standby database is converted back to a physical standby database.

    SQL> create table mytest as select * from sales;
    
    Table created.
    
    SQL> update mytest set prod_id=1;
    
    918843 rows updated.
    

    On the standby site, we will now convert the shapshot standby database to the original mode of physical standby database in Active Data Guard mode.

    Note that after the physical standby database is opened, changes that were made on the Primary database while it was open in shapshot standby mode are now applied as well.

    SQL> shutdown immediate;
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area  521936896 bytes
    Fixed Size                  2139784 bytes
    Variable Size             415238520 bytes
    Database Buffers           96468992 bytes
    Redo Buffers                8089600 bytes
    Database mounted.
    SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
    
    Database altered.
    
    SQL> shutdown immediate;
    ORA-01507: database not mounted
    
    ORACLE instance shut down.
    
    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area  521936896 bytes
    Fixed Size                  2139784 bytes
    Variable Size             415238520 bytes
    Database Buffers           96468992 bytes
    Redo Buffers                8089600 bytes
    Database mounted.
    SQL>  recover managed standby database disconnect;
    Media recovery complete.
    SQL> recover managed standby database cancel;
    Media recovery complete.
    SQL> alter database open;
    
    Database altered.
    
    Elapsed: 00:01:06.29
    SQL> recover managed standby database using current logfile disconnect;
    Media recovery complete.
    
    SQL> select count(*) from sh.mytest;
    
      COUNT(*)
    ----------
        918843
    
    SQL> select distinct prod_id from sh.mytest;
    
       PROD_ID
    ----------
             1
    

    Posted in High Availability, Oracle 11g | Tagged: , , | Leave a Comment »

    How to transport a tablespace from 10g ASM on Windows to 11g ASM on AIX

    Posted by Gavin Soorma on August 31, 2009

    This example demonstrates how to transport a tablespace from a 10g ASM on Windows environment to 11g ASM on AIX environment using the DBMS_FILE_TRANSFER package.

    The source database on Windows is GENUA1 and the target database is a 11g RAC database RACDB1.

    Create the tablespace and the test object in the source database

    SQL> create tablespace test_asm
      2  datafile size 100m;
    
    Tablespace created.
    
    SQL> select file_name from dba_data_files
      2  where tablespace_name='TEST_ASM';
    
    FILE_NAME
    --------------------------------------------------------------------------------
    
    +DATA/genua1/datafile/test_asm.376.696333127
    
    SQL> conn system/xxx
    Connected.
    
    SQL> create table myobjects
      2  tablespace test_asm
      3  as select * from dba_objects;
    
    Table created.
    
    SQL> select count(*) from system.myobjects;
    
      COUNT(*)
    ----------
         56504
    

    Create a database link from the source database to the target database

    SQL> CREATE DATABASE LINK  DBS2 CONNECT TO
      2  system identified by oracle
      3  using 'racdb1';
    
    Database link created.
    
    SQL> select * from dual@dbs2;
    
    D
    -
    X
    

    Create a directory to hold the converted datafile on the source

    SQL> create directory source_dir as '+DATA/genua1/xtransport/';
    
    Directory created.
    

    Create a directory to hold the tablespace meta data dump on the source

    SQL> create directory source_dir_2 as '+DATA';
    
    Directory created.
    

    Create an O/S level directory for the data pump export logfile

    SQL> create directory test_asm_log as 'd:\oracle\';
    
    Directory created.
    

    Create a directory on the target to hold the transported datafile

    SQL> CREATE OR REPLACE DIRECTORY target_dir  AS '+DATA';
    
    Directory created.
    

    Create an O/S level directory for the data pump import logfile

     create directory test_asm_log as '/u01/oracle/';
    
    Directory created.
    

    Make the tablespace which is going to be transported read only

    SQL> alter tablespace test_asm read only;
    
    Tablespace altered.
    

    Export the tablespace meta data using Data Pump

    C:\Documents and Settings\bwdba_cbduat>expdp directory=source_dir_2 dumpfile=test_asm.dmp transport_tablespaces=TEST_ASM transport_full_check=Y logfile=test_asm_log:exp.log

    Export: Release 10.2.0.4.0 - Production on Monday, 31 August, 2009 10:41:34
    
    Copyright (c) 2003, 2007, Oracle.  All rights reserved.
    
    Username: system
    Password:
    
    Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Produc
    tion
    With the Partitioning, Real Application Clusters, OLAP, Data Mining
    and Real Application Testing options
    Starting "SYSTEM"."SYS_EXPORT_TRANSPORTABLE_01":  system/******** directory=sour
    ce_dir_2 dumpfile=test_asm.dmp transport_tablespaces=TEST_ASM transport_full_che
    ck=Y logfile=test_asm_log:exp.log
    Processing object type TRANSPORTABLE_EXPORT/PLUGTS_BLK
    Processing object type TRANSPORTABLE_EXPORT/TABLE
    Processing object type TRANSPORTABLE_EXPORT/INDEX
    Processing object type TRANSPORTABLE_EXPORT/INDEX_STATISTICS
    Processing object type TRANSPORTABLE_EXPORT/POST_INSTANCE/PLUGTS_BLK
    Master table "SYSTEM"."SYS_EXPORT_TRANSPORTABLE_01" successfully loaded/unloaded
    
    ******************************************************************************
    Dump file set for SYSTEM.SYS_EXPORT_TRANSPORTABLE_01 is:
      +DATA/test_asm.dmp
    Job "SYSTEM"."SYS_EXPORT_TRANSPORTABLE_01" successfully completed at 10:42:16
    

    Use DBMS_FILE_TRANSFER to transfer the export dump file to the target

    SQL> begin
      2   dbms_file_transfer.put_file
      3   (source_directory_object => 'source_dir_2',
      4   source_file_name => 'test_asm.dmp',
      5   destination_directory_object => 'target_dir',
      6  destination_file_name => 'test_asm.dmp',
      7   destination_database => 'dbs2');
      8   end;
      9  /
    
    PL/SQL procedure successfully completed.
    

    Use RMAN to convert the tablespace to the target AIX 64 bit format

    Note that the converted datafile will reside in the directory we created in an earlier step - +DATA/genua1/xtransport
    
    RMAN> convert tablespace test_asm to
    2> platform 'AIX-Based Systems (64-bit)'
    3> format '+DATA';
    
    Starting backup at 31/AUG/09
    configuration for DISK channel 2 is ignored
    configuration for DISK channel 3 is ignored
    configuration for DISK channel 4 is ignored
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=470 instance=genua11 devtype=DISK
    channel ORA_DISK_1: starting datafile conversion
    input datafile fno=00224 name=+DATA/genua1/datafile/test_asm.376.696333127
    converted datafile=+DATA/genua1/xtransport/test_asm.266.696338369
    channel ORA_DISK_1: datafile conversion complete, elapsed time: 00:00:07
    Finished backup at 31/AUG/09
    

    Use DBMS_FILE_TRANSFER to copy the converted datafile to the target

    SQL> begin
      2    dbms_file_transfer.put_file
      3    (source_directory_object => 'source_dir',
      4     source_file_name => 'test_asm.266.696338369',
      5    destination_directory_object => ' target_dir',
      6    destination_file_name => 'tts1_db1.dbf',
      7     destination_database => 'dbs2');
      8    end;
      9     /
    
    PL/SQL procedure successfully completed.
    

    On the target import the datafile meta data using Data Pump

    The imp.par import parameter file has the following contents 
    
    directory=target_dir
    dumpfile=test_asm.dmp
    logfile=test_asm_log:imp.log
    TRANSPORT_DATAFILES='+DATA1/tts1_db1.dbf'
    keep_master=y
    
    middba1:/u01/oracle> impdp parfile=imp.par
    
    Import: Release 11.1.0.6.0 - 64bit Production on Monday, 31 August, 2009 11:39:32
    
    Copyright (c) 2003, 2007, Oracle.  All rights reserved.
    
    Username: system
    Password:
    
    Connected to: Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
    With the Partitioning, Real Application Clusters and Real Application Testing options
    Master table "SYSTEM"."SYS_IMPORT_TRANSPORTABLE_02" successfully loaded/unloaded
    Starting "SYSTEM"."SYS_IMPORT_TRANSPORTABLE_02":  system/******** parfile=imp.par
    Processing object type TRANSPORTABLE_EXPORT/PLUGTS_BLK
    Processing object type TRANSPORTABLE_EXPORT/TABLE
    Processing object type TRANSPORTABLE_EXPORT/INDEX
    Processing object type TRANSPORTABLE_EXPORT/INDEX_STATISTICS
    Processing object type TRANSPORTABLE_EXPORT/POST_INSTANCE/PLUGTS_BLK
    Job "SYSTEM"."SYS_IMPORT_TRANSPORTABLE_02" successfully completed at 11:39:48
    

    On the target database RACDB1, check that the TEST_ASM tablespace has been plugged in and the MYOBJECTS table is present

    middba1:/u01/oracle> sql
    
    SQL*Plus: Release 11.1.0.6.0 - Production on Mon Aug 31 11:41:29 2009
    
    Copyright (c) 1982, 2007, Oracle.  All rights reserved.
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
    With the Partitioning, Real Application Clusters and Real Application Testing options
    
    SQL> select file_name from dba_data_files where tablespace_name='TEST_ASM';
    
    FILE_NAME
    --------------------------------------------------------------------------------
    +DATA/tts1_db1.dbf
    
    SQL> select count(*) from system.myobjects;
    
      COUNT(*)
    ----------
         56504
    

    Posted in Administration, Oracle 11g | Tagged: , , , | Leave a Comment »

    Duplicating a database from an ASM to a Non-ASM file system

    Posted by Gavin Soorma on August 26, 2009

    By default, when we are using ASM with a Flash Recovery Area configured, backups to disk will go to the “backupset” subdirectory – unless we use the FORMAT clause while taking the backup to point the backups to another location on disk.

    When we try to duplicate a database which is hosted on an ASM storage to a conventional O/S file system, the restore will fail as it will look for the ASM diskgroups where the backup was taken on the target node as well.

    We will see an error like …..

    channel ORA_AUX_DISK_1: reading from backup piece +DBATEAM/bwdba/backupset/2009_08_25/nnndf0_tag20090825t131155_0.388.695826727
    channel ORA_AUX_DISK_1: ORA-19870: error while restoring backup piece +DBATEAM/bwdba/backupset/2009_08_25/nnndf0_tag20090825t131155_0.388.695826727
    ORA-19505: failed to identify file “+DBATEAM/bwdba/backupset/2009_08_25/nnndf0_tag20090825t131155_0.388.695826727″
    ORA-17503: ksfdopn:2 Failed to open file +DBATEAM/bwdba/backupset/2009_08_25/nnndf0_tag20090825t131155_0.388.695826727
    ORA-15001: diskgroup “DBATEAM” does not exist or is not mounte
    failover to previous backup

    In this case we will need to use the BACKUPSET clause of the BACKUP command to relocate that backup from the ASM storage to a O/S file system. We then ftp this backupset to the target node ensuring that the same directory file structure is available on the target node as compared to the source node where the ASM disk backup was restored. We can then proceed as normal with the RMAN Duplicate database procedure.

    RMAN> list backup of database;
    
    List of Backup Sets
    ===================
    
    BS Key  Type LV Size       Device Type Elapsed Time Completion Time
    ------- ---- -- ---------- ----------- ------------ ---------------
    14      Full    1.40G      DISK        00:00:47     26-AUG-09
            BP Key: 14   Status: AVAILABLE  Compressed: NO  Tag: TAG20090826T102022
            Piece Name: +DBATEAM/bwdba/backupset/2009_08_26/nnndf0_tag20090826t102022_0.397.695902837
      List of Datafiles in backup set 14
      File LV Type Ckp SCN    Ckp Time  Name
      ---- -- ---- ---------- --------- ----
      1       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/system.256.695305997
      2       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/sysaux.257.695305999
      3       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/undotbs1.258.695305999
      4       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/users.259.695305999
      5       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/example.267.695306155
      6       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/undotbs2.268.695306391
    
    RMAN> backup device type disk format '/u01/oracle/backup/bwdba_bkp.%u'
    2> backupset 14
    3> ;
    
    Starting backup at 26-AUG-09
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: SID=108 instance=bwdba1 device type=DISK
    input backup set count=15 STAMP=695902823 creation_time=26-AUG-09
    channel ORA_DISK_1: starting piece 1 at 26-AUG-09
    channel ORA_DISK_1: backup piece +DBATEAM/bwdba/backupset/2009_08_26/nnndf0_tag20090826t102022_0.397.695902837
    piece handle=/u01/oracle/backup/bwdba_bkp.0fknl8j7 comment=NONE
    channel ORA_DISK_1: finished piece 1 at 26-AUG-09
    channel ORA_DISK_1: backup set complete, elapsed time: 00:00:25
    Finished backup at 26-AUG-09
    

    We will see now that the backup location has changed

    RMAN> list backup of database;
    
    List of Backup Sets
    ===================
    
    BS Key  Type LV Size       Device Type Elapsed Time Completion Time
    ------- ---- -- ---------- ----------- ------------ ---------------
    14      Full    1.40G      DISK        00:00:47     26-AUG-09
            BP Key: 18   Status: AVAILABLE  Compressed: NO  Tag: TAG20090826T102022
            Piece Name: /u01/oracle/backup/bwdba_bkp.0fknl8j7
      List of Datafiles in backup set 14
      File LV Type Ckp SCN    Ckp Time  Name
      ---- -- ---- ---------- --------- ----
      1       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/system.256.695305997
      2       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/sysaux.257.695305999
      3       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/undotbs1.258.695305999
      4       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/users.259.695305999
      5       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/example.267.695306155
      6       Full 2765425    26-AUG-09 +DBATEAM/bwdba/datafile/undotbs2.268.695306391
    

    Posted in Backup and Recovery | Tagged: , , , | Leave a Comment »

    11g Adaptive Cursor Sharing

    Posted by Gavin Soorma on August 21, 2009

    While one of the performance best practices that is always recommended is the use of bind variables over the literals, Optimizer plan unstability was one of the problems faced by many Oracle 10g shops. When queries that normally perform well, suddenly stopped performing, very often it is found that this bad performance is being observed only for a few or specific parameters or values. And this is particularly so on tables with highly skewed data especially the columns which are being referenced via the WHERE clause.

    In Oracle 11g however, with Adaptive Cursor Sharing, if the optimizer observes a sub-optimal plan for a particular bind variable value, it will enable particular bind variables or range of bind variable values to use a different execution plan for the same SQL statement.

    To illustrate the 11g Adaptive Cursor Sharing, we will create a table called MYOBJECTS which is based on the ALL_OBJECTS view. The data in the OBJECT_TYPE column is skewed – for example there are more than 50,000 rows with the value ‘SYNONYM’ and only 1 row with the value ‘RULE’. The table has about 100,000 rows.

    SQL> conn sh/sh
    Connected.
    
    SQL> create table myobjects
      2  as select * from all_objects;
    
    Table created.
    
    SQL> create index myobject_ind on myobjects (object_type);
    
    Index created.
    

    We then gather statistics on the table and see that histograms have been created on the table which has the skewed data.

    SQL> exec dbms_stats.gather_table_stats(null,'MYOBJECTS', METHOD_OPT => 'FOR ALL COLUMNS SIZE 10');
    
    PL/SQL procedure successfully completed.
    
    SQL> SELECT column_name, histogram FROM user_tab_cols WHERE  table_name ='MYOBJECTS';
    
    COLUMN_NAME                    HISTOGRAM
    ------------------------------ ---------------
    OWNER                          HEIGHT BALANCED
    OBJECT_NAME                    HEIGHT BALANCED
    SUBOBJECT_NAME                 HEIGHT BALANCED
    OBJECT_ID                      HEIGHT BALANCED
    DATA_OBJECT_ID                 HEIGHT BALANCED
    OBJECT_TYPE                    HEIGHT BALANCED
    CREATED                        HEIGHT BALANCED
    LAST_DDL_TIME                  HEIGHT BALANCED
    TIMESTAMP                      HEIGHT BALANCED
    STATUS                         FREQUENCY
    TEMPORARY                      FREQUENCY
    GENERATED                      FREQUENCY
    SECONDARY                      FREQUENCY
    

    If we query the MYOBJECTS Table using the predicate ‘RULE’, we find that an INDEX RANGE SCAN using the MYOBJECTS_IND index is being carried out by the optimizer.

    If we change the predicate to ‘SYNONYM’ we find that a FULL TABLE SCAN is being performed instead which is the expected behaviour given the distribution of rows in the table.

    SQL> explain plan for
      2  select * from myobjects where object_type='RULE';
    
    Explained.
    
    SQL> set linesize 120
    SQL> select * from table(dbms_xplan.display);
    
    PLAN_TABLE_OUTPUT
    ---------------------------------------------------------------------------------------------
    Plan hash value: 2533318700
    
    --------------------------------------------------------------------------------------------
    | Id  | Operation                   | Name         | Rows  | Bytes | Cost (%CPU)| Time     |
    --------------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT            |              |  1612 |   147K|    56   (0)| 00:00:01 |
    |   1 |  TABLE ACCESS BY INDEX ROWID| MYOBJECTS    |  1612 |   147K|    56   (0)| 00:00:01 |
    |*  2 |   INDEX RANGE SCAN          | MYOBJECT_IND |  1612 |       |     5   (0)| 00:00:01 |
    --------------------------------------------------------------------------------------------
    
    SQL>  explain plan for
      2  select * from myobjects where object_type='SYNONYM';
    
    Explained.
    
    SQL> select * from table(dbms_xplan.display);
    
    PLAN_TABLE_OUTPUT
    ------------------------------------------------------------------------------------------------
    Plan hash value: 2581838392
    
    -------------------------------------------------------------------------------
    | Id  | Operation         | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
    -------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT  |           | 34398 |  3157K|   284   (2)| 00:00:04 |
    |*  1 |  TABLE ACCESS FULL| MYOBJECTS | 34398 |  3157K|   284   (2)| 00:00:04 |
    -------------------------------------------------------------------------------
    

    Let us now see how the optimizer handles bind variables and how it performs the task of bind variable peeking. In the first case, the variable v_obj_type is assigned the value ‘SYNONYM’ and in the second case the same variable is assigned the value ‘RULE’.

    Remember, when literals were being used, using the value ‘RULE’ in the WHERE clause resulted in an index range scan, while usage of the value ‘SYNONYM’ resulted in a full table scan.

    SQL> VARIABLE v_obj_type VARCHAR2(10)
    SQL> EXEC :v_obj_type := 'SYNONYM';
    
    PL/SQL procedure successfully completed.
    
    SQL> select * from myobjects where object_type=:v_obj_type;
    
    40156 rows selected.
    
    SQL> select * from table(dbms_xplan.display_cursor);
    
    PLAN_TABLE_OUTPUT
    -----------------------------------------------------------------------------------------------
    SQL_ID  5utj7m6zxjdsc, child number 0
    -------------------------------------
    select * from myobjects where object_type=:v_obj_type
    
    Plan hash value: 2581838392
    
    -------------------------------------------------------------------------------
    | Id  | Operation         | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
    -------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT  |           |       |       |   284 (100)|          |
    |*  1 |  TABLE ACCESS FULL| MYOBJECTS | 34398 |  3157K|   284   (2)| 00:00:04 |
    -------------------------------------------------------------------------------
    

    Although the predicate is now the value ‘RULE’ based on the bind variable that is being used here, we find that the optimizer is still performing a FULL TABLE SCAN, while in fact it should be actually performing an index scan.

    But since bind peeking is being performed here, the optimizer will use the same plan even though the value of the variable is different. If the data is skewed as in this case, it can cause a lot of performance problems and optimizer unstability depending on which bind variable value is assigned at hard parse.

    SQL> VARIABLE v_obj_type VARCHAR2(10)
    SQL> EXEC :v_obj_type := 'RULE';
    
    PL/SQL procedure successfully completed.
    
    SQL> select * from myobjects where object_type=:v_obj_type;
    
    no rows selected
    
    SQL> /
    
    no rows selected
    
    SQL> /
    
    no rows selected
    
    SQL>  select * from table(dbms_xplan.display_cursor);
    
    PLAN_TABLE_OUTPUT
    ----------------------------------------------------------------------------------------------
    SQL_ID  5utj7m6zxjdsc, child number 0
    -------------------------------------
    select * from myobjects where object_type=:v_obj_type
    
    Plan hash value: 2581838392
    
    -------------------------------------------------------------------------------
    | Id  | Operation         | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
    -------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT  |           |       |       |   284 (100)|          |
    |*  1 |  TABLE ACCESS FULL| MYOBJECTS | 34398 |  3157K|   284   (2)| 00:00:04 |
    -------------------------------------------------------------------------------
    

    But if we query the V$SQL view, we find that for this particular SQL_ID, the column IS_BIND_SENSITIVE shows ‘Y’ but the column ‘IS_BIND_AWARE’ still shows ‘N’.

    This means that Oracle is aware that this query can require different execution plans based on the bind variable values but has not yet acted on this at this stage.

    SQL> SELECT sql_id, child_number, is_bind_sensitive, is_bind_aware
     from v$sql
     WHERE  sql_text ='select * from myobjects where object_type=:v_obj_type';  2    3
    
    SQL_ID        CHILD_NUMBER I I
    ------------- ------------ - -
    5utj7m6zxjdsc            0 Y N
    

    We then execute the same SQL statement once again using the same bind variable value, and we find that this time a more optimal execution plan has been chosen which in this case is an INDEX RANGE SCAN

    SQL>  VARIABLE v_obj_type VARCHAR2(10)
    SQL> EXEC :v_obj_type := 'RULE';
    
    PL/SQL procedure successfully completed.
    
    SQL> select * from myobjects where object_type=:v_obj_type;
    
    no rows selected
    
    SQL> select * from table(dbms_xplan.display_cursor);
    
    PLAN_TABLE_OUTPUT
    -----------------------------------------------------------------------------------------------
    SQL_ID  5utj7m6zxjdsc, child number 1
    -------------------------------------
    select * from myobjects where object_type=:v_obj_type
    
    Plan hash value: 2745750972
    
    ---------------------------------------------------------------------------------------------
    | Id  | Operation                   | Name          | Rows  | Bytes | Cost (%CPU)| Time     |
    ---------------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT            |               |       |       |    30 (100)|          |
    |   1 |  TABLE ACCESS BY INDEX ROWID| MYOBJECTS     |   823 | 83946 |    30   (0)| 00:00:01 |
    
    PLAN_TABLE_OUTPUT
    -------------------------------------------------------------------------------------------------
    |*  2 |   INDEX RANGE SCAN          | MYOBJECTS_IND |   823 |       |     3   (0)| 00:00:01 |
    ---------------------------------------------------------------------------------------------
    

    If we now query the V$SQL view, we find that this optimizer change is also reflected in the column IS_BIND_SENSITIVE showing the value ‘Y’ instead of ‘N’ which was the earlier value.

    SQL> SELECT sql_id, child_number, is_bind_sensitive, is_bind_aware
     from v$sql
     WHERE  sql_text ='select * from myobjects where object_type=:v_obj_type';  2    3
    
    SQL_ID        CHILD_NUMBER I I
    ------------- ------------ - -
    5utj7m6zxjdsc            0 Y N
    5utj7m6zxjdsc            1 Y Y
    

    Posted in Oracle 11g | Tagged: , , | Leave a Comment »

    DBUPGDIAG script to check integrity of database before 11g upgrade

    Posted by Arjun Raja on August 20, 2009

    Script to check integrity of a 9i or 10g database before upgrade to 11g.

    Run this script connected as sysdba in the 9i or 10g database.

    NAME: DBUPGDIAG.SQL

    This script can be downloaded from Oracle Metalink note 556610.1

    – PURPOSE:
    – This script is intended to provide a user friendly output to diagonise
    – the status of the database before (or) after upgrade. The script will
    – create a file called db_upg_diag__.log in your local
    – working directory. This does not make any DDL / DML modifications.

    – This script will work in both Windows and Unix platforms from database
    – version 9.2 or higher.

    col TODAY NEW_VALUE _DATE
    col VERSION NEW_VALUE _VERSION
    set termout off
    select to_char(SYSDATE,’fmMonth DD, YYYY’) TODAY from DUAL;
    select version from v$instance;
    set termout on
    set echo off
    set feedback off
    set head off
    set verify off
    Prompt
    PROMPT Enter location for Spooled output:
    Prompt
    DEFINE log_path = &1
    column timecol new_value timestamp
    column spool_extension new_value suffix
    SELECT to_char(sysdate,’dd-Mon-yyyy_hhmi’) timecol,’.log’ spool_extension FROM
    sys.dual;
    column output new_value dbname
    SELECT value || ‘_’ output FROM v$parameter WHERE name = ‘db_name’;
    spool &log_path/db_upg_diag_&&dbname&&timestamp&&suffix
    set linesize 150
    set pages 100
    set trim on
    set trims on
    col Compatible for a35
    col comp_id for a12
    col comp_name for a40
    col org_version for a11
    col prv_version for a11
    col owner for a12
    col object_name for a40
    col object_type for a40
    col Wordsize for a25
    col Metadata for a8
    col ‘Initial DB Creation Info’ for a35
    col ‘Total Invalid JAVA objects’ for a45
    col ‘Role’ for a30
    col ‘User Existence’ for a27
    col “JAVAVM TESTING” for a15
    Prompt
    Prompt
    set feedback off head off
    select LPAD(‘*** Start of LogFile ***’,50) from dual;
    select LPAD(‘Oracle Database Upgrade Diagnostic Utility’,44)||
    LPAD(TO_CHAR(SYSDATE, ‘MM-DD-YYYY HH24:MI:SS’),26) from dual;
    Prompt
    Prompt ===============
    Prompt Database Uptime
    Prompt ===============
    SELECT to_char(startup_time, ‘HH24:MI DD-MON-YY’) “Startup Time”
    FROM v$instance;
    Prompt
    Prompt =================
    Prompt Database Wordsize
    Prompt =================
    SELECT distinct(‘This is a ‘ || (length(addr)*4) || ‘-bit database’) “WordSize”
    FROM v$process;
    Prompt
    Prompt ================
    Prompt Software Verison
    Prompt ================
    SELECT * FROM v$version;
    Prompt
    Prompt =============
    Prompt Compatibility
    Prompt =============
    SELECT ‘Compatibility is set as ‘||value Compatible
    FROM v$parameter WHERE name =’compatible’;
    Prompt
    Prompt ================
    Prompt Component Status
    Prompt ================
    Prompt
    SET SERVEROUTPUT ON;
    DECLARE

    ORG_VERSION varchar2(12);
    PRV_VERSION varchar2(12);
    P_VERSION VARCHAR2(10);

    BEGIN

    SELECT version INTO p_version
    FROM registry$ WHERE cid=’CATPROC’ ;

    IF SUBSTR(p_version,1,5) = ‘9.2.0′ THEN

    DBMS_OUTPUT.PUT_LINE(RPAD(‘Comp ID’, 8) ||RPAD(‘Component’,35)||
    RPAD(‘Status’,10) ||RPAD(‘Version’, 15));

    DBMS_OUTPUT.PUT_LINE(RPAD(‘ ‘,8,’-') ||RPAD(‘ ‘,35,’-')||
    RPAD(‘ ‘,10,’-') ||RPAD(‘ ‘,15,’-'));

    FOR x in (SELECT SUBSTR(dr.comp_id,1,8) comp_id,
    SUBSTR(dr.comp_name,1,35) comp_name,
    dr.status Status,SUBSTR(dr.version,1,15) version
    FROM dba_registry dr,registry$ r
    WHERE dr.comp_id=r.cid and dr.comp_name=r.cname
    ORDER BY 1)

    LOOP

    DBMS_OUTPUT.PUT_LINE(RPAD(SUBSTR(x.comp_id,1,8),8) ||
    RPAD(SUBSTR(x.comp_name,1,35),35)||
    RPAD(x.status,10) || RPAD(x.version, 15));
    END LOOP;

    ELSIF SUBSTR(p_version,1,5) != ‘9.2.0′ THEN

    DBMS_OUTPUT.PUT_LINE(RPAD(‘Comp ID’, 8) ||RPAD(‘Component’,35)||
    RPAD(‘Status’,10) ||RPAD(‘Version’, 15)||
    RPAD(‘Org_Version’,15)||RPAD(‘Prv_Version’,15));

    DBMS_OUTPUT.PUT_LINE(RPAD(‘ ‘,8,’-') ||RPAD(‘ ‘,35,’-')||
    RPAD(‘ ‘,10,’-')||RPAD(‘ ‘,15,’-')||RPAD(‘ ‘,15,’-')||
    RPAD(‘ ‘,15,’-'));

    FOR y in (SELECT SUBSTR(dr.comp_id,1,8) comp_id,
    SUBSTR(dr.comp_name,1,35) comp_name, dr.status Status,
    SUBSTR(dr.version,1,11) version,org_version,prv_version
    FROM dba_registry dr,registry$ r
    WHERE dr.comp_id=r.cid and dr.comp_name=r.cname
    ORDER BY 1)

    LOOP

    DBMS_OUTPUT.PUT_LINE(RPAD(substr(y.comp_id,1,8), 8) ||
    RPAD(substr(y.comp_name,1,35),35)||RPAD(y.status,10) ||
    RPAD(y.version, 15)||RPAD(y.org_version,15)||RPAD(y.prv_version,15));

    END LOOP;

    END IF;
    END;
    /
    SET SERVEROUTPUT OFF
    Prompt
    Prompt
    Prompt ======================================================
    Prompt List of Invalid Database Objects Owned by SYS / SYSTEM
    Prompt ======================================================
    Prompt
    set head on
    SELECT case count(object_name)
    WHEN 0 THEN ‘There are no Invalid Objects’
    ELSE ‘There are ‘||count(object_name)||’ Invalid objects’
    END “Number of Invalid Objects”
    FROM dba_objects
    WHERE status=’INVALID’
    AND owner in (‘SYS’,'SYSTEM’);
    Prompt
    DOC
    ################################################################

    If there are no Invalid objects below will result in zero rows.

    ################################################################
    #
    Prompt
    set feedback on
    SELECT owner,object_name,object_type
    FROM dba_objects
    WHERE status=’INVALID’
    AND owner in (‘SYS’,'SYSTEM’)
    ORDER BY owner,object_type;
    set feedback off
    Prompt
    Prompt ================================
    Prompt List of Invalid Database Objects
    Prompt ================================
    Prompt
    set head on
    SELECT case count(object_name)
    WHEN 0 THEN ‘There are no Invalid Objects’
    ELSE ‘There are ‘||count(object_name)||’ Invalid objects’
    END “Number of Invalid Objects”
    FROM dba_objects
    WHERE status=’INVALID’
    AND owner not in (‘SYS’,'SYSTEM’);
    Prompt
    DOC
    ################################################################

    If there are no Invalid objects below will result in zero rows.

    ################################################################
    #
    Prompt
    set feedback on
    SELECT owner,object_name,object_type
    FROM dba_objects
    WHERE status=’INVALID’
    AND owner not in (‘SYS’,'SYSTEM’)
    ORDER BY owner,object_type;
    set feedback off
    Prompt
    Prompt ==============================================================
    Prompt Identifying whether a database was created as 32-bit or 64-bit
    Prompt ==============================================================
    Prompt
    DOC
    ###########################################################################

    Result referencing the string ‘B023′ ==> Database was created as 32-bit
    Result referencing the string ‘B047′ ==> Database was created as 64-bit
    When String results in ‘B023′ and when upgrading database to 10.2.0.3.0
    (64-bit) , For known issue refer below articles

    Note 412271.1 ORA-600 [22635] and ORA-600 [KOKEIIX1] Reported While
    Upgrading Or Patching Databases To 10.2.0.3
    Note 579523.1 ORA-600 [22635], ORA-600 [KOKEIIX1], ORA-7445 [KOPESIZ] and
    OCI-21500 [KOXSIHREAD1] Reported While Upgrading To 11.1.0.6

    ###########################################################################
    #
    Prompt
    SELECT SUBSTR(metadata,109,4) “Metadata”,
    CASE SUBSTR(metadata,109,4)
    WHEN ‘B023′ THEN ‘Database was created as 32-bit’
    WHEN ‘B047′ THEN ‘Database was created as 64-bit’
    ELSE ‘Metadata not Matching’
    END “Initial DB Creation Info”
    FROM sys.kopm$;
    Prompt
    Prompt ===================================================
    Prompt Number of Duplicate Objects Owned by SYS and SYSTEM
    Prompt ===================================================
    Prompt
    Prompt Counting duplicate objects ….
    Prompt
    SELECT count(1)
    FROM dba_objects
    WHERE object_name||object_type in
    (SELECT object_name||object_type
    from dba_objects
    where owner = ‘SYS’)
    and owner = ‘SYSTEM’;
    Prompt
    Prompt =========================================
    Prompt Duplicate Objects Owned by SYS and SYSTEM
    Prompt =========================================
    Prompt
    Prompt Querying duplicate objects ….
    Prompt
    SELECT object_name, object_type
    FROM dba_objects
    WHERE object_name||object_type in
    (SELECT object_name||object_type
    FROM dba_objects
    WHERE owner = ‘SYS’)
    AND owner = ‘SYSTEM’;
    Prompt
    DOC

    ################################################################################

    If any objects found please follow below article.
    Note 1030426.6 How to Clean Up Duplicate Objects Owned by SYS and SYSTEM schema
    Read the Exceptions carefully before taking actions.

    ################################################################################
    #
    Prompt
    Prompt ================
    Prompt JVM Verification
    Prompt ================
    Prompt
    SET SERVEROUTPUT ON
    DECLARE

    V_CT NUMBER;
    P_VERSION VARCHAR2(10);

    BEGIN

    – If so, get the version of the JAVAM component
    EXECUTE IMMEDIATE ‘SELECT version FROM registry$ WHERE cid=”JAVAVM”
    AND status 99′ INTO p_version;

    SELECT count(*) INTO v_ct FROM dba_objects
    WHERE object_type LIKE ‘%JAVA%’ AND owner=’SYS’;

    IF SUBSTR(p_version,1,5) = ‘8.1.7′ THEN
    IF v_ct>=6787 THEN
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Installed properly’);
    ELSE
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Not Installed properly’);
    END IF;
    ELSIF SUBSTR(p_version,1,5) = ‘9.0.1′ THEN
    IF v_ct>=8585 THEN
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Installed properly’);
    ELSE
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Not Installed properly’);
    END IF;
    ELSIF SUBSTR(p_version,1,5) = ‘9.2.0′ THEN
    IF v_ct>=8585 THEN
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Installed properly’);
    ELSE
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Not Installed properly’);
    END IF;
    ELSIF SUBSTR(p_version,1,6) = ‘10.1.0′ THEN
    IF v_ct>=13866 THEN
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Installed properly’);
    ELSE
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Not Installed properly’);
    END IF;
    ELSIF SUBSTR(p_version,1,6) = ‘10.2.0′ THEN
    IF v_ct>=14113 THEN
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Installed properly’);
    ELSE
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – Not Installed properly’);
    END IF;
    END IF;

    EXCEPTION WHEN NO_DATA_FOUND THEN
    DBMS_OUTPUT.PUT_LINE(‘JAVAVM – NOT Installed. Below results can be ignored’);

    END;
    /
    SET SERVEROUTPUT OFF
    Prompt
    Prompt ================================================
    Prompt Checking Existence of Java-Based Users and Roles
    Prompt ================================================
    Prompt
    DOC

    ################################################################################

    There should not be any Java Based users for database version 9.0.1 and above.
    If any users found, it is faulty JVM.

    ################################################################################
    #

    Prompt
    SELECT CASE count(username)
    WHEN 0 THEN ‘No Java Based Users’
    ELSE ‘There are ‘||count(*)||’ JAVA based users’
    END “User Existence”
    FROM dba_users WHERE username LIKE ‘%AURORA%’ AND username LIKE ‘%OSE%’;
    Prompt
    DOC

    ###############################################################

    Healthy JVM Should contain Six Roles.
    If there are more or less than six role, JVM is inconsistent.

    ###############################################################
    #

    Prompt
    SELECT CASE count(role)
    WHEN 0 THEN ‘No JAVA related Roles’
    ELSE ‘There are ‘||count(role)||’ JAVA related roles’
    END “Role”
    FROM dba_roles
    WHERE role LIKE ‘%JAVA%’;
    Prompt
    Prompt Roles
    Prompt
    SELECT role FROM dba_roles WHERE role LIKE ‘%JAVA%’;
    set head off
    Prompt
    Prompt =========================================
    Prompt List of Invalid Java Objects owned by SYS
    Prompt =========================================
    SELECT CASE count(*)
    WHEN 0 THEN ‘There are no SYS owned invalid JAVA objects’
    ELSE ‘There are ‘||count(*)||’ SYS owned invalid JAVA objects’
    END “Total Invalid JAVA objects”
    FROM dba_objects
    WHERE object_type LIKE ‘%JAVA%’
    AND status=’INVALID’
    AND owner=’SYS’;
    Prompt
    DOC

    #################################################################

    Check the status of the main JVM interface packages DBMS_JAVA
    and INITJVMAUX and make sure it is VALID.
    If there are no Invalid objects below will result in zero rows.

    #################################################################
    #
    Prompt
    set feedback on
    SELECT owner,object_name,object_type
    FROM dba_objects
    WHERE object_type LIKE ‘%JAVA%’
    AND status=’INVALID’
    AND owner=’SYS’;
    set feedback off
    Prompt
    Prompt INFO: Below query should succeed with ‘foo’ as result.
    set heading on
    select dbms_java.longname(‘foo’) “JAVAVM TESTING” from dual;
    set heading off
    Prompt

    set feedback off head off
    select LPAD(‘*** End of LogFile ***’,50) from dual;
    set feedback on head on
    Prompt
    spool off
    Prompt
    set heading off
    set heading off
    set feedback off
    select ‘Upload db_upg_diag_&&dbname&&timestamp&&suffix from “&log_path” directory’
    from dual;
    set heading on
    set feedback on
    Prompt
    – – - – - – - – - – - – - – - – Script ends here – - – - – - – - – - – - – -

    Posted in Scripts | Tagged: , | Leave a Comment »

    ORA-27504 error creating ASM instance on 11g RAC

    Posted by Gavin Soorma on August 19, 2009

    While creating the ASM instance using DBCA in an 11g RAC environment on 64 bit AIX 5L, we got the ORA-27504 error along with others as shown below.

    asmdb7

    The problem was caused by incorrect UDP and TCP packet settings and is documented in Metalink note 300956.1

    As root, the following changes need to be made

    # no -o tcp_sendspace=262144
    # no -o tcp_recvspace=262144
    # no -o udp_sendspace=65536
    # no -o udp_recvspace=262144
    # no -o rfc1323=1

    Adding these entries to the /etc/rc.net will ensure that these parameters take effect on every machine reboot.

    Posted in Oracle 11g, Real Application Clusters | Tagged: , , , | Leave a Comment »

    Using FLASHBACK to rollback a TRUNCATE

    Posted by Gavin Soorma on August 18, 2009

    This scenario will show how we can use a combination of FLASHBACK database and also recovery to take a database back in time to undo a TRUNCATE operation and then roll forward the database after the flashback operation to bring it to the current point in time.

    INSERT ROWS INTO TABLE MYOBJ – THIS WILL BE TRUNCATED

    SQL> insert into scott.myobj select * from all_objects;
    
    50496 rows created.
    
    SQL> /
    
    50496 rows created.
    
    SQL> select count(*) from scott.myobj;
    
      COUNT(*)
    ----------
        100992
    

    OBTAIN THE CURRENT SCN – FLASHBACK WILL HAPPEN TO THIS SCN

    SQL> select current_scn from v$database;
    
              CURRENT_SCN
    ---------------------
              15633908021
    
    TRUNCATE THE TABLE
    
    SQL> truncate table scott.myobj;
    
    Table truncated.
    
    SQL> select count(*) from scott.myobj;
    
      COUNT(*)
    ----------
             0
    


    AT THE SAME TIME OTHER CHANGES ARE HAPPENING IN THE DATABASE AND THESE CHANGES WILL BE RECOVERED AFTER THE FLASHBACK IS DONE

    SQL> insert into scott.myobj2 select * from scott.myobj2;
    
    356874 rows created.
    
    SQL> /
    
    713748 rows created.
    
    SQL> commit;
    
    Commit complete.
    

    SHUTDOWN THE DATABASE AND PERFORM THE FLASHBACK TO THE SCN BEFORE THE TRUNCATE WAS DONE

    SQL> shutdown immediate;
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area  469762048 bytes
    Fixed Size                  2084880 bytes
    Variable Size             377491440 bytes
    Database Buffers           83886080 bytes
    Redo Buffers                6299648 bytes
    Database mounted.
    
    SQL> FLASHBACK DATABASE TO SCN 15633908021;
    
    Flashback complete.
    

    OPEN THE DATABASE IN READ ONLY MODE AND EXPORT THE TABLE THAT WAS TRUNCATED EARLIER. THIS TABLE WILL BE IMPORTED AFTER THE RECOVERY IS DONE

    SQL> alter database open read only;
    
    Database altered.
    
    SQL>  select count(*) from scott.myobj;
    
      COUNT(*)
    ----------
             100992
    
    SQL> quit
    Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    crashdb:/u03/oradata/crashdb/arch> exp file=scott.dmp tables=myobj
    
    Export: Release 10.2.0.4.0 - Production on Fri Feb 6 09:53:00 2009
    
    Copyright (c) 1982, 2007, Oracle.  All rights reserved.
    
    Username: scott
    Password:
    
    Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    Export done in US7ASCII character set and AL16UTF16 NCHAR character set
    server uses WE8ISO8859P1 character set (possible charset conversion)
    
    About to export specified tables via Conventional Path ...
    . . exporting table                          MYOBJ          100992 rows exported
    Export terminated successfully without warnings.
    

    NOW SHUTDOWN THE DATABASE,STARTUP MOUNT AND PERFORM THE RECOVERY

    SQL> shutdown immediate
    Database closed.
    Database dismounted.
    ORACLE instance shut down.
    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area  696254464 bytes
    Fixed Size                  2086616 bytes
    Variable Size             184551720 bytes
    Database Buffers          503316480 bytes
    Redo Buffers                6299648 bytes
    Database mounted.
    SQL> recover database;
    Media recovery complete.
    SQL> alter database open;
    
    Database altered.
    

    AS EXPECTED TABLE MYOBJ WHICH WAS TRUNCATED NOW AGAIN HAS 0 ROWS AFTER THE RECOVERY – WE CAN IMPORT THE DUMP WE TOOK AFTER THE FLASHBACK

    SQL> select count(*) from scott.myobj;
    
      COUNT(*)
    ----------
             0
    

    CONFIRM THAT OTHER COMMITTED CHANGES IN THE DATABASE HAVE BEEN RECOVERED

    SQL> select count(*) from scott.myobj2;
    
      COUNT(*)
    ----------
      713748
    

    Posted in Backup and Recovery, High Availability | Tagged: , , | Leave a Comment »

    11g RAC Software Relink Error on AIX

    Posted by Gavin Soorma on August 18, 2009

    While relinking the Oracle 11g RAC software on 64 bit AIX platform, we got the error as shown below:

    ora11

    Looking at the installation log we found the following lines:

    ld: 0706-006 Cannot find or open library file: -l ha_gs_r
    ld:open(): A file or directory in the path name does not exist.
    ld: 0706-006 Cannot find or open library file: -l ha_em_r
    ld:open(): A file or directory in the path name does not exist.
    make: 1254-004 The error code from the last command is 255

    The cause of this error is that a particluar fileset rsct.basic.rte which is required specifically for a 11g RAC installation has not been installed at the OS level. After the fileset was installed we could relink the software without the same ld errors as mentioned above.

    We can check if the fileset has been installed by running the command ‘lslpp’ as shown below.

    testdb:/u02/oradata/testdb> lslpp -l rsct.basic.rte
    lslpp: Fileset rsct.basic.rte not installed.

    After the fileset is installed we can confirm the same…

    middba1:/u01/oracle/dump> lslpp -l rsct.basic.rte
     Fileset                      Level  State      Description
      ----------------------------------------------------------------------------
    Path: /usr/lib/objrepos
      rsct.basic.rte            2.4.10.0  COMMITTED  RSCT Basic Function
    

    This package is part of the OS Patch IY98287 which is one of the mandatory required patches to be installed on AIX if we are configuring a 11g RAC environment.

    This is also documented in Metalink Note 471701.1

    Posted in Oracle 11g, Real Application Clusters | Tagged: , , , | Leave a Comment »

    11g RAC Transparent Application Failover (TAF)

    Posted by Gavin Soorma on August 14, 2009

    This example illustrates Transparent Application Failover (TAF) in a two node AIX 11g RAC configuration.

    The database name is racdb and the two instances are racdb1 and racdb2.

    Node 1 – middba1 which hosts instance racdb1.
    Node2 – middba2 which hosts instance racdb2

    We have defined two services racdb1 and racdb2. For service racdb1 the preferred instance is racdb1 and
    for service racdb2 the preferred instance is racdb2.

    The following srvctl commands illustrates the same.

    middba1:/u01/oracle/dump> srvctl status service -d racdb -s racdb1
    Service racdb1 is running on instance(s) racdb1

    middba1:/u01/oracle/dump> srvctl config service -d racdb -s racdb1
    racdb1 PREF: racdb1 AVAIL: racdb2

    middba1:/u01/oracle/dump> srvctl status service -d racdb -s racdb2
    Service racdb2 is running on instance(s) racdb2

    middba1:/u01/oracle/dump> srvctl config service -d racdb -s racdb2
    racdb2 PREF: racdb2 AVAIL: racdb1

    From a SQL*PLUS client, we establish a session as user SYSTEM using the service racdb1. Note the machine name where this service is running from.

    testdb:/u01/oracle> sqlplus system/xxx@racdb1
    
    SQL*Plus: Release 11.1.0.6.0 - Production on Fri Aug 14 10:37:03 2009
    
    Copyright (c) 1982, 2007, Oracle.  All rights reserved.
    
    Connected to:
    Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production
    With the Partitioning, Real Application Clusters and Real Application Testing options
    
    SQL> select host_name from v$instance;
    
    HOST_NAME
    ----------------------------------------------------------------
    middba1
    

    We then execute a long running SELECT statement

    SQL> select * from sh.sales;
    

    While this statement is running we reboot the node middba1 where this service is currently connected to.

    We will note that the screen will momentarily freeze while the service gets relocated to the other functional node racdb2 and then the query continues from where it got interrupted. The session will not get disconnected – so for the end user it is transparent from which instance the query is being serviced.

    If we check the status of the cluster resources, we will find that the VIP from node middba1 has now relocated to middba2. Other resources that were originally running from node middba1 are now offline.

    middba2:/u01/oracle> crs_stat -t
    Name           Type           Target    State     Host
    ------------------------------------------------------------
    ora....SM1.asm application    ONLINE    OFFLINE
    ora....A1.lsnr application    ONLINE    OFFLINE
    ora....ba1.gsd application    ONLINE    OFFLINE
    ora....ba1.ons application    ONLINE    OFFLINE
    ora....ba1.vip application    ONLINE    ONLINE    middba2
    ora....SM2.asm application    ONLINE    ONLINE    middba2
    ora....A2.lsnr application    ONLINE    ONLINE    middba2
    ora....ba2.gsd application    ONLINE    ONLINE    middba2
    ora....ba2.ons application    ONLINE    ONLINE    middba2
    ora....ba2.vip application    ONLINE    ONLINE    middba2
    ora.racdb.db   application    ONLINE    ONLINE    middba2
    ora....cdb1.cs application    ONLINE    ONLINE    middba2
    ora....b1.inst application    ONLINE    OFFLINE
    ora....db1.srv application    ONLINE    ONLINE    middba2
    ora....cdb2.cs application    ONLINE    ONLINE    middba2
    ora....b2.inst application    ONLINE    ONLINE    middba2
    ora....db2.srv application    ONLINE    ONLINE    middba2
    

    If we check the status of the service, we will find that the service racdb1 is now running on node middba2 connecting to the instance racdb2.

    middba2:/u01/oracle> srvctl status service -d racdb -s racdb1
    Service racdb1 is running on instance(s) racdb2

    Once the node middba1 which was rebooted finally does come online, we will find that the VIP will relocate
    to its original node middba1 once the cluster services are brought online after the reboot.

    middba2:/u01/oracle> crs_stat -t
    Name           Type           Target    State     Host
    ------------------------------------------------------------
    ora....SM1.asm application    ONLINE    ONLINE    middba1
    ora....A1.lsnr application    ONLINE    ONLINE    middba1
    ora....ba1.gsd application    ONLINE    ONLINE    middba1
    ora....ba1.ons application    ONLINE    ONLINE    middba1
    ora....ba1.vip application    ONLINE    ONLINE    middba1
    ora....SM2.asm application    ONLINE    ONLINE    middba2
    ora....A2.lsnr application    ONLINE    ONLINE    middba2
    ora....ba2.gsd application    ONLINE    ONLINE    middba2
    ora....ba2.ons application    ONLINE    ONLINE    middba2
    ora....ba2.vip application    ONLINE    ONLINE    middba2
    ora.racdb.db   application    ONLINE    ONLINE    middba2
    ora....cdb1.cs application    ONLINE    ONLINE    middba2
    ora....b1.inst application    ONLINE    ONLINE    middba1
    ora....db1.srv application    ONLINE    ONLINE    middba2
    ora....cdb2.cs application    ONLINE    ONLINE    middba2
    ora....b2.inst application    ONLINE    ONLINE    middba2
    ora....db2.srv application    ONLINE    ONLINE    middba2
    

    Note however, that the service racdb1 will continue running on node middba2 (instance racdb2)even though the original node which it was running on middba1 has been brought online.

    We will have to relocate the service back to its original node using the srvctl command

    middba1:/u01/oracle> srvctl relocate service -d racdb -s racdb1 -i racdb2 -t racdb1

    Posted in High Availability, Oracle 11g, Real Application Clusters | Tagged: , , , | Leave a Comment »

    READ ONLY Tablespace Restore and Recovery

    Posted by Gavin Soorma on August 12, 2009

    Keeping static or historical data in read only tablespaces is a good practice especially for data warehouse type environments.

    Using the RMAN SKIP READONLY command, we can reduce the backup window and overhead as well by excluding these read only tablespaces from the database backupsets.

    But we need to keep in mind that we need to take at least one backup of the tablespace after it has been made read only and thereafter we can use the SKIP READONLY command to exclude these tablespaces from the daily or weekly database backups.

    However, while doing a restore we need to use the CHECK READONLY keywords otherwise by default the read only tablespaces will not be restored and hence the recovery will also bypass these tablespaces. Subssequent attempts to open the database will fail.

    Let us illustrate the same with an example where we have made the USERS tablespace read only and then simulated a media failure by deleting the three datafiles of the USERS tablespace at the OS level.

    We will first attempt a normal restore and recovery and see how that fails. We follow that by using the CHECK READONLY keywords and see how the restore and recovery succeeds.

     restore database;
    
    Starting restore at 12-AUG-09
    using channel ORA_SBT_TAPE_1
    using channel ORA_DISK_1
    
    datafile 4 not processed because file is read-only
    datafile 6 not processed because file is read-only
    datafile 9 not processed because file is read-only
    channel ORA_SBT_TAPE_1: starting datafile backupset restore
    channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
    restoring datafile 00001 to /u02/oradata/testdb/system01.dbf
    restoring datafile 00002 to /u02/oradata/testdb/undotbs01.dbf
    restoring datafile 00003 to /u02/oradata/testdb/sysaux01.dbf
    restoring datafile 00005 to /u02/oradata/testdb/example01.dbf
    channel ORA_SBT_TAPE_1: reading from backup piece 08kmb8f7_1_1
    channel ORA_SBT_TAPE_1: restored backup piece 1
    piece handle=08kmb8f7_1_1 tag=TAG20090810T120039
    channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:10:36
    Finished restore at 12-AUG-09
    
    RMAN> recover database;
    
    Starting recover at 12-AUG-09
    using channel ORA_SBT_TAPE_1
    using channel ORA_DISK_1
    datafile 4 not processed because file is read-only
    datafile 6 not processed because file is read-only
    datafile 9 not processed because file is read-only
    channel ORA_SBT_TAPE_1: starting incremental datafile backupset restore
    channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
    destination for restore of datafile 00001: /u02/oradata/testdb/system01.dbf
    .....
    .......
    
    archive log filename=/u02/oradata/testdb/arch/arch.36.1.694170424.log thread=1 sequence=36
    archive log filename=/u02/oradata/testdb/arch/arch.37.1.694170424.log thread=1 sequence=37
    archive log filename=/u02/oradata/testdb/arch/arch.38.1.694170424.log thread=1 sequence=38
    media recovery complete, elapsed time: 00:00:43
    Finished recover at 12-AUG-09
    
    RMAN> alter database open;
    
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of alter db command at 08/12/2009 12:55:30
    ORA-01157: cannot identify/lock data file 4 - see DBWR trace file
    ORA-01110: data file 4: '/u02/oradata/testdb/users03.dbf'
    

    Now, the read only tableaspaces are restored as well, but we notice the recovery stage will skip these tablespaces since no recovery is required as no changes have happened on these tablespaces since the last backup of these tablespaces after they were made read only.

    RMAN> restore database check readonly;
    
    Starting restore at 12-AUG-09
    using channel ORA_SBT_TAPE_1
    using channel ORA_DISK_1
    
    channel ORA_SBT_TAPE_1: starting datafile backupset restore
    channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
    restoring datafile 00001 to /u02/oradata/testdb/system01.dbf
    restoring datafile 00002 to /u02/oradata/testdb/undotbs01.dbf
    restoring datafile 00003 to /u02/oradata/testdb/sysaux01.dbf
    restoring datafile 00004 to /u02/oradata/testdb/users03.dbf
    restoring datafile 00005 to /u02/oradata/testdb/example01.dbf
    restoring datafile 00006 to /u02/oradata/testdb/users02.dbf
    restoring datafile 00009 to /u02/oradata/testdb/users01.dbf
    channel ORA_SBT_TAPE_1: reading from backup piece 08kmb8f7_1_1
    channel ORA_SBT_TAPE_1: restored backup piece 1
    piece handle=08kmb8f7_1_1 tag=TAG20090810T120039
    channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:05:25
    Finished restore at 12-AUG-09
    
    RMAN> recover database;
    Starting recover at 12-AUG-09
    using channel ORA_SBT_TAPE_1
    using channel ORA_DISK_1
    datafile 4 not processed because file is read-only
    datafile 6 not processed because file is read-only
    datafile 9 not processed because file is read-only
    channel ORA_SBT_TAPE_1: starting incremental datafile backupset restore
    channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
    destination for restore of datafile 00001: /u02/oradata/testdb/system01.dbf
    destination for restore of datafile 00002: /u02/oradata/testdb/undotbs01.dbf
    destination for restore of datafile 00003: /u02/oradata/testdb/sysaux01.dbf
    destination for restore of datafile 00005: /u02/oradata/testdb/example01.dbf
    channel ORA_SBT_TAPE_1: reading from backup piece 0akmb8uu_1_1
    channel ORA_SBT_TAPE_1: restored backup piece 1
    piece handle=0akmb8uu_1_1 tag=TAG20090810T120901
    channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:07
    
    starting media recovery
    
    archive log thread 1 sequence 36 is already on disk as file /u02/oradata/testdb/arch/arch.36.1.694170424.log
    archive log thread 1 sequence 37 is already on disk as file /u02/oradata/testdb/arch/arch.37.1.694170424.log
    archive log thread 1 sequence 38 is already on disk as file /u02/oradata/testdb/arch/arch.38.1.694170424.log
    archive log thread 1 sequence 39 is already on disk as file /u02/oradata/testdb/arch/arch.39.1.694170424.log
    archive log thread 1 sequence 40 is already on disk as file /u02/oradata/testdb/arch/arch.40.1.694170424.log
    archive log filename=/u02/oradata/testdb/arch/arch.36.1.694170424.log thread=1 sequence=36
    archive log filename=/u02/oradata/testdb/arch/arch.37.1.694170424.log thread=1 sequence=37
    archive log filename=/u02/oradata/testdb/arch/arch.38.1.694170424.log thread=1 sequence=38
    media recovery complete, elapsed time: 00:00:39
    Finished recover at 12-AUG-09
    
    RMAN> alter database open;
    
    database opened
    

    Posted in Backup and Recovery | Tagged: , , , | Leave a Comment »

    CRS-0184 error and its resolution

    Posted by Gavin Soorma on August 12, 2009

    While installing the 11g Clusterware on 64 bit AIX 5L, we  encountered the CRS-0184 error while running the root.sh on the second node.

    The error was …

    Cluster Synchronization Services is active on all the nodes.
    
    Waiting for the Oracle CRSD and EVMD to start
    
    Oracle CRS stack installed and running under init(1M)
    
    Running vipca(silent) for configuring nodeapps
    
    Creating VIP application resource on (2) nodes0:CRS-0184: Cannot communicate with the CRS daemon
    

     
    The crsd.log file showed the following lines ….

    2009-08-10 14:18:36.059: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11
    2009-08-10 14:19:12.143: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11
    2009-08-10 14:20:15.037: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11
    2009-08-10 14:21:11.379: [ CRSMAIN][10286] Failed to spawn a thread for UI connection. error=-11
    

    The fix for this error was to increase the Shell Limits for the user ‘root’ as the Clusterware processes are started as root.

    The documentation does mention that we need to so the same for the ‘oracle’ user account and fails to mention that we need to do it for the root account as well.

    After changing the shell limits for root (on BOTH nodes) the CRS-0184 problem was resolved.

    middba1 # ulimit -a
    time(seconds) unlimited
    file(blocks) unlimited
    data(kbytes) unlimited
    stack(kbytes) 4194304
    memory(kbytes) unlimited
    coredump(blocks) 2097151
    nofiles(descriptors) unlimited
    

    Note that since the root.sh on the second node will internally launch VIPCA and this part did not run to successful completion, we had to manually run the vipca command as root to get the vip,ons amd gsd resources online.
     

    Posted in Administration, Oracle 11g, Real Application Clusters | Tagged: , , , | Leave a Comment »

    Flashback database – with and without resetlogs.

    Posted by Arjun Raja on August 11, 2009

    FLASHBACK DATABASE – IMPACT OF RESETLOGS-

    A flashback database can be completed and the database can either be opened for READ ONLY or opened with a RESETLOGS option.

    As long as the RESETLOGS option is not used, a flashback can be rolled back .

    Example -

    Create table test – insert some records -

    select count (*) from test ;
    

    COUNT(*)
    ———-
    100000

    Select current_scn from v$database;
    

    653026

    Switch a couple of logfiles -

    alter system switch logfile;
    

    Insert another 50000 records into table test.

    select count (*) from test ;
    

    COUNT(*)
    ———-
    150000

    select first_change# ,to_char(first_time,'DD-MON-YYYY HH24:MI:SS'),sequence# from V$LOG_HISTORY ORDER BY 1 ;
    

    FIRST_CHANGE# TO_CHAR(FIRST_TIME,’ SEQUENCE#
    ————- ——————– ———-
    538113 11-AUG-2009 10:58:17 1
    576251 11-AUG-2009 11:03:35 2
    653024 11-AUG-2009 11:18:11 3
    653026 11-AUG-2009 11:18:12 4
    653029 11-AUG-2009 11:18:17 5

    Now shutdown immediate and flashback database to SCN 653026 – Before the latest insert to table test – at which point the count(*) of test is 100,000.

    shutdown immediate;

    startup mount ;

    Now flashback database to SCN 653026.

    SQL> flashback database to scn 653026;
    

    Now open the database READ ONLY -

    alter database read only;
    
    select count(*) from test;
    

    COUNT(*)
    ———-
    100000

    If you are satisfied with the results, the database can be opened with the RESETLOGS option -

    Shutdown immediate;
    
    startup mount;
    
    alter database open resetlogs;
    

    If not satisfied and you want to redo the flashback or get the database back to it’s current state-

    Shutdown immediate;
    
    startup mount;
    
    recover database;
    

    Media recovery complete.

    alter database open;
    

    Database altered.

    select count(*) from test;
    

    COUNT(*)
    ———-
    150000

    Therefore as long as a RESETLOGS is not mentioned, the flashback of a database can be reversed.

    Ex – if you use RESETLOGS -

    Shutdown immediate;

    Startup mount;

    flashback database to scn 653026;

    Flashback complete.

    alter database open resetlogs;

    Database altered.

    select count(*) from test;

    COUNT(*)
    ———-
    100000

    shutdown immediate;

    startup mount;

    recover database;

    SQL> ORA-00283: recovery session canceled due to errors
    ORA-00264: no recovery required

    alter database open;

    Database altered.

    select count(*) from test;

    COUNT(*)
    ———-
    100000

    Therefore the RESETLOGS option does not allow you to reverse the FLASHBACK of the database.

    Posted in Administration | Tagged: , | Leave a Comment »

    How to check the UTL_FILE_DIR parameter?

    Posted by Arjun Raja on August 7, 2009

    CREATE OR REPLACE procedure utl_file_test_write1 (
    path in varchar2,
    filename in varchar2,
    firstline in varchar2,
    secondline in varchar2)
    is
    output_file utl_file.file_type;
    begin
    output_file := utl_file.fopen (path,filename, ‘W’);

    utl_file.put_line (output_file, firstline);
    utl_file.put_line (output_file, secondline);
    utl_file.fclose(output_file);

    –exception
    – when others then null;
    end;
    /

    begin
    utl_file_test_write1 (
    ‘/u03/users/ofsa/ofsatest/oracle_utl_data’,
    ‘utl_file_test’,
    ‘first line’,
    ’second line’
    );
    end;
    /

    This works too -

    SET SERVEROUTPUT ON
    DECLARE
    fid UTL_FILE.FILE_TYPE;
    v VARCHAR2(32767);
    PROCEDURE recNgo (str IN VARCHAR2)
    IS
    BEGIN
    DBMS_OUTPUT.PUT_LINE (‘UTL_FILE error ‘ || str);

    UTL_FILE.FCLOSE (fid);
    END;
    BEGIN
    /* Change the directory name to one to which you at least
    || THINK you have read/write access.
    */
    fid := UTL_FILE.FOPEN (‘/u03/users/ofsa/ofsatest/oracle_utl_data’, ‘utl_file_test’, ‘R’);
    UTL_FILE.GET_LINE (fid, v);
    dbms_output.put_line (v);
    UTL_FILE.FCLOSE (fid);

    fid := UTL_FILE.FOPEN (‘/u03/users/ofsa/ofsatest/oracle_utl_data’, ‘utl_file_test_10′, ‘W’);
    UTL_FILE.PUT_LINE (fid, v);
    UTL_FILE.FCLOSE (fid);
    EXCEPTION
    WHEN UTL_FILE.INVALID_PATH
    THEN recNgo (‘invalid_path’);
    WHEN UTL_FILE.INVALID_MODE
    THEN recNgo (‘invalid_mode’);
    WHEN UTL_FILE.INVALID_FILEHANDLE
    THEN recNgo (‘invalid_filehandle’);
    WHEN UTL_FILE.INVALID_OPERATION
    THEN recNgo (‘invalid_operation’);
    WHEN UTL_FILE.READ_ERROR
    THEN recNgo (‘read_error’);
    WHEN UTL_FILE.WRITE_ERROR
    THEN recNgo (‘write_error’);
    WHEN UTL_FILE.INTERNAL_ERROR
    THEN recNgo (‘internal_error’);
    END;
    /

    message for success below.

    first line

    PL/SQL procedure successfully completed.

    SQL> exit

    Posted in Administration | Tagged: , | Leave a Comment »

    Troubleshooting Agent Startup and Upload

    Posted by Arjun Raja on August 7, 2009

    If the agent in GRID control in not uploading to the OMS -

    cd $AGENT_HOME/bin

    oracle(DATABASE)@prdu030:./emctl start agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    Starting agent ………. started.

    oracle(DATABASE)@prdu030:./emctl status agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    —————————————————————
    Agent Version : 10.2.0.4.0
    OMS Version : 10.2.0.4.0
    Protocol Version : 10.2.0.4.0
    Agent Home : /u01/oracle/agent10g
    Agent binaries : /u01/oracle/agent10g
    Agent Process ID : 856310
    Parent Process ID : 897132
    Agent URL : https://prdu030.bankwest.com:3872/emd/main/
    Repository URL : https://prdu010.bankwest.com:1159/em/upload
    Started at : 2009-08-07 12:27:29
    Started by user : oracle
    Last Reload : 2009-08-07 12:27:29
    Last successful upload : (none)
    Last attempted upload : (none)
    Total Megabytes of XML files uploaded so far : 0.00
    Number of XML files pending upload : 5004
    Size of XML files pending upload(MB) : 20.53
    Available disk space on upload filesystem : 69.35%
    Collection Status : Disabled by Upload Manager
    Last successful heartbeat to OMS : 2009-08-07 12:27:39
    —————————————————————
    Agent is Running and Ready

    Steps to fix this issue -

    cd $AGENT_HOME/sysman/emd

    rm lastupld.xml agntstmp.txt

    cd $AGENT_HOME/sysman/emd/upload

    rm *.*

    cd $AGENT_HOME/sysman/emd/state

    rm *.*

    No need to remove storage directory under $AGENT_HOME/sysman/emd/state

    cd $AGENT_HOME/bin

    ./emctl start agent

    ./emctl status agent

    If the Last successful heartbeat to OMS is not shown as successful, then secure the agent .

    cd $AGENT_HOME/bin

    ./emctl secure agent

    Make sure the password prompted for matches the OMS password – usually the sysman password for the OMS URL -

    Once agent is secured -

    oracle(DATABASE)@prdu030:./emctl status agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    —————————————————————
    Agent is Not Running

    oracle(DATABASE)@prdu030:./emctl start agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    Starting agent ……… started.

    oracle(DATABASE)@prdu030:./emctl status agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    —————————————————————
    Agent Version : 10.2.0.4.0
    OMS Version : 10.2.0.4.0
    Protocol Version : 10.2.0.4.0
    Agent Home : /u01/oracle/agent10g
    Agent binaries : /u01/oracle/agent10g
    Agent Process ID : 217220
    Parent Process ID : 864280
    Agent URL : https://prdu030.bankwest.com:3872/emd/main/
    Repository URL : https://prdu010.bankwest.com:1159/em/upload
    Started at : 2009-08-07 12:55:43
    Started by user : oracle
    Last Reload : 2009-08-07 12:55:43
    Last successful upload : 2009-08-07 12:56:00
    Total Megabytes of XML files uploaded so far : 2.00
    Number of XML files pending upload : 6
    Size of XML files pending upload(MB) : 3.96
    Available disk space on upload filesystem : 69.31%
    Last successful heartbeat to OMS : 2009-08-07 12:55:52—————————————————————
    Agent is Running and Ready
    oracle(DATABASE)@prdu030:

    Posted in EM Grid Control | Tagged: , , , | Leave a Comment »

    FLASHBACK DATABASE TO SCN

    Posted by Arjun Raja on August 4, 2009

    FLASHBACK DATABASE TO SCN

    Time to FLASHBACK DATABASE TO – 06:30 a.m on 31 July.

    Get oldest flashback time –

    SELECT OLDEST_FLASHBACK_SCN,to_char(OLDEST_FLASHBACK_TIME,'DD-MON-YYYY HH24:MI:SS') FROM V$FLASHBACK_DATABASE_LOG;

    If there are logs up until 06:30 on 31 Jul, you can flashback.

    Get the SCN number to flashback to -

    col first_change# format 99999999999
    select first_change# ,to_char(first_time,'DD-MON-YYYY HH24:MI:SS'),sequence# from V$LOG_HISTORY ORDER BY 1 ;
    

    23856899540 31-JUL-2009 04:30:09 119695
    23856965501 31-JUL-2009 06:30:10 119696
    23857031498 31-JUL-2009 08:30:09 119697
    23859233947 31-JUL-2009 10:00:29 119704
    23859242647 31-JUL-2009 10:31:34 119705

    Note down the SCN at the time you want to flashback to which is the FIRST_CHANGE# in the V$LOG_HISTORY view –

    Example here the SCN is 23856965501 at 06:30 – the time you want to flashback to.

    Restore a few archivelogs prior to and after the flashback time – the log sequence is in the last column above….

    Ex – to restore archivelogs from 119695 to 119702

    rman target / catalog rman/rman@rman

    RMAN > restore archivelog from sequence 119695 until sequence 119705;

    Once the logs are restored make sure no automatic RMAN backup job of archivelogs run which delete old archivelogs from disk –

    Next flashback the database –

    SHUTDOWN IMMEDIATE;

    STARTUP MOUNT;

    FLASHBACK DATABASE TO SCN 23856965501;

    Once FLASHBACK is complete-

    Verify the data before opening with a resetlogs -

    ALTER DATABASE OPEN READ ONLY;

    If satisfied with the data -

    Shutdown immediate;
    Startup mount ;
    Alter database open resetlogs;

    Posted in High Availability | Tagged: | Leave a Comment »

    RMAN TSPITR Limitations

    Posted by Gavin Soorma on August 4, 2009

    While the RMAN Tablespace Point In Time Recovery (TSPITR) automates the process of creating an auxiliary instance and exporting and importing objects from the auxiliary instance to the target database, it still has many limitations and we encountered bugs which apparently have been only fixed in 11g.

    Not only do we need to allocate temporary disk space for the SYSTEM, UNDO and the tablespace that we want to be recovered, it cannot be used to recover from a case of a dropped tablespace. Further, the tablespace needs to be self contained and cannot contain any objects owned by SYS or any undo or rollback segments. In case of partitioned tables spread over a number of tablespaces, all the referenced tablespaces will need to be recovered together.

    While running the following command to do a TSPITR, we encountered the errors shown below

    run {
    recover tablespace test_tspitr
    until logseq 7 thread 1
    auxiliary destination ‘/u02/oradata/stage/’;
    }

    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of recover command at 08/04/2009 11:28:52
    RMAN-03015: error occurred in stored script Memory Script
    RMAN-06026: some targets not found – aborting restore
    RMAN-06100: no channel to restore a backup or copy of datafile 2
    RMAN-06100: no channel to restore a backup or copy of datafile 1

    Metalink Note 391785.1 indicates that this error is because the automatically configured Auxiliary SBT channel is released during the TSPITR process and not reused later on in the process. This is also seen in the log and the workaround suggested is to explicitly allocate an auxiliary channel.

    released channel: ORA_SBT_TAPE_1
    released channel: ORA_AUX_SBT_TAPE_1
    released channel: ORA_DISK_1
    released channel: ORA_AUX_DISK_1

    if we change the command to allocate the channel as shown below, we are getting another error that it is not able to connect to an auxiliary instance. This means we need to startup and connect to an auxiliary instance first before running the TSPITR command. Not very good.

    run {
    allocate auxiliary channel c1 device type ‘SBT_TAPE’ ;
    recover tablespace test_tspitr
    until logseq 7 thread 1
    auxiliary destination ‘/u02/oradata/stage/’;
    }

    RMAN-03002: failure of allocate command at 08/04/2009 12:08:48
    RMAN-06174: not connected to auxiliary database

    This has been documented in Bug 4768353 which is apparently fixed in 11g. The issue is not faced if backups are available on disk and default device type is disk.

    Posted in Backup and Recovery | Tagged: , , | Leave a Comment »

    RMAN Recovering a Dropped Tablespace

    Posted by Gavin Soorma on August 4, 2009

    To recover from a case of a dropped tablespace, the Tablespace Point In Time Recovery (TSPITR) method cannot be used. When you drop a tablespace, the controlfile will then no longer have any records of the tablespace which has been dropped. Attempts to use the RMAN RECOVER TABLESPACE command will return the RMAN error RMAN-06019 – “could not translate tablespace name” as shown below.

    SQL> drop tablespace rman10 including contents and datafiles;
    
    Tablespace dropped.
    
    testdb:/u01/oracle/diag/rdbms/apex/apex/trace> rman target /
    
    Recovery Manager: Release 10.2.0.4.0 - Production on Mon Aug 3 11:53:58 2009
    
    Copyright (c) 1982, 2007, Oracle.  All rights reserved.
    
    connected to target database: TESTDB (DBID=2469552796)
    
    RMAN> restore tablespace rman10;
    
    Starting restore at 03-AUG-09
    using target database control file instead of recovery catalog
    allocated channel: ORA_SBT_TAPE_1
    channel ORA_SBT_TAPE_1: sid=141 devtype=SBT_TAPE
    channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.5.1.0
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=140 devtype=DISK
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of restore command at 08/03/2009 11:54:11
    RMAN-20202: tablespace not found in the recovery catalog
    

    So to recover from a dropped tablespace, we have two options:

    1) Do a point in time recovery of the whole database until the time the tablespace was dropped.
    2) Create a clone of the database from a valid backup, export the required tables from the tablespace which has been dropped, recreate the tablespace and then import the tables from the clone.

    The first option will require an outage of the entire database and the entire database will be rolled back in tine in order to recover the tablespace. The second option can be peformed online, but we will need to factor in the disk space requirements to create a clone of the database from which the tablespace has been dropped.

    Let us examine the first option using the example shown below:

    In this example, CONTROLFILE AUTOBACKUP has been turned on and Flashback has been enabled for the database.

    With Flashback enabled, the db_recovery_file_dest will have a sub-directory ‘autobackup’ as shown below for each day

    ttestdb:/u02/oradata/testdb/TESTDB/autobackup/2009_08_03> ls -lrt
    total 63040
    -rw-r-----    1 oracle   dba         6455296 Aug  3 10:22 o1_mf_s_693915680_57dlgcqh_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 11:49 o1_mf_s_693920955_57dqkw0j_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 13:28 o1_mf_s_693926889_57dxcbdx_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 14:18 o1_mf_s_693928526_57f094n9_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 14:20 o1_mf_s_693930026_57f0fbo2_.bkp
    

    When we drop the tablespace we are changing the structure of the database and since controlfile autobackup has been turned on, we see another backup file has been created in the autobackup location in the flash recovery area on disk.

    SQL> drop tablespace arul including contents and datafiles;
    
    Tablespace dropped.
    
    SQL> quit
    Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
    With the Partitioning, OLAP, Data Mining and Real Application Testing options
    testdb:/u02/oradata/testdb/TESTDB/autobackup/2009_08_03> ls -lrt
    total 75648
    -rw-r-----    1 oracle   dba         6455296 Aug  3 10:22 o1_mf_s_693915680_57dlgcqh_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 11:49 o1_mf_s_693920955_57dqkw0j_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 13:28 o1_mf_s_693926889_57dxcbdx_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 14:18 o1_mf_s_693928526_57f094n9_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 14:20 o1_mf_s_693930026_57f0fbo2_.bkp
    -rw-r-----    1 oracle   dba         6455296 Aug  3 14:38 o1_mf_s_693931114_57f1hbmo_.bkp
    

    We then shutdown the database, startup in nomount mode and attempt to restore the controlfile from autobackup.

    The most recent controlfile autobackup has been restored, but since this has been taken after the tablespace was dropped, the tablespace which has been dropped (ARUL) is not referenced in the control file that we just restored. If we try to restore and recover the database, the dropped tablespace will not be restored.

    SQL> startup nomount;
    ORACLE instance started.
    
    Total System Global Area  264241152 bytes
    Fixed Size                  2083304 bytes
    Variable Size             142607896 bytes
    Database Buffers          113246208 bytes
    Redo Buffers                6303744 bytes
    
    RMAN> restore controlfile from autobackup;
    
    Starting restore at 03-AUG-09
    using target database control file instead of recovery catalog
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=156 devtype=DISK
    
    recovery area destination: /u02/oradata/testdb/
    database name (or database unique name) used for search: TESTDB
    channel ORA_DISK_1: autobackup found in the recovery area
    channel ORA_DISK_1: autobackup found: /u02/oradata/testdb/TESTDB/autobackup/2009_08_03/o1_mf_s_693931114_57f1hbmo_.bkp
    channel ORA_DISK_1: control file restore from autobackup complete
    output filename=/u02/oradata/testdb/control01.ctl
    output filename=/u02/oradata/testdb/control02.ctl
    output filename=/u02/oradata/testdb/control03.ctl
    Finished restore at 03-AUG-09
    
    RMAN> alter database mount;
    
    database mounted
    released channel: ORA_DISK_1
    
    RMAN> report schema;
    ...
    
    List of Permanent Datafiles
    ===========================
    File Size(MB) Tablespace           RB segs Datafile Name
    ---- -------- -------------------- ------- ------------------------
    1    1230     SYSTEM               ***     /u02/oradata/testdb/system01.dbf
    2    1700     UNDOTBS1             ***     /u02/oradata/testdb/undotbs01.dbf
    3    370      SYSAUX               ***     /u02/oradata/testdb/sysaux01.dbf
    4    280      USERS                ***     /u02/oradata/backup/bkp.04klgv2b
    5    131      EXAMPLE              ***     /u02/oradata/testdb/example01.dbf
    6    150      USERS                ***     /u02/oradata/backup/bkp.06klgv3k
    9    100      USERS                ***     /u02/oradata/backup/bkp.08klgv4i
    

    We will need to restore a backup of the controlfile which contains records for the tablespace ARUL. We use the RESTORE CONTROLFILE FROM command to restore a specific controlfile autobackup.

    RMAN>  restore controlfile from '/u02/oradata/testdb/TESTDB/autobackup/2009_08_03/o1_mf_s_693930026_57f0fbo2_.bkp';
    
    Starting restore at 03-AUG-09
    using channel ORA_DISK_1
    
    channel ORA_DISK_1: restoring control file
    channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
    output filename=/u02/oradata/testdb/control01.ctl
    output filename=/u02/oradata/testdb/control02.ctl
    output filename=/u02/oradata/testdb/control03.ctl
    Finished restore at 03-AUG-09
    
    RMAN> report schema;
    .....
    
    List of Permanent Datafiles
    ===========================
    File Size(MB) Tablespace           RB segs Datafile Name
    ---- -------- -------------------- ------- ------------------------
    1    1230     SYSTEM               ***     /u02/oradata/testdb/system01.dbf
    2    1700     UNDOTBS1             ***     /u02/oradata/testdb/undotbs01.dbf
    3    370      SYSAUX               ***     /u02/oradata/testdb/sysaux01.dbf
    4    280      USERS                ***     /u02/oradata/backup/bkp.04klgv2b
    5    131      EXAMPLE              ***     /u02/oradata/testdb/example01.dbf
    6    150      USERS                ***     /u02/oradata/backup/bkp.06klgv3k
    7    0        ARUL                 ***     /u02/oradata/testdb/arul01.dbf
    9    100      USERS                ***     /u02/oradata/backup/bkp.08klgv4i
    

    The alert log will also show the time when the tablespace was dropped. We can also see that a controlfile autobackup has taken place after the tablespace was dropped.

    drop tablespace arul including contents and datafiles
    Mon Aug  3 14:38:34 2009
    Deleted file /u02/oradata/testdb/arul01.dbf
    Starting control autobackup
    Control autobackup written to DISK device
            handle '/u02/oradata/testdb/TESTDB/autobackup/2009_08_03/o1_mf_s_693931114_57f1hbmo_.bkp'
    Completed: drop tablespace arul including contents and datafiles
    

    Now that we know the time the tablespace was dropped, we can do a point in time recovery of the DATABASE in order to recover the tablespace which has been dropped.

    RMAN> run {
    2> set until time "to_date('03-AUG-2009 14:38:00','DD-MON-YYYY HH24:Mi:SS')";
    3> restore database;
    4> recover database;
    5> }
    
    executing command: SET until clause
    using target database control file instead of recovery catalog
    
    Starting restore at 03-AUG-09
    allocated channel: ORA_SBT_TAPE_1
    channel ORA_SBT_TAPE_1: sid=159 devtype=SBT_TAPE
    channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.5.1.0
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=155 devtype=DISK
    
    channel ORA_DISK_1: restoring datafile 00004
    input datafile copy recid=14 stamp=693929215 filename=/u02/oradata/testdb/users01.dbf
    destination for restore of datafile 00004: /u02/oradata/backup/bkp.04klgv2b
    channel ORA_SBT_TAPE_1: starting datafile backupset restore
    channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
    restoring datafile 00001 to /u02/oradata/testdb/system01.dbf
    restoring datafile 00002 to /u02/oradata/testdb/undotbs01.dbf
    restoring datafile 00003 to /u02/oradata/testdb/sysaux01.dbf
    restoring datafile 00005 to /u02/oradata/testdb/example01.dbf
    restoring datafile 00007 to /u02/oradata/testdb/arul01.dbf
    channel ORA_SBT_TAPE_1: reading from backup piece 0gkloo6p_1_1
    channel ORA_DISK_1: copied datafile copy of datafile 00004
    output filename=/u02/oradata/backup/bkp.04klgv2b recid=21 stamp=693932732
    channel ORA_DISK_1: restoring datafile 00006
    input datafile copy recid=13 stamp=693929146 filename=/u02/oradata/testdb/users02.dbf
    destination for restore of datafile 00006: /u02/oradata/backup/bkp.06klgv3k
    channel ORA_DISK_1: copied datafile copy of datafile 00006
    output filename=/u02/oradata/backup/bkp.06klgv3k recid=23 stamp=693932755
    channel ORA_DISK_1: restoring datafile 00009
    input datafile copy recid=10 stamp=693929108 filename=/u02/oradata/testdb/users03.dbf
    destination for restore of datafile 00009: /u02/oradata/backup/bkp.08klgv4i
    channel ORA_DISK_1: copied datafile copy of datafile 00009
    output filename=/u02/oradata/backup/bkp.08klgv4i recid=26 stamp=693932809
    channel ORA_SBT_TAPE_1: restored backup piece 1
    piece handle=0gkloo6p_1_1 tag=TAG20090803T113241
    channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:02:40
    Finished restore at 03-AUG-09
    
    Starting recover at 03-AUG-09
    using channel ORA_SBT_TAPE_1
    using channel ORA_DISK_1
    
    starting media recovery
    
    archive log thread 1 sequence 8 is already on disk as file /u02/oradata/testdb/arch/arch.8.1.693662800.log
    archive log thread 1 sequence 9 is already on disk as file /u02/oradata/testdb/arch/arch.9.1.693662800.log
    archive log thread 1 sequence 10 is already on disk as file /u02/oradata/testdb/arch/arch.10.1.693662800.log
    archive log thread 1 sequence 1 is already on disk as file /u02/oradata/testdb/redo01.log
    archive log thread 1 sequence 2 is already on disk as file /u02/oradata/testdb/redo02.log
    archive log filename=/u02/oradata/testdb/arch/arch.8.1.693662800.log thread=1 sequence=8
    archive log filename=/u02/oradata/testdb/arch/arch.9.1.693662800.log thread=1 sequence=9
    archive log filename=/u02/oradata/testdb/arch/arch.10.1.693662800.log thread=1 sequence=10
    archive log filename=/u02/oradata/testdb/redo01.log thread=1 sequence=1
    archive log filename=/u02/oradata/testdb/redo02.log thread=1 sequence=2
    media recovery complete, elapsed time: 00:00:06
    Finished recover at 03-AUG-09
    
    RMAN>
    
    RMAN> alter database open resetlogs;
    
    database opened
    

    We can now see that the tablespace which has been dropped has been recovered

    SQL> select file_name,bytes from dba_data_files where
      2  tablespace_name='ARUL';
    
    FILE_NAME
    --------------------------------------------------------------------------------
         BYTES
    ----------
    /u02/oradata/testdb/arul01.dbf
      37748736
    

    Posted in Backup and Recovery | Tagged: , , , | Leave a Comment »

    11g ADR – Viewing and Maintaining the alert log file

    Posted by Gavin Soorma on August 3, 2009

    In Oracle 11g, the init.ora parameters like user_dump_dest and background_dump_dest are deprecated. They have been replaced by the single parameter DIAGNOSTIC_DEST which identifies the location of the ADR.

    The ADR is the Automatic Diagnostic Repository which is a file based hierarchical respository of data and information related the the database alert, trace and dump files, health monitor reports (new in 11g), network trace and log information, diagnostic data related to the 11g Incident Packaging Service etc.

    The ADR can be managed via the 11g Enterprise Manager GUI (Database Control and not Grid Control) or via the ADR command line interpreter adrci

    The DIAGNOSTIC_DEST parameter identifies the location of the ADR and it defaults to the $ORACLE_HOME/log or $ORACLE_BASE if this environment variable has been set.

    Let us assign this location the environment variable $ADR_DUMP.

    The alert log is stored in both XML as well as standard text format in the ADR. In this example the database name is ‘apex’.

    The XML alert.log is located at $ADR_DUMP/diag/rdbms/apex/apex/alert and is called ‘log.xml’. The current alert log contents will be found in the file ‘log.xml’ while the older entries are archived every 10 MB into files like ‘log_1.xml’, ‘log_2.xml’ and so on.

    The standard text alert log file (alert_apex.log) is located at $ADR_DUMP/diag/rdbms/apex/apex/trace and is mantained for backward compatability purposes.

    The view V$DIAG_INFO also has information on the location of the various components of the ADR.

    SQL> col value format a50
    SQL>  SELECT NAME,VALUE FROM v$diag_info
      2   WHERE NAME LIKE '%Alert%';
    
    NAME                 VALUE
    -------------------- --------------------------------------------------
    Diag Alert           /u01/oracle/diag/rdbms/apex/apex/alert
    

    Using the ADR CLI to view Alert Log Information

    The command ‘adrci’ will take us to the prompt.

    apex:/u01/oracle/diag/rdbms/apex/apex/alert> adrci

    ADRCI: Release 11.1.0.6.0 – Beta on Mon Aug 3 09:46:52 2009

    Copyright (c) 1982, 2007, Oracle. All rights reserved.

    ADR base = “/u01/oracle”
    adrci>

    Many of the adrci commands are very similar to the standard Unix vi commands. For example

    SHOW ALERT -TAIL (to view the last 10 entries)

    SHOW ALERT -TAIL 50 (to view the last 50 entries)

    SHOW ALERT -TAIL -F (keeps the alert log open to view as entries arrive until Control C is pressed)

    show alert -p “message_text like ‘%ORA-07445%’” (to show all occurences of the string ‘ORA-07445′ in the alert log.)

    Purging Alert Log Content

    The adrci command ‘purge’ can be used to purge entries from the alert log. Note that this purge will only apply to the XML based alert log and not the text file based alert log which still has to be maintained using OS commands.

    The purge command takes the input in minutes and specifies the number of minutes for which records should be retained.

    So to purge all alert log entries older than 7 days the following command will be used:

    adrci > purge -age 10080 -type ALERT

    Posted in Oracle 11g | Tagged: , , , | Leave a Comment »

    RMAN Recovery using the SWITCH DATABASE TO COPY command

    Posted by Gavin Soorma on July 31, 2009

    By using Optimized Incremental Backup to disk backup strategy, an up to date copy of the database is always available on disk. In the event of a failure, the SWITCH command will point the controlfile to the backup of the datafiles that are present on disk.

    In this way, we can significantly reduce the downtime related to restoring large datafiles from tape in the event of a media failure.

    The downside of this is that we need to consider the disk space (and cost) of additional disks which will be required to hold the recovered incremental copies of backups on disk. Also as we will see, the SWITCH DATABASE TO COPY command will alter the names of all the datafiles and there is manual work involved in renaming them back to their original datafile names.

    The following RMAN command can be used to take an optimized incremental backup to disk. Note in the first run, since no copy of any of the datafiles are found since this is the first backup, no recovery is performed.

    But subsequent backups will find that the backup copies of the datafiles will undergo recovery as changes which have occurred since the last incremental backup are applied to them making them ‘current’.

     run {
    allocate channel c1 device type disk format '/u02/oradata/backup/bkp.%u';
    recover copy of database with tag 'LEVEL0_BKP';
    backup incremental level 1 for recover of copy with tag 'LEVEL0_BKP' database;
    }
    
    RMAN>  run {
    allocate channel c1 device type disk format '/u02/oradata/backup/bkp.%u';
    recover copy of database with tag 'LEVEL0_BKP';
    backup incremental level 1 for recover of copy with tag 'LEVEL0_BKP' database;
    }
    2> 3> 4> 5>
    using target database control file instead of recovery catalog
    allocated channel: c1
    channel c1: sid=143 devtype=DISK
    
    Starting recover at 31-JUL-09
    no copy of datafile 1 found to recover
    no copy of datafile 2 found to recover
    no copy of datafile 3 found to recover
    no copy of datafile 4 found to recover
    no copy of datafile 5 found to recover
    no copy of datafile 6 found to recover
    no copy of datafile 7 found to recover
    no copy of datafile 8 found to recover
    no copy of datafile 9 found to recover
    Finished recover at 31-JUL-09
    
    Starting backup at 31-JUL-09
    no parent backup or copy of datafile 2 found
    no parent backup or copy of datafile 1 found
    no parent backup or copy of datafile 3 found
    no parent backup or copy of datafile 4 found
    no parent backup or copy of datafile 8 found
    no parent backup or copy of datafile 6 found
    .......
    .......
    
    RMAN>  run {
    allocate channel c1 device type disk format '/u02/oradata/backup/bkp.%u';
    recover copy of database with tag 'LEVEL0_BKP';
    backup incremental level 1 for recover of copy with tag 'LEVEL0_BKP' database;
    }2> 3> 4> 5>
    
    allocated channel: c1
    channel c1: sid=140 devtype=DISK
    
    Starting recover at 31-JUL-09
    channel c1: starting incremental datafile backupset restore
    channel c1: specifying datafile copies to recover
    recovering datafile copy fno=00001 name=/u02/oradata/backup/bkp.2qklgmt6
    recovering datafile copy fno=00002 name=/u02/oradata/backup/bkp.2pklgmpt
    recovering datafile copy fno=00003 name=/u02/oradata/backup/bkp.2rklgmvr
    recovering datafile copy fno=00004 name=/u02/oradata/backup/bkp.2sklgn0u
    recovering datafile copy fno=00005 name=/u02/oradata/backup/bkp.2vklgn30
    recovering datafile copy fno=00006 name=/u02/oradata/backup/bkp.2uklgn2h
    recovering datafile copy fno=00007 name=/u02/oradata/backup/bkp.31klgn3f
    recovering datafile copy fno=00008 name=/u02/oradata/backup/bkp.2tklgn1o
    recovering datafile copy fno=00009 name=/u02/oradata/backup/bkp.30klgn37
    channel c1: reading from backup piece /u02/oradata/backup/bkp.33klgnda
    channel c1: restored backup piece 1
    piece handle=/u02/oradata/backup/bkp.33klgnda tag=TAG20090731T103002
    channel c1: restore complete, elapsed time: 00:00:08
    Finished recover at 31-JUL-09
    
    Starting backup at 31-JUL-09
    channel c1: starting incremental level 1 datafile backupset
    channel c1: specifying datafile(s) in backupset
    input datafile fno=00002 name=/u02/oradata/testdb/undotbs01.dbf
    input datafile fno=00001 name=/u02/oradata/testdb/system01.dbf
    input datafile fno=00003 name=/u02/oradata/testdb/sysaux01.dbf
    input datafile fno=00004 name=/u02/oradata/testdb/users01.dbf
    input datafile fno=00008 name=/u02/oradata/testdb/rman10.dbf
    input datafile fno=00006 name=/u02/oradata/testdb/users02.dbf
    input datafile fno=00005 name=/u02/oradata/testdb/example01.dbf
    input datafile fno=00009 name=/u02/oradata/testdb/users03.dbf
    ......
    ......
    

    Let us simulate a database failure by removing all the datafiles at the OS level.

    We will then use the SWITCH DATABASE TO COPY command to point the database to the backups which are available on disk and note how the recovery is performed in a few seconds without any any backups having to be restored from tape and no archive log files also being applied.

    testdb:/u02/oradata> cd /u02/oradata/testdb/
    testdb:/u02/oradata/testdb> rm users*.dbf

    We then shutdown and mount the database

    SQL> shutdown abort
    ORACLE instance shut down.
    SQL> startup mount;
    ORACLE instance started.

    Total System Global Area 264241152 bytes
    Fixed Size 2083304 bytes
    Variable Size 146802200 bytes
    Database Buffers 109051904 bytes
    Redo Buffers 6303744 bytes
    Database mounted.

    RMAN> switch database to copy;
    
    using target database control file instead of recovery catalog
    datafile 1 switched to datafile copy "/u02/oradata/backup/bkp.2qklgmt6"
    datafile 2 switched to datafile copy "/u02/oradata/backup/bkp.2pklgmpt"
    datafile 3 switched to datafile copy "/u02/oradata/backup/bkp.2rklgmvr"
    datafile 4 switched to datafile copy "/u02/oradata/backup/bkp.2sklgn0u"
    datafile 5 switched to datafile copy "/u02/oradata/backup/bkp.2vklgn30"
    datafile 6 switched to datafile copy "/u02/oradata/backup/bkp.2uklgn2h"
    datafile 7 switched to datafile copy "/u02/oradata/backup/bkp.31klgn3f"
    datafile 8 switched to datafile copy "/u02/oradata/backup/bkp.2tklgn1o"
    datafile 9 switched to datafile copy "/u02/oradata/backup/bkp.30klgn37"
    
    RMAN> recover database;
    
    Starting recover at 31-JUL-09
    allocated channel: ORA_SBT_TAPE_1
    channel ORA_SBT_TAPE_1: sid=156 devtype=SBT_TAPE
    channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.5.1.0
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=155 devtype=DISK
    channel ORA_DISK_1: starting incremental datafile backupset restore
    channel ORA_DISK_1: specifying datafile(s) to restore from backup set
    destination for restore of datafile 00001: /u02/oradata/backup/bkp.2qklgmt6
    destination for restore of datafile 00002: /u02/oradata/backup/bkp.2pklgmpt
    destination for restore of datafile 00003: /u02/oradata/backup/bkp.2rklgmvr
    destination for restore of datafile 00004: /u02/oradata/backup/bkp.2sklgn0u
    destination for restore of datafile 00005: /u02/oradata/backup/bkp.2vklgn30
    destination for restore of datafile 00006: /u02/oradata/backup/bkp.2uklgn2h
    destination for restore of datafile 00007: /u02/oradata/backup/bkp.31klgn3f
    destination for restore of datafile 00008: /u02/oradata/backup/bkp.2tklgn1o
    destination for restore of datafile 00009: /u02/oradata/backup/bkp.30klgn37
    channel ORA_DISK_1: reading from backup piece /u02/oradata/backup/bkp.35klgnj6
    channel ORA_DISK_1: restored backup piece 1
    piece handle=/u02/oradata/backup/bkp.35klgnj6 tag=TAG20090731T103309
    channel ORA_DISK_1: restore complete, elapsed time: 00:00:02
    
    starting media recovery
    media recovery complete, elapsed time: 00:00:01
    
    Finished recover at 31-JUL-09
    
    RMAN> alter database open;
    
    database opened
    

    After the SWITCH command has completed, we can now see that the datafile names have changed.

    I have not been able to identify an easy way to rename them back to their original names other than recreating the controlfile and renaming all the datafiles at the OS level as well. This could be an issue when we have hundreds of datafiles to rename.

     1* select name,status from v$datafile
    SQL> /
    
    NAME                                     STATUS
    ---------------------------------------- -------
    /u02/oradata/backup/bkp.2qklgmt6         SYSTEM
    /u02/oradata/backup/bkp.2pklgmpt         ONLINE
    /u02/oradata/backup/bkp.2rklgmvr         ONLINE
    /u02/oradata/backup/bkp.2sklgn0u         ONLINE
    /u02/oradata/backup/bkp.2vklgn30         ONLINE
    /u02/oradata/backup/bkp.2uklgn2h         ONLINE
    /u02/oradata/backup/bkp.31klgn3f         ONLINE
    /u02/oradata/backup/bkp.2tklgn1o         ONLINE
    /u02/oradata/backup/bkp.30klgn37         ONLINE
    

    Similarly, we can also use the SWITCH TABLEPACE TO COPY command to quickly recover from the loss of all datafiless at the tablespace level.

    We simulate this by removing all the datafiles for the USERS tablespace at the OS level.

    testdb:/u02/oradata/testdb> rm users*.dbf

    SQL> conn scott/tiger
    Connected.
    SQL> create table emp2
    2 tablespace users
    3 as select * from emp;
    as select * from emp
    *
    ERROR at line 3:
    ORA-01116: error in opening database file 4
    ORA-01110: data file 4: ‘/u02/oradata/testdb/users01.dbf’
    ORA-27041: unable to open file
    IBM AIX RISC System/6000 Error: 2: No such file or directory
    Additional information: 3

    RMAN>  run {
    2> sql 'alter tablespace users offline immediate';3>
    }
    
    sql statement: alter tablespace users offline immediate
    
    RMAN> switch tablespace users to copy;
    
    datafile 4 switched to datafile copy "/u02/oradata/backup/bkp.04klgv2b"
    datafile 6 switched to datafile copy "/u02/oradata/backup/bkp.06klgv3k"
    datafile 9 switched to datafile copy "/u02/oradata/backup/bkp.08klgv4i"
    
    RMAN> recover tablespace users;
    
    Starting recover at 31-JUL-09
    allocated channel: ORA_SBT_TAPE_1
    channel ORA_SBT_TAPE_1: sid=143 devtype=SBT_TAPE
    channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.5.1.0
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=141 devtype=DISK
    
    starting media recovery
    media recovery complete, elapsed time: 00:00:00
    
    Finished recover at 31-JUL-09
    
    RMAN> sql 'alter tablespace users online';
    
    sql statement: alter tablespace users online
    
    SQL> select file_name from dba_data_files where tablespace_name='USERS';
    
    FILE_NAME
    --------------------------------------------------------------------------------
    /u02/oradata/backup/bkp.08klgv4i
    /u02/oradata/backup/bkp.06klgv3k
    /u02/oradata/backup/bkp.04klgv2b
    

    Posted in Backup and Recovery | Tagged: , , , , | Leave a Comment »

    11g Optimizer Plan Stabilty using SQL Plan Baselines

    Posted by Gavin Soorma on July 30, 2009

    In Oracle 11g, we can ensure that the same proven execution plans are used by the Optimizer regardless of any change which can effect the optimizer like a version change, index drop or recreate, init.ora parameter change etc.
    Once plans are marked as enabled and accepted, the optimizer will not use any new plans unless there is proven performance benefits of adopting a new plan. Plans can also be manually adopted or “evolved”.

    One of the dangers of upgrading from 10g to 11g (or between any release) is that execution plans may change in the new version which could affect application performance.

    But with this new 11g feature, if we have tried and trusted 10g execution plans in place, we can ensure that the same plans will be used by the optimizer even after the migration unless new 11g plans are found to offer better performance. This can be achieved by capturing 10g plans in SQL tuning sets (STS) and exporting those to the 11g database.

    Let us demonstrate the same with a small example using the SALES table in the SH schema.

    We need to enable the use of SQL Plan management by setting the parameter optimizer_capture_sql_plan_baselines to TRUE. The default value is FALSE.

    Initially the OPTIMIZER_MODE is set to FIRST_ROWS which should optimize the statement for best response time. So in this case the index on the SALES table is used.

    SQL> ALTER SESSION SET optimizer_mode='FIRST_ROWS';
    
    Session altered.
    
    SQL> EXPLAIN PLAN FOR
      2  SELECT * FROM SALES
      3  WHERE QUANTITY_SOLD > 40 ORDER BY PROD_ID;
    
    Explained.
    
    SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY);
    
    | Id  | Operation                           | Name            | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
    ---------------------------------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT                    |                 |    29 |  5063   (2)| 00:01:01 |       |       |
    |   1 |  SORT ORDER BY                      |                 |    29 |  5063   (2)| 00:01:01 |       |       |
    |   2 |   PARTITION RANGE ALL               |                 |    29 |  5062   (2)| 00:01:01 |     1 |    28 |
    |*  3 |    TABLE ACCESS BY LOCAL INDEX ROWID| SALES           |    29 |  5062   (2)| 00:01:01 |     1 |    28 |
    |   4 |     BITMAP CONVERSION TO ROWIDS     |                 |            |          |       |       |
    |   5 |      BITMAP INDEX FULL SCAN         | SALES_PROMO_BIX |       |            |          |     1 |    28 |
    ---------------------------------------------------------------------------------------------------------------

    if we change the OPTIMIZER_MODE to ALL_ROWS, we now see from the explain plan that a FULL TABLE SCAN is being used as the statement is optimized for best throughput.

    SQL> ALTER SESSION SET optimizer_mode='ALL_ROWS';
    
    Session altered.
    
    SQL> EXPLAIN PLAN FOR
      2  SELECT * FROM SALES
      3  WHERE QUANTITY_SOLD > 40 ORDER BY PROD_ID;
    
    Explained.
    
    SQL> SELECT * FROM TABLE(DBMS_XPLAN.DISPLAY);
    
    PLAN_TABLE_OUTPUT
    ----------------------------------------------------------------------------------------------
    Plan hash value: 3147563666
    
    ----------------------------------------------------------------------------------------------
    | Id  | Operation            | Name  | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |
    ----------------------------------------------------------------------------------------------
    |   0 | SELECT STATEMENT     |       |     1 |    29 |  1196   (3)| 00:00:15 |       |       |
    |   1 |  SORT ORDER BY       |       |     1 |    29 |  1196   (3)| 00:00:15 |       |       |
    |   2 |   PARTITION RANGE ALL|       |     1 |    29 |  1195   (3)| 00:00:15 |     1 |    28 |
    |*  3 |    TABLE ACCESS FULL | SALES |     1 |    29 |  1195   (3)| 00:00:15 |     1 |    28 |
    ----------------------------------------------------------------------------------------------
    

    We will run the query – initially with OPTIMIZER_MODE set to FIRST_ROWS and then the same query with OPTIMIZER_MODE set to ALL_ROWS. We will use a label ‘GAVIN’ in the SQL statement to help us easily identify it from the other SQL statements in the shared pool.

    We explicitly flush the shared pool to force a hard parse the next time the same query is run but with the ALL_ROWS optimizer session setting.

    SQL> ALTER SESSION SET optimizer_mode='FIRST_ROWS';
    
    Session altered.
    
    SQL>  SELECT /* GAVIN */
      2   *
      3   FROM SALES WHERE QUANTITY_SOLD > 40
      4  ORDER BY PROD_ID;
    
    no rows selected
    
    SQL> ALTER SYSTEM FLUSH SHARED_POOL;
    
    System altered.
    
    SQL> ALTER SESSION SET optimizer_mode='ALL_ROWS';
    
    Session altered.
    
    SQL>  SELECT /* GAVIN */
      2   *
      3   FROM SALES WHERE QUANTITY_SOLD > 40
      4  ORDER BY PROD_ID;
    
    no rows selected
    

    We will notice that the second execution of the query has encountered a change in the execution plan. But it has not been ‘accepted as yet even though it has been ‘enabled’.

    SQL> SELECT sql_handle, plan_name, enabled, accepted FROM dba_sql_plan_baselines
    WHERE sql_text like '%GAVIN%';  2
    
                                                                  Ena-
    SQL_HANDLE                     PLAN_NAME                      bled Acpt
    ------------------------------ ------------------------------ ---- ----
    SYS_SQL_6f3dcd88c7488035       SYS_SQL_PLAN_c748803511df68d0  YES  YES
    SYS_SQL_6f3dcd88c7488035       SYS_SQL_PLAN_c748803554bc8843  YES  NO
    

    We can use the EVOLVE_SQL_PLAN_BASELINE function to compare the performance between the two plans. We note that the second plan has not been automatically evolved or accepted because it does not pass the performance improvement criteria which has been laid down

    SQL> SET LONG 10000
    SQL> SELECT DBMS_SPM.evolve_sql_plan_baseline(sql_handle => 'SYS_SQL_6f3dcd88c7488035') FROM dual;
    
    DBMS_SPM.EVOLVE_SQL_PLAN_BASELINE(SQL_HANDLE=>'SYS_SQL_6F3DCD88C7488035')
    --------------------------------------------------------------------------------
    
    -------------------------------------------------------------------------------
                            Evolve SQL Plan Baseline Report
    -------------------------------------------------------------------------------
    
    Inputs:
    -------
      SQL_HANDLE = SYS_SQL_6f3dcd88c7488035
      PLAN_NAME  =
      TIME_LIMIT = DBMS_SPM.AUTO_LIMIT
      VERIFY     = YES
      COMMIT     = YES
    
    Plan: SYS_SQL_PLAN_c748803554bc8843
    -----------------------------------
      Plan was verified: Time used 1.457 seconds.
      Failed performance criterion: Compound improvement ratio <= 1.16.
    
                          Baseline Plan      Test Plan     Improv. Ratio
                          -------------      ---------     -------------
      Execution Status:        COMPLETE       COMPLETE
      Rows Processed:                 0              0
      Elapsed Time(ms):            1036            304              3.41
      CPU Time(ms):                1020            300               3.4
      Buffer Gets:                 1929           1727              1.12
      Disk Reads:                     0              0
      Direct Writes:                  0              0
      Fetches:                        0              0
      Executions:                     1              1
    
    -------------------------------------------------------------------------------
                                     Report Summary
    -------------------------------------------------------------------------------
    Number of SQL plan baselines verified: 1.
    Number of SQL plan baselines evolved: 0.
    

    We can also manually evolve the plan if we ourselves feel that we know that the plan is a better one

    SQL> var spm number;
    
    SQL> exec :spm := dbms_spm.alter_sql_plan_baseline(sql_handle =>'SYS_SQL_6f3dcd88c7488035',
    - plan_name =>'SYS_SQL_PLAN_c748803554bc8843', attribute_name => 'ACCEPTED',attribute_value => 'YES');
    
    PL/SQL procedure successfully completed.
    
    SQL> SELECT sql_handle, plan_name, enabled, accepted FROM dba_sql_plan_baselines
      2  WHERE sql_text like '%GAVIN%';
    
                                                                  Ena-
    SQL_HANDLE                     PLAN_NAME                      bled Acpt
    ------------------------------ ------------------------------ ---- ----
    SYS_SQL_6f3dcd88c7488035       SYS_SQL_PLAN_c748803511df68d0  YES  YES
    SYS_SQL_6f3dcd88c7488035       SYS_SQL_PLAN_c748803554bc8843  YES  YES
    

    Posted in Oracle 11g, Performance Tuning | Tagged: , , , , | Leave a Comment »

    Drop and Recreate Online Redolog Files

    Posted by Arjun Raja on July 29, 2009

    Method to drop and recreate online redolog files with 2 members to each group.

    Firstly ORACLE will never allow you to drop the current ONLINE redolog file -

    Ex :

    SQL> select GROUP#,THREAD#,STATUS from v$log;
    

    GROUP# THREAD# STATUS
    ———- ———- —————-
    1 1 CURRENT
    2 1 UNUSED
    3 1 INACTIVE
    4 1 INACTIVE

    SQL> alter database drop logfile group 1;
    

    alter database drop logfile group 1
    *
    ERROR at line 1:
    ORA-01623: log 1 is current log for instance test (thread 1) – cannot drop
    ORA-00312: online log 1 thread 1: ‘/u03/oradata/testdb/redo01.log’

    Now to drop and recreate online redolog files in the same directory -

    SQL> select  member from v$logfile;
    

    MEMBER
    ——————————————————————————–
    /u50/oradata/test/redo4a.log
    /u51/oradata/test/redo4b.log
    /u50/oradata/test/redo3a.log
    /u51/oradata/test/redo3b.log
    /u50/oradata/test/redo2a.log
    /u51/oradata/test/redo2b.log
    /u50/oradata/test/redo1a.log
    /u51/oradata/test/redo1b.log

    8 rows selected.

    Intention is to drop and recreate these online logs with a different size.

    SQL> select GROUP#,THREAD#,STATUS from v$log;
    

    GROUP# THREAD# STATUS
    ———- ———- —————-
    1 1 INACTIVE
    2 1 INACTIVE
    3 1 CURRENT
    4 1 INACTIVE

    Begin dropping the redolog groups with STATUS=’INACTIVE’ – As mentioned earlier Oracle will not allow you to drop a current online redolog file.

    sql > alter database drop logfile group 1;
    

    cd /u50/oradata/test

    rm redo1a.log

    cd /u51/oradata/test

    rm redo1b.log

    Recreate group with both members -

    sql > alter database add logfile group 1('/u50/oradata/test/log1a.ora','/u51/oradata/test/log1b.ora')size 2048m;
    

    Do the same for the other three groups -

    To change the status of a log group from CURRENT to INACTIVE, simply switch a logfile with this command-

    sql> alter system switch logfile;

    Run the command 2 or 3 times if the group status does not change.

    To check status of group remember the command is

    sql > select GROUP#,THREAD#,STATUS from v$log;
    

    Also make sure the logfile is physically removed from the disk with the rm command – if you don’t do this and try to recreate a logfile an error will appear similar to this -

    Ex: You dropped log group 3 but did not remove the physical file redo03.log from disk /u03/oradata/test and tried to recreate the group.

    SQL> alter database add logfile group3 '/u03/oradata/test/redo03.log' size 50m;
    

    alter database add logfile ‘/u03/oradata/test/redo03.log’ size 50m
    *
    ERROR at line 1:
    ORA-01577: cannot add log file ‘/u03/oradata/test/redo03.log’ – file already
    part of database

    Posted in Administration | Tagged: , , | Leave a Comment »

    RMAN-06563: control file or SPFILE must be restored using FROM AUTOBACKUP

    Posted by Gavin Soorma on July 28, 2009

    Backup and Recovery best practices dictate that we must use a RMAN recovery catalog and also have the controlfile AUTOBACKUP enabled.

    If we do not do either and we lose all the controlfiles, we cannot restore the controlfiles even if we have taken a backup to tape as shown in the case below.

    We will encounter the RMAN-06563 error even if we set the DBID or explicitly alllocate a channel for a tape device.

      set dbid=693232013;
    
    executing command: SET DBID
    
    RMAN> run {
    2> restore controlfile;
    3> recover database;
    4> }
    
    Starting restore at 28-JUL-09
    using target database control file instead of recovery catalog
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=159 devtype=DISK
    
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of restore command at 07/28/2009 12:17:19
    RMAN-06563: control file or SPFILE must be restored using FROM AUTOBACKUP
    
    RMAN>  run {
    2>  allocate channel c1 device type  sbt_tape;
    3> restore controlfile;
    4> alter database mount;
    5> recover database;
    6> }
    
    released channel: ORA_DISK_1
    allocated channel: c1
    channel c1: sid=159 devtype=SBT_TAPE
    channel c1: Data Protection for Oracle: version 5.5.1.0
    
    Starting restore at 28-JUL-09
    
    released channel: c1
    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of restore command at 07/28/2009 12:19:36
    RMAN-06563: control file or SPFILE must be restored using FROM AUTOBACKUP
    

    However, if we connect to a RMAN recovery catalog, we can restore a controlfile without using the AUTOBACKUP keyword.

    $ rman target / catalog rman9p/xxx@rcatp
    
    Recovery Manager: Release 9.2.0.8.0 - 64bit Production
    
    Copyright (c) 1995, 2002, Oracle Corporation.  All rights reserved.
    
    connected to target database: ofsap (not mounted)
    connected to recovery catalog database
    
    RMAN> run {
    2> allocate channel ch1 type 'sbt_tape' parms 'ENV=(TDPO_OPTFILE=/usr/tivoli/tsm/client/oracle/bin64/tdpo.opt)';
    3> restore controlfile;
    4> release channel ch1;
    5> }
    
    allocated channel: ch1
    channel ch1: sid=8 devtype=SBT_TAPE
    channel ch1: Data Protection for Oracle: version 5.4.1.0
    
    Starting restore at 27-JUN-08
    
    channel ch1: starting datafile backupset restore
    channel ch1: restoring controlfile
    output filename=/u04/oradata/ofsap/control01.ctl
    channel ch1: restored backup piece 1
    piece handle=c-2764499561-20080627-08 tag=null params=NULL
    channel ch1: restore complete
    replicating controlfile
    input filename=/u04/oradata/ofsap/control01.ctl
    output filename=/u04/oradata/ofsap/control02.ctl
    Finished restore at 27-JUN-08
    
    released channel: ch1
    

    Posted in Backup and Recovery | Tagged: , , , , , | Leave a Comment »

    Script – Check RMAN Backup Status

    Posted by Arjun Raja on July 28, 2009

    Scripts to check backup status and timings of database backups -

    This script will be run in the database, not the catalog.

    Login as sysdba -

    This script will report on all backups – full, incremental and archivelog backups -

    col STATUS format a9
    col hrs format 999.99
    select
    SESSION_KEY, INPUT_TYPE, STATUS,
    to_char(START_TIME,'mm/dd/yy hh24:mi') start_time,
    to_char(END_TIME,'mm/dd/yy hh24:mi')   end_time,
    elapsed_seconds/3600                   hrs
    from V$RMAN_BACKUP_JOB_DETAILS
    order by session_key;
    

    This script will report all on full and incremental backups, not archivelog backups -

    col STATUS format a9
    col hrs format 999.99
    select
    SESSION_KEY, INPUT_TYPE, STATUS,
    to_char(START_TIME,'mm/dd/yy hh24:mi') start_time,
    to_char(END_TIME,'mm/dd/yy hh24:mi')   end_time,
    elapsed_seconds/3600                   hrs
    from V$RMAN_BACKUP_JOB_DETAILS
    where input_type='DB INCR'
    order by session_key;
    

    Posted in Administration, Backup and Recovery, Scripts | Tagged: , , | Leave a Comment »

    Troubleshooting Grid Control Agent issues with startup

    Posted by Arjun Raja on July 28, 2009

    Sometimes the GRID CONTROL agent will not start because an old HTTP process is still running on the host – this usually happens if the agent has crashed for any reason.

    Example -

    Check status of agent.

    oracle(DATABASE)@hostname:./emctl status agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    —————————————————————
    Agent is Not Running

    START AGENT -

    oracle(DATABASE)@fhostname:./emctl start agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    Starting agent …… failed.
    Failed to start HTTP listener.
    Consult the log files in: /u01/oracle/agent10g/sysman/log

    To solve the problem.

    CHECK IF PORT 3872 – PORT USED BY AGENT IS IN USE.

    oracle(DATABASE)@hostname:netstat -an | grep 3872
    tcp4 0 0 *.3872 *.* LISTEN

    oracle(DATABASE)@hostname:ps -ef | grep emagent
    oracle 864486 1 0 Jan 27 – 5:36 /u01/oracle/agent10g/perl/bin/perl /u01/oracle/agent10g/bin/emwd.pl agent /u01/oracle/agent10g/sysman/log/emagent.nohup
    oracle 1593344 864486 0 Jan 27 – 97:38 /u01/oracle/agent10g/bin/emagent

    KILL ALL EMAGENT PROCESSES STILL RUNNING -

    oracle(DATABASE)@hostname:ps -ef | grep emagent | awk ‘ {print $2}’ | xargs kill -9

    oracle(DATABASE)@hostname:ps -ef | grep emagent| grep -v grep

    No emagent process running now.

    cd $AGENT_HOME/bin

    oracle(DATABASE)@hostname:pwd
    /u01/oracle/agent10g/bin

    START AGENT

    oracle(DATABASE)@hostname:./emctl start agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.

    AGENT WILL START.

    Posted in EM Grid Control | Tagged: , | Leave a Comment »

    Alert Log Test Message

    Posted by Arjun Raja on July 28, 2009

    FORCE MESSAGE TO BE WRITTEN TO THE ALERT LOG

    If you wish to force an alert to be written to the alert log- for example ORA-600 to test the functioning of a monitoring system you can use this command to force an ORA-600 alert to be written to the alert log.

    Login as sysdba

    exec dbms_system.ksdwrt(2,’ORA-00600: This is a test error message for monitoring and can be ignored.’);

    To check -

    cd $BDUMP

    view alert_SID.log and you will find the alert written to the log.

    Tue Jul 28 10:06:28 2009 ORA-00600: ” This is a test error message for monitoring and can be ignored “

    Posted in Administration | Tagged: , , | Leave a Comment »

    Exporting and Importing AWR snapshot data

    Posted by Gavin Soorma on July 25, 2009

    The AWR tables contains a wealth of important performance data which can be very useful in performance tuning trend analysis and also when comparing performance between two seperate periods of time.

    AWR data is stored in the WRH$ and DBA_HIST tables in the SYSAUX tablespace. There could be performance implications if these tables were to grow too large in size or if the retention was increased beyond the default of 7 days.

    A good solution is to have a central repository and move statistical AWR data periodically to this central repository database using the Oracle supplied awrextr.sql and awrload.sql scripts which can be found in the $ORACLE_HOME/rdbms/admin directory.

    The AWR History is by default maintained for 7 days and the data is gathered in the AWR repository tables every hour by default.

    The current snapshot retention settings and data gathering frequency can be determined by the query shown below. Note in this case the default settings of 7 days and 1 hour is displayed.

    SQL> select to_char(snap_interval,'DD'),to_char(retention,'DD') FROM dba_hist_wr_control;
    
    TO_CHAR(SNAP_INTER TO_CHAR(RETENTION,
    ------------------ ------------------
    +00000 01:00:00.0  +00007 00:00:00.0;
    

    The AWR default settings can be modified using the DBMS_WORKLOAD_REPOSITORY package as shown below. In this case the retention is being increased to 30 days and the interval to every 30 minutes.

    BEGIN
    DBMS_WORKLOAD_REPOSITORY.modify_snapshot_settings(
    retention => 43200,
    interval => 30);
    END;
    /
    Extracting AWR data

    Create a directory

    SQL> CREATE DIRECTORY AWR_DATA AS
    2 ‘/u01/oracle/’;

    Directory created.

    SQL> @?/rdbms/admin/awrextr.sql

    The script will list the information we need to provide to it

    AWR EXTRACT
    ~~~~~~~~~~~~~
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ~  This script will extract the AWR data for a range of snapshots  ~
    ~  into a dump file.  The script will prompt users for the         ~
    ~  following information:                                          ~
    ~     (1) database id                                              ~
    ~     (2) snapshot range to extract                                ~
    ~     (3) name of directory object                                 ~
    ~     (4) name of dump file                                        ~
    

    After entering the range of snapshot ids, we will need to provide the directory location where the data pump export file will be located. We need to also enter the dumpfile name as well.

    Note: the script will autiomatically append a ‘.dmp’ to the data punp export file name. So we need to just enter the dumpfile name without any extension.

    Specify the Directory Name
    ~~~~~~~~~~~~~~~~~~~~~~~~~~
    
    Directory Name                 Directory Path
    ------------------------------ -------------------------------------------------
    ADMIN_DIR                      /u01/oracle/product/10.2.0/rmand/md/admin
    AWR_DATA                       /u01/oracle/
    DATA_PUMP_DIR                  /u01/oracle/product/10.2.0/rmand/admin/rmand/dpdu
                                   mp/
    
    ORACLE_OCM_CONFIG_DIR          /u01/oracle/product/10.2.0.4/rmand/ccr/state
    WORK_DIR                       /u01/oracle/product/10.2.0/rmand/work
    
    Choose a Directory Name from the above list (case-sensitive).
    
    Enter value for directory_name: AWR_DATA
    
    Using the dump directory: AWR_DATA
    
    Specify the Name of the Extract Dump File
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    The prefix for the default dump file name is awrdat_11369_11383.
    To use this name, press  to continue, otherwise enter
    an alternative.
    
    Enter value for file_name: awrexp
    

    After the export is complete, we will need to ftp the data pump dump file awrexp.dmp to the target server where our central repository database is located.

    Loading AWR Data

    On the repository database, we will create a directiory AWR_DATA as well and the ensure that the DIRECTORY_PATH corresponds to the directory where the awrexport.dmp file is located.

    Also, a temporary staging schema AWR_STAGE is created. Objects are first imported into this staging schema and then inserted into the WR$ and DBA_HIST AWR historical tables.

    We will need to provide information about the directory location, dump file and the staging schema name

    @?/rdbms/admin/awrload.sql
    
    AWR LOAD
    ~~~~~~~~~~
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    ~  This script will load the AWR data from a dump file. The   ~
    ~  script will prompt users for the following information:    ~
    ~     (1) name of directory object                            ~
    ~     (2) name of dump file                                   ~
    ~     (3) staging schema name to load AWR data into           ~
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

    While specifying the dumpfile name we need to only provide the file name without the .dmp extension

    Specify the Name of the Dump File to Load
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Please specify the prefix of the dump file (.dmp) to load:
    

    We need to specify the default tablespace and temporary tablespace for this staging user. Note – this user will be dropped once the load is completed.

    Choose the AWR_STAGE users's default tablespace.  This is the
    tablespace in which the AWR data will be staged.
    
    TABLESPACE_NAME                CONTENTS  DEFAULT TABLESPACE
    ------------------------------ --------- ------------------
    MGMT_ECM_DEPOT_TS              PERMANENT
    MGMT_TABLESPACE                PERMANENT
    PATROL                         PERMANENT
    SYSAUX                         PERMANENT *
    USERS                          PERMANENT
    
    Pressing  will result in the recommended default
    tablespace (identified by *) being used.
    
    Enter value for default_tablespace:
    
    Using tablespace SYSAUX as the default tablespace for the AWR_STAGE
    
    Choose the Temporary tablespace for the AWR_STAGE user
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Choose the AWR_STAGE user's temporary tablespace.
    
    TABLESPACE_NAME                CONTENTS  DEFAULT TEMP TABLESPACE
    ------------------------------ --------- -----------------------
    TEMP                           TEMPORARY *
    
    Pressing  will result in the database's default temporary
    tablespace (identified by *) being used.
    
    Enter value for temporary_tablespace:
    
    Using tablespace TEMP as the temporary tablespace for AWR_STAGE
    
    ... Creating AWR_STAGE user
    

    if we look at the import log file, we will see that data is first imported into the AWR_STAGE schema and then from here it is inserted into the WRH$ and other DBA_HIST tables.

    Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
    . . imported "AWR_STAGE"."WRH$_SQL_PLAN"                 3.845 MB   11494 rows
    . . imported "AWR_STAGE"."WRH$_SQLTEXT"                  1.012 MB     569 rows
    . . imported "AWR_STAGE"."WRH$_SYSMETRIC_SUMMARY"        156.8 KB    2025 rows
    .......
    ........
    	Append Data for "AWR_STAGE".WRH$_SGASTAT.
    INSERT /*+ APPEND */ INTO SYS.WRH$_SGASTAT (SNAP_ID, DBID, INSTANCE_NUMBER, NAME, POOL, BYTES) SELECT SNAP_ID,
    3228342000, INSTANCE_NUMBER, NAME, POOL, BYTES FROM "AWR_STAGE".WRH$_SGASTAT WHERE DBID = :c_dbid
    ... appended 388 rows
    

    If we now query the DBA_HIST_SNAPSHOT table, we see that it contains the data for two DBID’s – this shows that AWR history data is available in the repository database for two databases.

    SQL> select distinct dbid from dba_hist_snapshot;
    
          DBID
    ----------
    3228342000
    3892233981
    

    We can query the repository AWR tables based on DBID values to obtain data for a specific database.

    Posted in Administration, Performance Tuning | Tagged: , , , , | Leave a Comment »

    RMAN Validate Backup

    Posted by Arjun Raja on July 23, 2009

    The validate command for backup and restore is useful to confirm that backup and restoration is posssible and valid without actually backing up or restoring datafiles.

    We can validate the restore of a spfile,controlfile,archivelog or even the whole database backup.

    The following example enables us to specify a point in time upto which we need to test the validity of a backup.

    RMAN> run {
    2> set until time “to_date(‘23-JUL-2009 17:00:00′,’DD-MON-YYYY HH24:MI:SS’)”;
    3> restore database validate;4> }

    executing command: SET until clause
    using target database control file instead of recovery catalog

    Starting restore at 23-JUL-09
    allocated channel: ORA_SBT_TAPE_1
    channel ORA_SBT_TAPE_1: sid=159 devtype=SBT_TAPE
    channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.5.1.0
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=143 devtype=DISK

    channel ORA_SBT_TAPE_1: starting validation of datafile backupset
    channel ORA_SBT_TAPE_1: reading from backup piece 1pkkbkop_1_1
    channel ORA_SBT_TAPE_1: restored backup piece 1
    piece handle=1pkkbkop_1_1 tag=TAG20090717T085801
    channel ORA_SBT_TAPE_1: validation complete, elapsed time: 00:04:27
    Finished restore at 23-JUL-09

    We can validate spfile and controlfile backups as well as shown below

    RMAN> restore validate spfile;
    Starting restore at 23-JUL-09
    using channel ORA_SBT_TAPE_1
    using channel ORA_DISK_1

    channel ORA_DISK_1: starting validation of datafile backupset
    channel ORA_DISK_1: reading from backup piece /u02/oradata/testdb/TESTDB/autobackup/2009_07_22/o1_mf_s_692887841_56f6o2jr_.bkp
    channel ORA_DISK_1: restored backup piece 1
    piece handle=/u02/oradata/testdb/TESTDB/autobackup/2009_07_22/o1_mf_s_692887841_56f6o2jr_.bkp tag=TAG20090722T125041
    channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
    Finished restore at 23-JUL-09

    RMAN> restore validate controlfile;

    Starting restore at 23-JUL-09
    using channel ORA_SBT_TAPE_1
    using channel ORA_DISK_1

    channel ORA_DISK_1: starting validation of datafile backupset
    channel ORA_DISK_1: reading from backup piece /u02/oradata/testdb/TESTDB/autobackup/2009_07_22/o1_mf_s_692887841_56f6o2jr_.bkp
    channel ORA_DISK_1: restored backup piece 1
    piece handle=/u02/oradata/testdb/TESTDB/autobackup/2009_07_22/o1_mf_s_692887841_56f6o2jr_.bkp tag=TAG20090722T125041
    channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
    Finished restore at 23-JUL-09

    Let us now simulate a case where the restore validate will fail because either a backup or backupset is not available or because a file is not available on the disk itself.

    We will rename one of the most recent archivelog files and then attempt a restore validate of archivellogs for the past day.

    $ mv arch.355.1.692887607.log arch.355.1.692887607.log.old

    RMAN> restore validate archivelog from time ’sysdate -1′;

    Starting restore at 23-JUL-09
    using target database control file instead of recovery catalog
    allocated channel: ORA_SBT_TAPE_1
    channel ORA_SBT_TAPE_1: sid=140 devtype=SBT_TAPE
    channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.5.1.0
    allocated channel: ORA_DISK_1
    channel ORA_DISK_1: sid=141 devtype=DISK

    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03002: failure of restore command at 07/23/2009 21:22:21
    RMAN-06026: some targets not found – aborting restore
    RMAN-06025: no backup of log thread 1 seq 355 lowscn 5192098 found to

    This command can be safely run at anytime .

    Also worth noting that even if you simply forget to enter validate and type for example -

    RMAN > restore database ;

    RMAN will never restore on top of an ‘OPEN’ database -

    An error similar to the one below will be received if you try to overwrite files of an open database.

    ORA-19870: error while restoring backup piece 16kermqo_1_1
    ORA-19573: cannot obtain exclusive enqueue for datafile 2

    Posted in Administration | Tagged: , | Leave a Comment »

    Unix – tar and gzip commands

    Posted by Arjun Raja on July 23, 2009

    tar and compress a bunch of datafiles and then untar and uncompress them

    cd /u02/oradata/test- (area of database files )

    Now tar and compress and copy the files to backup area – /u02/oradata/test_bkup and name the file as test.tar.gz

    tar cvf – * |gzip -c >/u02/oradata/test_bkup/test.tar.gz –

    cd /u02/oradata/test_bkup > ls -lrt

    -rw-r–r– 1 ofsad1 dba 105952962 Feb 26 11:31 test.tar.gz

    Now to untar and uncompress the files back to the original area -

    cd /u02/oradata/test

    gzip -dc < /u02/oradata/test_bkup/test.tar.gz | tar xvf -

    Posted in Unix | Tagged: , , | Leave a Comment »

    Flashback Database after a Data Guard Failover

    Posted by Gavin Soorma on July 22, 2009

    After a data guard failover operation has been performed, instead of recreating the original primary database as the new standby database, we can use the FLASHBACK DATABASE to SCN command instead. This is particulary important in case the size of the primary database would mean a long backup and restore RMAN operation to recreate the standby during which we have no DR in place.

    In this test case we will simulate a failure on the Primary node by shutting down the current Primary database.

    We will then perform the standard Failover procedure. Before performing a Failover, we first try and ensure that there is no archive log gap between the Primary and Standby databases. If there is one (and if the Primary server is still accessible), we can register any ‘missing’ archivelog files using the ALTER DATABASE REGISTER PHYSICAL LOGFILE command.

    SQL> SELECT THREAD#, LOW_SEQUENCE#, HIGH_SEQUENCE# FROM V$ARCHIVE_GAP;
    
    no rows selected
    
    SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE FINISH FORCE;
    
    Database altered.
    

    These are the relevant lines taken from the alert log ….

    Identified End-Of-Redo for thread 1 sequence 19
    Terminal Recovery: Updated next available block for thread 1 sequence 19 lno 4 to value 92
    Wed Jul 22 12:43:53 2009
    Incomplete recovery applied all redo ever generated.
    Recovery completed through change 5003771
    Wed Jul 22 12:43:53 2009
    MRP0: Media Recovery Complete (testdb)
    
     ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;
    
    Database altered.
    

    The alert log will confirm the switchover operation ….

    Standby terminal recovery start SCN: 5003412
    RESETLOGS after complete recovery through change 5003771
    Standby became primary SCN: 5003411
    

    Complete the switchover operation by shutting down and starting the database

    SQL> shutdown immediate;
    ORA-01109: database not open
    
    Database dismounted.
    ORACLE instance shut down.
    SQL> startup;
    ORACLE instance started.
    
    Total System Global Area  264241152 bytes
    Fixed Size                  2083304 bytes
    Variable Size             142607896 bytes
    Database Buffers          113246208 bytes
    Redo Buffers                6303744 bytes
    Database mounted.
    Database opened.
    

    Note the SCN at which the former standby has now become the primary

    SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;
    
    TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)
    ----------------------------------------
    5003411
    

    On the original primary, we will now flashback the database using the SCN obtained above.

    SQL> flashback database to scn 5003411;
    
    Flashback complete.
    

    from the alert log ….

    Incomplete Recovery applied until change 5003412
    Flashback Media Recovery Complete
    Completed: flashback database to scn 5003411
    

    We will now turn off the flashback logging to delete existing falshback log files. Since we will be restoring a standby control file as well in the next step, these flashback logs will become obsolete.

    SQL> ALTER DATABASE FLASHBACK OFF;
    
    Database altered.
    

    From the alert log we can see that the flashback logs are being deleted.

    Stopping background process RVWR
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56dydp02_.flb
    .......
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56f0pz1n_.flb
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56f0q51v_.flb
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56f0qmw4_.flb
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56f0qsk9_.flb
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56f0r312_.flb
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56f0rkro_.flb
    Deleted Oracle managed file /u02/oradata/testdb/TESTDB_DG/flashback/o1_mf_56f0rtqq_.flb
    Flashback Database Disabled
    

    We will now create a standby control file and overwrite the existing control files with the standby control file.

    SQL>  ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/tmp/control01.ctl';
    
    Database altered.
    
    SQL> shutdown immediate;
    ORA-01109: database not open
    
    Database dismounted.
    ORACLE instance shut down.
    
    [DEV] testdb:/tmp > cp control01.ctl /u02/oradata/testdb/control01.ctl
    [DEV] testdb:/tmp > cp control01.ctl /u02/oradata/testdb/control02.ctl
    [DEV] testdb:/tmp > cp control01.ctl /u02/oradata/testdb/control03.ctl
    

    We then mount the standby database, turn on flashback logging and the managed recovery process on the standby database

    SQL> startup mount;
    ORACLE instance started.
    
    Total System Global Area  390070272 bytes
    Fixed Size                  2084272 bytes
    Variable Size             364905040 bytes
    Database Buffers           16777216 bytes
    Redo Buffers                6303744 bytes
    Database mounted.
    SQL> alter database flashback on;
    
    Database altered.
    
    SQL> recover managed standby database disconnect;
    Media recovery complete.
    

    Posted in High Availability | Tagged: , , , | 1 Comment »

    EM Grid Control target discovery after a clone

    Posted by Arjun Raja on July 22, 2009

    If a database is cloned or recreated on another machine with the same name, the new database will not be discovered on the GRID.

    There is a reason behind this .

    Example

    Database called test already discovered on GRID – database on ABC machine.

    Create or clone a database with the same name test on another machine XYZ.

    Try to discover this new database on the GRID using the manual option or the option to monitor via grid control while creating the database.

    The new database test will not be discovered on the GRID and cannot be seen in the list of targets on the GRID .

    That is because a databse with the same name the test database has also has been discovered and runs from another box.

    Only when a manual install was attempted and the name of the instance was issued as test.domain instead of just test that it worked .

    As you can see below – the repository ( sysman in emrep database ) has a table called mgmt_targets which maintains these details and does not allow duplicates( so a little tweaking is necessary )

    The agent tried to upload test automatically when installed on the new box and start it up- and it failed to do so as this entry exists in the table.

    Currently both databases are disovered – run this command in the emrep ( repository database) as sysman.

    SQL> select target_name, from mgmt_targets where target_name like ‘%TEST%’;

    TARGET_NAME
    test
    test.domain

    Posted in EM Grid Control | Tagged: , | Leave a Comment »

    EM Grid Control Agent Target Discovery

    Posted by Arjun Raja on July 22, 2009

    Sometimes , you may install the 10g OEM GRID agent before you actually create a new database or you may add a new database to an existing box.

    This new database will have to be discovered by GRID CONTROL.

    I created a database called test on a box which already had a Grid agent running on it.

    This database has to be discovered by the agent and it’s details uploaded to the GRID.

    Make sure the ORAINVENTORY location in /etc/oraInst.loc matches the location when you installed the AGENT.

    Go to the new AGENT_HOME/bin and issue command agentca- d

    test:/u01/ofsap/agent10g/bin> ./agentca -d
    Stopping the agent using /u01/ofsap/agent10g/bin/emctl stop agent
    Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
    Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
    Stopping agent … stopped.
    Running agentca using /u01/ofsap/agent10g/oui/bin/runConfig.sh
    ORACLE_HOME=/u01/ofsap/agent10g ACTION=Configure MODE=Perform
    RESPONSE_FILE=/u01/ofsap/agent10g/response_file RERUN=TRUE
    INV_PTR_LOC=/etc/oraInst.loc
    COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
    Perform – mode finished for action: Configure
    Perform – mode finished for action: Configure

    You can see the log file:
    /u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log
    test:/u01/ofsap/agent10g/bin>

    Output of log -
    /u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log

    The action configuration is performing
    ——————————————————
    The plug-in Agent Configuration Assistant is running

    Performing free port detection on host=prdu024.bankwest.com
    Performing targets discovery and agent configuration
    Starting the agent
    AgentPlugIn:agent configuration finished with status = true

    The plug-in Agent Configuration Assistant has successfully been performed
    ——————————————————
    The action configuration has successfully completed
    ###################################################

    Now login to the database as sys, unlock the dbsnmp user and also change
    the password of dbsnmp user – for example to temp.

    Login to the GRID , and click on targets – choose the new box and then the
    new database and configure the dbsmp user .

    Once this is complete, the GRID screen will show both the agent and
    database on the targets page.

    Posted in EM Grid Control | Tagged: , , , , , | Leave a Comment »

    10g Online Table Redefinition using DBMS_REDEFINTION

    Posted by Gavin Soorma on July 19, 2009

    In this example we will perform an online definition of the EMP table in the SCOTT schema by performing the following tasks online:

    1) Range partition the EMP table based on empno
    2) Change the column name of SALARY to SAL

    Verify that the table is a candidate for online redefinition

    SQL> conn / as sysdba
    Connected.
    SQL> BEGIN
    DBMS_REDEFINITION.CAN_REDEF_TABLE(‘SCOTT’,'EMP’);
    END;
    /

    PL/SQL procedure successfully completed.

    SQL> conn scott/tiger
    Connected.

    SQL> desc emp
    Name Null? Type
    —————————————– ——– —————————-
    EMPNO NOT NULL NUMBER(4)
    ENAME VARCHAR2(10)
    JOB VARCHAR2(9)
    MGR NUMBER(4)
    DOJ DATE
    SALARY NUMBER(7,2)
    COMM NUMBER(7,2)
    DEPTNO NUMBER(2)

    Create an Interim Table

    SQL> CREATE TABLE ONLINE_EMP
    2 (empno NUMBER(5) NOT NULL,
    3 ename VARCHAR2(15) NOT NULL,
    4 job VARCHAR2(10),
    5 mgr NUMBER(5),
    6 doj DATE DEFAULT (sysdate),
    7 sal NUMBER(7,2),
    8 comm NUMBER(7,2),
    9 deptno NUMBER(3) NOT NULL)
    10 PARTITION BY RANGE(empno)
    11 (PARTITION emp1 VALUES LESS THAN (7800) ,
    12 PARTITION emp2 VALUES LESS THAN (8000) );

    Table created.

    Start the table Redefinition process

    SQL> conn / as sysdba
    Connected.

    SQL> BEGIN
    DBMS_REDEFINITION.START_REDEF_TABLE(‘SCOTT’, ‘EMP’,'ONLINE_EMP’,
    ‘empno empno, ename ename, job job, mgr mgr, doj doj, salary sal, comm comm, deptno deptno’,
    dbms_redefinition.cons_use_pk);
    END;
    /

    PL/SQL procedure successfully completed.

    While the redefinition process in progress, dependant objects are still VALID and are in use

    SQL> conn scott/tiger
    Connected.
    SQL> select get_sal(7934) from dual;

    GET_SAL(7934)
    ————-
    1950

    Automatically create any triggers, indexes and constraints that were originally defined on EMP on interim table ONLINE_EMP

    SQL> declare x binary_Integer;
    BEGIN
    DBMS_REDEFINITION.COPY_TABLE_DEPENDENTS(‘SCOTT’,'EMP’,'ONLINE_EMP’,
    1, TRUE, TRUE, TRUE,FALSE,x);
    END;
    /

    PL/SQL procedure successfully completed.

    Perform some DML on the table while redefinition is in progress

    SQL> update emp set comm =comm +salary;

    14 rows updated.

    SQL> commit;

    Commit complete.

    Synchronize the Interim and Original Tables

    SQL> conn / as sysdba
    Connected.
    SQL> BEGIN
    DBMS_REDEFINITION.SYNC_INTERIM_TABLE(‘SCOTT’, ‘EMP’, ‘ONLINE_EMP’);
    END;
    /

    PL/SQL procedure successfully completed.

    Complete the redefinition

    SQL> BEGIN
    DBMS_REDEFINITION.FINISH_REDEF_TABLE(‘SCOTT’, ‘EMP’, ‘ONLINE_EMP’);
    END;
    /

    PL/SQL procedure successfully completed.

    The table ONLINE_EMP is locked in the exclusive mode only for a small window toward the end of this step. After this call the table EMP is redefined such that it has all the attributes of the ONLINE_EMP table

    Drop the Interim Table

    SQL> conn scott/tiger
    Connected.
    SQL> drop table online_emp;

    Table dropped.

    Check the Online Redefinition process

    Check the table column names

    SQL> desc emp
    Name Null? Type
    —————————————– ——– —————————-
    EMPNO NOT NULL NUMBER(5)
    ENAME NOT NULL VARCHAR2(15)
    JOB VARCHAR2(10)
    MGR NUMBER(5)
    DOJ DATE
    SAL (Originally SALARY) NUMBER(7,2)
    COMM NUMBER(7,2)
    DEPTNO NOT NULL NUMBER(3)

    Confirm the table is now partitioned

    SQL> select count(*) from emp partition (emp1);

    COUNT(*)
    ———-
    8

    SQL> select count(*) from emp partition (emp2);

    COUNT(*)
    ———-
    6

    Check all constraints originally defined on EMP are still present

    SQL> select constraint_name from user_constraints
    2 where table_name=’EMP’;

    CONSTRAINT_NAME
    ——————————
    SYS_C0017251
    SYS_C0017252
    SYS_C0017253
    PK_EMP
    FK_DEPTNO

    The function however will have to be recreated as the code still contained the original column name “SALARY”

    SQL> alter function get_sal compile;

    Warning: Function altered with compilation errors.

    SQL> sho err
    Errors for FUNCTION GET_SAL:

    LINE/COL ERROR
    ——– —————————————————————–
    6/8 PL/SQL: SQL Statement ignored
    6/15 PL/SQL: ORA-00904: “SALARY”: invalid identifier
    SQL>

    Posted in Administration | Tagged: | Leave a Comment »

    Unix for the Oracle DBA – Part 1

    Posted by Gavin Soorma on July 17, 2009

    Locating Files under a particular directory

    find . -print |grep -i test.sql

    Using AWK in UNIX

    To remove a specific column of output from a UNIX command – for example to determine the UNIX process Ids for all Oracle processes on server (second column)

    ps -ef |grep -i oracle |awk ‘{ print $2 }’

    Changing the standard prompt for Oracle Users

    Edit the .profile for the oracle user

    PS1=”`hostname`*$ORACLE_SID:$PWD>”

    Display top 10 CPU consumers using the ps command

    /usr/ucb/ps auxgw | head –11

    Show number of active Oracle dedicated connection users for a particular ORACLE_SID

    ps -ef | grep $ORACLE_SID|grep -v grep|grep -v ora_|wc –l

    Display the number of CPU’s in Solaris

    psrinfo -v | grep “Status of processor”|wc –l

    Display the number of CPU’s in AIX

    lsdev –C | grep Process|wc –l

    Display RAM Memory size on Solaris

    prtconf |grep -i mem

    Display RAM memory size on AIX

    First determine name of memory device

    lsdev -C |grep mem

    then assuming the name of the memory device is ‘mem0’

    lsattr -El mem0

    Swap space allocation and usage

    Solaris : swap -s or swap -l

    Aix : lsps -a

    Total number of semaphores held by all instances on server

    ipcs -as | awk ‘{sum += $9} END {print sum}’

    View allocated RAM memory segments

    ipcs -pmb

    Manually deallocate shared memeory segments

    ipcrm -m ‘ID’

    Show mount points for a disk in AIX

    lspv -l hdisk13

    Display amount of occupied space (in KB) for a file or collection of files in a directory or sub-directory

    du -ks * | sort -n| tail

    Display total file space in a directory

    du -ks .

    Cleanup any unwanted trace files more than seven days old

    find . *.trc -mtime +7 -exec rm {} \;

    Locate Oracle files that contain certain strings

    find . -print | xargs grep rollback

    Locate recently created UNIX files (in the past one day)

    find . -mtime -1 –print

    Finding large files on the server (more than 100MB in size)

    find . -size +102400 -print

    Crontab :

    To submit a task every Tuesday (day 2) at 2:45PM

    45 14 2 * * /opt/oracle/scripts/tr_listener.sh > /dev/null 2>&1

    To submit a task to run every 15 minutes on weekdays (days 1-5)

    15,30,45 * 1-5 * * /opt/oracle/scripts/tr_listener.sh > /dev/null 2>&1

    To submit a task to run every hour at 15 minutes past the hour on weekends (days 6 and 0)

    15 * 0,6 * * opt/oracle/scripts/tr_listener.sh > /dev/null 2>&1

    Posted in Unix | Tagged: , , , , , | Leave a Comment »

    Performance Tuning Tips and Techniques

    Posted by Gavin Soorma on July 17, 2009

    Checks to be performed at the machine level (note the example is Red Hat Linux specific)

    run queue should be ideally not more than the number of CPU’s on the machine

    At the maximum it should never be more than twice the number of CPU’s.

    This is denoted by the column ‘r’ in the vmstat output shown below

    vmstat – 5
    
    face:/opt/oracle>vmstat 5
    procs                      memory      swap          io     system         cpu
     r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
     4  1 488700 245704 178276 12513572    0    1    10    17   48  1365 40 12 43  5
     2  0 488700 302568 178312 12514904    0    0   157   686 3354  4342 55 22 22  1
     3  1 488700 257500 178508 12517896    0    0   616   221 1352  2132 21  7 64  9
     2  2 488700 232348 178820 12525392    0    0  1550   274 3632  6091 29  9 42 20
     3  0 488700 225040 178880 12527336    0    0   346   452 2494  3300 45 13 38  4
     5  0 488700 225680 178916 12527884    0    0    79   269 2005  3847 29 11 58  2
    

    CPU idle% < 10 ( id column) could indicate a machine that is having CPU resource issues

    Note: How to find number of CPU’s on a LINUX machine?

    cat /proc/cpuinfo |grep -w “processor” |wc –l

    Swap columns si and so should ideally be 0 to indicate no swapping activity

    face:/opt/oracle>free -m
                 total       used       free     shared    buffers     cached
    Mem:         15635      15476        159          0        173      12196
    -/+ buffers/cache:       3106      12528
    Swap:         3999        477       3522
    
    We should be looking at the free and used values in the row denoted by
    “-/+ buffers/cache”
    

    The ‘top’ command will help us identify the load average on the machine as well as any process that is consuming excessive CPU

    face:/opt/oracle>top
    09:04:01  up 243 days, 13:01, 10 users,  load average: 2.80, 3.14, 3.13
    813 processes: 797 sleeping, 8 running, 8 zombie, 0 stopped
    CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
               total   40.7%    0.0%   12.7%   0.0%     0.3%    2.3%   43.7%
               cpu00   42.8%    0.0%   13.4%   0.3%     1.1%    1.9%   40.0%
               cpu01   37.3%    0.0%   12.6%   0.0%     0.0%    1.7%   48.2%
               cpu02   47.5%    0.0%   14.1%   0.0%     0.1%    1.1%   36.9%
               cpu03   35.1%    0.0%   10.5%   0.0%     0.0%    4.5%   49.7%
    
    Mem:  16010560k av, 15808688k used,  201872k free,       0k shrd,  181324k buff
                       11368660k actv, 2257548k in_d,  265628k in_c
    Swap: 4095984k av,  488700k used, 3607284k free                 12551192k cache
    
      PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
    24062 oracle    18   0 1342M 1.3G 1328M S    14.0  8.5 370:13   2 oracle
     6129 oracle    18   0  958M 953M  923M R    12.3  6.0 172:14   0 oracle
    10077 oracle    25   0  149M 149M  145M R     4.5  0.9   0:00   2 oracle
     9924 oracle    16   0  156M 155M  150M S     2.6  0.9   0:00   2 oracle
    10038 oracle    17   0  140M 140M  136M S     2.5  0.8   0:00   1 oracle
    10055 oracle    24   0  136M 135M  132M S     2.4  0.8   0:00   0 oracle
    25529 oracle    17   0  782M 782M  775M S     1.7  5.0   3:58   0 oracle
     8245 oracle    16   0  274M 274M  268M S     1.6  1.7   0:03   1 oracle
     9751 oracle    16   0 70484  67M 62952 S     1.3  0.4   0:01   2 oracle
    31879 oracle    16   0  830M 828M  815M R     1.0  5.2  46:06   3 oracle
     9210 oracle    15   0  229M 229M  225M S     0.7  1.4   0:01   3 oracle
    

    A load average greater than 5 or 10 could indicate a heavily utilized machine

    CPU information for each CPU is also provided via the top command as well as information on the physical as well as virtual Memory that is available on the machine.

    Information on the top CPU as well as Memory consuming processes is also provided along with the Process ID (PID). In a later section we will use this PID as a parameter for a SQL query to identify the SQL being executed by the same CPU consuming process.

    Identify if any single PID is constantly appearing in the top output

    The iowait column can also help us identify if there s any resource contention from the IO perspective.

    A value above 40-50% would indicate I/O resource issues and require further investigation into the process that is causing this high I/O – or it could indicate a case of inefficient disk sub system or file layout in the database.

    We can also view the state of the machine at a particular time of the day by running the sar command which will provide the system utilisation report since 12:00 AM on that particular day.

    We can use the sar command to identify the machine state even for a particular day of the month

    For example sar -f /var/log/sa/sa03 will report for the 3rd of the month

    Checks to be performed at the database level

    Identify with the user if the problem is a slow response or a hanging situation.

    Establish a connection via SQL*NET using a non SYSDBA account to confirm that the listener is accepting client connections and the hanging is not due to the archive area getting 100% full.

    Examine the alert log file for ‘Checkpoint Not Completed’ errors recorded at the time the performance problem is reported – this could indicate an I/O contention issue or inadequately sized redo log files which can also cause an application hang while the checkpoint completes.

    Ensure that the mount point on which the Oracle software is not 100% full or the disks holding the controlfiles are also not 100% full.

    Check the listener.log file if it exists and ensure that it is not > 2GB – on some Operating Systems like LINUX, there is a file size upper limit for the listener.log file after which client connections will not be accepted by the listener.

    Check for locked sessions (see script check_lock.sql).

    If a PID has been identified as a top CPU consuming process, check the SQL being executed by that particular PID (see script check_pid_sql.sql)

    If the user provides a particular SID where a possible performance issue exists, check the SQL being executed by that SID ( see script check_sid_sql.sql)

    If the user provides a particular Oracle username where a possible performance issue exists, check the SQL being executed by that Oracle user ( see script check_username_sql.sql)

    Very Important – check the major wait events in the database (see script wait_events.sql)

    Check the SID along with the events that each SID is waiting on (see script wait_events_sid.sql) – based on the SID, the SQL being executed by the waiting sessions can be obtained as well ( see script check_sid_sql.sql)

    Refer document Resolving common Oracle Wait Events using the Wait Interface for details on important wait events along with their possible causes and resolution.

    Check for any sessions continuously waiting for on a particular latch (see script check_latch.sql) and also refer document “Resolving common Oracle Wait Events using the Wait Interface.doc”.

    What has changed?

    Is there a measurable baseline regarding the “problem” query – when did it last perform well?

    Has the database been upgraded recently?

    Has any modifications been done to the database in terms of init.ora parameters?

    Have any new indexes been added to the table or has the table structure changed?

    Has the platform or database version changed?

    Is this a period of unusual business activity? – like a monthly data load or one-off batch job

    Appendix

    check_pid_sql.sql
    
    SET PAGESIZE 500
    set long 500000
    set head off
    select
           s.username su,
           substr(sa.sql_text,1,540) txt
    from v$process p,
         v$session s,
         v$sqlarea sa
    where    p.addr=s.paddr
    and      s.username is not null
    and      s.sql_address=sa.address(+)
    and      s.sql_hash_value=sa.hash_value(+)
    and spid=;
    
    check_sid_sql.sql
    
    SET PAGESIZE 500
    PROMPT=============================================================
    PROMPT Current SQL statement this session executes
    PROMPT=============================================================
    col sql_text for a70 hea "Current SQL"
    select q.sql_text
    from v$session s
    ,    v$sql     q
    WHERE s.sql_address = q.address
    and   s.sql_hash_value + DECODE
                     (SIGN(s.sql_hash_value), -1, POWER( 2, 32), 0) = q.hash_value
    AND   s.sid= ;
    
    check_username_sql.sql
    
    set long 500000
    SET PAGESIZE 500
    select
           s.username su,
           substr(sa.sql_text,1,540) txt
    from v$process p,
         v$session s,
         v$sqlarea sa
    where    p.addr=s.paddr
    and      s.username is not null
    and      s.sql_address=sa.address(+)
    and      s.sql_hash_value=sa.hash_value(+)
    and s.username=upper('&username');
    
    check_lock.sql
    
    set linesize 500
    SET PAGESIZE 500
    col waiting_session format 99999 heading 'Waiting|Session'
    col holding_session format 99999 heading 'Holding|Session'
    col mode_held format a20 heading 'Mode|Held'
    col mode_requested format a20 heading 'Mode|Requested'
    col lock_type format a20 heading 'Lock|Type'
    prompt blocked objects from V$LOCK and SYS.OBJ$
    
    set lines 132
    col BLOCKED_OBJ format a35 trunc
    
    select /*+ ORDERED */
        l.sid
    ,   l.lmode
    ,   TRUNC(l.ctime/60) min_blocked
    ,   u.name||'.'||o.NAME blocked_obj
    from (select *
          from v$lock
          where type='TM'
          and sid in (select sid
                      from v$lock
                      where block!=0)) l
    ,     sys.obj$ o
    ,     sys.user$ u
    where o.obj# = l.ID1
    and   o.OWNER# = u.user#
    ;
    
    prompt blocked sessions from V$LOCK
    
    select /*+ ORDERED */
       blocker.sid blocker_sid
    ,  blocked.sid blocked_sid
    ,  TRUNC(blocked.ctime/60) min_blocked
    ,  blocked.request
    from (select *
          from v$lock
          where block != 0
          and type = 'TX') blocker
    ,    v$lock        blocked
    where blocked.type='TX'
    and blocked.block = 0
    and blocked.id1 = blocker.id1
    ;
    
    prompt blockers session details from V$SESSION
    
    set lines 132
    col username format a10 trunc
    col osuser format a12 trunc
    col machine format a15 trunc
    col process format a15 trunc
    col action format a50 trunc
    
    SELECT sid
    ,      serial#
    ,      username
    ,      osuser
    ,      machine
    FROM v$session
    WHERE sid IN (select sid
          from v$lock
          where block != 0
          and type = 'TX')
    ;
    
    wait_events.sql
    
    SELECT count(*), event FROM v$session_wait
    WHERE wait_time = 0
    AND event NOT IN
    ('smon timer','pmon timer','rdbms ipc message',
    'SQL*Net message from client')
    GROUP BY event ORDER BY 1 DESC
    ;
    
    wait_events_sid.sql
    
    col username format a12
    col sid format 9999
    col state format a15
    col event format a45
    col wait_time format 99999999
    set pagesize 800
    set linesize 800
    select s.sid, s.username, se.event
    from v$session s, v$session_wait se
    where s.sid=se.sid
    and se.event not like 'SQL*Net%'
    and se.event not like '%rdbms%'
    and s.username is not null
    order by 3;
    
    check_latch.sql
    
    select count(*), name latchname from v$session_wait, v$latchname
    where event='latch free' and state='WAITING' and p2=latch#
    group by name order by 1 desc;
    

    Posted in Performance Tuning, Unix | Tagged: , , , , , , , , , , | Leave a Comment »

    Data Guard Log Shipping Report

    Posted by Gavin Soorma on July 16, 2009

    If we have a number of Data Guard physical standby database environments to manage, the following report can help us quickly identify the log transported and log applied status of the Primary and Standby databases in our environment and if any standby database is lagging behind the primary as well.

    The script can also be customised so as to email an alert notification if the standby and the primary go out of sync by say 5 log files. Drop me a line if you need that customisation script.

    This report is based on a Unix shell script (check_logship.sh) which in turn calls a SQL script (check_logship.sql).

    The script requires a user MONITOR to be created in each target database with the CONNECT and SELECT ANY DICTIONARY privileges. We also have a config file (in our case bw_dg.lst) which will contain the list of all the TNS aliases of the Primary databases which we need to monitor.

    [PROD] emrep:/u01/oracle/scripts > ./check_logship.sh
    
    #######################################################################################
    	Data Guard Log Shipping Summary Report:  Thu Jul 16 14:22:02 WAUST 2009
    #######################################################################################
    
    DB_NAME  HOSTNAME     LOG_ARCHIVED LOG_APPLIED APPLIED_TIME  LOG_GAP
    -------- ------------ ------------ ----------- ------------  -------
    
    GENPRD   CBDORCA201          16742       16742 16-JUL/14:12       0
    
    CPSPRD   PRDU009N1           11494       11494 16-JUL/14:10       0
    
    LN1P     CBDORCA101          51173       51171 16-JUL/12:25       2
    
    LA1P     CBDORCA105          76971       76970 16-JUL/13:10       1
    
    #######################################################################################
    

    check_logship.sql

    SET PAGESIZE 124
    SET HEAD OFF
    COL DB_NAME FORMAT A8
    COL HOSTNAME FORMAT A12
    COL LOG_ARCHIVED FORMAT 999999
    COL LOG_APPLIED FORMAT 999999
    COL LOG_GAP FORMAT 9999
    COL APPLIED_TIME FORMAT A12
    SELECT DB_NAME, HOSTNAME, LOG_ARCHIVED, LOG_APPLIED,APPLIED_TIME,
    LOG_ARCHIVED-LOG_APPLIED LOG_GAP
    FROM
    (
    SELECT NAME DB_NAME
    FROM V$DATABASE
    ),
    (
    SELECT UPPER(SUBSTR(HOST_NAME,1,(DECODE(INSTR(HOST_NAME,’.'),0,LENGTH(HOST_NAME),
    (INSTR(HOST_NAME,’.')-1))))) HOSTNAME
    FROM V$INSTANCE
    ),
    (
    SELECT MAX(SEQUENCE#) LOG_ARCHIVED
    FROM V$ARCHIVED_LOG WHERE DEST_ID=1 AND ARCHIVED=’YES’
    ),
    (
    SELECT MAX(SEQUENCE#) LOG_APPLIED
    FROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND APPLIED=’YES’
    ),
    (
    SELECT TO_CHAR(MAX(COMPLETION_TIME),’DD-MON/HH24:MI’) APPLIED_TIME
    FROM V$ARCHIVED_LOG WHERE DEST_ID=2 AND APPLIED=’YES’
    );

    check_logship.sh

    if [ -f /tmp/dataguard1.out ]
    then
    rm /tmp/dataguard1.out
    fi

    if [ -f /tmp/dataguard2.out ]
    then
    rm /tmp/dataguard2.out
    fi

    export SCRPT=/u01/app/scripts

    for i in `cat $SCRPT/bw_dg.lst`
    do
    sqlplus -s monitor/xxx@$i <> /tmp/dataguard2.out
    @/$SCRPT/check_logship.sql
    EOF
    echo “#######################################################################################” > /tmp/dataguard1.out
    echo ” Data Guard Log Shipping Summary Report: `date ` ” >> /tmp/dataguard1.out
    echo “#######################################################################################” >> /tmp/dataguard1.out
    echo >> /tmp/dataguard1.out
    echo “DB_NAME HOSTNAME LOG_ARCHIVED LOG_APPLIED APPLIED_TIME LOG_GAP” >> /tmp/dataguard1.out
    echo “——– ———— ———— ———– ———— ——-” >> /tmp/dataguard1.out
    cat /tmp/dataguard2.out >> /tmp/dataguard1.out
    done
    cat /tmp/dataguard1.out

    Posted in High Availability, Unix | Tagged: , , , , , | Leave a Comment »