Oracle RAC Training with ASM

For Oracle RAC Training with ASM Check the following URL:

http://www.youtube.com/watch?v=y9Wgo0XpuUs&feature=colike

Oracle RAC clusterware startup sequence (11gR2) - 2

ohasd = Oracle High Availability Services Daemon  

Link to Oracle Clusterware components

Here is the Oracle 11gR2 clusterware startup sequence:

ohasd -> orarootagent -> ora.cssdmonitor : Monitors CSSD and node health (along with the cssdagent). Try to restart the node if the node is unhealthy.

-> ora.ctssd : Cluster Time Synchronization Services Daemon ->

                                                    ora.crsd   -> oraagent  -> ora.LISTENER.lsnr

                                                                             -> ora.LISTENER_SCAN.lsnr

                                                                             -> ora.ons

                                                                             -> ora.eons

                                                                             -> ora.asm

                                                                             -> ora.DB.db

                                                         ->orarootagent -> ora.nodename.vip

                                                                                 -> ora.net1.network

                                                                                 -> ora.gns.vip

                                                                                 -> ora.gnsd

                                                                                 -> ora.SCANn.vip

         -> cssdagent -> ora.cssd : Cluster Synchronization Services

 -> oraagent -> ora.mdnsd : Used for DNS lookup
                            -> ora.evmd

                            -> ora.asmd

                            -> ora.gpnpd : Grid Plug and Play = adding a node to the cluster is easier (we need less configuration for the new node)

If a resource is written using blue & bold font => resource owned by root. The other resources are owner by oracle. (all this on UNIX environment)

When a resource is managed by root, we need to run the command crsctl as root or oracle.

Clusterware Resource Status Check

$ crsctl status resource -t
 

--------------------------------------------------------------------------------
NAME                  TARGET  STATE      SERVER            STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr    

                      ONLINE  ONLINE    rac1.nitish.com
                      ONLINE  ONLINE    rac2.nitish.com
ora.asm              

                      OFFLINE OFFLINE   rac1.nitish.com
                      OFFLINE OFFLINE   rac2.nitish.com
ora.eons             

                      ONLINE  ONLINE    rac1.nitish.com
                      ONLINE  ONLINE    rac2.nitish.com
ora.gsd
                      OFFLINE OFFLINE   rac1.nitish.com
                      OFFLINE OFFLINE   rac2.nitish.com
ora.net1.network
                      ONLINE  ONLINE    rac1.nitish.com
                      ONLINE  ONLINE    rac2.nitish.com
ora.ons
                      ONLINE  ONLINE    rac1.nitish.com
                      ONLINE  ONLINE    rac2.nitish.com
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1                     ONLINE  ONLINE    rac1.nitish.com
ora.LISTENER_SCAN2.lsnr
1                     ONLINE  ONLINE    rac2.nitish.com
ora.LISTENER_SCAN3.lsnr
1                     ONLINE  ONLINE    rac2.nitish.com
ora.oc4j
1                     OFFLINE OFFLINE
ora.scan1.vip
1                     ONLINE  ONLINE    rac1.nitish.com
ora.scan2.vip
1                     ONLINE  ONLINE    rac2.nitish.com
ora.scan3.vip
1                     ONLINE  ONLINE    rac2.nitish.com
ora.NITI.db
1                     ONLINE  ONLINE    rac1.nitish.com         Open
2                     ONLINE  ONLINE    rac2.nitish.com
ora.rac1.nitish.com.vip
1                     ONLINE  ONLINE    rac1.nitish.com
ora.rac2.nitish.com.vip
1                     ONLINE  ONLINE    rac2.nitish.com
 

crsctl start has    -> start all the clusterware services/ resources (including the database server and the listener);

crsctl stop has    -> stop all the clusterware services/ resources (including the database server and the listener);

crsctl check has -> to check if ohasd is running/ stopped

crsctl check has
CRS-4638: Oracle High Availability Services is online
>> the ohasd daemon is running => the clusterware is (must be) up and running (if no error occur).

crsctl check has
CRS-4639: Could not contact Oracle High Availability Services

>> the ohasd daemon is NOT running => the clusterware is DOWN (stopped).

crsctl enable has  -> enable Oracle High Availability Services autostart

crsctl disable has -> disable Oracle High Availability Services autostart

crsctl config has -> check if Oracle High Availability Services autostart is enabled/ disabled.

crsctl config has
CRS-4622: Oracle High Availability Services autostart is enabled.
 

Cloning Oracle Application 11i /R12 with Rapid Clone using Hot Backup

You need to follow the "pre-requisite" and "preparation" steps from section 1 and 2 of the cloning note applicable to your EBS Version:

Release 11i  : Note 230672.1 - 'Cloning Oracle Applications Release 11i with Rapid Clone'
Release 12  : Note 406982.1 - 'Cloning Oracle Applications Release 12 with Rapid Clone'


This Note helps you to implement the steps mentioned in "Appendix B: Recreating database control files manually in Rapid Clone" as per above Clone Note.

Step 1: Ensure adpreclone.pl has been run

Step 2: Obtain a trace file script to recreate the controlfile. On the source database issue the following
            command:

SQL> ALTER DATABASE BACKUP CONTROLFILE TO TRACE;


The trace file script will be put into the user_dump_dest directory. The name of the trace file script will be something like PRODUCTION_ora_99999.trc.
Compare the date and time of the new trace file script to the time in which you entered the ALTER DATABASE BACKUP CONTROLFILE TO TRACE command.
This will ensure you will be using the most recent trace file script, the one you created in this step


Step 3: In the create controlfile script just created in step 2 change:
CREATE CONTROLFILE REUSE DATABASE "PRODUCTION" NORESETLOGS

to:

CREATE CONTROLFILE DATABASE "PRODUCTION" RESETLOGS ARCHIVELOG
If you want to change the Database Name , You need to use the clause SET DATABASEin the create control file script as follows:
CREATE CONTROLFILE SET DATABASE "newdbname" RESETLOGS NOARCHIVELOG
You must specify RESETLOGS.

The ARCHIVELOG mode may be changed to NOARCHIVELOG if you wish to run the copied database in noarchive log mode. Change all directories in the create controlfile clause to point to the correct directories for the new target database, if necessary.

Leave "only" the CREATE CONTROLFILE clause. The other statements, like the recover command, will be done manually. Be sure you also remove the STARTUP NOMOUNT command.
Note:  Please ensure that there is no new datafile/tablespace added to Database after you generate controlfile script as above

Step 4: On the source database make an online copy of all datafiles using:
SQL> ALTER TABLESPACE BEGIN BACKUP;

Copy all datafiles within tablespace to the new directory. 
On Unix systems, this can be done with the cp command.

Then do:

SQL> ALTER TABLESPACE END BACKUP;

Do NOT copy the controlfiles and redo log files as they will be recreated. You must copy the datafiles only after the ALTER .. BEGIN BACKUP command has been executed, otherwise the datafiles may be corrupted.

The names of the datafiles and tablespaces to which the datafiles belong can be obtained using the following command:

SQL> SELECT FILE_NAME, TABLESPACE_NAME FROM DBA_DATA_FILES;

Note: Instead of performing step 4 and step 5 to create a new online backup, you may instead use a previously taken online backup of your database.If you choose to use a previous online backup be sure to copy the required archived redo logs taken with the the previous online backup.

Step 5: After all datafiles have been copied and the tablespaces taken out of backup mode issue the following command:

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

You will need all of the archivelog files from the start of datafile copy commands including the one just created with the command ALTER SYSTEM ARCHIVE LOG CURRENT.

Step 6: Copy the database (DBF) files,controlfile script and archive log files from the source to the target
           system

Step 7: So, As mentioned in 230672.1 (Appendix B), Replace section 2.2a (Configure the target system database server) with the following steps:

Execute the following commands to configure the target system. You will be prompted for the target system specific values (SID, Paths, Ports, etc)

· Log on to the target system as the ORACLE user
· Configure the 
cd /appsutil/clone/bin
perl adcfgclone.pl dbTechStack

Step 7: On the target system issue STARTUP NOMOUNT command. For example:
     
SQL> startup nomount pfile=initTESTING.ora


Step 8: Run the prepared script created in step 3 to create the new controlfile. For example:

SQL> @PRODUCTION_ora_99999.trc

Step 9: Issue the command:
          
SQL> RECOVER DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE


You will be prompted to apply all of the archived redo logs from the source database including
the last archive redo log file created with the ALTER DATABASE ARCHIVE LOG CURRENT 
command from step 5.

After applying all of these archive log files issue the CANCEL command.

Step 10: Open the database with resetlogs:
         
SQL>ALTER DATABASE OPEN RESETLOGS

 At this point the target database will have been successfully cloned and opened.

Step 11:  Create Temporary Tablespace if not created in Source, else you shall add Temporary tablespace

You can check if tablespace TEMP has tempfiles or datafiles using the following SQL:

SQL> SELECT FILE_NAME,TABLESPACE_NAME,STATUS,AUTOEXTENSIBLE from DBA_TEMP_FILES where TABLESPACE_NAME like 'TEMP';

SQL> SELECT FILE_NAME,TABLESPACE_NAME, STATUS,AUTOEXTENSIBLE FROM DBA_DATA_FILES WHERE TABLESPACE_NAME LIKE 'TEMP';

If Temporary Tablespace is not created from above query,
SQL> create temporary tablespace temp add tempfile 'xxxx.dbf' size xx

Or

SQL> alter tablespace TEMP add tempfile  'xxxx.dbf' size xx


Step 12:  Run the library update script against the database

· cd /appsutil/install/
· sqlplus "/ as sysdba" @adupdlib.sql 
      where is "sl" for HP-UX, "so" for any other UNIX platform and 
      not required for Windows.

Step 13: Configure the target database (the database must be open)

· cd /appsutil/clone/bin
· perl adcfgclone.pl dbconfig 
     where target context file is:
     /appsutil/.xml 


Finally, refer back to cloning notes and the following sections:

· Copy the Application Tier File System
· Configure the Target System Application Tier Server Nodes
· Finishing Tasks
Release 11i : Note 230672.1 - 'Cloning Oracle Applications Release 11i with Rapid Clone'
Release 12 : Note 406982.1 - 'Cloning Oracle Applications Release 12 with Rapid Clone'

Could not validate ASMSNMP password due to following error- "ORA-01031: insufficient privileges".

  Recently i have come across this error. If you get this error when configuring RAC database.

           Please do the following:

          As for as 11GR2 is concern, ASM is installed along with the GRID.

          Please login as user grid

         [grid]$ orapwd file=/u01/app/11.2.0/grid/dbs/orapwasm password=asm


         Using SCP copy ASM password file to rac2

         [grid]$ scp orapwasm rac2:/u01/app/11.2.0/grid/dbs/

       
        Once SCP is done 

        Login in to ASM database as 

        [grid]sqlplus / as sysasm

        SQL>create user asmsnmp identified by asm;

        SQL> grant sysdba to asmsnmp;

       The reason for creating above user is to collect the statistics for ASM. 

Dependencies of 11g R2 clusterware and ASM

In Oracle 10g RAC and 11gR1 RAC,  Oracle clusterware and ASM are installed in the different Oracle homes, and the Clusterware has to be  up before ASM instance can be started because ASM instance uses the clusterware to access the shared storage.  Oracle 11g R2 introduced the  grid infrastructure home which combines Oracle clusterware and ASM.  The OCR and votingdisk of 11g R2 clusterware can be stored in ASM.  So it seems that ASM needs the clusterware up first to access the shared storage  and the clusterware needs ASM up first before it can access its key data structure: OCR and votingdisk.  So really clusterware and ASM, which one needs to be up first, and which one has to wait for other? This seemed to be the chicken or the ego problem.

 Oracle’s solution to this problem is to combines  the clusterware and ASM  into a single Grid Infrastructure home and  comes up a  procedure with  a complex  start up sequence which  mixes  the different components of clusterware and ASM  instance in order.  Oracle Metalink note 11gR2 Clusterware and Grid Home – What You Need to Know [ID 1053147.1] gave the following  startup sequence:

Although the clusterware startup command  $GI_HOME/bin/crsctl start crs follows this sequence to bring both clusterware and ASM online, but this command really doesn’t echo back each milestone of the startup process and we really can’t see how the startup was done.  A workaround is to look at the some of outputs  of root.sh command during the initial Grid infrastructure installation process as follow:

CRS-4123: Oracle High Availability Services has been started.

ohasd is starting

CRS-2672: Attempting to start ‘ora.gipcd’ on ‘owirac1′

CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘owirac1′

CRS-2676: Start of ‘ora.mdnsd’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.gipcd’ on ‘owirac1′

CRS-2676: Start of ‘ora.gipcd’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘owirac1′

CRS-2676: Start of ‘ora.gpnpd’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘owirac1′

CRS-2676: Start of ‘ora.cssdmonitor’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.cssd’ on ‘owirac1′

CRS-2672: Attempting to start ‘ora.diskmon’ on ‘owirac1′

CRS-2676: Start of ‘ora.diskmon’ on ‘owirac1′ succeeded

CRS-2676: Start of ‘ora.cssd’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.ctssd’ on ‘owirac1′

CRS-2676: Start of ‘ora.ctssd’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.asm’ on ‘owirac1′

CRS-2676: Start of ‘ora.asm’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.crsd’ on ‘owirac1′

CRS-2676: Start of ‘ora.crsd’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.evmd’ on ‘owirac1′

CRS-2676: Start of ‘ora.evmd’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.asm’ on ‘owirac1′

CRS-2676: Start of ‘ora.asm’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.OCRVOTDSK.dg’ on ‘owirac1′

CRS-2676: Start of ‘ora.OCRVOTDSK.dg’ on ‘owirac1′ succeeded

CRS-2672: Attempting to start ‘ora.registry.acfs’ on ‘owirac1′

CRS-2676: Start of ‘ora.registry.acfs’ on ‘owirac1′ succeeded

This  sequence shows  the   ASM instance startup  is just one step in middle of  the entire sequence : Some of  CRS components such as CSSD, CTSS get started before ASM, while other components such as CRSD,  EVEMD, ACFS are up after the ASM starts.  This sequence can be also confirmed by the  timestamps and log messages in  clusterware log files  (alter<hostname>.log, cssd.log and crsd.log)  and ASM instance log like  alert_+ASM1.log . Here are the sequences of messages and their timestamps: during the startup of 11g R2 clusterware and ASM instance:

OLR service started  : 2012-07-15 16:33:13.678

Starting CSS daemon 2012-07-15 16:33:18.684:

Fetching asmlib disk :ORCL:OCR1 : 2012-07-15 16:33:24.825

Read ASM header off dev:ORCL:OCR3:224:256

Opened hdl:0x1d485110 for dev:ORCL:OCR1: 2012-07-15 16:33:24.829

Successful discovery for disk ORCL:OCR1 : 2012-07-15 16:33:24.837

Successful discovery of 5 disks: 2012-07-15 16:33:24.838

CSSD voting file is online: ORCL:OCR1:  2012-07-15 16:33:50.047

CSSD Reconfiguration complete: 2012-07-15 16:34:07.729

The Cluster Time Synchronization Service started:  2012-07-15 16:34:12.333

Note: ** CSSD and CTSSD got up before ASM .  Votingdisks were discovered  by reading the header of the ASM disks (OCRL:OCR1) of  the votingdisk diskgroup without using ASM instance **

Starting ASM: Jan 17 16:34:13 2011 

CRS Daemon Starting 2012-07-15 16:34:30.329:  

Checking the OCR device : 2012-07-15 16:34:30.331

Initializing OCR 2012-07-15 16:34:30.337

diskgroup OCRVOTDSK was mounted : Jan 17 16:34:30 2011

OCRVOTDSK was mounted : Mon Jan 17 16:34:30 2011

The OCR service started : 2012-07-15 16:34:30.835

Verified ocr1-5: 2012-07-15 16:33:50.128

Cluster Time Synchronization Service started:  2012-07-15 16:34:12.333

The OCR service started : 2012-07-15 16:34:30.835 

CRSD started: 2012-07-15 16:34:31.902

Note: CRS server started  after ASM is up and the diskgroup for OCR and votingdisks are mounted

From this sequence of the log message and timestamp, we get some understanding about the sequence of clusterware and ASM instance:

1)      CSSD and CTSSD are up before ASM

2)      Votingdisks used by CSSD are discovered by reading the header of the disks, not throught ASM

3)      Startup of CRS service has to wait until ASM instance is up and the diskgroup for OCR and votingdisk is mounted.