Friday, May 31, 2013

What Happens when root.sh run in RAC installation?

Node 1
Initial Setup
Setup OLR for storing Oracle local registry data
Setup GPnP wallet and profile
Setup and copy files for OHASD daemon (startup scripts rc.d etc)
Start OHASD Daemon
Copy required CRS resources for OHASD to start
Start in Exclusive Mode and Configure Diskgroup
Push GPnP Profile to Remote Node(s)
Start Full Clusterware Stack
Adding Clusterware Resources

Other Nodes
Initial Setup
Get GPnP Profile
Setup OLR for storing Oracle local registry data
Setup and copy files for OHASD daemon (startup scripts rc.d etc)
Start OHASD Daemon
Copy required CRS resources for OHASD to start
Start Full Clusterware Stack
Adding Clusterware Resources

Again from Suresh Gandhi..Thank you

11G SCAN Listener and Node listener explained

A nice article presented by SureshGandhi on SCAN listener:

11gR2 RAC: Why SCAN & Node Listeners in 11gRAC

Hello,
A long notes (confusing) on listeners in 11gR2 RAC.
You all aware of we have two listeners running in database servers in Grid Infrastructure (Aka RAC) namely scan listeners and node listeners.
How does they work?
The SCAN works by being able to resolve to multiple IP addresses reflecting multiple listeners in the cluster handling public client connections. When a client submits a request, the SCAN listener listening on a SCAN IP address and the SCAN port is contracted on a client’s behalf. Because all services on the cluster are registered with the SCAN listener, the SCAN listener replies with the address of the local listener on the least-loaded node where the service is currently being offered. Finally, the client establishes connection to the service through the listener on the node where service is offered. All of these actions take place transparently to the client without any explicit configuration required in the client.

    A picture can save 100 words of explanation
image
    We have 3 scan listeners , registering the IPC(TCP) end points as the RAC instances (remote_listener) registered with PMON of rac instances and the PMON also registers to node listeners (local_listener) which actually points to your database.
  • Indeed they both run from the same home (Grid Home), but for different functionality. From 11gR2 onwards these listeners are part of clusterware and managed by oraagent. This oraagent take care of listeners. This oraagent spawned by crsd takes care of our listeners in terms of configuration and monitoring.
  • Especially when the listener is started by oraagent you can see the lines in the listener.ora file #line added by agent.
  • Another important aspect of the listener.ora files are these files are managed by oraagent, whenever there is a clusterware command is used to manage this listeners. for example srvctl etc.
  • Further, the listeners will be monitored by oraagent process every 60 seconds, which you can find in the  $GRID_HOME/log/nodename}/agent/crsd/oraagent_oracle/oraagent_oracle.log
  • You must set the local_listener and remote_listener parameter to scan listener, description or address list of the same. Since these is the only way now to register your database/instance to scan listener. Even if you dont specify the oraagent will automatically add local_listener values , but you must exclusively set remote_listener parameter to cluster scan name which will provide you the TAF. 
    For example,
If you delete the listener.ora and restart the listener with srvctl start listener, a listener.ora will reappear. The original configuration will reappear in listener.ora and the manually modified listener.ora will be renamed (with a timestamp suffix) 
  • The agent also creates and maintains the file: endpoints_listener.ora this file is there for backward compatibility,
  • Oraagent comes into action also at instance startup, when the instance is started with srvctl (as opposed to ‘manually’ started instance from sqlplus) and sets LOCAL_LISTENER  parameter, dynamically (this is done with an alter system command and only if the parameter has not been set on spfile).
You may also has observed in listener.ora file there were no tcp/ip settings, Where are the TCP/IP settings of my listeners in listener.ora? and Only IPC endpoints are listed in listener.ora,  As you see below the listener.ora contains only scan listener which contain IPC protocol 
Notes:- Before proceed further
What is dynamic listening end point?
dynamic listening endpoint=the address or connection point to the listener. The most known endpoint in the oracle DB world being TCP, port 1521.
Oraagent connects to the listener via IPC and activates the TCP (TCPS, etc) endpoints as specified in the clusterware configuration
Example Listener.ora settings:-
[oracle@rac1]$ cat /u01/app/11.2.0.2/grid/network/admin/listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
WALLET_LOCATION =
(SOURCE =
(METHOD = FILE)
(METHOD_DATA =
(DIRECTORY = /u01/app/11.2.0.2/grid/network/admin/cost)
)
So here no SID, no Port, no host address configured, this is due to the settings whole managed in the clusterware.
SCAN Listeners has an endpoints to the scan listeners in endpoint_listener.ora via IPC or TCP looks like.
endpoints_listener.ora
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=strac201a-vip)(PORT=1522))(ADDRESS=(PROTOCOL=TCP)(HOST=172.24.21.89)(PORT=1522)(IP=FIRST)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=strac201a-vip)(PORT=1521))(ADDRESS=(PROTOCOL=TCP)(HOST=172.24.21.89)(PORT=1521)(IP=FIRST)))) # line added by Agent
tnsnames.ora

MYDB1522 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = strac201-scan.frxntnyc.frx2.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = MYDB1522)
)
)
As you can see the listener.ora files points to IPC scan listeners end points of vip/hostnames in endpoint_listener.ora and once these listeners started , the database pmon registers the sid/services to this listeners (see picture above) as per tnsentries or the local_listener parameter, the pmon register to both node and scan listeners.
For example,
When you just start a listener ,
$GRID_HOME/bin/lsnrctl stop listener; $GRID_HOME/bin/lsnrctl start listener;
This command starts only IPC endpoint
However oraagent is posted at listener startup and makes active the rest of the endpoints (notably listening on the TCP port), this can be seen for example by running the following a few seconds after listener restart: $GRID_HOME/bin/lsnrctl status listener (which will list all the active endpoints), the active endpoints are infact the database instances. This is possible due to the listener.ora parameter you usually see ENABLE_GLOBAL_DYNAMIC_ENDPOINT_{LISTENER_NAME}=ON 
Another check,
when you disable the ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=OFF in listener.ora
and manually stop the listener (manually means not using srvctl)
$GRID_HOME/bin/lsnrctl stop listener 
$GRID_HOME/bin/lsnrctl start listener
When you check listener.ora and check that the parameter edited above has not changed, that is  in this case the TCP endpoint will not be started, that is the listener will be listening only on IPC.
Check with: $GRID_HOME/bin/lsnrctl status listener
If we try do the same exercise by stopping and starting the listener with srvctl, as it would be the typical way to do it, we will see that the parameter ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER in listener.ora will be set again to ON. 
Now, morale of the post,
Always, use srvctl to add listeners,stop,start
And Always, Use srvctl to start instances/db as it set the local_listeners parameter automatically to endpoints.
So that the listener.ora file manages promptly/neatly by oraagent process.
-Thanks

11g RAC over 10g RAC

From: https://sites.google.com/site/oraclerac009/b-rac-install-upgrade/11gr2-rac/11g-new-features-over-10g


11g RAC new features over 10g RAC

1) SCAN

The single client access name (SCAN) is the address used by all clients connecting to the cluster. The SCAN
name is a domain name registered to three IP addresses, either in the domain name service (DNS) or the Grid
Naming Service (GNS). The SCAN name eliminates the need to change clients when nodes are added to or
removed from the cluster. Clients using SCAN names can also access the cluster using EZCONNECT.

A) The Single Client Access Name (SCAN) is a domain name that resolves to all the addresses allocated
for the SCAN name. Three IP addresses should be provided (in DNS) to use for SCAN name mapping
as this ensures high availability. During Oracle Grid Infrastructure installation, listeners are created
for each of the SCAN addresses, and Oracle Grid Infrastructure controls which server responds to a
SCAN address request.

B) The SCAN addresses need to be on the same subnet as the VIP addresses for nodes in the cluster.

C) The SCAN domain name must be unique within your corporate network.




2) GNS

In the past, the host and VIP names and addresses were defined in the DNS or locally in a hosts file. GNS can
simplify this setup by using DHCP. To use GNS, DHCP must be configured in the subdomain in which the
cluster resides.


3) OCR and Voting on ASM storage

The ability to use ASM (Automatic Storage Management) diskgroups for Clusterware OCR and Voting disks
is a new feature in the Oracle Database 11g Release 2 Grid Infrastructure. If you choose this option and ASM
is not yet configured, OUI launches ASM configuration assistant to configure ASM and a diskgroup.


4) Passwordless Automatic SSH Connectivity

If SSH has not been configured prior to the Installation, you can prompt the installer to do this for you. The
configuration can be tested as well.




5) Intelligent Platform Management interface (IPMI)

Intelligent Platform Management Interface (IPMI) provides a set of common interfaces to computer hardware
and firmware that administrators can use to monitor system health and manage the system.
With Oracle Database 11g Release 2, Oracle Clusterware can integrate IPMI to provide failure isolation
support and to ensure cluster integrity. You must have the following hardware and software configured to
enable cluster nodes to be managed with IPMI:

A) Each cluster member node requires a Baseboard Management Controller (BMC) running firmware
compatible with IPMI version 1.5, which supports IPMI over LANs, and is configured for remote control.

B) Each cluster member node requires an IPMI driver installed on each node.
The cluster requires a management network for IPMI. This can be a shared network, but Oracle
recommends that you configure a dedicated network.

C) Each cluster node's ethernet port used by BMC must be connected to the IPMI management network.

If you intend to use IPMI, then you must provide an administration account username and password when
prompted during installation.


6) Time Sync

Oracle Clusterware 11g release 2 (11.2) requires time synchronization across all nodes within a cluster when
Oracle RAC is deployed. To achieve this you should have your OS configured network time protocol (NTP).
The new Oracle Cluster Time Synchronization Service is designed for organizations whose Oracle RAC
databases are unable to access NTP services.


7) Clusterware and ASM share the same Oracle Home

The clusterware and ASM share the same home thus it is known as the Grid Infrastructure home (prior to
11gR2, ASM and RDBMS could be installed either in the same Oracle home or in separate Oracle homes).



8) Hangchecktimer and oprocd are replaced

Oracle Clusterware 11g release 2 (11.2) replaces the oprocd and Hangcheck processes with the cluster
synchronization service daemon Agent and Monitor to provide more accurate recognition of hangs and to
avoid false termination.


9) Rebootless Restart

The fencing mechanism has changed in 11gR2. Oracle Clusterware aims to achieve a node eviction without
rebooting a node. CSSD starts a graceful shutdown mechanism after seeing a failure. Thereafter, OHASD will
try to restart the stack. It is only if the cleanup (of a failed subcomponent) fails that the node is rebooted in
order to perform a forced cleanup.


10) HAIP


In 11.2.0.2 the new HAIP (redundant Interconnect) facility is active and multiple interface selection will
support load balancing and failover. You can select more than 4 interfaces for private interconnect at install
time or add them dynamically using oifcfg.

Oracle 11g R2 RAC Startup Sesquence

Following is the picture representing RAC startup sequence. Got it from one of the links by Suresh Gandhi's .Thanks a ton for providing this...

http://oracle-info.com/









Without much hassle or discussion with bottom to top approach, I understand now,
OHASD Phase:-
OHASD Agent Phase:-
  • OHASD Agent starts and in turn this will start
gipcd
Grid interprocess communication daemon, used for monitoring cluster interconnect
mdnsd
Multicast DNS service It resolves DNS requests on behalf of GNS
gns
The Grid Naming Service (GNS), a gateway between DNS and mdnsd, resolves DNS requests
gpnpd
Grid Plug and Play Daemon, Basically a profile similar like OCR contents stored in XML format in $GI_HOME/gpnp/profiles/<peer> etc., this is where used by OCSSD also to read the ASM disk locations to start up with out having ASM to be up, moreover this also provides the plug and play profile where this can be distributed across nodes to cluster
evmd/
evmlogger
Evm service will be provided by evmd daemon, which is a information about events happening in cluster, stop node,start node, start instance etc.
  • cssdagent (cluster synchronization service agent), in turn starts
ocssd
Cluster synchronization service daemon which manages node membership in the cluster
If cssd found that ocssd is down, it will reboot the node to protect the data integrity.

  • cssdmonitor (cluster synchronization service monitor), replaces oprocd and provides I/O fencing

  • OHASD orarootagent starts and in turn starts
crsd.bin
Cluster ready services, which manages high availability of cluster resources , like stopping , starting, failing over etc.
diskmon.bin
disk monitor (diskdaemon monitor) provides I/O fencing for exadata storage
octssd.bin
Cluster synchronization time services , provides Network time protocol services but manages its own rather depending on OS
CRSD Agent Phase:- crsd.bin starts two more agents
crsd orarootagent(Oracle root agent) starts and in turn this will start
gns
Grid interprocess communication daemon, used for monitoring cluster interconnect
gns vip
Multicast DNS service It resolves DNS requests on behalf of GNS
Network
Monitor the additional networks to provide HAIP to cluster interconnects
Scan vip
Monitor the scan vip, if found fail or unreachable failed to other node
Node vip
Monitor the node vip, if found fail or unreachable failed to other node
crsd oraagent(Oracle Agent) starts and in turn it will start (the same functionality in 11gr1 and 10g managed by racgmain and racgimon background process) which is now managed by crs Oracle agent itself.
·
ASM & disk groups
Start & monitor local asm instance
ONS
FAN feature, provides notification to interested client
eONS
FAN feature, provides notification to interested client
SCAN Listener
Start & Monitor scan listener
Node Listener
Start & monitor the node listener (rdbms?)










Clusterware Miscellaneous commands

1). Start/stop ohasd process (HighAvailability Service)
crsctl start/stop has.
2). Start clusterware components
crsctl start resource ora.crsd -init

crsctl check has
crsctl check crs

crsctl check cluster -all

crsctl stop has
The above command will bring down all the HA services and CRS services on node where the command fired.

crsctl stop crs
will bring all cluster ware stack and has down

crsctl start crs
will bring all has and clusterware stack up. We need to wait for sometime to see the services.
crsctl stop cluster -all --> Stops CRS/CSS/EVM on all nodes.

crsctl status resource <resourcename> -v --> List out the status of resource where it was last running and the current status of resource.

Wednesday, May 29, 2013

ORA-12526: TNS:listener: all appropriate instances are in restricted mode

ORA-12526: TNS:listener: all appropriate instances are in restricted mode

Nothing special... just tested some about Database in Restricted Mode.

Opening the Database in Restricted Mode,that will only allow users with special privileges (RESTRICTED SESSION)
SQL> select * from dba_sys_privs where privilege='RESTRICTED SESSION';
GRANTEE PRIVILEGE ADM
------------------------------ ---------------------------------------- ---
DBA RESTRICTED SESSION YES
SYS RESTRICTED SESSION NO
On Server: just opened instance in restricted mode.
SQL> startup restrict
On client: TNSNAMES.ORA file
orcl =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = x.x.x.x)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)

DBA role: connected by using user who has DBA role.
$ sqlplus system@orcl

Enter password:
ERROR:
ORA-12526: TNS:listener: all appropriate instances are in restricted mode
Found!!! ORA-12526: TNS:listener: all appropriate instances are in restricted mode
Then solved it by changed TNSNAMES.ORA: using "(UR=A)"

orcl =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = x.x.x.x)(PORT = 1521))
(CONNECT_DATA =
(UR=A)
(SERVER = DEDICATED)
(SERVICE_NAME = orcl)
)
)

Tested for connection again.
$ sqlplus system@orcl

Enter password:

SQL>
OK... It's work. In case with NONE-DBA role:
$ sqlplus nondba@orcl

Enter password:
ERROR:
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege
What did I learn? When instance is restricted mode.
$ lsnrctl status
.
.
Services Summary...
Service "orcl" has 1 instance(s).
Instance "orcl", status RESTRICTED, has 1 handler(s) for this service...
PMON updates the listener with that information and blocks new connection(establish).

The (UR=A) clause for TNS connect strings in TNSNAMES.ORA file. This clause insert into:
(CONNECT_DATA =
(UR=A)It allows a privileged or administrative user to connect via the listener even when the service handler is blocking connections for non-privileged users.

However, On database LISTENER.ORA file, what can we do? Static service!!! by using SID_DISC= 
Example:
$ lsnrctl status
.
.
Services Summary...
Service "orcl" has 1 instance(s).
Instance "orcl", status RESTRICTED, has 1 handler(s) for this service...

$ sqlplus system@orcl

Enter password:
ERROR:
ORA-12526: TNS:listener: all appropriate instances are in restricted mode
Then change listener.ora file:
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC = (ORACLE_HOME = /oracle/11gR2) (SID_NAME = orcl ))
)
$ lsnrctl reload

$ lsnrctl status
Services Summary...
Service "orcl" has 2 instance(s).
Instance "orcl", status UNKNOWN, has 1 handler(s) for this service...
Instance "orcl", status RESTRICTED, has 1 handler(s) for this service...
What did we see? "status UNKNOWN" and "status RESTRICTED" ... then tested...
$ sqlplus system@orcl

Enter password:

SQL>
It's work... however, test with some user (NON-DBA role)
$ sqlplus nondba@orcl

Enter password:
ERROR:
ORA-01035: ORACLE only available to users with RESTRICTED SESSION privilege
^ ^ Now!!! We can use "DBA" role to do something with database by using LISTENER, when the Database open in Restricted Mode.

How to find Edition for Installed Oracle (EE/SE)

Once invoking sqlplus and connecting to database, we can use:

sql> SELECT * FROM VERSION;
SQL> SELELCT BANNER FROM V$VERSION;

But some condition "sqlplus" will not help in that case use the below procedure to check
Go $ORACLE_HOME -->Inventory (Directory) ---> Components21(Directory) ---> oracle.server (Directory) --->[version_name] (directory) --->context.xml (xml file)
Open the context.xml file and search for ""s_serverInstallType"" and check "Val" field value
If it is Val=EE mean Enterprise Edition Or if it is Val=SE mean Standard Edition

ORACLE ASM metadata Space as per 457485.1

Normal Redundancy Max. ~150MB.
External Redundancy Max. ~50MB.

1. Normal Redundancy.
Number of disks.Metadata space reservation.
2102
3153
4150
5157
6159

2.External Redundancy.
Number of disks.Metadata space reservation.
150
252
354
455
558

The above values are not always correct but they are just to give an estimate of how much space will be reserved for the metadata incase of different disk numbers.

Oracle ASM and Cluster related Important Links

http://www.pythian.com/blog/oracle-11g-asm-diskgroup-compatibility/
https://twiki.cern.ch/twiki/bin/view/PDBService/ASM_Internals
http://asmsupportguy.blogspot.in/2010/04/asm-metadata.html
http://www.datadisk.co.uk/html_docs/oracle/asm.htm

To BottomTo Bottom



Questions and Answers


How To Downgrade ASM Instances From 11.1 Release To 10.2 (RAC or Standalone)? 

Please refer to steps outlined in document
 How To Downgrade ASM Instances From 11.1 Release To 10.2 (RAC or Standalone)?  Document 1288664.1

How can I ensure that I use the same disk names across platforms in a Unix RAC cluster? 

Please refer to document
How To Create Alias (mknod) For Candidate or Member ASM Disks on Linux/Unix. Document 1313687.1

How can I verify that new candidate disks are visible cluster wide before trying to add them to a diskgroup? 

Please review
How To Verify If A New Candidate ASM Disk Is Cluster Wide Visible ORA-15075 / ORA-15080 Document 1316936.1

How can I create a new ACFS volume and set the filesystem ownership to non-Grid OS user? 

Please follow the steps outlined in
How To Create A New ACFS Volume And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User? Document 1322383.1

Is there a comprehensive document that outlines steps to install Grid Infrastructure in single instance mode to use ASM? 

Please refer to
ASM 11.2 Configuration KIT (ASM 11gR2 Installation & Configuration, Deinstallation, Upgrade, ASM Job Role Separation (Standalone Only). Document 1092213.1
There are PDF files attached to this document that discuss installation, setting up the environment to separate the roles for ASM from the database as well as steps to deinstall cleanly. 

How can I move a database from a file system to ASM or from ASM to a file system? 

Please review
Steps To Migrate/Move a Database From Non-ASM to ASM And Vice-Versa Document 252219.1

https://communities.oracle.com/portal/server.pt/community/storage_management/288

References

NOTE:1316936.1 - How To Verify If A New Candidate ASM Disk Is Cluster Wide Visible ORA-15075 / ORA-15080
NOTE:1322383.1 - How To Create A New ACFS Volume & Filesystem And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User?
NOTE:252219.1 - Steps To Migrate/Move a Database From Non-ASM to ASM And Vice-Versa
NOTE:1092213.1 - ASM 11.2 Configuration KIT (ASM 11gR2 Installation & Configuration, Deinstallation, Upgrade, ASM Job Role Separation (Standalone Only).
NOTE:1288664.1 - How To Downgrade ASM Instances From 11.1 Release To 10.2 (RAC or Standalone)?
NOTE:1313687.1 - How To Create Alias (mknod) For Candidate Or Member ASM Disks On Linux/Unix.
====================================================================
ACFS:

Note:? 1369107.1? ACFS Supported On OS Platforms.


ASM:

Note:? 452924.1?? How to Prepare Storage for ASM.
Note:? 1187723.1? Master Note for Automatic Storage Management (ASM).
Note:? 1345562.1? Master Note For Automatic Storage Management (ASM) For Operating System (OS) Administrators.
ACFS:

Note:? 603210.1?? ACFS: How To Unmount/Dismount An ACFS Filesystem?
Note:? 1173978.1? How To Resize An ACFS Filesystem/ASM Volume (ADVM).
Note:? 885363.1?? What diagnostic information to collect for ADVM/ACFS related issues?


ASM:

Note:? 557348.1?? How To Add a New Disk(s) to An Existing Diskgroup on RAC (Best Practices).
Note:? 946213.1?? How To Add Back An ASM Disk or Failgroup (Normal or High Redundancy) After A Transient Failure Occurred (11.1 & 10.2) Or When The DISK_REPAIR_TIME Attribute Expired (10.1 & 11.2)?
Note:? 1382840.1? 11.2.0.3 ASM Grid Infrastructure Installation Allows AU Size Setting Thru The OUI Interface.
Note:? 1313687.1? How To Create Alias (mknod) For Candidate Or Member ASM Disks On Linux/Unix.
Note:? 1366159.1? New 11.2 Grid Infrastructure Installation (root.sh) Is Removing Candidate mknod devices Or Symbolic Link Devices Created At /dev/asm Directory.
ACFS:
Note: 1322383.1? How To Create A New ACFS Volume & Filesystem And Set The ACFS Filesystem Ownership To A Non-Grid/Oracle OS User?
Note: 1371067.1? How To Install/Reinstall Or Deinstall ACFS Modules/Installation Manually?
Note: 1347365.1? How To Manually Create An ACFS Snapshot.
Note: 1350188.1? How To Setup 11.2.0.X ACFS On Windows (64-bit) On RAC.
?
ASM:
?
Note:? 837308.1? Exact Steps To Migrate ASM Diskgroups To Another SAN Without Downtime.
Note:? 567508.1? How To Migrate ASMLIB Devices To Block Devices (Non-ASMLIB)?
Note:? 461718.1? How To Install ASMLIB API via the Unbreakable Linux Network
Note:? 726915.1? ASMLIB Installation Demo & ASMLIB Troubleshooting
Note:? 566676.1? How To Determinate If An EMCPOWER Partition Is Valid For ASMLIB?
Note:? 1349613.1? How To Gather The OS Logs For Each Specific OS Platform.
Note:? 1349705.1? ASM Installation Logs Location.
Note:? 1180491.1? KFED Tool For Windows OS.
Note:? 1395851.1? ASMCA Can Not Discover All The ASMLIB Disks On The Second RAC Node (ASM 11.2).
Note:? 745271.1? How To Reconfigure the Cluster Synchronization Service (CSS) on Non-RAC & Trace It (Linux/Unix)?
Note:? 948040.1? How To Rename A Diskgroup On ASM 11gR2?
Note:? 470211.1? How To Gather/Backup ASM Metadata In A Formatted Manner? ?
Note:? 1355977.1? Steps To Shutdown/Startup The CRS, OHAS, ASM & RDBMS Services on RAC 11.2 Configuration.
Note:? 603210.1? How To Identify If A Disk/Partition Is Still Used By ASM, Has Been Used by ASM Or Has Not Been Used by ASM (Unix/Linux)?
Note:? 1382372.1? Unable To Open Database Due To Diskgroup Used To Multiplex The RedoLogs & Controlfiles & Archivelogs Got Corrupted.
Note:? 1422415.1? Alternative Way To Upgrade An ASM RAC Configuration From Release 10.2 Or 11.1 To Release 11.2.
Note:? 1434351.1? Alternative Way To Upgrade An ASM Standalone Configuration From Release 11.2.0.1 or 11.2.0.2 to release 11.2.0.3.
 -------------------------------------------------------------------------------------------------------------------------
Cluster Info Blogs:
http://oracle-info.com/2013/03/19/rac-find-when-the-crs-resource-was-last-started-or-status-changes/

http://aprakash.wordpress.com/2011/02/23/crs-11gr2/

http://dbatrain.wordpress.com/2010/08/13/get-started-with-oracle-restart/ --> Cluster startup sequence.