ORA-15032: not all alterations performed ORA-15063: ASM discovered an insufficient number of disks for diskgroup

Issue:

We had server reboot, After that we are not able to bring back the ASM disk,
We are getting below error.

[root@Node-1 bin]# ./crs_start ora.ASM_FRA01.dg
Attempting to start `ora.ASM_FRA01.dg` on member `za-fdbs-dr-01`
CRS-5017: The resource action "ora.ASM_FRA01.dg start" encountered the following error:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "ASM_FRA01" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "ASM_FRA01"


Start of `ora.ASM_FRA01.dg` on member `Node-1` failed.
Attempting to stop `ora.ASM_FRA01.dg` on member `Node-1`
Stop of `ora.ASM_FRA01.dg` on member `Node-1` succeeded.
CRS-0215: Could not start resource 'ora.ASM_FRA01.dg'.


Solution:

We had multipath configured in the server,So we blacklisted the ASM lun's in the /etc/multipath.conf.
then we mount the disk in ASM.

Steps
++ To find list of multipath disk
[root@Node-1 ~]#multipath -ll

mpath5 (360060160260d2900621e1d393121e411) dm-5 DGC,RAID 5
[size=500G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=6][active]
 \_ 15:0:0:4 sdad 65:208 [active][ready]
 \_ 16:0:0:4 sdag 66:0   [active][ready]
 \_ 18:0:0:4 sdak 66:64  [active][ready]
 \_ 21:0:0:4 sdal 66:80  [active][ready]
 \_ 22:0:0:4 sdam 66:96  [active][ready]
 \_ 20:0:0:4 sdan 66:112 [active][ready]

++ Black list in /etc/multipath.conf the wwid.

blacklist {
        wwid 360060160260d2900d2f242623020e411
  
++ Restart the services.

service multipathd restart

++ Now mount the disk in ASM.

root@Node-1 bin]# ./crs_start ora.ASM_FRA01.dg
Attempting to start `ora.ASM_FRA01.dg` on member `Node-1`
Start of `ora.ASM_FRA01.dg` on member `Node-1` succeeded.

oracleasm createdisk marking disk FAILED with Unable to open device Device or resource busy

While adding new lun on the ASM storage we are getting below error in /var/log/oracleasm

Error:
/etc/init.d/oracleasm createdisk ASM_DISK02 /dev/sdf1
Marking disk "ASM_DATA02" as an ASM disk: [FAILED]

and in /var/log/oracleasm we are getting below error

Unable to open device "/dev/sdf1": Device or resource busy
Unable to open device "/dev/sdf1": Device or resource busy


Solution:

We are using multipath storage for backup.So in /etc/multipath.conf ,we blacklist the wwid of the new luns.

Then we are able to createdisk at asm level.

11.2 GRID CRS is not starting after rebooting a cluster node: crsctl start has

In the Node alert crs log, we are getting below error:

2014-11-27 21:03:31.095
[/u01/app/11.2.0/grid/bin/oraagent.bin(15001)]CRS-5818:Aborted command 'start for resource: ora.gpnpd 1 1' for resource 'ora.gpnpd'. Details at (:CRSAGF00           113:) {0:0:2} in /u01/app/11.2.0/grid/log/hostname/agent/ohasd/oraagent_grid/oraagent_grid.log.
2014-11-27 21:03:35.100
[ohasd(14885)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/           grid/log/hostname/ohasd/ohasd.log.
2014-11-27 21:03:35.504
[mdnsd(15013)]CRS-5602:mDNS service stopping by request.
2014-11-27 21:03:36.716
[/u01/app/11.2.0/grid/bin/orarootagent.bin(16118)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/acfsload" spawned by agent "/u01/app/11.2.0/grid/bin/oraroota           gent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/za-rac-prd-01/agent/ohasd/orarootagent_root/orarootagent_root           .log"


[  clsdmc][1111984448]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)(KEY=DBG_GPNPD))][6]
2014-11-27 23:37:00.921: [ora.gpnpd][1111984448] {0:0:2} [start] Error = error 6 encountered when sending messages to GPNPD
2014-11-27 23:37:01.922: [ora.gpnpd][1111984448] {0:0:2} [start] without returnbuf
2014-11-27 23:37:01.922: [ COMMCRS][1111984448]clscsendx: (0x2aaaac068830) Connection not active


lient(10786)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/node-01/client/emcrsp.log.
2014-11-27 20:17:15.234
[client(11061)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-11-27 20:17:15.236
[client(11061)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/node-01/client/emcrsp.log.


011-08-17 17:51:27.601: [  OCRASM][2931596144]proprasmo: Failed to open file in dirty mode
2011-08-17 17:51:27.601: [  OCRASM][2931596144]proprasmo: Error in open/create file in dg [OCR_VOTE]
[  OCRASM][2931596144]SLOS : SLOS: cat=8, opn=kgfolclcpi1, dep=204, loc=kgfokge

2011-08-17 17:51:27.601: [  OCRASM][2931596144]ASM Error Stack : ORA-15077: could not locate ASM instance serving a required diskgroup

2011-08-17 17:51:27.756: [  OCRASM][2931596144]proprasmo: kgfoCheckMount returned [6]
2011-08-17 17:51:27.756: [  OCRASM][2931596144]proprasmo: The ASM disk group OCR_VOTE is not found or not mounted
2011-08-17 17:51:27.761: [  OCRRAW][2931596144]proprioo: Failed to open [+OCR_VOTE]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2011-08-17 17:51:27.761: [  OCRRAW][2931596144]proprioo: No OCR/OLR devices are usable
2011-08-17 17:51:27.761: [  OCRASM][2931596144]proprasmcl: asmhandle is NULL
2011-08-17 17:51:27.761: [  OCRRAW][2931596144]proprinit: Could not open raw device
2011-08-17 17:51:27.761: [  OCRASM][2931596144]proprasmcl: asmhandle is NULL
2011-08-17 17:51:27.761: [ default][2931596144]a_init:7!: Backend init unsuccessful : [26]
2011-08-20 01:32:09.691: [ CSSCLNT][2931596144]clsssinit: clsssInitNative returned unsuccesfully, but client did not specify native css connection, rc(3)
2011-08-20 01:32:09.691: [ CSSCLNT][2931596144]clsssinit: clsssInitNative returned unsuccesfully, but client did not specify native css connection, rc(3)
===================================

[root@node-01 gpnpd]# ls -ltr
total 744
-rw-r--r-- 1 root root          6 Nov 27 19:06 node-01.pid
-rw-r--r-- 1 grid oinstall  11700 Nov 28 00:23 gpnpdOUT.log
-rw-r--r-- 1 grid oinstall 739913 Nov 28 00:23 gpnpd.log
[root@node-01 gpnpd]# pwd
/u01/app/11.2.0/grid/log/node-01/gpnpd
[root@node-01 gpnpd]# tail -100f gpnpd.log
2014-11-28 00:23:48.075: [ default][1640935600]gpnpd START pid=2462 Oracle Grid Plug-and-Play Daemon
2014-11-28 00:23:48.076: [    GPNP][1640935600]clsgpnp_Init: [at clsgpnp0.c:582] '/u01/app/11.2.0/grid' in effect as GPnP home base.
2014-11-28 00:23:48.076: [    GPNP][1640935600]clsgpnp_Init: [at clsgpnp0.c:616] GPnP pid=2462, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
2014-11-28 00:23:48.080: [    GPNP][1640935600]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /u01/app/11.2.0/grid/gpnp/node-01/wallets/peer/
[   CLWAL][1640935600]clsw_Initialize: OLR initlevel [70000]
[  clsdmt][1101838656]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=node-01DBG_GPNPD))
2014-11-28 00:23:48.096: [  clsdmt][1101838656]PID for the Process [2462], connkey 10
2014-11-28 00:23:48.096: [  clsdmt][1101838656]Creating PID [2462] file for home /u01/app/11.2.0/grid host node-01 bin gpnp to /u01/app/11.2.0/grid/gpnp/init/
2014-11-28 00:23:48.096: [  clsdmt][1101838656]Error3 -2 writing PID [2462] to the file []
2014-11-28 00:23:48.096: [  clsdmt][1101838656]Failed to record pid for GPNPD
2014-11-28 00:23:48.096: [  clsdmt][1101838656]Terminating process
2014-11-28 00:23:48.096: [    GPNP][1101838656]CLSDM requested exit
2014-11-28 00:23:48.097: [ default][1101838656]GPNPD on node node-01 shutdown.
Solution:
 
After following ASM check << http://oracletechdba.blogspot.com/2014/12/ocrcheck-proc-26-error-while-accessing.html >>
We narrow down that the issue is not with ASM and with the help of friends Rijesh and vignesh,Manage to find the root cause.
we are not having file permission on <node_name>.<date>.orig

So followed the below solution as suggested in Note:Unable To Start CRS 11gR2 Due to Missing Files in $GRID_HOME/<daemon>/init Directory (Doc ID 1146055.1)

cp $GRID_HOME/<daemon>/init/<node_name> $GRID_HOME/<daemon>/init/<node_name>.<date>.orig
OR
cp $GRID_HOME/<daemon>/init/<node_name>.pid $GRID_HOME/<daemon>/init/<node_name>.pid.<date>.orig

Then restarted the CRS.
./crsctl start crs

It worked :)

ocrcheck PROC-26: Error while accessing the physical storage

After server restart on node-1 CRS is not coming up, But during dia we are getting not able to access physical storage,But in real that could not be the issue.
To verify  gpnpd was able to read the profile in OCR disk.
from the node verify the below.

++ OCR check fails

[root@ node-1 bin]# ./ocrcheck
PROT-602: Failed to retrieve data from the cluster registry
PROC-26: Error while accessing the physical storage
ORA-29701: unable to connect to Cluster Synchronization Service

++ But we are able to access the Disk.
[root@ node-1 bin]# ls -lr /dev/oracleasm/disks/*
brw-rw---- 1 grid asmadmin  8, 225 Nov 27 23:31 /dev/oracleasm/disks/OCR_VOTE02
brw-rw---- 1 grid asmadmin  8, 241 Nov 27 23:31 /dev/oracleasm/disks/ASM_FRA02
brw-rw---- 1 grid asmadmin 65,   1 Nov 27 20:12 /dev/oracleasm/disks/ASM_DATA05
brw-rw---- 1 grid asmadmin 65,  17 Nov 27 20:12 /dev/oracleasm/disks/ASM_DATA04

[root@ node-1 bin]#  /etc/init.d/oracleasm scandisks
Scanning the system for Oracle ASMLib disks:               [  OK  ]

[root@ node-1 bin]# /etc/init.d/oracleasm listdisks
ASM_DATA04
ASM_DATA05
ASM_FRA02
OCR_VOTE02
[root@ node-1 bin]#
[root@ node-1 bin]# /etc/init.d/oracleasm querydisk OCR_VOTE02
Disk "OCR_VOTE02" is a valid ASM disk
[root@ node-1 bin]#

++We can able to read the data as well usinf kfed.
[root@ node-1 bin]# ./kfed  read  /dev/oracleasm/disks/OCR_VOTE02 |more
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483649 ; 0x008: TYPE=0x8 NUMB=0x1
kfbh.check:                   344414986 ; 0x00c: 0x14875b0a
kfbh.fcn.base:                     1473 ; 0x010: 0x000005c1
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000

So the issue is some thing less follow the next post. 
<<http://oracletechdba.blogspot.com/2014/12/112-grid-crs-is-not-starting-after.html>>

Oracle ASM Disk Group free space/used space monitoring shell script and alerting through mail

#!/bin/bash
# This script is to send mail for alerting ASM Disk Group free space and used space status.
# Assumption smtb confirgured,created DIR's
# Note : you have to have SSH connectivity between monitoring server and the ASM server,Enter the server details in asm_serv.txt:as format "+ASM1 10.95.10.101 oracle"


WORK_DIR=/u01/app/oracle/util/scripts
LOG_DIR=/u01/app/oracle/util/log
Server_details=$WORK_DIR/asm_serv.txt
temp_file=$WORK_DIR/asm_report_temp.log
final_report=$WORK_DIR/asm_full_report.html
export PATH=$PATH:/usr/sbin
CURR_DATE=`date '+%m/%d/%y_%H:%M'`

if [ -e $temp_file ]
  then
    `rm   $temp_file `
fi
if [ -e $final_report ]
  then
    `rm $final_report `
fi


html_head="Subject: ASM Alert REPORT for ${CURR_DATE}
\nMIME-Version: 1.0 \nContent-Type: text/html \nContent-Disposition: inline \n

Standalone Oracle Grid Infrastructure in EXADATA

Standalone Oracle Grid Infrastructure in EXADATA

1st set up exadata server using my previous post 
<< http://oracletechdba.blogspot.com/2014/06/building-exadata-in-vm-for-learning.html#more >>

Now preparing the server for installation.

Pre check and setup for

------------------------

1./etc/sysctl.conf

fs.aio-max-nr = 1048576

fs.file-max = 6815744

kernel.shmmni = 4096

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default=262144

net.core.rmem_max=4194304

net.core.wmem_default=262144

net.core.wmem_max=1048586

kernel.core_uses_pid = 1

fs.suid_dumpable = 1

/sbin/sysctl -p

---------------------

2./etc/security/limits.conf

oracle  soft    nofile  131072

oracle  hard    nofile  131072

oracle  soft    nproc   131072

oracle  hard    nproc   131072

oracle  soft    core    unlimited

oracle  hard    core    unlimited

oracle  soft    memlock 3500000

oracle  hard    memlock 3500000

grid    soft    nofile  131072

grid    hard    nofile  131072

grid    soft    nproc   131072

grid    hard    nproc   131072

grid    soft    core    unlimited

grid    hard    core    unlimited

grid    soft    memlock 3500000

grid    hard    memlock 3500000

grid soft nproc 2047

------------------------------------

Deinstall oracle Grid standalone installtion after failed instal

Deinstall Oracle Grid Infrastructure Standalone Install
At the end of the installation i get the error stating.

INFO: Executing ASMCA
INFO: Command /u01/app/grid/bin/asmca -silent -oui_internal -configureASM -diskString 'o/*/*' -diskGroupName MGMT -diskList o/192.168.124.129/DATA_CD_DISK01_localhost,o
/192.168.124.129/DATA_CD_DISK02_localhost -redundancy NORMAL
INFO: ... GenericInternalPlugIn.handleProcess() entered.
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
INFO: ... GenericInternalPlugIn: checking secretArguments.
INFO: ... GenericInternalPlugIn: starting read loop.
INFO: Read: SYS_PASSWORD_PROMPT
INFO: Processing: SYS_PASSWORD_PROMPT for argument tag -sysAsmPassword
INFO: Read: ASMSNMP_PASSWORD_PROMPT
INFO: Processing: ASMSNMP_PASSWORD_PROMPT for argument tag -asmMonitorPassword
INFO: End of argument passing to stdin
INFO: Read:
INFO: Read: Configuring ASM failed with the following message:
INFO: Read: One or more disk group(s) creation failed as below:
INFO: Read: Disk Group MGMT creation failed with the following message:
INFO: Read: ORA-15018: diskgroup cannot be created
INFO: Read: ORA-15072: command requires at least 2 regular failure groups, discovered only 1

++ So i choose to De-install and Reinstall with proper ASM disk group option.

++ To Deinstall.

PRVF-5184 : check of following Udev attributes

Error:
======
PRVF-5184 : check of following Udev attributes of /dev/oracleasm/disks/ORC_VOTE01 failed:
"[Owner: found='root' Expected='grid',Group:found='root' Explected='asmadmin',permissions:found='0600'
Expected='0660']"

Solution:
=========
Ignore this and proceeds with installation

PRVF-4661 : Found inconsistent hosts entry in /etc/nsswitch.conf

RAC GRID - Post installation screen fail with PRVF-4661 : Found inconsistent hosts entry in /etc/nsswitch.conf
Error:
======
Oracle rac 11gr2 fails with Configure oracle grid infrastructure for a cluster.
Oracle rac 11gr2 oracle cluster verification utility failed

INFO: Post-check for cluster services setup was unsuccessful.
INFO: Checks did not pass for the following node(s):
INFO:   Node-01,Node-02
INFO:
WARNING:
INFO: Completed Plugin named: Oracle Cluster Verification Utility

Checks did not pass for the following node Post-check for cluster services setup was unsuccessful

ERROR:
PRVF-4661 : Found inconsistent hosts entry in /etc/nsswitch.conf on node Node-01
Solution:
=========
Change the setting of hosts in the file /etc/nsswitch.conf,It should be simitar to both the nodes as well.
Like below.
 hosts:    files   dns   nis

Then run the post cluster verify as below.

 ./cluvfy stage -post crsinst -n all -verbose

Rac GRID instal hangs at oracle.cluster.deployment.ractrans.ClientHandlerSupervisor.threadCleanup

Rac grid installation hangs as below
 
Error:
======
Node-2 hangs as below
 
Exception in thread "Install API Thread" java.lang.NullPointerException
        at oracle.cluster.deployment.ractrans.ClientHandlerSupervisor.threadCleanup(ClientHandlerSupervisor.java:981)
        at oracle.cluster.deployment.ractrans.RACTransfer.cleanup(RACTransfer.java:1757)
        at oracle.cluster.deployment.ractrans.RACTransfer.transferDirStructureToNodes(RACTransfer.java:747)
        at oracle.cluster.deployment.ractrans.RACTransfer.transferDirToNodes(RACTransfer.java:253)
        at oracle.ops.mgmt.cluster.ClusterCmd.transferDirToNodes(ClusterCmd.java:3119)
        at oracle.ops.mgmt.cluster.ClusterCmd.transferDirToNodes(ClusterCmd.java:3038)
        at oracle.sysman.oii.oiip.oiipg.OiipgClusterOps.transferDirToNodes(OiipgClusterOps.java:947)
        at oracle.sysman.oii.oiif.oiifw.OiifwClusterCopyWCCE.doOperation(OiifwClusterCopyWCCE.java:544)
        at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
        at oracle.sysman.oii.oiif.oiifw.OiifwActionsPhaseWCDE.doOperation(OiifwActionsPhaseWCDE.java:633)
        at oracle.sysman.oii.oiif.oiifb.OiifbLinearIterator.iterate(OiifbLinearIterator.java:147)
        at oracle.sysman.oii.oiic.OiicInstallAPISession$OiicAPISelCompsInstall.doOperation(OiicInstallAPISession.java:1072)
        at oracle.sysman.oii.oiif.oiifb.OiifbCondIterator.iterate(OiifbCondIterator.java:171)
        at oracle.sysman.oii.oiic.OiicInstallAPISession.doInstallAction(OiicInstallAPISession.java:656)
        at oracle.sysman.oii.oiic.OiicInstallAPISession.access$000(OiicInstallAPISession.java:91)
        at oracle.sysman.oii.oiic.OiicInstallAPISession$OiicActionsThread.run(OiicInstallAPISession.java:948)

Solution:
=========
Confirm that you have disabled the "iptables"

Check if firewall is running
# service iptables status
If it is running, stop it and retry instalation.
# service iptables stop

ntcontab.o - Error in Grid control agent instal

Error in Grid control agent installation:

Error:
./runInstaller -CLUSTER_NAME=rac_prd_cluster CLUSTER_NODES="{node-01,node-02}"

ntcontab.o network/lib/ins_net_client.mk

Solution:
------
Install 32 as well as 64 bit packages as below

rpm -ivh glibc-devel-2.5-24.i386.rpm
rpm -ivh  libstdc++-4.1.2-51.el5.x86_64.rpm
rpm -ivh libstdc++-devel-4.1.2-51.el5.x86_64.rpm
rpm -ivh compat-libstdc++-33-3.2.3-61.x86_64.rpm
rpm -ivh compat-libstdc++-33-3.2.3-61.i386.rpm
rpm -ivh compat-libstdc++-296-2.96-138.i386.rpm
rpm -ivh  glibc-devel-2.5-24.i386.rpm
rpm -ivh  glibc-devel-2.5-24.x86_64.rpm
rpm -ivh libXp-1.0.0-8.x86_64.rpm
rpm -ivh libXp-1.0.0-8.i386.rpm


The retry
./runInstaller -CLUSTER_NAME=rac_prd_cluster CLUSTER_NODES="{node-01,node-02}"

Ref: PROBLEM: Installation of the Grid Control Agent 10.2.0.4/5 on Linux x86_64 Fails: Error in invoking target 'ntcontab.o' of makefile [ID 861347.1]

INS 40912 - Rac Grid installation fail with

Rac Grid installation fail with INS 40912

Error:
--------
INS 40912 virtual host name is assigned to another system on the network


Solution:
--------
This could be due Virtual host name is used by some one or wrong configuration of vitual host
ping <vitual hostname> or <vitual ip>
If ping was working check the network configuration and Restart the newtork

#service network restart

Root.sh fails with crsconfig_lib.pm line 1016

Rac installtion,Root.sh fails with crsconfig_lib.pm line 1016
Error:
=========
Node-2: root.sh fails as below.

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node za-rac-uat-01, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Failed to start Oracle Clusterware stack
Failed to start Cluster Synchorinisation Service in clustered mode at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1016.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

Solution:
========
As per note: Oracle Grid Infrastructure 11.2.0.2 Installation or Upgrade may fail due to Multicasting Requirement [ID 1212703.1]
Apply patch 9974223
Download the patch and apply as below.
As root user:
#/u01/app/11.2.0/grid/crs/install/rootcrs.pl -unlock
#cd <GRID_HOME>/Opatch/
#./opatch napply -local -oh /u01/app/11.2.0/grid -id 9974223
 
#opatch lsinventory -detail -oh /u01/app/11.2.0/grid
#./opatch lsinventory -oh /u01/app/11.2.0/grid

As a root run below.
#<GRID_HOME>/crs/install/rootcrs.pl -patch

Repeat this on Node-2.Then check the CRS status.
crsctl stat res -t

OS Kernal upgrade on RAC ASM system

OS Kernal upgrade on RAC ASM system

How upgrade OS with out RAC  Grid infrastructure reinstall.

1.Stop the database and GRID from the OS Upgrading node.
    <GRID_HOME>/bin/crsctl stop crs
2.Take OCR back up
    <GRID_HOME>/bin/ocrconfig -export
3.Disable auto start of Grid infra
    $GRID_HOME/bin/crsctl disable crs
4.upgrade the OS(Server team will take care)
       onces the system is up check and apply the compatible ASMLib for the kernal which you has been upgraded.

Adding ASM disk group in VM lab

Adding ASM disk group in VM lab

We received the following message

ERROR at line 1:
ORA-01119: error in creating database file '+DATA'
ORA-17502: ksfdcre:4 Failed to create file +DATA
ORA-15041: diskgroup space exhausted

Check the diskspace in ASM:
SQL> select GROUP_NUMBER, NAME,TOTAL_MB, FREE_MB, USABLE_FILE_MB from V$ASM_DISKGROUP;

GROUP_NUMBER NAME TOTAL_MB FREE_MB USABLE_FILE_MB
------------ ------------------------------ ---------- ---------- --------------
1 ARCHIVELOGS 1349 1283 1283
2 DATA 2039 4 4
3 ONLINELOGS 705 581 581

SQL> select substr(name,1,10) name,substr(path,1,20) path, REDUNDANCY, TOTAL_MB, os_mb, free_mb from V$ASM_DISK where GROUP_NUMBER = 2;

NAME PATH REDUNDA TOTAL_MB OS_MB FREE_MB
---------- -------------------- ------- ---------- ---------- ----------
VOL1 ORCL:VOL1 UNKNOWN 2039 2039 4

First try to extend current diskgroup.

SQL> ALTER DISKGROUP data RESIZE DISK VOL1 size 4G;
ALTER DISKGROUP data RESIZE DISK VOL1 size 4G
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15289: ASM disk VOL1 cannot be resized beyond 2039 M

Diskgroup cannot be extended. Only solution is to add a new disk.
After adding the new disk, the new disk device is /dev/sde

List all current disks on Linux:
sfdisk -l

Now create a partition on /dev/sde to span the whole disk.

[root@dbvisit32 /]# fdisk /dev/sde

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-261, default 261):
Using default value 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
[root@dbvisit32 /]# ls -la /dev/sde*
brw-r----- 1 root disk 8, 64 Jun 7 11:59 /dev/sde
brw-r----- 1 root disk 8, 65 Jun 7 11:59 /dev/sde1

Partition /dev/sde1 is now created.

Now make the disk available to ASM.

[root@dbvisit31 ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
[root@dbvisit31 ~]# /etc/init.d/oracleasm createdisk VOL4 /dev/sde1
Marking disk "/dev/sde1" as an ASM disk: [ OK ]
[root@dbvisit31 ~]# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4

Go back to ASM and add the new disk.

SQL> select substr(name,1,10) name,substr(path,1,20) path, REDUNDANCY, TOTAL_MB, os_mb, free_mb from V$ASM_DISK;

NAME PATH REDUNDA TOTAL_MB OS_MB FREE_MB
---------- -------------------- ------- ---------- ---------- ----------
ORCL:VOL4 UNKNOWN 0 2047 0
VOL1 ORCL:VOL1 UNKNOWN 2039 2039 215
VOL2 ORCL:VOL2 UNKNOWN 705 705 606
VOL3 ORCL:VOL3 UNKNOWN 1349 1349 1214

SQL> ALTER DISKGROUP DATA ADD DISK 'ORCL:VOL4';

SQL> select substr(name,1,10) name,substr(path,1,20) path, REDUNDANCY, TOTAL_MB, os_mb, free_mb from V$ASM_DISK where GROUP_NUMBER = 2;

NAME PATH REDUNDA TOTAL_MB OS_MB FREE_MB
---------- -------------------- ------- ---------- ---------- ----------
VOL1 ORCL:VOL1 UNKNOWN 2039 2039 368
VOL4 ORCL:VOL4 UNKNOWN 2047 2047 1892

All done. Disk group DATA now has an extra 2G.