In the Node alert crs log, we are getting below error:
2014-11-27 21:03:31.095
[/u01/app/11.2.0/grid/bin/oraagent.bin(15001)]CRS-5818:Aborted command 'start for resource: ora.gpnpd 1 1' for resource 'ora.gpnpd'. Details at (:CRSAGF00 113:) {0:0:2} in /u01/app/11.2.0/grid/log/hostname/agent/ohasd/oraagent_grid/oraagent_grid.log.
2014-11-27 21:03:35.100
[ohasd(14885)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/ grid/log/hostname/ohasd/ohasd.log.
2014-11-27 21:03:35.504
[mdnsd(15013)]CRS-5602:mDNS service stopping by request.
2014-11-27 21:03:36.716
[/u01/app/11.2.0/grid/bin/orarootagent.bin(16118)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/acfsload" spawned by agent "/u01/app/11.2.0/grid/bin/oraroota gent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/za-rac-prd-01/agent/ohasd/orarootagent_root/orarootagent_root .log"
[ clsdmc][1111984448]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)(KEY=DBG_GPNPD))][6]
2014-11-27 23:37:00.921: [ora.gpnpd][1111984448] {0:0:2} [start] Error = error 6 encountered when sending messages to GPNPD
2014-11-27 23:37:01.922: [ora.gpnpd][1111984448] {0:0:2} [start] without returnbuf
2014-11-27 23:37:01.922: [ COMMCRS][1111984448]clscsendx: (0x2aaaac068830) Connection not active
lient(10786)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/node-01/client/emcrsp.log.
2014-11-27 20:17:15.234
[client(11061)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-11-27 20:17:15.236
[client(11061)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/node-01/client/emcrsp.log.
011-08-17 17:51:27.601: [ OCRASM][2931596144]proprasmo: Failed to open file in dirty mode
2011-08-17 17:51:27.601: [ OCRASM][2931596144]proprasmo: Error in open/create file in dg [OCR_VOTE]
[ OCRASM][2931596144]SLOS : SLOS: cat=8, opn=kgfolclcpi1, dep=204, loc=kgfokge
2011-08-17 17:51:27.601: [ OCRASM][2931596144]ASM Error Stack : ORA-15077: could not locate ASM instance serving a required diskgroup
2011-08-17 17:51:27.756: [ OCRASM][2931596144]proprasmo: kgfoCheckMount returned [6]
2011-08-17 17:51:27.756: [ OCRASM][2931596144]proprasmo: The ASM disk group OCR_VOTE is not found or not mounted
2011-08-17 17:51:27.761: [ OCRRAW][2931596144]proprioo: Failed to open [+OCR_VOTE]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2011-08-17 17:51:27.761: [ OCRRAW][2931596144]proprioo: No OCR/OLR devices are usable
2011-08-17 17:51:27.761: [ OCRASM][2931596144]proprasmcl: asmhandle is NULL
2011-08-17 17:51:27.761: [ OCRRAW][2931596144]proprinit: Could not open raw device
2011-08-17 17:51:27.761: [ OCRASM][2931596144]proprasmcl: asmhandle is NULL
2011-08-17 17:51:27.761: [ default][2931596144]a_init:7!: Backend init unsuccessful : [26]
2011-08-20 01:32:09.691: [ CSSCLNT][2931596144]clsssinit: clsssInitNative returned unsuccesfully, but client did not specify native css connection, rc(3)
2011-08-20 01:32:09.691: [ CSSCLNT][2931596144]clsssinit: clsssInitNative returned unsuccesfully, but client did not specify native css connection, rc(3)
===================================
[root@node-01 gpnpd]# ls -ltr
total 744
-rw-r--r-- 1 root root 6 Nov 27 19:06 node-01.pid
-rw-r--r-- 1 grid oinstall 11700 Nov 28 00:23 gpnpdOUT.log
-rw-r--r-- 1 grid oinstall 739913 Nov 28 00:23 gpnpd.log
[root@node-01 gpnpd]# pwd
/u01/app/11.2.0/grid/log/node-01/gpnpd
[root@node-01 gpnpd]# tail -100f gpnpd.log
2014-11-28 00:23:48.075: [ default][1640935600]gpnpd START pid=2462 Oracle Grid Plug-and-Play Daemon
2014-11-28 00:23:48.076: [ GPNP][1640935600]clsgpnp_Init: [at clsgpnp0.c:582] '/u01/app/11.2.0/grid' in effect as GPnP home base.
2014-11-28 00:23:48.076: [ GPNP][1640935600]clsgpnp_Init: [at clsgpnp0.c:616] GPnP pid=2462, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
2014-11-28 00:23:48.080: [ GPNP][1640935600]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /u01/app/11.2.0/grid/gpnp/node-01/wallets/peer/
[ CLWAL][1640935600]clsw_Initialize: OLR initlevel [70000]
[ clsdmt][1101838656]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=node-01DBG_GPNPD))
2014-11-28 00:23:48.096: [ clsdmt][1101838656]PID for the Process [2462], connkey 10
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Creating PID [2462] file for home /u01/app/11.2.0/grid host node-01 bin gpnp to /u01/app/11.2.0/grid/gpnp/init/
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Error3 -2 writing PID [2462] to the file []
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Failed to record pid for GPNPD
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Terminating process
2014-11-28 00:23:48.096: [ GPNP][1101838656]CLSDM requested exit
2014-11-28 00:23:48.097: [ default][1101838656]GPNPD on node node-01 shutdown.
Solution:
After following ASM check << http://oracletechdba.blogspot.com/2014/12/ocrcheck-proc-26-error-while-accessing.html >>
We narrow down that the issue is not with ASM and with the help of friends Rijesh and vignesh,Manage to find the root cause.
we are not having file permission on <node_name>.<date>.orig
So followed the below solution as suggested in Note:Unable To Start CRS 11gR2 Due to Missing Files in $GRID_HOME/<daemon>/init Directory (Doc ID 1146055.1)
cp $GRID_HOME/<daemon>/init/<node_name> $GRID_HOME/<daemon>/init/<node_name>.<date>.orig
OR
cp $GRID_HOME/<daemon>/init/<node_name>.pid $GRID_HOME/<daemon>/init/<node_name>.pid.<date>.orig
Then restarted the CRS.
./crsctl start crs
It worked :)
2014-11-27 21:03:31.095
[/u01/app/11.2.0/grid/bin/oraagent.bin(15001)]CRS-5818:Aborted command 'start for resource: ora.gpnpd 1 1' for resource 'ora.gpnpd'. Details at (:CRSAGF00 113:) {0:0:2} in /u01/app/11.2.0/grid/log/hostname/agent/ohasd/oraagent_grid/oraagent_grid.log.
2014-11-27 21:03:35.100
[ohasd(14885)]CRS-2757:Command 'Start' timed out waiting for response from the resource 'ora.gpnpd'. Details at (:CRSPE00111:) {0:0:2} in /u01/app/11.2.0/ grid/log/hostname/ohasd/ohasd.log.
2014-11-27 21:03:35.504
[mdnsd(15013)]CRS-5602:mDNS service stopping by request.
2014-11-27 21:03:36.716
[/u01/app/11.2.0/grid/bin/orarootagent.bin(16118)]CRS-5016:Process "/u01/app/11.2.0/grid/bin/acfsload" spawned by agent "/u01/app/11.2.0/grid/bin/oraroota gent.bin" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/11.2.0/grid/log/za-rac-prd-01/agent/ohasd/orarootagent_root/orarootagent_root .log"
[ clsdmc][1111984448]Failed to send meta message to connection [(ADDRESS=(PROTOCOL=ipc)(KEY=DBG_GPNPD))][6]
2014-11-27 23:37:00.921: [ora.gpnpd][1111984448] {0:0:2} [start] Error = error 6 encountered when sending messages to GPNPD
2014-11-27 23:37:01.922: [ora.gpnpd][1111984448] {0:0:2} [start] without returnbuf
2014-11-27 23:37:01.922: [ COMMCRS][1111984448]clscsendx: (0x2aaaac068830) Connection not active
lient(10786)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/node-01/client/emcrsp.log.
2014-11-27 20:17:15.234
[client(11061)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-11-27 20:17:15.236
[client(11061)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/node-01/client/emcrsp.log.
011-08-17 17:51:27.601: [ OCRASM][2931596144]proprasmo: Failed to open file in dirty mode
2011-08-17 17:51:27.601: [ OCRASM][2931596144]proprasmo: Error in open/create file in dg [OCR_VOTE]
[ OCRASM][2931596144]SLOS : SLOS: cat=8, opn=kgfolclcpi1, dep=204, loc=kgfokge
2011-08-17 17:51:27.601: [ OCRASM][2931596144]ASM Error Stack : ORA-15077: could not locate ASM instance serving a required diskgroup
2011-08-17 17:51:27.756: [ OCRASM][2931596144]proprasmo: kgfoCheckMount returned [6]
2011-08-17 17:51:27.756: [ OCRASM][2931596144]proprasmo: The ASM disk group OCR_VOTE is not found or not mounted
2011-08-17 17:51:27.761: [ OCRRAW][2931596144]proprioo: Failed to open [+OCR_VOTE]. Returned proprasmo() with [26]. Marking location as UNAVAILABLE.
2011-08-17 17:51:27.761: [ OCRRAW][2931596144]proprioo: No OCR/OLR devices are usable
2011-08-17 17:51:27.761: [ OCRASM][2931596144]proprasmcl: asmhandle is NULL
2011-08-17 17:51:27.761: [ OCRRAW][2931596144]proprinit: Could not open raw device
2011-08-17 17:51:27.761: [ OCRASM][2931596144]proprasmcl: asmhandle is NULL
2011-08-17 17:51:27.761: [ default][2931596144]a_init:7!: Backend init unsuccessful : [26]
2011-08-20 01:32:09.691: [ CSSCLNT][2931596144]clsssinit: clsssInitNative returned unsuccesfully, but client did not specify native css connection, rc(3)
2011-08-20 01:32:09.691: [ CSSCLNT][2931596144]clsssinit: clsssInitNative returned unsuccesfully, but client did not specify native css connection, rc(3)
===================================
[root@node-01 gpnpd]# ls -ltr
total 744
-rw-r--r-- 1 root root 6 Nov 27 19:06 node-01.pid
-rw-r--r-- 1 grid oinstall 11700 Nov 28 00:23 gpnpdOUT.log
-rw-r--r-- 1 grid oinstall 739913 Nov 28 00:23 gpnpd.log
[root@node-01 gpnpd]# pwd
/u01/app/11.2.0/grid/log/node-01/gpnpd
[root@node-01 gpnpd]# tail -100f gpnpd.log
2014-11-28 00:23:48.075: [ default][1640935600]gpnpd START pid=2462 Oracle Grid Plug-and-Play Daemon
2014-11-28 00:23:48.076: [ GPNP][1640935600]clsgpnp_Init: [at clsgpnp0.c:582] '/u01/app/11.2.0/grid' in effect as GPnP home base.
2014-11-28 00:23:48.076: [ GPNP][1640935600]clsgpnp_Init: [at clsgpnp0.c:616] GPnP pid=2462, GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:ORA_DAEMON_LOGGING_LEVELS, apitl:0, complog:1, tstenv:0, devenv:0, envopt:0, flags=3
2014-11-28 00:23:48.080: [ GPNP][1640935600]clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:399] Using FS Wallet Location : /u01/app/11.2.0/grid/gpnp/node-01/wallets/peer/
[ CLWAL][1640935600]clsw_Initialize: OLR initlevel [70000]
[ clsdmt][1101838656]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=node-01DBG_GPNPD))
2014-11-28 00:23:48.096: [ clsdmt][1101838656]PID for the Process [2462], connkey 10
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Creating PID [2462] file for home /u01/app/11.2.0/grid host node-01 bin gpnp to /u01/app/11.2.0/grid/gpnp/init/
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Error3 -2 writing PID [2462] to the file []
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Failed to record pid for GPNPD
2014-11-28 00:23:48.096: [ clsdmt][1101838656]Terminating process
2014-11-28 00:23:48.096: [ GPNP][1101838656]CLSDM requested exit
2014-11-28 00:23:48.097: [ default][1101838656]GPNPD on node node-01 shutdown.
Solution:
After following ASM check << http://oracletechdba.blogspot.com/2014/12/ocrcheck-proc-26-error-while-accessing.html >>
We narrow down that the issue is not with ASM and with the help of friends Rijesh and vignesh,Manage to find the root cause.
we are not having file permission on <node_name>.<date>.orig
So followed the below solution as suggested in Note:Unable To Start CRS 11gR2 Due to Missing Files in $GRID_HOME/<daemon>/init Directory (Doc ID 1146055.1)
cp $GRID_HOME/<daemon>/init/<node_name> $GRID_HOME/<daemon>/init/<node_name>.<date>.orig
OR
cp $GRID_HOME/<daemon>/init/<node_name>.pid $GRID_HOME/<daemon>/init/<node_name>.pid.<date>.orig
Then restarted the CRS.
./crsctl start crs
It worked :)
No comments:
Post a Comment