Oracle - Troubleshooting on Oracle Exadata Problem on slow crsctl command
The server is highly loaded to make "crsctl status res -t" to run slowly.
Oracle support requested the below checking:
Check image version
imageinfo
Kernel version: 2.6.18-194.3.1.0.4.el5 #1 SMP Sat Feb 19 03:38:37 EST 2011 x86_64
Image version: 11.2.2.3.2.110520
Image activated: 2011-07-18 12:16:20 +0800
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1
OS watcher
Os watcher logs are stored in below location:
ls -lt /opt/oracle.oswatcher/osw/archive
Above oswatcher logs are retained for 10 days.
Oracle support suggested to upload the OS watcher on the day when reboot issue happened using below way :
cd /opt/oracle.oswatcher/osw/archive
find . -name '*05.09*' -print -exec zip /tmp/osw_`hostname`.zip {} \;
Replace 05.09 with month and day if problem happened on a different day.
Check patch level
opatch lsinventory -details
DB01 :
top - 14:15:39 up 149 days, 11:11, 2 users, load average: 2.47, 2.57, 2.56
Tasks: 2300 total, 2 running, 2297 sleeping, 0 stopped, 1 zombie
Cpu(s): 6.7%us, 2.5%sy, 0.0%ni, 90.7%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 98848968k total, 98120440k used, 728528k free, 1918844k buffers
Swap: 25165816k total, 2983856k used, 22181960k free, 53953164k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10399 root 15 0 2538m 1.6g 25m S 99.7 1.7 106855:51 crsd.bin
8592 grid RT 0 411m 187m 52m S 4.9 0.2 5918:51 ocssd.bin
11062 oracle 15 0 1918m 103m 16m S 2.3 0.1 3378:21 oraagent.bin
6996 root 34 19 0 0 0 S 0.3 0.0 1134:58 kipmi0
8531 root 16 0 422m 49m 15m S 0.7 0.1 838:07.13 orarootagent.bi
10512 root 15 0 814m 24m 13m S 0.7 0.0 810:29.37 orarootagent.bi
Conclusion
matches symptoms as said in bugs 10019726, bug 12615394, and bug 12767563
as said in Doc ID 1348202.1
1) bug 10019726, bug 12615394, and bug 12767563 are fixed in 11.2.0.2 GI PSU4 and 11.2.0.3.
(11.2.0.2 BP16) - Patch 13837673 has 11.2.0.2 GI PSU6.
So Kindly apply (11.2.0.2 BP16) - Patch 13837673 or apply 11.2.0.3.
2) If pre-11.2 database exists, it's recommended to apply patch 12801833 to pre-11.2 homes.
Oracle support requested the below checking:
Check image version
imageinfo
Kernel version: 2.6.18-194.3.1.0.4.el5 #1 SMP Sat Feb 19 03:38:37 EST 2011 x86_64
Image version: 11.2.2.3.2.110520
Image activated: 2011-07-18 12:16:20 +0800
Image status: success
System partition on device: /dev/mapper/VGExaDb-LVDbSys1
OS watcher
Os watcher logs are stored in below location:
ls -lt /opt/oracle.oswatcher/osw/archive
Above oswatcher logs are retained for 10 days.
Oracle support suggested to upload the OS watcher on the day when reboot issue happened using below way :
cd /opt/oracle.oswatcher/osw/archive
find . -name '*05.09*' -print -exec zip /tmp/osw_`hostname`.zip {} \;
Replace 05.09 with month and day if problem happened on a different day.
Check patch level
opatch lsinventory -details
Check CRSD log
CRSD log of each node is located at :
$GRID_HOME/log/
Check top command (sorted by accumulated CPU time)
|
DB01 :
top - 14:15:39 up 149 days, 11:11, 2 users, load average: 2.47, 2.57, 2.56
Tasks: 2300 total, 2 running, 2297 sleeping, 0 stopped, 1 zombie
Cpu(s): 6.7%us, 2.5%sy, 0.0%ni, 90.7%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 98848968k total, 98120440k used, 728528k free, 1918844k buffers
Swap: 25165816k total, 2983856k used, 22181960k free, 53953164k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10399 root 15 0 2538m 1.6g 25m S 99.7 1.7 106855:51 crsd.bin
8592 grid RT 0 411m 187m 52m S 4.9 0.2 5918:51 ocssd.bin
11062 oracle 15 0 1918m 103m 16m S 2.3 0.1 3378:21 oraagent.bin
6996 root 34 19 0 0 0 S 0.3 0.0 1134:58 kipmi0
8531 root 16 0 422m 49m 15m S 0.7 0.1 838:07.13 orarootagent.bi
10512 root 15 0 814m 24m 13m S 0.7 0.0 810:29.37 orarootagent.bi
Conclusion
matches symptoms as said in bugs 10019726, bug 12615394, and bug 12767563
as said in Doc ID 1348202.1
Proposed Solution
(11.2.0.2 BP16) - Patch 13837673 has 11.2.0.2 GI PSU6.
So Kindly apply (11.2.0.2 BP16) - Patch 13837673 or apply 11.2.0.3.
2) If pre-11.2 database exists, it's recommended to apply patch 12801833 to pre-11.2 homes.
Comments