Standalone Oracle Grid Infrastructure in EXADATA
1st set up exadata server using my previous post
<< http://oracletechdba.blogspot.com/2014/06/building-exadata-in-vm-for-learning.html#more >>
Now preparing the server for installation.
Pre check and setup for
------------------------
1./etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048586
kernel.core_uses_pid = 1
fs.suid_dumpable = 1
/sbin/sysctl -p
---------------------
2./etc/security/limits.conf
oracle soft nofile 131072
oracle hard nofile 131072
oracle soft nproc 131072
oracle hard nproc 131072
oracle soft core unlimited
oracle hard core unlimited
oracle soft memlock 3500000
oracle hard memlock 3500000
grid soft nofile 131072
grid hard nofile 131072
grid soft nproc 131072
grid hard nproc 131072
grid soft core unlimited
grid hard core unlimited
grid soft memlock 3500000
grid hard memlock 3500000
grid soft nproc 2047
3.Add on "/etc/pam.d/login" file, if it does not already exist.
session required pam_limits.so
------------------------------------
4./etc/selinux/config" file, making sure the SELINUX flag is set as follows.
SELINUX=disabled
------------------------------------
5./etc/hosts
cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.124.129 hostname # eth0
------------------------------------
6.
/usr/sbin/groupadd -g 1000 oinstall
/usr/sbin/groupadd -g 1020 asmadmin
/usr/sbin/groupadd -g 1021 asmdba
/usr/sbin/groupadd -g 1022 asmoper
/usr/sbin/groupadd -g 1031 dba
/usr/sbin/useradd -u 1100 -g oinstall -G asmdba,dba,asmadmin,asmdba,asmoper grid
/usr/sbin/useradd -u 1101 -g oinstall -G asmdba,dba,asmadmin,asmdba,asmoper oracle
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/grid
mkdir -p /u01/app/oracle
chown grid:oinstall /u01/app/11.2.0/grid
chown grid:oinstall /u01/app/grid
chown oracle:oinstall /u01/app/oracle
chmod -R 775 /u01/
chown -R grid:oinstall /u01
++Set the password
passwd grid
passwd oracle
------------------------------------
7.
rpm -q binutils-2.17.50.0.6
rpm -q compat-libstdc++-33-3.2.3
rpm -q elfutils-libelf-0.125
rpm -q elfutils-libelf-devel-0.125
rpm -q gcc-4.1.2
rpm -q bgcc-c++-4.1.2
rpm -q glibc-2.5-24
rpm -q glibc-common-2.5
rpm -q glibc-devel-2.5
rpm -q glibc-headers-2.5
rpm -q ksh-20060214
rpm -q libaio-0.3.106
rpm -q libaio-devel-0.3.106
rpm -q libgcc-4.1.2
rpm -q libstdc++-4.1.2
rpm -q libstdc++-devel 4.1.2
rpm -q make-3.81
rpm -q numactl-devel-0.9.8.x86_64
rpm -q sysstat-7.0.2
rpm -q unixODBC-2.2.11
rpm -q unixODBC-devel-2.2.11
cd < to the rpm location >
yum install localhost package*
------------------------------------
8.
/etc/profile
ulimit -S -c 0 > /dev/null 2>&1
Change it to the following:
ulimit -S -c unlimited > /dev/null 2>&1
And
------------------------------------
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
umask 022
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
unset i
unset pathmunge
------------------------------------
9.
After grid software is Copied to server,
++
cvuqdisk-1.0.9-1.rpm /stage/cvu/cv/remenv/
[root@ stage]# rpm -iv cvuqdisk-1.0.9-1.rpm
Preparing packages for installation...
Using default group oinstall to install package
cvuqdisk-1.0.9-1
[root@ stage]# rpm -e cvuqdisks not installed
[root@ stage]# CVUQDISK_GRP=oinstall; export CVUQDISK_GRP
[root@ stage]# rpm -ivh cvuqdisk-1.0.9-1.rpm
++ create storage management IP in $OSSCONF/cellip.ora
cell="192.168.124.129"
cell="localhost"
And cross check the ip in $OSSCONF/cell_disk_config.xml file
<Attribute NAME="name" VALUE="localhost"></Attribute>
++ Now check the disk are available before installation using kfod tool in the software stage location.From the below output you can see the disks are available from ASM Disk group creation.
[root@localhost config]# export LD_LIBRARY_PATH=/u01/grid/stage/ext/lib
[root@localhost config]# /u01/grid/stage/ext/bin/kfod disks=all op=disks
--------------------------------------------------------------------------------
Disk Size Path User Group
================================================================================
1: 464 Mb o/192.168.124.129/DATA_CD_DISK01_localhost <unknown> <unknown>
2: 464 Mb o/192.168.124.129/DATA_CD_DISK02_localhost <unknown> <unknown>
3: 464 Mb o/192.168.124.129/DATA_CD_DISK03_localhost <unknown> <unknown>
4: 464 Mb o/192.168.124.129/DATA_CD_DISK04_localhost <unknown> <unknown>
5: 464 Mb o/192.168.124.129/DATA_CD_DISK05_localhost <unknown> <unknown>
6: 464 Mb o/192.168.124.129/DATA_CD_DISK06_localhost <unknown> <unknown>
7: 464 Mb o/192.168.124.129/DATA_CD_DISK07_localhost <unknown> <unknown>
8: 464 Mb o/192.168.124.129/DATA_CD_DISK08_localhost <unknown> <unknown>
9: 464 Mb o/192.168.124.129/DATA_CD_DISK09_localhost <unknown> <unknown>
10: 464 Mb o/192.168.124.129/DATA_CD_DISK10_localhost <unknown> <unknown>
11: 464 Mb o/192.168.124.129/DATA_CD_DISK11_localhost <unknown> <unknown>
12: 464 Mb o/192.168.124.129/DATA_CD_DISK12_localhost <unknown> <unknown>
[root@localhost config]#
++ Now we are starting the installation as normal.
cd /u01/stage/grid/
./runInstaller
[root@localhost grid]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@localhost grid]# /u01/app/grid/product/11.2.0/grid/root.sh
Running Oracle 11g root script...
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/grid/product/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/grid/product/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'grid', privgrp 'oinstall'..
Operation successful.
LOCAL ONLY MODE
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4664: Node localhost successfully pinned.
Adding daemon to inittab
ls: /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/usm/install/Oracle/EL5/x86_64: No such file or directory
ACFS-9301: ADVM/ACFS installation can not proceed:
ACFS-9303: No installation files found for OS kernel version 2.6.18-348.el5.
localhost 2014/06/24 07:38:47 /u01/app/grid/product/11.2.0/grid/cdata/localhost/backup_20140624_073847.olr
Successfully configured Oracle Grid Infrastructure for a Standalone Server
[root@localhost grid]#
tail -100f /u01/app/grid/product/11.2.0/grid/cfgtoollogs/oracle.assistants.asm_2014-06-24_07-42-59-AM.log
INFO: Executing ASMCA
INFO: Command /u01/app/grid/product/11.2.0/grid/bin/asmca -silent -oui_internal -configureASM -diskGroupName ASM_META -diskList o/192.168.124.130/DATA_CD_DISK01_localhost,o/192.168.124.130/DATA_CD_DISK02_localhost,o/192.168.124.130/DATA_CD_DISK03_localhost -redundancy EXTERNAL
INFO: ... GenericInternalPlugIn.handleProcess() entered.
INFO: ... GenericInternalPlugIn: getting configAssistantParmas.
INFO: ... GenericInternalPlugIn: checking secretArguments.
INFO: ... GenericInternalPlugIn: starting read loop.
INFO: Read: SYS_PASSWORD_PROMPT
INFO: Processing: SYS_PASSWORD_PROMPT for argument tag -sysAsmPassword
INFO: Read: ASMSNMP_PASSWORD_PROMPT
INFO: Processing: ASMSNMP_PASSWORD_PROMPT for argument tag -asmMonitorPassword
INFO: End of argument passing to stdin
INFO: Read:
INFO: Read: ASM created and started successfully.
INFO: Read:
INFO: Read: Disk Group ASM_META created successfully.
INFO: Read:
INFO: oracle.assistants.asm:The plug-in Automatic Storage Management Configuration Assistant has successfully been performed
Checking the installation
[grid@localhost ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM
The Oracle base has been set to /u01/app/grid
ORACLE_SID = [grid] ? +ASM
The Oracle base has been set to /u01/app/grid
[grid@localhost ~]$ ps -ef|grep pmon
grid 22570 1 0 07:48 ? 00:00:00 asm_pmon_+ASM
grid 24946 24834 0 08:04 pts/7 00:00:00 grep pmon
grid 22570 1 0 07:48 ? 00:00:00 asm_pmon_+ASM
grid 24946 24834 0 08:04 pts/7 00:00:00 grep pmon
[grid@localhost ~]$ asmcmd lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 4194304 1392 1288 0 1288 0 N ASM_META/
[grid@localhost ~]$
[grid@localhost ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....META.dg ora....up.type ONLINE ONLINE localhost
ora....ER.lsnr ora....er.type ONLINE ONLINE localhost
ora.asm ora.asm.type ONLINE ONLINE localhost
ora.cssd ora.cssd.type ONLINE ONLINE localhost
ora.diskmon ora....on.type ONLINE ONLINE localhost
ora.evmd ora.evm.type ONLINE ONLINE localhost
ora.ons ora.ons.type OFFLINE OFFLINE
[grid@localhost ~]$
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 4096 4194304 1392 1288 0 1288 0 N ASM_META/
[grid@localhost ~]$
[grid@localhost ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....META.dg ora....up.type ONLINE ONLINE localhost
ora....ER.lsnr ora....er.type ONLINE ONLINE localhost
ora.asm ora.asm.type ONLINE ONLINE localhost
ora.cssd ora.cssd.type ONLINE ONLINE localhost
ora.diskmon ora....on.type ONLINE ONLINE localhost
ora.evmd ora.evm.type ONLINE ONLINE localhost
ora.ons ora.ons.type OFFLINE OFFLINE
[grid@localhost ~]$
]$ sqlplus
SQL*Plus: Release 11.2.0.2.0 Production on Tue Jun 24 08:08:00 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter user-name: /as sysasm
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Automatic Storage Management option
SQL> set pages 30
set lines 300
col path form a45
col name form a25
select group_number, header_status, disk_number, name, create_date, path, total_mb, free_mb
from v$asm_disk where header_status='MEMBER' order by group_number, disk_number, name
/SQL> SQL> SQL> SQL> 2 3
GROUP_NUMBER HEADER_STATU DISK_NUMBER NAME CREATE_DA PATH TOTAL_MB FREE_MB
------------ ------------ ----------- ------------------------- --------- --------------------------------------------- ---------- ----------
1 MEMBER 0 DATA_CD_DISK01_LOCALHOST 24-JUN-14 o/192.168.124.130/DATA_CD_DISK01_localhost 464 420
1 MEMBER 1 DATA_CD_DISK02_LOCALHOST 24-JUN-14 o/192.168.124.130/DATA_CD_DISK02_localhost 464 444
1 MEMBER 2 DATA_CD_DISK03_LOCALHOST 24-JUN-14 o/192.168.124.130/DATA_CD_DISK03_localhost 464 424
SQL> select group_number, name, state, type, total_mb, free_mb from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB
------------ ------------------------- ----------- ------ ---------- ----------
1 ASM_META MOUNTED EXTERN 1392 1288
SQL>
Continue ....
<< http://oracletechdba.blogspot.com/2014/06/creating-asm-disk-group-from-exadata.html >>
<< http://oracletechdba.blogspot.com/2014/07/oracle-11gr2-installation-in-standalone.html >>
SQL*Plus: Release 11.2.0.2.0 Production on Tue Jun 24 08:08:00 2014
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter user-name: /as sysasm
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Automatic Storage Management option
SQL> set pages 30
set lines 300
col path form a45
col name form a25
select group_number, header_status, disk_number, name, create_date, path, total_mb, free_mb
from v$asm_disk where header_status='MEMBER' order by group_number, disk_number, name
/SQL> SQL> SQL> SQL> 2 3
GROUP_NUMBER HEADER_STATU DISK_NUMBER NAME CREATE_DA PATH TOTAL_MB FREE_MB
------------ ------------ ----------- ------------------------- --------- --------------------------------------------- ---------- ----------
1 MEMBER 0 DATA_CD_DISK01_LOCALHOST 24-JUN-14 o/192.168.124.130/DATA_CD_DISK01_localhost 464 420
1 MEMBER 1 DATA_CD_DISK02_LOCALHOST 24-JUN-14 o/192.168.124.130/DATA_CD_DISK02_localhost 464 444
1 MEMBER 2 DATA_CD_DISK03_LOCALHOST 24-JUN-14 o/192.168.124.130/DATA_CD_DISK03_localhost 464 424
SQL> select group_number, name, state, type, total_mb, free_mb from v$asm_diskgroup;
GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB
------------ ------------------------- ----------- ------ ---------- ----------
1 ASM_META MOUNTED EXTERN 1392 1288
SQL>
Continue ....
<< http://oracletechdba.blogspot.com/2014/06/creating-asm-disk-group-from-exadata.html >>
<< http://oracletechdba.blogspot.com/2014/07/oracle-11gr2-installation-in-standalone.html >>
Hi Karl,
ReplyDeleteI am finished with the EXADATA CELL server setup.
So now I am trying to install grid infrastructure software for ASM.
Did you use the same server(CELL server) for the grid Infrastructure software install ?
Thanks a lot
John Oh
Hi John Oh,
ReplyDeleteYes i used the same server since i am not able to run multi VM.
But in real case it will be difference server.
Regard
~Karl
Hi Karl,
ReplyDeleteSounds good to me.
I am going to implement the grid infrastructure installation using the different server.
So If I implement the ASM and database on top of the exadata simulator, can I try most of the
exadata features such as smart scan and so on?
Thanks a lot Karl
John OH
This comment has been removed by the author.
ReplyDeleteI followed your post and successfully created a cell.
ReplyDeleteWhile creating the dbserver,the cell disks are not visible.
kfod is not returning anything
launched runInstaller and set the disovery location to cellip(o//*),but no luck
Hi,
ReplyDeleteI am facing the issue,while discovering ASM disk on DB nodes
my setup:
DB node: 172.168.1.30
eth1 ip:192.168.178.201 -- eth1
CELL Node:192.168.178.200 --eth1
[root@exadb1 Server]# /dump/grid/stage/ext/bin/kfod disks=all op=disks
[root@exadb1 Server]#
This comment has been removed by the author.
ReplyDeleteHi Karl Thanks a lot for this good post
ReplyDeletekindly help me with grid install using this cell storage i am not able to get disk on the vm i made for grid and db
do we need to share disk on vm level ?