We had to change the hostnames on Exadata with one of the customers. It is a well known fact that RAC doesn’t like such changes – we had to reconfigure the cluster and after the whole process I found the funny thing:
[oracle@hubdb01 ~]$ . grid.env [oracle@hubdb01 ~]$ env | grep ORA ORACLE_SID=+ASM1 ORACLE_BASE=/u01/app/oracle ORACLE_HOME=/u01/app/11.2.0.3/grid [oracle@hubdb01 ~]$ skgxpinfo udp
So let’s try to turn RDS on 🙂
After shutting down all cluster resources, we have to recompile the binaries (I can never remember the needed options of the ins_rdbms.mk makefile, but when you use "grep", you will find all the necessary informations).
[root@hubdb02 lib]# grep "rds" ins_rdbms.mk ipc_rds: (if $(ORACLE_HOME)/bin/skgxpinfo | grep rds;\ $(MAKE) -f $(MAKEFILE) ipc_rds; \ [root@hubdb02 lib]# env | grep ORACLE_HOME ORACLE_HOME=/u01/app/11.2.0.3/grid [root@hubdb02 lib]# pwd /u01/app/11.2.0.3/grid/rdbms/lib [root@hubdb02 lib]# make -f ins_rdbms.mk ipc_rds ioracle rm -f /u01/app/11.2.0.3/grid/lib/libskgxp11.so (output removed for clarity)
And now there is a trick – ownership of the "/u01/app/11.2.0.3/grid/bin/oracle" binary has changed to root:root.
[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle -rwsr-s--x 1 root root 205208185 May 7 12:31 /u01/app/11.2.0.3/grid/bin/oracle
Because of that, the shared memory segment of the ASM instnce will have insufficient privileges for CRS deamon to access it. Therefore you will find the following errors in grid alertlog:
[crsd(126495)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage ORA-01031: insufficient privileges ]. Details at (:CRSD00111:) in /u01/app/11.2.0.3/grid/log/hubdb01/crsd/crsd.log.
So before starting the ASM instance, we have to change the privileges for the "oracle" binary file.
[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle -rwsr-s--x 1 root root 205208185 May 7 12:31 /u01/app/11.2.0.3/grid/bin/oracle [root@hubdb02 lib]# chown oracle:oinstall /u01/app/11.2.0.3/grid/bin/oracle [root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle -rwxr-x--x 1 oracle oinstall 205208185 May 7 12:31 /u01/app/11.2.0.3/grid/bin/oracle [root@hubdb02 lib]# chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle [root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle -rwsr-s--x 1 oracle oinstall 205208185 May 7 12:31 /u01/app/11.2.0.3/grid/bin/oracle
The same thing has to be done on the second node:
[root@hubdb01 ~]# cd $ORACLE_HOME/rdbms/lib [root@hubdb01 lib]# pwd /u01/app/11.2.0.3/grid/rdbms/lib [root@hubdb01 lib]# grep "rds" ins_rdbms.mk ipc_rds: (if $(ORACLE_HOME)/bin/skgxpinfo | grep rds;\ $(MAKE) -f $(MAKEFILE) ipc_rds; \ [root@hubdb01 lib]# make -f ins_rdbms.mk ipc_rds ioracle rm -f /u01/app/11.2.0.3/grid/lib/libskgxp11.so cp /u01/app/11.2.0.3/grid/lib//libskgxpr.so /u01/app/11.2.0.3/grid/lib/libskgxp11.so chmod 755 /u01/app/11.2.0.3/grid/bin - Linking Oracle rm -f /u01/app/11.2.0.3/grid/rdbms/lib/oracle gcc -o /u01/app/11.2.0.3/grid/rdbms/lib/oracle -m64 -L/u01/app/11.2.0.3/grid/rdbms/lib/ -L/u01/app/11.2.0.3/grid/lib/ -L/u01/app/11.2.0.3/grid/lib/stubs/ -Wl,-E /u01/app/11.2.0.3/grid/rdbms/lib/opimai.o /u01/app/11.2.0.3/grid/rdbms/lib/ssoraed.o /u01/app/11.2.0.3/grid/rdbms/lib/ttcsoi.o -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/11.2.0.3/grid/lib/nautab.o /u01/app/11.2.0.3/grid/lib/naeet.o /u01/app/11.2.0.3/grid/lib/naect.o /u01/app/11.2.0.3/grid/lib/naedhs.o /u01/app/11.2.0.3/grid/rdbms/lib/config.o -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11 -lrt -lplp11 -lserver11 -lclient11 -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/11.2.0.3/grid/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/11.2.0.3/grid/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11 -lrt -lplp11 -ljavavm11 -lserver11 -lwwg `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags` -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/11.2.0.3/grid/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged -lippsmerged -lippcore -lippcpemerged -lippcpmerged -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -Wl,-rpath,/u01/app/11.2.0.3/grid/lib -lm `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -ldl -lm -L/u01/app/11.2.0.3/grid/lib test ! -f /u01/app/11.2.0.3/grid/bin/oracle ||\ mv -f /u01/app/11.2.0.3/grid/bin/oracle /u01/app/11.2.0.3/grid/bin/oracleO mv /u01/app/11.2.0.3/grid/rdbms/lib/oracle /u01/app/11.2.0.3/grid/bin/oracle chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle [root@hubdb01 lib]# chown oracle:oinstall /u01/app/11.2.0.3/grid/bin/oracle [root@hubdb01 lib]# chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle [root@hubdb01 lib]#
Now we can start the cluster resources:
[root@hubdb01 lib]# crsctl start res ora.asm -init CRS-2672: Attempting to start 'ora.asm' on 'hubdb01' CRS-2676: Start of 'ora.asm' on 'hubdb01' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'hubdb01' CRS-2676: Start of 'ora.crsd' on 'hubdb01' succeeded [root@hubdb01 lib]# ssh hubdb02 Last login: Wed May 7 12:25:42 2014 from hub01.arrowecs.hub [root@hubdb02 ~]# . /home/oracle/grid.env [root@hubdb02 ~]# crsctl start res ora.asm -init CRS-2672: Attempting to start 'ora.asm' on 'hubdb02' CRS-2676: Start of 'ora.asm' on 'hubdb02' succeeded CRS-2672: Attempting to start 'ora.crsd' on 'hubdb02' CRS-2676: Start of 'ora.crsd' on 'hubdb02' succeeded
And that’s it. 🙂