Exadata – skgxpinfo returns UDP after reconfiguration


07.05.2014
by Kamil Stawiarski

We had to change the hostnames on Exadata with one of the customers. It is a well known fact that RAC doesn’t like such changes – we had to reconfigure the cluster and after the whole process I found the funny thing:

[oracle@hubdb01 ~]$ . grid.env 
[oracle@hubdb01 ~]$ env | grep ORA
ORACLE_SID=+ASM1
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/11.2.0.3/grid
[oracle@hubdb01 ~]$ skgxpinfo 
udp

So let’s try to turn RDS on 🙂
After shutting down all cluster resources, we have to recompile the binaries (I can never remember the needed options of the ins_rdbms.mk makefile, but when you use "grep", you will find all the necessary informations).

[root@hubdb02 lib]# grep "rds" ins_rdbms.mk 
ipc_rds:
	(if $(ORACLE_HOME)/bin/skgxpinfo | grep rds;\
	$(MAKE) -f  $(MAKEFILE) ipc_rds; \
[root@hubdb02 lib]# env | grep ORACLE_HOME
ORACLE_HOME=/u01/app/11.2.0.3/grid
[root@hubdb02 lib]# pwd
/u01/app/11.2.0.3/grid/rdbms/lib
[root@hubdb02 lib]# make -f ins_rdbms.mk ipc_rds ioracle
rm -f /u01/app/11.2.0.3/grid/lib/libskgxp11.so
(output removed for clarity)

And now there is a trick – ownership of the "/u01/app/11.2.0.3/grid/bin/oracle" binary has changed to root:root.

[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
-rwsr-s--x 1 root root 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle

Because of that, the shared memory segment of the ASM instnce will have insufficient privileges for CRS deamon to access it. Therefore you will find the following errors in grid alertlog:

[crsd(126495)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
ORA-01031: insufficient privileges
]. Details at (:CRSD00111:) in /u01/app/11.2.0.3/grid/log/hubdb01/crsd/crsd.log.

So before starting the ASM instance, we have to change the privileges for the "oracle" binary file.

[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
-rwsr-s--x 1 root root 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle
[root@hubdb02 lib]# chown oracle:oinstall /u01/app/11.2.0.3/grid/bin/oracle
[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
-rwxr-x--x 1 oracle oinstall 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle
[root@hubdb02 lib]# chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle
[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
-rwsr-s--x 1 oracle oinstall 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle

The same thing has to be done on the second node:

[root@hubdb01 ~]# cd $ORACLE_HOME/rdbms/lib
[root@hubdb01 lib]# pwd
/u01/app/11.2.0.3/grid/rdbms/lib
[root@hubdb01 lib]# grep "rds" ins_rdbms.mk 
ipc_rds:
	(if $(ORACLE_HOME)/bin/skgxpinfo | grep rds;\
	$(MAKE) -f  $(MAKEFILE) ipc_rds; \
[root@hubdb01 lib]# make -f ins_rdbms.mk ipc_rds ioracle
rm -f /u01/app/11.2.0.3/grid/lib/libskgxp11.so
cp /u01/app/11.2.0.3/grid/lib//libskgxpr.so /u01/app/11.2.0.3/grid/lib/libskgxp11.so
chmod 755 /u01/app/11.2.0.3/grid/bin

 - Linking Oracle 
rm -f /u01/app/11.2.0.3/grid/rdbms/lib/oracle
gcc  -o /u01/app/11.2.0.3/grid/rdbms/lib/oracle -m64 -L/u01/app/11.2.0.3/grid/rdbms/lib/ -L/u01/app/11.2.0.3/grid/lib/ -L/u01/app/11.2.0.3/grid/lib/stubs/   -Wl,-E /u01/app/11.2.0.3/grid/rdbms/lib/opimai.o /u01/app/11.2.0.3/grid/rdbms/lib/ssoraed.o /u01/app/11.2.0.3/grid/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/11.2.0.3/grid/lib/nautab.o /u01/app/11.2.0.3/grid/lib/naeet.o /u01/app/11.2.0.3/grid/lib/naect.o /u01/app/11.2.0.3/grid/lib/naedhs.o /u01/app/11.2.0.3/grid/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/11.2.0.3/grid/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/11.2.0.3/grid/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/11.2.0.3/grid/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -Wl,-rpath,/u01/app/11.2.0.3/grid/lib -lm    `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -ldl -lm   -L/u01/app/11.2.0.3/grid/lib
test ! -f /u01/app/11.2.0.3/grid/bin/oracle ||\
	   mv -f /u01/app/11.2.0.3/grid/bin/oracle /u01/app/11.2.0.3/grid/bin/oracleO
mv /u01/app/11.2.0.3/grid/rdbms/lib/oracle /u01/app/11.2.0.3/grid/bin/oracle
chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle
[root@hubdb01 lib]# chown oracle:oinstall /u01/app/11.2.0.3/grid/bin/oracle
[root@hubdb01 lib]# chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle
[root@hubdb01 lib]# 

Now we can start the cluster resources:

[root@hubdb01 lib]# crsctl start res ora.asm -init
CRS-2672: Attempting to start 'ora.asm' on 'hubdb01'
CRS-2676: Start of 'ora.asm' on 'hubdb01' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'hubdb01'
CRS-2676: Start of 'ora.crsd' on 'hubdb01' succeeded
[root@hubdb01 lib]# ssh hubdb02
Last login: Wed May  7 12:25:42 2014 from hub01.arrowecs.hub
[root@hubdb02 ~]# . /home/oracle/grid.env 
[root@hubdb02 ~]# crsctl start res ora.asm -init
CRS-2672: Attempting to start 'ora.asm' on 'hubdb02'
CRS-2676: Start of 'ora.asm' on 'hubdb02' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'hubdb02'
CRS-2676: Start of 'ora.crsd' on 'hubdb02' succeeded

And that’s it. 🙂

Search:


Kontakt

Database Whisperers sp. z o. o. sp. k.
Al. Jerozolimskie 200
Wejście B, III piętro/ pokój 342
02-486 Warszawa
NIP: 5272744987
REGON:362524978
+48 508 943 051
+48 661 966 009
info@ora-600.pl

Newsletter zapisz się żeby być na bieżąco