Exadata – skgxpinfo returns UDP after reconfiguration


07.05.2014
by Kamil Stawiarski

We had to change the hostnames on Exadata with one of the customers. It is a well known fact that RAC doesn’t like such changes – we had to reconfigure the cluster and after the whole process I found the funny thing:

1[oracle@hubdb01 ~]$ . grid.env
2[oracle@hubdb01 ~]$ env | grep ORA
3ORACLE_SID=+ASM1
4ORACLE_BASE=/u01/app/oracle
5ORACLE_HOME=/u01/app/11.2.0.3/grid
6[oracle@hubdb01 ~]$ skgxpinfo
7udp

So let’s try to turn RDS on 🙂
After shutting down all cluster resources, we have to recompile the binaries (I can never remember the needed options of the ins_rdbms.mk makefile, but when you use "grep", you will find all the necessary informations).

1[root@hubdb02 lib]# grep "rds" ins_rdbms.mk
2ipc_rds:
3    (if $(ORACLE_HOME)/bin/skgxpinfo | grep rds;\
4    $(MAKE) -f  $(MAKEFILE) ipc_rds; \
5[root@hubdb02 lib]# env | grep ORACLE_HOME
6ORACLE_HOME=/u01/app/11.2.0.3/grid
7[root@hubdb02 lib]# pwd
8/u01/app/11.2.0.3/grid/rdbms/lib
9[root@hubdb02 lib]# make -f ins_rdbms.mk ipc_rds ioracle
10rm -f /u01/app/11.2.0.3/grid/lib/libskgxp11.so
11(output removed for clarity)

And now there is a trick – ownership of the "/u01/app/11.2.0.3/grid/bin/oracle" binary has changed to root:root.

1[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
2-rwsr-s--x 1 root root 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle

Because of that, the shared memory segment of the ASM instnce will have insufficient privileges for CRS deamon to access it. Therefore you will find the following errors in grid alertlog:

1[crsd(126495)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
2ORA-01031: insufficient privileges
3]. Details at (:CRSD00111:) in /u01/app/11.2.0.3/grid/log/hubdb01/crsd/crsd.log.

So before starting the ASM instance, we have to change the privileges for the "oracle" binary file.

1[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
2-rwsr-s--x 1 root root 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle
3[root@hubdb02 lib]# chown oracle:oinstall /u01/app/11.2.0.3/grid/bin/oracle
4[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
5-rwxr-x--x 1 oracle oinstall 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle
6[root@hubdb02 lib]# chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle
7[root@hubdb02 lib]# ls -al /u01/app/11.2.0.3/grid/bin/oracle
8-rwsr-s--x 1 oracle oinstall 205208185 May  7 12:31 /u01/app/11.2.0.3/grid/bin/oracle

The same thing has to be done on the second node:

1[root@hubdb01 ~]# cd $ORACLE_HOME/rdbms/lib
2[root@hubdb01 lib]# pwd
3/u01/app/11.2.0.3/grid/rdbms/lib
4[root@hubdb01 lib]# grep "rds" ins_rdbms.mk
5ipc_rds:
6    (if $(ORACLE_HOME)/bin/skgxpinfo | grep rds;\
7    $(MAKE) -f  $(MAKEFILE) ipc_rds; \
8[root@hubdb01 lib]# make -f ins_rdbms.mk ipc_rds ioracle
9rm -f /u01/app/11.2.0.3/grid/lib/libskgxp11.so
10cp /u01/app/11.2.0.3/grid/lib//libskgxpr.so /u01/app/11.2.0.3/grid/lib/libskgxp11.so
11chmod 755 /u01/app/11.2.0.3/grid/bin
12 
13 - Linking Oracle
14rm -f /u01/app/11.2.0.3/grid/rdbms/lib/oracle
15gcc  -o /u01/app/11.2.0.3/grid/rdbms/lib/oracle -m64 -L/u01/app/11.2.0.3/grid/rdbms/lib/ -L/u01/app/11.2.0.3/grid/lib/ -L/u01/app/11.2.0.3/grid/lib/stubs/   -Wl,-E /u01/app/11.2.0.3/grid/rdbms/lib/opimai.o /u01/app/11.2.0.3/grid/rdbms/lib/ssoraed.o /u01/app/11.2.0.3/grid/rdbms/lib/ttcsoi.o  -Wl,--whole-archive -lperfsrv11 -Wl,--no-whole-archive /u01/app/11.2.0.3/grid/lib/nautab.o /u01/app/11.2.0.3/grid/lib/naeet.o /u01/app/11.2.0.3/grid/lib/naect.o /u01/app/11.2.0.3/grid/lib/naedhs.o /u01/app/11.2.0.3/grid/rdbms/lib/config.o  -lserver11 -lodm11 -lcell11 -lnnet11 -lskgxp11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 -lknlopt `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep xsyeolap.o > /dev/null 2>&1 ; then echo "-loraolap11" ; fi` -lslax11 -lpls11  -lrt -lplp11 -lserver11 -lclient11  -lvsn11 -lcommon11 -lgeneric11 `if [ -f /u01/app/11.2.0.3/grid/lib/libavserver11.a ] ; then echo "-lavserver11" ; else echo "-lavstub11"; fi` `if [ -f /u01/app/11.2.0.3/grid/lib/libavclient11.a ] ; then echo "-lavclient11" ; fi` -lknlopt -lslax11 -lpls11  -lrt -lplp11 -ljavavm11 -lserver11  -lwwg  `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11 -lmm -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lztkg11 `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnro11 `cat /u01/app/11.2.0.3/grid/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11 -lnnz11 -lzt11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 `if /usr/bin/ar tv /u01/app/11.2.0.3/grid/rdbms/lib/libknlopt.a | grep "kxmnsd.o" > /dev/null 2>&1 ; then echo " " ; else echo "-lordsdo11"; fi` -L/u01/app/11.2.0.3/grid/ctx/lib/ -lctxc11 -lctx11 -lzx11 -lgx11 -lctx11 -lzx11 -lgx11 -lordimt11 -lclsra11 -ldbcfg11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11  -loraz -llzopro -lorabz2 -lipp_z -lipp_bz2 -lippdcemerged -lippsemerged -lippdcmerged  -lippsmerged -lippcore  -lippcpemerged -lippcpmerged  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lsnls11 -lunls11  -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lasmclnt11 -lcommon11 -lcore11 -laio    `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -Wl,-rpath,/u01/app/11.2.0.3/grid/lib -lm    `cat /u01/app/11.2.0.3/grid/lib/sysliblist` -ldl -lm   -L/u01/app/11.2.0.3/grid/lib
16test ! -f /u01/app/11.2.0.3/grid/bin/oracle ||\
17       mv -f /u01/app/11.2.0.3/grid/bin/oracle /u01/app/11.2.0.3/grid/bin/oracleO
18mv /u01/app/11.2.0.3/grid/rdbms/lib/oracle /u01/app/11.2.0.3/grid/bin/oracle
19chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle
20[root@hubdb01 lib]# chown oracle:oinstall /u01/app/11.2.0.3/grid/bin/oracle
21[root@hubdb01 lib]# chmod 6751 /u01/app/11.2.0.3/grid/bin/oracle
22[root@hubdb01 lib]#

Now we can start the cluster resources:

1[root@hubdb01 lib]# crsctl start res ora.asm -init
2CRS-2672: Attempting to start 'ora.asm' on 'hubdb01'
3CRS-2676: Start of 'ora.asm' on 'hubdb01' succeeded
4CRS-2672: Attempting to start 'ora.crsd' on 'hubdb01'
5CRS-2676: Start of 'ora.crsd' on 'hubdb01' succeeded
6[root@hubdb01 lib]# ssh hubdb02
7Last login: Wed May  7 12:25:42 2014 from hub01.arrowecs.hub
8[root@hubdb02 ~]# . /home/oracle/grid.env
9[root@hubdb02 ~]# crsctl start res ora.asm -init
10CRS-2672: Attempting to start 'ora.asm' on 'hubdb02'
11CRS-2676: Start of 'ora.asm' on 'hubdb02' succeeded
12CRS-2672: Attempting to start 'ora.crsd' on 'hubdb02'
13CRS-2676: Start of 'ora.crsd' on 'hubdb02' succeeded

And that’s it. 🙂


Contact us

Database Whisperers sp. z o. o. sp. k.
al. Jerozolimskie 200, 3rd floor, room 342
02-486 Warszawa
NIP: 5272744987
REGON:362524978
+48 508 943 051
+48 661 966 009
info@ora-600.pl

Newsletter Sign up to be updated