ORACLE RAC 21C: RESTORE OCR AND VOTEDISK


13.05.2022
by Kamil Stawiarski

Six years ago I wrote a short instruction on how to restore OCR and Votedisk in RAC 12c after a diskgroup failure.

Yesterday I had a possibility of validating my procedure in 21c RAC environment and it has occurred that it is a bit more complicated. Who would have thought? 21 is just 12 backwards!

Most of the procedure is the same, but after you finish, you might face the following error:

ohasd_orarootagent_root_1.trc:2022-05-12 11:41:30.336 :CLSDYNAM:139822700111616: [ora.storage]{0:5:3} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 1017, other: ORA-01017: invalid username/password; logon denied

WTF?

The procedure for this kind of situation is described in MOS How to Recreate Shared ASM Password File in 19c Grid Infrastructure (GI) (Doc ID 2717306.1)

But when tried this fix, I got the following error:

[root@rac1 ~]# asmcmd --nocp credfix
credfix: Connected successfully using credentials for CRSUSER__ASM_001
op=credstoxml wrap=/tmp/new_creds.xml
credfix: OLR for rac1 has been fixed if credentials were created incorrectly.
credfix: Starting SSH session on node rac2.
credfix: OLR for rac2 has been fixed if credentials were created incorrectly. Exiting SSH session.
ASMCMD-8043: more than one credential in OCR
credfix: Attempted credverify to check if credentials were fixed; credentials remain incorrect.

credentials remain incorrect

Thank you, dear Oracle Support 🙂 You almost helped!

How to recover from this? Well, my solution was to brake it even more! Let’s find a username that CRS is using:

[root@rac1 ~]# ocrdump
[root@rac1 ~]# grep "ASM.CREDENTIALS" OCRDUMPFILE | grep CRSUSER
[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001]
[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001.CRSUSER]

So apparently the user used for ASM connection is CRSUSER__ASM_001.

It should exist in a password file after recreating it manually:

ASMCMD> lspwusr
        Username sysdba sysoper sysasm
             SYS   TRUE    TRUE   TRUE
CRSUSER__ASM_001   TRUE   FALSE   TRUE

Great, so now let’s mess it even more:

SQL> alter user CRSUSER__ASM_001 identified by oracle;

User altered.

And now we will try to use our MOS note once again:

[root@rac1 ~]# asmcmd --nocp credfix
credfix: Failed to connect using credentials for CRSUSER__ASM_001.
credfix: Failed to connect using credentials for CRSUSER__ASM_001.
op=addcrscreds wrap=/tmp/creds0.xml
credfix: Creating new credentials, no valid credentials in OCR.
credfix: New user CRSUSER__ASM_002 created.
op=credimport wrap=/tmp/creds0.xml olr=true force=true
credfix: OLR for rac1 has been fixed if credentials were created incorrectly.
credfix: Starting SSH session on node rac2.
credfix: OLR for rac2 has been fixed if credentials were created incorrectly. Exiting SSH session.
op=delcrscreds crs_user=CRSUSER__ASM_001
credfix: Deleted CRSUSER__ASM_001 from OCR.
op=delcrscreds crs_user=CRSUSER__ASM_001
credfix: Deleted CRSUSER__ASM_001 from OCR.
credverify: Credentials created correctly on rac1.
credverify: Starting SSH session on node rac2
credverify: Credentials created correctly on rac2. Exiting SSH session.
credfix: Credentials have been fixed if they were created incorrectly.

Now we are talking!

So the procedure is the same, but you have to remember about those ASM credentials. Oh, and OCR backup of FS is in a different place:

[root@rac1 ocr]# ls -al $ORACLE_BASE/crsdata/`hostname`/ocr
total 6540
drwxr-xr-x  2 oracle oinstall      42 May 11 15:51 .
drwxrwxr-x 27 oracle oinstall    4096 May 11 10:56 ..
-rw-------  1 root   root     3346432 May 11 15:51 backup00.ocr
-rw-------  1 root   root     3346432 May 11 15:51 day_.ocr

Cheers!


Contact us

Database Whisperers sp. z o. o. sp. k.
al. Jerozolimskie 200, 3rd floor, room 342
02-486 Warszawa
NIP: 5272744987
REGON:362524978
+48 508 943 051
+48 661 966 009
info@ora-600.pl

Newsletter Sign up to be updated