Six years ago I wrote a short instruction on how to restore OCR and Votedisk in RAC 12c after a diskgroup failure.
Yesterday I had a possibility of validating my procedure in 21c RAC environment and it has occurred that it is a bit more complicated. Who would have thought? 21 is just 12 backwards!
Most of the procedure is the same, but after you finish, you might face the following error:
ohasd_orarootagent_root_1.trc:2022-05-12 11:41:30.336 :CLSDYNAM:139822700111616: [ora.storage]{0:5:3} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 1017, other: ORA-01017: invalid username/password; logon denied
WTF?
The procedure for this kind of situation is described in MOS How to Recreate Shared ASM Password File in 19c Grid Infrastructure (GI) (Doc ID 2717306.1)
But when tried this fix, I got the following error:
[root@rac1 ~]# asmcmd --nocp credfix
credfix: Connected successfully using credentials for CRSUSER__ASM_001
op=credstoxml wrap=/tmp/new_creds.xml
credfix: OLR for rac1 has been fixed if credentials were created incorrectly.
credfix: Starting SSH session on node rac2.
credfix: OLR for rac2 has been fixed if credentials were created incorrectly. Exiting SSH session.
ASMCMD-8043: more than one credential in OCR
credfix: Attempted credverify to check if credentials were fixed; credentials remain incorrect.
credentials remain incorrect
Thank you, dear Oracle Support 🙂 You almost helped!
How to recover from this? Well, my solution was to brake it even more! Let’s find a username that CRS is using:
[root@rac1 ~]# ocrdump
[root@rac1 ~]# grep "ASM.CREDENTIALS" OCRDUMPFILE | grep CRSUSER
[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001]
[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001.CRSUSER]
So apparently the user used for ASM connection is CRSUSER__ASM_001.
It should exist in a password file after recreating it manually:
ASMCMD> lspwusr
Username sysdba sysoper sysasm
SYS TRUE TRUE TRUE
CRSUSER__ASM_001 TRUE FALSE TRUE
Great, so now let’s mess it even more:
SQL> alter user CRSUSER__ASM_001 identified by oracle;
User altered.
And now we will try to use our MOS note once again:
[root@rac1 ~]# asmcmd --nocp credfix
credfix: Failed to connect using credentials for CRSUSER__ASM_001.
credfix: Failed to connect using credentials for CRSUSER__ASM_001.
op=addcrscreds wrap=/tmp/creds0.xml
credfix: Creating new credentials, no valid credentials in OCR.
credfix: New user CRSUSER__ASM_002 created.
op=credimport wrap=/tmp/creds0.xml olr=true force=true
credfix: OLR for rac1 has been fixed if credentials were created incorrectly.
credfix: Starting SSH session on node rac2.
credfix: OLR for rac2 has been fixed if credentials were created incorrectly. Exiting SSH session.
op=delcrscreds crs_user=CRSUSER__ASM_001
credfix: Deleted CRSUSER__ASM_001 from OCR.
op=delcrscreds crs_user=CRSUSER__ASM_001
credfix: Deleted CRSUSER__ASM_001 from OCR.
credverify: Credentials created correctly on rac1.
credverify: Starting SSH session on node rac2
credverify: Credentials created correctly on rac2. Exiting SSH session.
credfix: Credentials have been fixed if they were created incorrectly.
Now we are talking!
So the procedure is the same, but you have to remember about those ASM credentials. Oh, and OCR backup of FS is in a different place:
[root@rac1 ocr]# ls -al $ORACLE_BASE/crsdata/`hostname`/ocr
total 6540
drwxr-xr-x 2 oracle oinstall 42 May 11 15:51 .
drwxrwxr-x 27 oracle oinstall 4096 May 11 10:56 ..
-rw------- 1 root root 3346432 May 11 15:51 backup00.ocr
-rw------- 1 root root 3346432 May 11 15:51 day_.ocr
Cheers!