Six years ago I wrote a short instruction on how to restore OCR and Votedisk in RAC 12c after a diskgroup failure.
Yesterday I had a possibility of validating my procedure in 21c RAC environment and it has occurred that it is a bit more complicated. Who would have thought? 21 is just 12 backwards!
Most of the procedure is the same, but after you finish, you might face the following error:
1ohasd_orarootagent_root_1.trc:2022-05-12 11:41:30.336 :CLSDYNAM:139822700111616: [ora.storage]{0:5:3} [start] (null) category: 7, operation: kgfoAl06, loc: kgfokge, OS error: 1017, other: ORA-01017: invalid username/password; logon denied
WTF?
The procedure for this kind of situation is described in MOS How to Recreate Shared ASM Password File in 19c Grid Infrastructure (GI) (Doc ID 2717306.1)
But when tried this fix, I got the following error:
2credfix: Connected successfully using credentials for CRSUSER__ASM_001
3op=credstoxml wrap=/tmp/new_creds.xml
4credfix: OLR for rac1 has been fixed if credentials were created incorrectly.
5credfix: Starting SSH session on node rac2.
6credfix: OLR for rac2 has been fixed if credentials were created incorrectly. Exiting SSH session.
7ASMCMD-8043: more than one credential in OCR
8credfix: Attempted credverify to check if credentials were fixed; credentials remain incorrect.
credentials remain incorrect
Thank you, dear Oracle Support 🙂 You almost helped!
How to recover from this? Well, my solution was to brake it even more! Let’s find a username that CRS is using:
2[root@rac1 ~]# grep "ASM.CREDENTIALS" OCRDUMPFILE | grep CRSUSER
3[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001]
4[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001.CRSUSER]
So apparently the user used for ASM connection is CRSUSER__ASM_001.
It should exist in a password file after recreating it manually:
2 Username sysdba sysoper sysasm
4CRSUSER__ASM_001 TRUE FALSE TRUE
Great, so now let’s mess it even more:
1SQL> alter user CRSUSER__ASM_001 identified by oracle;
And now we will try to use our MOS note once again:
2credfix: Failed to connect using credentials for CRSUSER__ASM_001.
3credfix: Failed to connect using credentials for CRSUSER__ASM_001.
4op=addcrscreds wrap=/tmp/creds0.xml
5credfix: Creating new credentials, no valid credentials in OCR.
6credfix: New user CRSUSER__ASM_002 created.
7op=credimport wrap=/tmp/creds0.xml olr=true force=true
8credfix: OLR for rac1 has been fixed if credentials were created incorrectly.
9credfix: Starting SSH session on node rac2.
10credfix: OLR for rac2 has been fixed if credentials were created incorrectly. Exiting SSH session.
11op=delcrscreds crs_user=CRSUSER__ASM_001
12credfix: Deleted CRSUSER__ASM_001 from OCR.
13op=delcrscreds crs_user=CRSUSER__ASM_001
14credfix: Deleted CRSUSER__ASM_001 from OCR.
15credverify: Credentials created correctly on rac1.
16credverify: Starting SSH session on node rac2
17credverify: Credentials created correctly on rac2. Exiting SSH session.
18credfix: Credentials have been fixed if they were created incorrectly.
Now we are talking!
So the procedure is the same, but you have to remember about those ASM credentials. Oh, and OCR backup of FS is in a different place:
3drwxr-xr-x 2 oracle oinstall 42 May 11 15:51 .
4drwxrwxr-x 27 oracle oinstall 4096 May 11 10:56 ..
5-rw------- 1 root root 3346432 May 11 15:51 backup00.ocr
6-rw------- 1 root root 3346432 May 11 15:51 day_.ocr
Cheers!
