After a memory upgrade in our RAC servers, the Cluster wasn’t coming up after a reboot. We first tried to start the CRS in init mode but even that failed. Upon checking the CRS alert log we discovered that the ASM was failing to start due to below error.
CRS-5017: The resource action “ora.asm start” encountered the following error:
ORA-01017: invalid username/password; logon denied
. For details refer to “(:CLSN00107:)” in “/u01/app/grid/diag/crs/<hostname>/crs/trace/ohasd_oraagent_grid.trc”
We discovered the above error in the CRS alert log. After knocking our heads around, trying to start crsd in init mode without any success. We eventually discovered it was related to SQLNET.AUTHENTICATION_SERVICES parameter in sqlnet.ora file in the Grid Home. We removed the entry in the sqlnet.ora file from both nodes of the cluster and restarted the cluster. Everything was back to normal after that. Another day of troubleshooting CRS without any help from Oracle Support.