Oh, sometimes it’s just (own or close-people’s) PEBKAC that costs you time and gray hair… Patching 11.2.0.4.x to PSU2 on Linux was such an issue.
Problem
“opatch auto” with all comfort worked well, applied the patch, but re-starting the clusterware failed with:
Starting CRS... Installing Trace File Analyzer CRS-4123: Oracle High Availability Services has been started. Oracle Grid Infrastructure stack start initiated but failed to complete at /u01/app/11.2.0/grid_2/crs/install/crsconfig_lib.pm line 11814.
What happens here? OHASD tries to start up CRSD, but it fails with an exception: permission check failed. Mh, might come from most $GRID_HOME/bin/*.bin files have no executable (“x”) flags set. WTF?
So what.. who’s the culprit? Spent a whole weekend on investigating and re-trying, just to learn that the scripts in the PSU won’t change a flag in bin directory. What do the file parameters look like in the unzipped patch directory. The same. Grml.
Wait… The PSU2 README.html says: “unzip the patch as the grid home owner”. How did the patch directory get to the server…? Oh yes, it was unzipped on the downloading machine (automatically). Woe …
Solution
- Download the PSU in zipped format
- Copy the ZIP container to the system you’d like to patch
- “Unzip the patch as the grid home owner”
- Don’t find a “failed to complete at crsconfig_lib.pm line 11814” in your console output
- Have an early beer for reading correctly, and not a late one for finding your own mistakes
Be an extremely careful reader and stay safe
Martin