The Problem
In the last week, I had to set up an Oracle RAC (and thus, the Clusterware) version 12.2.0.1 on SuSE Linux Enterprise Server 12 SP2 (SLES12 Sp2). The gridSetup.sh failed latest in root.sh of the first node with a kinda non-intuitive error:
CRS-5804: Communication error with agent process
CRS-4000: Command Start failed, or completed with errors.
2017/07/10 10:18:49 CLSRSC-119: Start of the exclusive mode cluster failed
Died at /u01/app/12.2.0/grid_1/crs/install/crsinstall.pm line 2053.
The command ‘/u01/app/12.2.0/grid_1/perl/bin/perl -I/u01/app/12.2.0/grid_1/perl/lib -I/u01/app/12.2.0/grid_1/crs/install /u01/app/12.2.0/grid_1/crs/install/rootcrs.pl ‘ execution failed
Thank you – for nothing.
The rootcrs.log in /tmp/GridAction<date> directory was a bit more enlightening, but not much:
CRS-5804: Communication error with agent process CRS-4000: Command Start failed, or completed with errors. The exlusive mode cluster start failed, see Clusterware alert log for more information Executing cmd: /u01/app/12.2.0/grid_1/bin/clsecho -p has -f clsrsc -m 119 Command output: > CLSRSC-119: Start of the exclusive mode cluster failed >End Command output CLSRSC-119: Start of the exclusive mode cluster failed ###### Begin DIE Stack Trace ###### Package File Line Calling --------------- -------------------- ---- ---------- 1: main rootcrs.pl 287 crsutils::dietrap 2: crsinstall crsinstall.pm 2053 main::__ANON__ 3: crsinstall crsinstall.pm 1963 crsinstall::perform_initial_config 4: crsinstall crsinstall.pm 653 crsinstall::perform_init_config 5: crsinstall crsinstall.pm 813 crsinstall::init_config 6: crsinstall crsinstall.pm 380 crsinstall::CRSInstall 7: main rootcrs.pl 446 crsinstall::new ####### End DIE Stack Trace #######
Eh… yes.
The error in the cluster alert log finally was better, and made me curious:
[ORAROOTAGENT(20948)]
CRS-8503: Oracle Clusterware process ORAROOTAGENT with operating system process ID 20948 experienced fatal signal or exception code 11.
Errors in file /u01/app/oracle/diag/crs/myserver08/crs/trace/ohasd_orarootagent_root.trc (incident=1):
CRS-8503 [__lll_unlock_elision()+48] [Signal/Exception: 11] [Instruction Addr: 0x7f5df148a4a0] [Memory Addr: (nil)] [] [] [] [] [] [] [] []
Ah. Of course. 🙂
The System
CPU:
Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz
Kernel:
# uname -a
Linux myhost08 4.4.59-92.24-default #1 SMP Thu Jun 22 14:29:09 UTC 2017 (d11a83a) x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/SuSE-release
SUSE Linux Enterprise Server 12 (x86_64)
VERSION = 12
PATCHLEVEL = 2
First Conclusions
This unlock_elision thing was unusual enough to raise suspicions. First, a little research about the term: Seems that Lock Elision is a part of Intel® Transactional Synchronization Extensions (Intel® TSX) and is described here. Long story short: A CPU extension to work with mutexes more efficiently, dealing with transactional serialisation on the chip. Cool stuff.
And for one or other reason, Oracle does not like that on this SLES12 SP2 system with this E5-26xx chip. Oracle came back with a short and somewhat cryptic hint: “Would it be possible to disable HLE and retry again?” Guess HLE means “Hardware Lock Elision”.
If we can’t have it, how can we get rid of it, to free our way?
SLES12 – How to disable Hardware Lock Elision Support
Lock Elision on Linux is used by the libpthread libraries, and thankfully, the distributor has prepared a plan B for people who are not blessed with this performance booster: in /lib64 is a directory “noelision” containing a symlink to an appropriately pre-configured libpthread-x.xx.so. All we have to do now is to ensure that dynamically linked binaries will use this instead of the “normal” one. I decided not to use ld.so.conf directly, but to use a “distributor-proof” approach and generate a noelision.conf in /etc/ld.so.conf.d, just containing:
/lib64/noelision
And, of course run “ldconfig -v” afterwards. Check its output for the new libpthread symlink, should be in the upper part.
There’s no point in checking the linker search path now (with gcc -v or ld itself), they will not change. Since we talk about the effect of dynamically linked libaries on started programs, all you can do is to look into the output of ldconfig -v as described above to check before re-testing gridSetup.sh.
Yes, that did the trick, and I am grateful.
Be careful how you choose your hardware, folks… Best regards
Martin
Thanks and Acknowledgements
- My customer for patience, peace and peanuts. 🙂
- A big Thanks to RAC PM Anil Nair (@RACMasterPM) for two short but VERY helpful tips that meant the breakthrough.
- The Oracle RAC 12.2 designers for implementing the setup process in a restartable and dynamic fashion, especially helpful were the root.sh checkpoints and the “add-nodes-later feature”, both to reduce effort in re-starting, re-testing and simplifying test cases.
- Last but not least, helpful web resources: