<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Running two VMs on ESXi 5.1.0 and trying to get global heart beat (HB) working with no luck (on about my 20th rebuild and redo)<div class=""><br class=""></div><div class=""><b class="">Environment:</b></div><div class=""><br class=""></div><div class="">Two VMware based VMs running</div><div class=""><br class=""></div><div class=""># cat /etc/oracle-release</div><div class=""><br class=""></div><div class="">Oracle Linux Server release 6.5</div><div class=""><br class=""></div><div class=""># uname -r</div><div class=""><br class=""></div><div class="">2.6.32-400.36.8.el6uek.x86_64</div><div class=""><br class=""></div><div class=""># yum list installed | grep ocfs</div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">ocfs2-tools.x86_64 1.8.0-11.el6 @oel-latest </div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""># yum list installed | grep uek</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""><div style="margin: 0px;" class="">kernel-uek.x86_64 2.6.32-400.36.8.el6uek @oel-latest </div><div style="margin: 0px;" class="">kernel-uek-firmware.noarch 2.6.32-400.36.8.el6uek @oel-latest </div><div style="margin: 0px;" class="">kernel-uek-headers.x86_64 2.6.32-400.36.8.el6uek @oel-latest </div></div><div class=""><br class=""></div><div class=""><b class="">Configuration:</b></div><div class=""><br class=""></div><div class="">The shared data stores (HB and mounted OCFS) are setup in a similar way as described by VMWare and Oracle for shared RAC VMWare based data stores. All blogs, wikis and VMWare KB docs show similar setup, VM shared SCSI settings [multi-writer], shared disk [independant + persistent] etc. such as:</div><div class=""><br class=""></div><div class=""><a href="http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1034165" class="">http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1034165</a> ) </div><div class=""><br class=""></div><div class="">The devices can be seen by both VMs after in the OS. I have used the same configuration to run an OCFS2 setup with local heartbeat, and that works fine (cluster starts up and the OCFS2 file system mounts with no issues) </div><div class=""><br class=""></div><div class="">I followed similar procedures as show in an Oracle blog + docs: <a href="https://docs.oracle.com/cd/E37670_01/E37355/html/ol_instcfg_ocfs2.html" class="">https://docs.oracle.com/cd/E37670_01/E37355/html/ol_instcfg_ocfs2.html</a> and <a href="https://blogs.oracle.com/wim/entry/ocfs2_global_heartbeat" class="">https://blogs.oracle.com/wim/entry/ocfs2_global_heartbeat</a> with no luck. </div><div class=""><br class=""></div><div class="">The shared SCSI controllers are VMware paravirtual and set to “shared none” as suggested by the VMware RAC shared disk KB (previously mentioned)</div><div class=""><br class=""></div><div class="">After the shared Linux devices have been added to both VMs and are seen by both VMs in the OS (ls /dev/sd* shows the devices on each) I format the global HB devices in a way similar to the following from one VM:</div><div class=""><br class=""></div><div class=""># mkfs.ocfs2 -b 4K -C 4K -J size=4M -N 4 -L ocfs2vol1 --cluster-name=test --cluster-stack=o2cb --global-heartbeat /dev/sdc</div><div class=""># mkfs.ocfs2 -b 4K -C 4K -J size=4M -N 4 -L ocfs2vol2 --cluster-name=test --cluster-stack=o2cb --global-heartbeat /dev/sdd</div><div class=""><br class=""></div><div class="">From both VMs you can run the following and see:</div><div class=""><br class=""></div><div class=""># mounted.ocfs2 -d</div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">Device Stack Cluster F UUID Label</div><div style="margin: 0px;" class="">/dev/sdc o2cb test G 5620F19D43D840C7A46523019AE15A96 ocfs2vol1</div><div style="margin: 0px;" class="">/dev/sdd o2cb test G 9B9182279ABD4FD99F695F91488C94C1 ocfs2vol2</div></div><div class=""><br class=""></div><div class="">I then add the global HB devices to the ocfs config file with similar commands:</div><div class=""><br class=""></div><div class=""># o2cb add-heartbeat test 5620F19D43D840C7A46523019AE15A96</div><div class=""># o2cb add-heartbeat test 9B9182279ABD4FD99F695F91488C94C1</div><div class=""><br class=""></div><div class="">Thus far looking good (heh, but then all we’ve done is format ocfs2 with options and updated a text file) - then I do the following:</div><div class=""><br class=""></div><div class=""># o2cb heartbeat-mode test global</div><div class=""><br class=""></div><div class="">All this being done on one node in the cluster I copy the following to the other node (with hostnames changed here, though the actual hostname = output of the <i class="">hostname</i> command on each node):</div><div class=""><br class=""></div><div class=""># cat /etc/ocfs2/cluster.conf </div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">node:</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>name = <a href="http://clusterhost1.mydomain.com" class="">clusterhost1.mydomain.com</a></div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>number = 0</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_address = 10.143.144.12</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_port = 7777</div><div style="margin: 0px; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px;" class="">node:</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>name = <a href="http://clusterhost2.mydomain.com" class="">clusterhost2.mydomain.com</a></div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>number = 1</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_address = 10.143.144.13</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_port = 7777</div><div style="margin: 0px; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px;" class="">cluster:</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>name = test</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>heartbeat_mode = global</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>node_count = 2</div><div style="margin: 0px; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px;" class="">heartbeat:</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>region = 5620F19D43D840C7A46523019AE15A96</div><div style="margin: 0px; min-height: 14px;" class=""><br class=""></div><div style="margin: 0px;" class="">heartbeat:</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test</div><div style="margin: 0px;" class=""><span class="Apple-tab-span" style="white-space:pre"> </span>region = 9B9182279ABD4FD99F695F91488C94C1</div></div><div class=""><br class=""></div><div class="">The same config works fine with heartbeat_mode set to local and the global heartbeat devices removed, and I can mount a shared FS - the local HB interfaces are IPv4 on a private L2 non routed VLAN, are up and each node can ping each other.</div><div class=""><br class=""></div><div class="">Once the config is copied to each node and have already run:</div><div class=""><br class=""></div><div class=""># service o2cb configure</div><div class=""><br class=""></div><div class="">Which completes in local heartbeat mode fine, so the cluster will start on boot and the params are default for timeouts etc. </div><div class=""><br class=""></div><div class="">I check that the service on both nodes unloads and loads modules with no issues:</div><div class=""><br class=""></div><div class=""># service o2cb unload</div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">Clean userdlm domains: OK</div><div style="margin: 0px;" class="">Unmounting ocfs2_dlmfs filesystem: OK</div><div style="margin: 0px;" class="">Unloading module "ocfs2_dlmfs": OK</div><div style="margin: 0px;" class="">Unloading module "ocfs2_stack_o2cb": OK</div><div style="margin: 0px;" class="">Unmounting configfs filesystem: OK</div><div style="margin: 0px;" class="">Unloading module "configfs": OK</div></div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""># service o2cb load</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""><div style="margin: 0px;" class="">Loading filesystem "configfs": OK</div><div style="margin: 0px;" class="">Mounting configfs filesystem at /sys/kernel/config: OK</div><div style="margin: 0px;" class="">Loading stack plugin "o2cb": OK</div><div style="margin: 0px;" class="">Loading filesystem "ocfs2_dlmfs": OK</div><div style="margin: 0px;" class="">Mounting ocfs2_dlmfs filesystem at /dlm: OK</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""># mount -v</div><div style="margin: 0px;" class="">…</div><div style="margin: 0px;" class="">….</div><div style="margin: 0px;" class=""><div style="margin: 0px;" class="">debugfs on /sys/kernel/debug type debugfs (rw)</div><div style="margin: 0px;" class="">….</div><div style="margin: 0px;" class=""><div style="margin: 0px;" class="">ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""># lsmod | grep ocfs</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""><div style="margin: 0px;" class="">ocfs2_dlmfs 18026 1 </div><div style="margin: 0px;" class="">ocfs2_stack_o2cb 3606 0 </div><div style="margin: 0px;" class="">ocfs2_dlm 196778 1 ocfs2_stack_o2cb</div><div style="margin: 0px;" class="">ocfs2_nodemanager 202856 3 ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm</div><div style="margin: 0px;" class="">ocfs2_stackglue 11283 2 ocfs2_dlmfs,ocfs2_stack_o2cb</div><div style="margin: 0px;" class="">configfs 25853 2 ocfs2_nodemanager</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class="">Looks good on both nodes…. then (sigh)</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""># service o2cb enable</div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class=""><div style="margin: 0px;" class="">Writing O2CB configuration: OK</div><div style="margin: 0px;" class="">Setting cluster stack "o2cb": OK</div><div style="margin: 0px;" class="">Registering O2CB cluster "test": Failed</div><div style="margin: 0px;" class="">o2cb: Unable to access cluster service while registering heartbeat mode 'global'</div><div style="margin: 0px;" class="">Unregistering O2CB cluster "test": OK</div><div class=""><br class=""></div><div class="">I have searched for the error string and have come up with a huge ZERO on help - and the local OS log messages are equally unhelpful:</div><div class=""><br class=""></div><div class=""># tail /var/log/messages</div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">Nov 12 21:54:53 clusterhost1 o2cb.init: online test</div><div style="margin: 0px;" class="">Nov 13 00:58:38 clusterhost1 o2cb.init: online test</div><div style="margin: 0px;" class="">Nov 13 01:00:06 clusterhost1 o2cb.init: offline test 0</div><div style="margin: 0px;" class="">Nov 13 01:00:06 clusterhost1 kernel: ocfs2: Unregistered cluster interface o2cb</div><div style="margin: 0px;" class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 Node Manager 1.6.3</div><div style="margin: 0px;" class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 DLM 1.6.3</div><div style="margin: 0px;" class="">Nov 13 01:01:14 clusterhost1 kernel: ocfs2: Registered cluster interface o2cb</div><div style="margin: 0px;" class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 DLMFS 1.6.3</div><div style="margin: 0px;" class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 User DLM kernel interface loaded</div><div style="margin: 0px;" class="">Nov 13 01:03:32 clusterhost1 o2cb.init: online test</div></div><div class=""><br class=""></div><div class="">Dmesg shows the same:</div><div class=""><br class=""></div><div class=""># dmesg</div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">OCFS2 Node Manager 1.6.3</div><div style="margin: 0px;" class="">OCFS2 DLM 1.6.3</div><div style="margin: 0px;" class="">ocfs2: Registered cluster interface o2cb</div><div style="margin: 0px;" class="">OCFS2 DLMFS 1.6.3</div><div style="margin: 0px;" class="">OCFS2 User DLM kernel interface loaded</div><div style="margin: 0px;" class="">Slow work thread pool: Starting up</div><div style="margin: 0px;" class="">Slow work thread pool: Ready</div><div style="margin: 0px;" class="">FS-Cache: Loaded</div><div style="margin: 0px;" class="">FS-Cache: Netfs 'nfs' registered for caching</div><div style="margin: 0px;" class="">eth0: no IPv6 routers present</div><div style="margin: 0px;" class="">eth1: no IPv6 routers present</div><div style="margin: 0px;" class="">ocfs2: Unregistered cluster interface o2cb</div><div style="margin: 0px;" class="">OCFS2 Node Manager 1.6.3</div><div style="margin: 0px;" class="">OCFS2 DLM 1.6.3</div><div style="margin: 0px;" class="">ocfs2: Registered cluster interface o2cb</div><div style="margin: 0px;" class="">OCFS2 DLMFS 1.6.3</div><div style="margin: 0px;" class="">OCFS2 User DLM kernel interface loaded</div><div style="margin: 0px;" class="">ocfs2: Unregistered cluster interface o2cb</div><div style="margin: 0px;" class="">OCFS2 Node Manager 1.6.3</div><div style="margin: 0px;" class="">OCFS2 DLM 1.6.3</div><div style="margin: 0px;" class="">ocfs2: Registered cluster interface o2cb</div><div style="margin: 0px;" class="">OCFS2 DLMFS 1.6.3</div><div style="margin: 0px;" class="">OCFS2 User DLM kernel interface loaded</div></div><div class=""><br class=""></div></div></div></div></div></div><div class="">The filesystem looks fine and this can be run from both hosts in the cluster:</div><div class=""><br class=""></div><div class=""># fsck.ocfs2 -n /dev/sdc </div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">fsck.ocfs2 1.8.0</div><div style="margin: 0px;" class="">Checking OCFS2 filesystem in /dev/sdc:</div><div style="margin: 0px;" class=""> Label: ocfs2vol1</div><div style="margin: 0px;" class=""> UUID: 5620F19D43D840C7A46523019AE15A96</div><div style="margin: 0px;" class=""> Number of blocks: 524288</div><div style="margin: 0px;" class=""> Block size: 4096</div><div style="margin: 0px;" class=""> Number of clusters: 524288</div><div style="margin: 0px;" class=""> Cluster size: 4096</div><div style="margin: 0px;" class=""> Number of slots: 4</div></div><div class=""><br class=""></div><div class=""># fsck.ocfs2 -n /dev/sdd</div><div class=""><br class=""></div><div class=""><div style="margin: 0px;" class="">fsck.ocfs2 1.8.0</div><div style="margin: 0px;" class="">Checking OCFS2 filesystem in /dev/sdd:</div><div style="margin: 0px;" class=""> Label: ocfs2vol2</div><div style="margin: 0px;" class=""> UUID: 9B9182279ABD4FD99F695F91488C94C1</div><div style="margin: 0px;" class=""> Number of blocks: 524288</div><div style="margin: 0px;" class=""> Block size: 4096</div><div style="margin: 0px;" class=""> Number of clusters: 524288</div><div style="margin: 0px;" class=""> Cluster size: 4096</div><div style="margin: 0px;" class=""> Number of slots: 4</div></div><div style="margin: 0px;" class=""><br class=""></div><div style="margin: 0px;" class="">What am I missing? I’ve re-done this, re-created the devices a few too many times (thinking I may have missed something) but I am mystified. From all outer appearances I have two VMs that can see and in local heartbeat mode mount a shared OCFS2 filesystem and access it (have it running in local heartbeat mode for a cluster of rsyslog servers that are being load balanced by an F5 LTM VS with no issues) I am stumped on how to get global HB devices setup, though I have read and re-read the user guides, troubleshooting guides and wikis/blogs on how to make that work until my eyes hurt. </div><div class=""><br class=""></div><div class="">Mounted the debugfs and ran the debugfs.ocfs2 utility but am unfamiliar of what I should be looking for there (or if this is where I would look for cluster not coming online errors) </div><div class=""><br class=""></div><div class="">As the oc2b/ocfs modules are all kernel based I am not 100% sure how to increase debug information without digging into the source code and mucking around there.</div><div class=""><br class=""></div><div class="">Any guidance or lessons learned (or things to check) would be super :) and if works warrant a happy scream of joy from my frustrated cube!</div><div class=""><br class=""><div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline">Warm regards,</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Jon<br class=""></div></div></div></div></body></html>