<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks! Srinivas was helping me troubleshoot that last night with a series of strace(es) that started to point to the Kernel issues - and then he realized that it was the wrong version! I updated the version to 2.6.39-400.215.10.el6uek.x86_64 on both of my ocfs boxes and service o2cb enable brought the global HB online with no issues!<div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline">Warm regards and thanks!</div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Jon<br class="Apple-interchange-newline"></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Nov 13, 2014, at 12:00 PM, <a href="mailto:ocfs2-users-request@oss.oracle.com" class="">ocfs2-users-request@oss.oracle.com</a> wrote:</div><br class="Apple-interchange-newline"><div class="">Send Ocfs2-users mailing list submissions to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span><a href="mailto:ocfs2-users@oss.oracle.com" class="">ocfs2-users@oss.oracle.com</a><br class=""><br class="">To subscribe or unsubscribe via the World Wide Web, visit<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>https://oss.oracle.com/mailman/listinfo/ocfs2-users<br class="">or, via email, send a message with subject or body 'help' to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ocfs2-users-request@oss.oracle.com<br class=""><br class="">You can reach the person managing the list at<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ocfs2-users-owner@oss.oracle.com<br class=""><br class="">When replying, please edit your Subject line so it is more specific<br class="">than "Re: Contents of Ocfs2-users digest..."<br class=""><br class=""><br class="">Today's Topics:<br class=""><br class=""> 1. Re: Ocfs2-users Digest, Vol 130, Issue 1 (Richard Sibthorp)<br class=""><br class=""><br class="">----------------------------------------------------------------------<br class=""><br class="">Message: 1<br class="">Date: Thu, 13 Nov 2014 14:16:05 +0000<br class="">From: Richard Sibthorp <richard.sibthorp@oracle.com><br class="">Subject: Re: [Ocfs2-users] Ocfs2-users Digest, Vol 130, Issue 1<br class="">To: ocfs2-users@oss.oracle.com<br class="">Message-ID: <5464BD25.8010003@oracle.com><br class="">Content-Type: text/plain; charset=ISO-8859-1; format=flowed<br class=""><br class="">Hi Jon,<br class=""><br class="">The kernel you are using includes the ocfs2 kernel modules at version <br class="">1.6.3. The global heartbeat feature was introduced in ocfs2 1.8.<br class=""><br class="">I haven't checked whether any of the 2.6.32-based uek's include ocfs2 <br class="">1.8, but certainly the 2.6.39 and later (aka uek2, uek3) ones do.<br class=""><br class="">I assume from below you have an Oracle support license - at least for <br class="">rdbms if not Oracle Linux. When using ocfs2 for rdbms resources, your <br class="">rdbms license entitles you to ocfs2 support via MOS, though for <br class="">general-purpose ocfs2 issues an Oracle Linux Support contract needs to <br class="">be in place. This would have a separate CSI from that of licensed <br class="">products - obviously open-source products are not licensed, but if you <br class="">require support you need a support contract.<br class=""><br class="">You may also want to review MOS documents 1552519.1 and 1553162.1<br class=""><br class="">Best regards,<br class="">Richard.<br class=""><br class="">On 13/11/2014 02:27, ocfs2-users-request@oss.oracle.com wrote:<br class=""><blockquote type="cite" class="">Send Ocfs2-users mailing list submissions to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ocfs2-users@oss.oracle.com<br class=""><br class="">To subscribe or unsubscribe via the World Wide Web, visit<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>https://oss.oracle.com/mailman/listinfo/ocfs2-users<br class="">or, via email, send a message with subject or body 'help' to<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ocfs2-users-request@oss.oracle.com<br class=""><br class="">You can reach the person managing the list at<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ocfs2-users-owner@oss.oracle.com<br class=""><br class="">When replying, please edit your Subject line so it is more specific<br class="">than "Re: Contents of Ocfs2-users digest..."<br class=""><br class=""><br class="">Today's Topics:<br class=""><br class=""> 1. OCFS2 v1.8 on VMware VMs global heartbeat woes (Jon Norris)<br class=""><br class=""><br class="">----------------------------------------------------------------------<br class=""><br class="">Message: 1<br class="">Date: Wed, 12 Nov 2014 18:26:51 -0800<br class="">From: Jon Norris <jon_norris@apple.com><br class="">Subject: [Ocfs2-users] OCFS2 v1.8 on VMware VMs global heartbeat woes<br class="">To: ocfs2-users@oss.oracle.com<br class="">Message-ID: <FFF084CA-12F0-4FA1-BB38-B70F5A4A6695@apple.com><br class="">Content-Type: text/plain; charset="utf-8"<br class=""><br class="">Running two VMs on ESXi 5.1.0 and trying to get global heart beat (HB) working with no luck (on about my 20th rebuild and redo)<br class=""><br class="">Environment:<br class=""><br class="">Two VMware based VMs running<br class=""><br class=""># cat /etc/oracle-release<br class=""><br class="">Oracle Linux Server release 6.5<br class=""><br class=""># uname -r<br class=""><br class="">2.6.32-400.36.8.el6uek.x86_64<br class=""><br class=""># yum list installed | grep ocfs<br class=""><br class="">ocfs2-tools.x86_64 1.8.0-11.el6 @oel-latest<br class=""><br class=""># yum list installed | grep uek<br class=""><br class="">kernel-uek.x86_64 2.6.32-400.36.8.el6uek @oel-latest<br class="">kernel-uek-firmware.noarch 2.6.32-400.36.8.el6uek @oel-latest<br class="">kernel-uek-headers.x86_64 2.6.32-400.36.8.el6uek @oel-latest<br class=""><br class="">Configuration:<br class=""><br class="">The shared data stores (HB and mounted OCFS) are setup in a similar way as described by VMWare and Oracle for shared RAC VMWare based data stores. All blogs, wikis and VMWare KB docs show similar setup, VM shared SCSI settings [multi-writer], shared disk [independant + persistent] etc. such as:<br class=""><br class="">http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1034165 <http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1034165> )<br class=""><br class="">The devices can be seen by both VMs after in the OS. I have used the same configuration to run an OCFS2 setup with local heartbeat, and that works fine (cluster starts up and the OCFS2 file system mounts with no issues)<br class=""><br class="">I followed similar procedures as show in an Oracle blog + docs: https://docs.oracle.com/cd/E37670_01/E37355/html/ol_instcfg_ocfs2.html <https://docs.oracle.com/cd/E37670_01/E37355/html/ol_instcfg_ocfs2.html> and https://blogs.oracle.com/wim/entry/ocfs2_global_heartbeat <https://blogs.oracle.com/wim/entry/ocfs2_global_heartbeat> with no luck.<br class=""><br class="">The shared SCSI controllers are VMware paravirtual and set to ?shared none? as suggested by the VMware RAC shared disk KB (previously mentioned)<br class=""><br class="">After the shared Linux devices have been added to both VMs and are seen by both VMs in the OS (ls /dev/sd* shows the devices on each) I format the global HB devices in a way similar to the following from one VM:<br class=""><br class=""># mkfs.ocfs2 -b 4K -C 4K -J size=4M -N 4 -L ocfs2vol1 --cluster-name=test --cluster-stack=o2cb --global-heartbeat /dev/sdc<br class=""># mkfs.ocfs2 -b 4K -C 4K -J size=4M -N 4 -L ocfs2vol2 --cluster-name=test --cluster-stack=o2cb --global-heartbeat /dev/sdd<br class=""><br class=""><blockquote type="cite" class="">From both VMs you can run the following and see:<br class=""></blockquote><br class=""># mounted.ocfs2 -d<br class=""><br class="">Device Stack Cluster F UUID Label<br class="">/dev/sdc o2cb test G 5620F19D43D840C7A46523019AE15A96 ocfs2vol1<br class="">/dev/sdd o2cb test G 9B9182279ABD4FD99F695F91488C94C1 ocfs2vol2<br class=""><br class="">I then add the global HB devices to the ocfs config file with similar commands:<br class=""><br class=""># o2cb add-heartbeat test 5620F19D43D840C7A46523019AE15A96<br class=""># o2cb add-heartbeat test 9B9182279ABD4FD99F695F91488C94C1<br class=""><br class="">Thus far looking good (heh, but then all we?ve done is format ocfs2 with options and updated a text file) - then I do the following:<br class=""><br class=""># o2cb heartbeat-mode test global<br class=""><br class="">All this being done on one node in the cluster I copy the following to the other node (with hostnames changed here, though the actual hostname = output of the hostname command on each node):<br class=""><br class=""># cat /etc/ocfs2/cluster.conf<br class=""><br class="">node:<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>name = clusterhost1.mydomain.com<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>number = 0<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_address = 10.143.144.12<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_port = 7777<br class=""><br class="">node:<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>name = clusterhost2.mydomain.com<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>number = 1<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_address = 10.143.144.13<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>ip_port = 7777<br class=""><br class="">cluster:<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>name = test<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>heartbeat_mode = global<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>node_count = 2<br class=""><br class="">heartbeat:<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>region = 5620F19D43D840C7A46523019AE15A96<br class=""><br class="">heartbeat:<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>cluster = test<br class=""><span class="Apple-tab-span" style="white-space:pre"> </span>region = 9B9182279ABD4FD99F695F91488C94C1<br class=""><br class="">The same config works fine with heartbeat_mode set to local and the global heartbeat devices removed, and I can mount a shared FS - the local HB interfaces are IPv4 on a private L2 non routed VLAN, are up and each node can ping each other.<br class=""><br class="">Once the config is copied to each node and have already run:<br class=""><br class=""># service o2cb configure<br class=""><br class="">Which completes in local heartbeat mode fine, so the cluster will start on boot and the params are default for timeouts etc.<br class=""><br class="">I check that the service on both nodes unloads and loads modules with no issues:<br class=""><br class=""># service o2cb unload<br class=""><br class="">Clean userdlm domains: OK<br class="">Unmounting ocfs2_dlmfs filesystem: OK<br class="">Unloading module "ocfs2_dlmfs": OK<br class="">Unloading module "ocfs2_stack_o2cb": OK<br class="">Unmounting configfs filesystem: OK<br class="">Unloading module "configfs": OK<br class=""><br class=""># service o2cb load<br class=""><br class="">Loading filesystem "configfs": OK<br class="">Mounting configfs filesystem at /sys/kernel/config: OK<br class="">Loading stack plugin "o2cb": OK<br class="">Loading filesystem "ocfs2_dlmfs": OK<br class="">Mounting ocfs2_dlmfs filesystem at /dlm: OK<br class=""><br class=""># mount -v<br class="">?<br class="">?.<br class="">debugfs on /sys/kernel/debug type debugfs (rw)<br class="">?.<br class="">ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)<br class=""><br class=""># lsmod | grep ocfs<br class=""><br class="">ocfs2_dlmfs 18026 1<br class="">ocfs2_stack_o2cb 3606 0<br class="">ocfs2_dlm 196778 1 ocfs2_stack_o2cb<br class="">ocfs2_nodemanager 202856 3 ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm<br class="">ocfs2_stackglue 11283 2 ocfs2_dlmfs,ocfs2_stack_o2cb<br class="">configfs 25853 2 ocfs2_nodemanager<br class=""><br class="">Looks good on both nodes?. then (sigh)<br class=""><br class=""># service o2cb enable<br class=""><br class="">Writing O2CB configuration: OK<br class="">Setting cluster stack "o2cb": OK<br class="">Registering O2CB cluster "test": Failed<br class="">o2cb: Unable to access cluster service while registering heartbeat mode 'global'<br class="">Unregistering O2CB cluster "test": OK<br class=""><br class="">I have searched for the error string and have come up with a huge ZERO on help - and the local OS log messages are equally unhelpful:<br class=""><br class=""># tail /var/log/messages<br class=""><br class="">Nov 12 21:54:53 clusterhost1 o2cb.init: online test<br class="">Nov 13 00:58:38 clusterhost1 o2cb.init: online test<br class="">Nov 13 01:00:06 clusterhost1 o2cb.init: offline test 0<br class="">Nov 13 01:00:06 clusterhost1 kernel: ocfs2: Unregistered cluster interface o2cb<br class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 Node Manager 1.6.3<br class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 DLM 1.6.3<br class="">Nov 13 01:01:14 clusterhost1 kernel: ocfs2: Registered cluster interface o2cb<br class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 DLMFS 1.6.3<br class="">Nov 13 01:01:14 clusterhost1 kernel: OCFS2 User DLM kernel interface loaded<br class="">Nov 13 01:03:32 clusterhost1 o2cb.init: online test<br class=""><br class="">Dmesg shows the same:<br class=""><br class=""># dmesg<br class=""><br class="">OCFS2 Node Manager 1.6.3<br class="">OCFS2 DLM 1.6.3<br class="">ocfs2: Registered cluster interface o2cb<br class="">OCFS2 DLMFS 1.6.3<br class="">OCFS2 User DLM kernel interface loaded<br class="">Slow work thread pool: Starting up<br class="">Slow work thread pool: Ready<br class="">FS-Cache: Loaded<br class="">FS-Cache: Netfs 'nfs' registered for caching<br class="">eth0: no IPv6 routers present<br class="">eth1: no IPv6 routers present<br class="">ocfs2: Unregistered cluster interface o2cb<br class="">OCFS2 Node Manager 1.6.3<br class="">OCFS2 DLM 1.6.3<br class="">ocfs2: Registered cluster interface o2cb<br class="">OCFS2 DLMFS 1.6.3<br class="">OCFS2 User DLM kernel interface loaded<br class="">ocfs2: Unregistered cluster interface o2cb<br class="">OCFS2 Node Manager 1.6.3<br class="">OCFS2 DLM 1.6.3<br class="">ocfs2: Registered cluster interface o2cb<br class="">OCFS2 DLMFS 1.6.3<br class="">OCFS2 User DLM kernel interface loaded<br class=""><br class="">The filesystem looks fine and this can be run from both hosts in the cluster:<br class=""><br class=""># fsck.ocfs2 -n /dev/sdc<br class=""><br class="">fsck.ocfs2 1.8.0<br class="">Checking OCFS2 filesystem in /dev/sdc:<br class=""> Label: ocfs2vol1<br class=""> UUID: 5620F19D43D840C7A46523019AE15A96<br class=""> Number of blocks: 524288<br class=""> Block size: 4096<br class=""> Number of clusters: 524288<br class=""> Cluster size: 4096<br class=""> Number of slots: 4<br class=""><br class=""># fsck.ocfs2 -n /dev/sdd<br class=""><br class="">fsck.ocfs2 1.8.0<br class="">Checking OCFS2 filesystem in /dev/sdd:<br class=""> Label: ocfs2vol2<br class=""> UUID: 9B9182279ABD4FD99F695F91488C94C1<br class=""> Number of blocks: 524288<br class=""> Block size: 4096<br class=""> Number of clusters: 524288<br class=""> Cluster size: 4096<br class=""> Number of slots: 4<br class=""><br class="">What am I missing? I?ve re-done this, re-created the devices a few too many times (thinking I may have missed something) but I am mystified. From all outer appearances I have two VMs that can see and in local heartbeat mode mount a shared OCFS2 filesystem and access it (have it running in local heartbeat mode for a cluster of rsyslog servers that are being load balanced by an F5 LTM VS with no issues) I am stumped on how to get global HB devices setup, though I have read and re-read the user guides, troubleshooting guides and wikis/blogs on how to make that work until my eyes hurt.<br class=""><br class="">Mounted the debugfs and ran the debugfs.ocfs2 utility but am unfamiliar of what I should be looking for there (or if this is where I would look for cluster not coming online errors)<br class=""><br class="">As the oc2b/ocfs modules are all kernel based I am not 100% sure how to increase debug information without digging into the source code and mucking around there.<br class=""><br class="">Any guidance or lessons learned (or things to check) would be super :) and if works warrant a happy scream of joy from my frustrated cube!<br class=""><br class=""><br class="">Warm regards,<br class=""><br class="">Jon<br class="">-------------- next part --------------<br class="">An HTML attachment was scrubbed...<br class="">URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20141112/68b903fe/attachment.html<br class=""><br class="">------------------------------<br class=""><br class="">_______________________________________________<br class="">Ocfs2-users mailing list<br class="">Ocfs2-users@oss.oracle.com<br class="">https://oss.oracle.com/mailman/listinfo/ocfs2-users<br class=""><br class="">End of Ocfs2-users Digest, Vol 130, Issue 1<br class="">*******************************************<br class=""></blockquote><br class=""><br class=""><br class=""><br class="">------------------------------<br class=""><br class="">_______________________________________________<br class="">Ocfs2-users mailing list<br class="">Ocfs2-users@oss.oracle.com<br class="">https://oss.oracle.com/mailman/listinfo/ocfs2-users<br class=""><br class="">End of Ocfs2-users Digest, Vol 130, Issue 2<br class="">*******************************************<br class=""></div></blockquote></div><br class=""></body></html>