[Ocfs2-users] Another node is heartbeating in our slot! errors with LUN removal/addition
Daniel Keisling
daniel.keisling at austin.ppdi.com
Tue Nov 25 08:15:25 PST 2008
All nodes except the node that I run snapshots on have the correct
number of o2hb threads running. However, node 2, the node that has
daily snapshots taken has _way_ too many threads:
[root at ausracdb03 ~]# ps aux | grep o2hb | wc -l
79
root at ausracdb03 ~]# ps aux | grep o2hb | head -n 10
root 1166 0.0 0.0 0 0 ? S< Nov20 0:47
[o2hb-00EFECD3FF]
root 1216 0.0 0.0 0 0 ? S< Oct25 4:14
[o2hb-5E0C4AD17C]
root 1318 0.0 0.0 0 0 ? S< Nov01 3:18
[o2hb-98697EE8BC]
root 1784 0.0 0.0 0 0 ? S< Nov15 1:25
[o2hb-A7DBDA5C27]
root 2293 0.0 0.0 0 0 ? S< Nov18 1:05
[o2hb-FBA96061AD]
root 2410 0.0 0.0 0 0 ? S< Oct23 4:49
[o2hb-289FD53333]
root 2977 0.0 0.0 0 0 ? S< Nov21 0:00
[o2hb-58CB9EA8F0]
root 3038 0.0 0.0 0 0 ? S< Nov21 0:00
[o2hb-D33787D93D]
root 3150 0.0 0.0 0 0 ? S< Oct25 4:38
[o2hb-3CB2E03215]
root 3302 0.0 0.0 0 0 ? S< Nov09 2:22
[o2hb-F78E8BF89E]
What's the best way to proceed?
Is this being caused by unpresenting/presenting snapshotted LUNs back to
the system? Those steps include:
- unmount the snapshot dir
- unmap the snapshot lun
- take a SAN-based snapshot
- present snapshot lun (same SCSI ID/WWNN) back to server
- force a uuid reset with tunefs.ocfs2 on the snapshot filesystem
- change the label with tunefs.ocfs2 on the snapshot filesystem
- fsck the snapshot filesystem
- mount the snapshot filesystem
I am using tunefs.ocfs2 v1.2.7 because the --force-uuid-reset is not in
the v1.4.1 release.
My two node development cluster, which is exactly the same as above, is
exhibiting the same behavior. My single node cluster, which is exactly
the same as above, is NOT exhibiting the same behavior.
Another single-node Oracle RAC cluster that is nearly the same (using
Qlogic HBA drivers for SCSI devices instead of device-mapper) does not
exhibit the o2hb thread issue.
Daniel
> -----Original Message-----
> From: Sunil Mushran [mailto:sunil.mushran at oracle.com]
> Sent: Monday, November 24, 2008 6:17 PM
> To: Daniel Keisling
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] Another node is heartbeating in
> our slot! errors with LUN removal/addition
>
> This is strange. There is something weird going on in your setup.
> When heartbeat starts, it generates a random generation number
> that remains constant as long as the heartbeat region is alive.
>
> The log file shows that the hb gen is not constant for node 0.
> Meaning, some other node is running the heartbeat thread claiming
> itself to be node 0. Node 1's hb gen remains the same.
>
> What is strange is that there are 8 different hb generations numbers.
> That explains the error message saying a different node is hbing in
> that slot.
>
> However, you have to find stray o2hb threads that have not
> been stopped.
> They could be running on node 0 itself. Or, some other nodes that you
> had setup as node 0 before. I am guessing here. Go thru each
> node that is
> physically connected to the storage.
>
> You can start with running "ps | aux | grep o2hb" to see how
> many of those
> threads are running on your nodes. It is should be 1 per
> mount per node.
>
> Sunil
>
>
> Fri Nov 21 13:28:56 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270bf7 44588ab5b5bb4ddc e9c92a6c
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:04 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c00 9bf2a853e46fba5f b3736682
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:05 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c01 44588ab5b5bb4ddc 49371c11
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:06 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c02 3e47b90030af5dfd 7786e320
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:07 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c03 44588ab5b5bb4ddc 69d2aa41
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:08 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c04 58b983437004059c 79b4a671
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:10 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c05 44588ab5b5bb4ddc 08fc70b1
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:11 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c07 e6b901ad46180797 adc0a96e
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:12 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c07 44588ab5b5bb4ddc 2819c6e1
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:22 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c12 f87033cdd779681d e63030fb
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:23 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c13 44588ab5b5bb4ddc b58e1e80
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:24 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c14 bf0c82508b3f190d 93d006f3
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:25 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c15 44588ab5b5bb4ddc d4a0c470
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
> Fri Nov 21 13:29:27 CST 2008
> node: node seq generation checksum
> 0: 0 0000000049270c16 2535047a999203a9 5e896498
> 1: 1 0000000049266fa3 bc1f697ec3c0e60b a7c6f487
>
>
>
>
>
> Daniel Keisling wrote:
> > Sorry for the delay in getting back to you.
> >
> > I never catch a core during the segfault of the umount.
> >
> > I tried to delete the heartbeat again and the command completed
> > successfully, but the messages are still appearing in syslog. A
> > subsequent issue of the command brings:
> >
> > [root at ausracdbd01 ~]# ocfs2_hb_ctl -K -d /dev/dm-30 o2cb
> > ocfs2_hb_ctl: Unable to access cluster service while
> stopping heartbeat
> >
> >
> > Please see attached for the log you requested.
> >
> > Daniel
> >
> >
> >> -----Original Message-----
> >> From: Sunil Mushran [mailto:sunil.mushran at oracle.com]
> >> Sent: Thursday, October 30, 2008 5:42 PM
> >> To: Daniel Keisling
> >> Cc: ocfs2-users at oss.oracle.com
> >> Subject: Re: [Ocfs2-users] Another node is heartbeating in
> >> our slot! errors with LUN removal/addition
> >>
> >> So manually stopping the heartbeat worked.
> >>
> >> Did you catch a coredump for the segfault during umount?
> >>
> >> There is a small difference in the stop heartbeat that is called
> >> as part of umount and the one called by hand. But have not
> >> been able to figure out the source of the segfault.
> >>
> >> The above coredump will help.
> >>
> >> The other thing you could do is run the following command when you
> >> see the "another node heartbeating..." message.
> >> $ for i in `seq 30` ; do date >>/tmp/hb.out ;
> debugfs.ocfs2 -R "hb"
> >> /dev/dmX >>/tmp/hb.out ; sleep 1; done;
> >>
> >> Replace the device name with the one that is in the logs.
> >> Email me the
> >> output.
> >>
> >> Sunil
> >>
>
>
______________________________________________________________________
This email transmission and any documents, files or previous email
messages attached to it may contain information that is confidential or
legally privileged. If you are not the intended recipient or a person
responsible for delivering this transmission to the intended recipient,
you are hereby notified that you must not read this transmission and
that any disclosure, copying, printing, distribution or use of this
transmission is strictly prohibited. If you have received this transmission
in error, please immediately notify the sender by telephone or return email
and delete the original transmission and its attachments without reading
or saving in any manner.
More information about the Ocfs2-users
mailing list