[Ocfs2-users] Another node is heartbeating in our slot! errors with LUN removal/addition

[Daniel Keisling]

Sunil,

I manually killed the heartbeat via 'ocfs2_hb_ctl -K -d ... o2cb' but it
did not generate a core.  It did, however, stop the "ERROR: Device
"dm-30": another node is heartbeating in our slot!" messages.

After remounting the device, the "ERROR: Device "dm-30": another node is
heartbeating in our slot!" messages return to syslog.  Unmounting the
device brings a segfault:

Oct 30 10:59:13 ausracdbd01 multipathd: dm-30: umount map (uevent)
Oct 30 10:59:14 ausracdbd01 kernel: ocfs2_hb_ctl[11351]: segfault at
0000000000000000 rip 0000000000428fa0 rsp 00007fffe6710138 error 4
Oct 30 10:59:14 ausracdbd01 kernel: ocfs2: Unmounting device (253,30) on
(node 0)

The "ERROR: Device "dm-30": another node is heartbeating in our slot!"
messages keep flowing through syslog until I manually remove the
heartbeat.

I can reproduce this over and over.

TIA,

Daniel

> -----Original Message-----
> From: Sunil Mushran [mailto:sunil.mushran at oracle.com] 
> Sent: Friday, October 24, 2008 2:46 PM
> To: Daniel Keisling
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] Another node is heartbeating in 
> our slot! errors with LUN removal/addition
> 
> So that's the problem. The heartbeat is not stopping because of the
> segfault. I reviewed the code change in this tool (1.2.7 to 1.4.1)
> and it is quite limited. As in, I have no idea as to why it is
> segfaulting.
> 
> Now you could stop the heartbeat manually. You have to be careful
> though because stopping it for a mounted volume will be very
> problematic to say the least. But one good reason to do it manually
> would be to catch the coredump.... which could help tell us what the
> problem is.
> 
> If you are up for it, do:
> $ ulimit -c unlimited
> $ ocfs2_hb_ctl -K -d /dev/sdX o2cb
> 
> Run it on a umounted volume that has a reference left over. -I is
> to view the number of references.
> 
> Sunil
> 
> Daniel Keisling wrote:
> > Oct 23 08:53:21 ausracdb03 kernel: 
> (2410,3):o2hb_do_disk_heartbeat:770
> > ERROR: Device "dm-28": another node is heartbeating in our slot!
> >
> > [root at ausracdb03 ~]# ocfs2_hb_ctl -I -d /dev/dm-28
> > 289FD533334645C5A88FD715FC0EEF85: 1 refs
> >   
> > Yes, it segfaults every night (I take two snapshots per night):
> >
> > [root at ausracdb03 log]# grep segfault /var/log/messages
> > Oct 21 03:15:47 ausracdb03 kernel: ocfs2_hb_ctl[4197]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fffefd623e8 error 4
> > Oct 21 03:17:43 ausracdb03 kernel: ocfs2_hb_ctl[8002]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fff1a8f9318 error 4
> > Oct 21 16:43:30 ausracdb03 kernel: ocfs2_hb_ctl[16933]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fff816aa558 error 4
> > Oct 21 16:43:31 ausracdb03 kernel: ocfs2_hb_ctl[16950]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fffcb162b88 error 4
> > Oct 22 03:15:44 ausracdb03 kernel: ocfs2_hb_ctl[7721]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fff88a7efb8 error 4
> > Oct 22 03:17:46 ausracdb03 kernel: ocfs2_hb_ctl[11294]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fff85549f68 error 4
> > Oct 23 03:15:51 ausracdb03 kernel: ocfs2_hb_ctl[32555]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fff8fefe498 error 4
> > Oct 23 03:17:40 ausracdb03 kernel: ocfs2_hb_ctl[3756]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fff99bb25d8 error 4
> > Oct 24 03:15:47 ausracdb03 kernel: ocfs2_hb_ctl[15664]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007ffff4254aa8 error 4
> > Oct 24 03:17:43 ausracdb03 kernel: ocfs2_hb_ctl[18029]: segfault at
> > 0000000000000000 rip 0000000000428fa0 rsp 00007fff75055a78 error 4
> >
> >
> > This began when I upgraded to v1.4.1-1 from v1.2.8.
> >
> > Thanks,
> >
> > Daniel
> >   
> 
> 
> 

______________________________________________________________________
This email transmission and any documents, files or previous email
messages attached to it may contain information that is confidential or
legally privileged. If you are not the intended recipient or a person
responsible for delivering this transmission to the intended recipient,
you are hereby notified that you must not read this transmission and
that any disclosure, copying, printing, distribution or use of this
transmission is strictly prohibited. If you have received this transmission
in error, please immediately notify the sender by telephone or return email
and delete the original transmission and its attachments without reading
or saving in any manner.