<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=iso-8859-1">
<META content="MSHTML 6.00.2800.1561" name=GENERATOR>
<STYLE></STYLE>
</HEAD>
<BODY>
<DIV><FONT face=Arial size=2>Additional info - node had not ANY active OCFSv2
operations (OCFSv2 used for backups only and from another node only). So, if
system just SUSPEND all FS operations and try to rejoin to the cluster, it all
could work (moreover, connection to the disk system was intact, so it could
close file sytem gracefully).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>It reveals 3 problems at once:</FONT></DIV>
<DIV><FONT face=Arial size=2>- single heartbeat link (instead of multiple
links)</FONT></DIV>
<DIV><FONT face=Arial size=2>- timeout too short (ethernet can't guarantee 10
seconds, it can guarantee 1 minute minimum);</FONT></DIV>
<DIV><FONT face=Arial size=2>- fencing even if system is passive and can remount
/ reconnect instead of rebooting.</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>All we did in the lab was _disconnect 1 of trunks
between switches for a few seconds, then insert it back into the socket_. No one
other application failed</FONT></DIV>
<DIV><FONT face=Arial size=2>(including heartbeat clusters). Database cluster
was not doing anything on OCFS in time of failure (even backups).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>I will try heartbeat between loopback interfaces
(and OCFS protocol) next time (I am just curios if it can provide 10 seconds for
network reconfiguration).</FONT></DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><STRONG>...</STRONG></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><STRONG>Feb 1 12:19:13
testrac12 kernel: o2net: connection to node testrac11 (num 0) at
10.254.32.111:7777 has been idle for <FONT color=#ff0000>10 seconds</FONT>,
shutting it down. <BR>Feb 1 12:19:13 testrac12 kernel:
(13,3):o2net_idle_timer:1310 here are some times that might help debug the
situation: (tmr 1170361135.521061 now 1170361145.520476 dr 1170361141.852795 adv
1170361135.521063:1170361135.521064 func (c4378452:505)
1170361067.762941:1170361067.762967) <BR>Feb 1 12:19:13 testrac12 kernel:
o2net: no longer connected to node testrac11 (num 0) at 10.254.32.111:7777
<BR>Feb 1 12:19:13 testrac12 kernel:
(1855,3):dlm_send_remote_convert_request:398 ERROR: status = -107 <BR>Feb
1 12:19:13 testrac12 kernel: (1855,3):dlm_wait_for_node_death:371
5AECFF0BBCF74F069A3B8FF79F09FB5A: waiting 5000ms for notification of death of
node 0 <BR>Feb 1 12:19:13 testrac12 kernel:
(1855,1):dlm_send_remote_convert_request:398 ERROR: status = -107 <BR>Feb
1 12:19:13 testrac12 kernel: (1855,1):dlm_wait_for_node_death:371
5AECFF0BBCF74F069A3B8FF79F09FB5A: waiting 5000ms for notification of death of
node 0 <BR>Feb 1 12:22:22 testrac12 kernel:
(1855,2):dlm_send_remote_convert_request:398 ERROR: status = -107 <BR>Feb
1 12:22:22 testrac12 kernel: (1855,2):dlm_wait_for_node_death:371
5AECFF0BBCF74F069A3B8FF79F09FB5A: waiting 5000ms for notification of death of
node 0 <BR>Feb 1 12:22:27 testrac12 kernel: (13,3):o2quo_make_decision:144
ERROR: fencing this node because it is connected to a half-quorum of 1 out of 2
nodes which doesn't include the lowest active node 0 <BR>Feb 1 12:22:27
testrac12 kernel: (13,3):o2hb_stop_all_regions:1889 ERROR: stopping heartbeat on
all active regions. <BR>Feb 1 12:22:27 testrac12 kernel: Kernel panic:
ocfs2 is very sorry to be fencing this system by panicing <BR>Feb 1
12:22:27 testrac12 kernel: <BR>Feb 1 12:22:28 testrac12 su: pam_unix2:
session finished for user oracle, service su <BR>Feb 1 12:22:29 testrac12
logger: Oracle CSSD failure. Rebooting for cluster integrity. <BR>Feb
1 12:22:32 testrac12 su: pam_unix2: session finished for user oracle,
service su <BR>...</STRONG></FONT></DIV></BODY></HTML>