<table cellspacing='0' cellpadding='0' border='0' ><tr><td style='font: inherit;'><br> If you expect one of the switches to remain alive you should configure your bonding driver timeout low to force it to failover soon to the other switch.<br><br> What kind of bonding are you using? There are several different modes on Linux, for different types of switch configurations. I have used balance-tlb with good results, also active-backup could work too. The main problem I see is the time the switch takes to identify the different path.<br><br> I had some problems with virtual IP failover due to the kernel not broadcasting something to the switch if forwarding is not enabled. Although virtual IPs are very different from network bonding you might want to try enabling ip_forward to see if the switches reconfigure faster. (echo 1 > /proc/sys/net/ipv4/ip_forward). <br><br> Some people here have asked the O2CB to allow for multiple
heartbeat interfaces, it would provide an alternative to this problem, as the switches would not need to reconfigure.<br><br>Regards,<br>Luis<br><br>--- On <b>Tue, 4/22/08, Brendan Beveridge <i><brendan@sitesuite.com.au></i></b> wrote:<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;">From: Brendan Beveridge <brendan@sitesuite.com.au><br>Subject: Re: [Ocfs2-users] Node reboot during network outage<br>To: <br>Cc: "ocfs2-users@oss.oracle.com" <ocfs2-users@oss.oracle.com><br>Date: Tuesday, April 22, 2008, 8:06 PM<br><br><pre>STP Shouldnt have anything to do with the nodes still seeing each other <br>when the switch fails<br><br>Have you checked that your bonding config is correct?<br>ie<br><br>cat /proc/net/bonding/bond0<br>and check that it fails over to the other eth when the switch goes down<br><br>Cheers<br>Brendan<br><br><br>Sunil Mushran wrote:<br>> The issue is not the time the
switch takes to reboot. The issue is the<br>> amount of time the secondary switch takes to find a unique path.<br>><br>> http://en.wikipedia.org/wiki/Spanning_tree_protocol<br>><br>> Mick Waters wrote:<br>> <br>>> Thanks Sunil,<br>>><br>>> The network switch is brand new but has a fairly complex configuration<br>due to us running a number of VLANs - however, we have found that it has always<br>taken quite a while to reboot.<br>>><br>>> I'll try increasing the idle timeout as suggested and let you know<br>what happens. However, surely this is only treating the symptoms of what is,<br>after all, a contrived scenario. Rebooting the switch is supposed to test what<br>would happen if we had a real network outage. What if the switch were to stay<br>down?<br>>><br>>> My issue is that we have an alternative route via the other NIC in the<br>bond and the other switch. The affected nodes in cluster
shouldn't fence<br>because they should still be able to see all of the other nodes in the cluster<br>via this other route.<br>>><br>>> Does this make sense?<br>>><br>>> Regards,<br>>><br>>> Mick.<br>>><br>>> -----Original Message-----<br>>> From: Sunil Mushran [mailto:Sunil.Mushran@oracle.com]<br>>> Sent: 22 April 2008 17:40<br>>> To: Mick Waters<br>>> Cc: ocfs2-users@oss.oracle.com<br>>> Subject: Re: [Ocfs2-users] Node reboot during network outage<br>>><br>>> The interface died at 14:25:44 and recovered at 14:27:43.<br>>> That's two minutes.<br>>><br>>> One solution is to increase o2cb_idle_timeout to > 2mins.<br>>><br>>> Better solution would be to look into your router setup to determine<br>why it is taking 2 minutes for the router to reconfigure.<br>>><br>>> Mick Waters wrote:<br>>> <br>>>
<br>>>> Hi, my company is in the process of moving our web and database<br>>>> servers to new hardware. We have a HP EVA 4100 SAN which is being<br>>>> used by two database servers running in an Oracle 10g cluster and<br>that<br>>>> works fine. We have gone to extreme lengths to ensure high<br>>>> availability. The SAN has twin disk arrays, twin controllers, and<br>all<br>>>> servers have dual fibre interfaces. Networking is (should be)<br>>>> similarly redundant with bonded NICs connected in two-switch<br>>>> configuration, two firewalls and so on.<br>>>><br>>>> We also want to share regular Linux filesystems between our<br>servers -<br>>>> HP DL580 G5s running RedHat AS 5 (kernel 2.6.18-53.1.14.el5) and<br>we<br>>>> chose OCFS2 (1.2.8) to manage the cluster.<br>>>><br>>>> As stated, each server in the 4 node cluster has a
bonded<br>interface<br>>>> set up as bond0 in a two-switch configuration (each NIC in the<br>bond is<br>>>> connected to a different switch). Because this is a two-switch<br>>>> configuration, we are running the bond in active-standby mode and<br>this<br>>>> works just fine.<br>>>><br>>>> Our problem occurred when we were doing failover testing where we<br>>>> simulated the loss of one of the network switches by powering it<br>off.<br>>>> The result was that the servers rebooted and this make a mockery<br>of<br>>>> our attempts at a HA solution.<br>>>><br>>>> Here is a short section from /var/log/messages following a reboot<br>of<br>>>> one of the switches to simulate an outage:<br>>>><br>>>><br>----------------------------------------------------------------------<br>>>> ---- Apr 22 14:25:44 mtkws01p1 kernel: bonding:
bond0: backup<br>>>> interface eth0 is now down Apr 22 14:25:44 mtkws01p1 kernel: bnx2:<br>>>> eth0 NIC Link is Down Apr 22 14:26:13 mtkws01p1 kernel: o2net:<br>>>> connection to node mtkdb01p2 (num 1) at 10.1.3.50:7777 has been<br>idle<br>>>> for 30.0 seconds, shutting it down.<br>>>> Apr 22 14:26:13 mtkws01p1 kernel: (0,12):o2net_idle_timer:1426<br>here<br>>>> are some times that might help debug the situation: (tmr<br>>>> 1208870743.673433 now 1208870773.673192 dr 1208870743.673427 adv<br>>>> 1208870743.673433:1208870743.673434 func (97690d75:2)<br>>>> 1208870697.670758:1208870697.670760)<br>>>> Apr 22 14:26:13 mtkws01p1 kernel: o2net: no longer connected to<br>node<br>>>> mtkdb01p2 (num 1) at 10.1.3.50:7777<br>>>> Apr 22 14:27:38 mtkws01p1 kernel: bnx2: eth0 NIC Link is Up, 1000<br>Mbps<br>>>> full duplex Apr 22 14:27:43 mtkws01p1
kernel: bonding: bond0:<br>backup<br>>>> interface eth0 is now up Apr 22 14:28:35 mtkws01p1 kernel:<br>>>> (5234,9):dlm_do_master_request:1418<br>>>> ERROR: link to 1 went down!<br>>>> Apr 22 14:28:35 mtkws01p1 kernel:<br>(5234,9):dlm_get_lock_resource:995<br>>>> ERROR: status = -107<br>>>> Apr 22 14:28:35 mtkws01p1 kernel:<br>(5234,9):ocfs2_broadcast_vote:731<br>>>> ERROR: status = -107<br>>>> Apr 22 14:28:35 mtkws01p1 kernel:<br>(5234,9):ocfs2_do_request_vote:804<br>>>> ERROR: status = -107<br>>>> Apr 22 14:28:35 mtkws01p1 kernel: (5234,9):ocfs2_unlink:843 ERROR:<br>>>> status = -107<br>>>> Apr 22 14:29:29 mtkws01p1 kernel: o2net: connection to node<br>mtkdb02p2<br>>>> (num 2) at 10.1.3.51:7777 has been idle for 30.0 seconds, shutting<br>it<br>>>> down.<br>>>> Apr 22 14:29:29 mtkws01p1 kernel:
(0,12):o2net_idle_timer:1426<br>here<br>>>> are some times that might help debug the situation: (tmr<br>>>> 1208870939.955991 now 1208870969.956343 dr 1208870939.955984 adv<br>>>> 1208870939.955992:1208870939.955993 func (97690d75:2)<br>>>> 1208870697.670916:1208870697.670918)<br>>>> Apr 22 14:29:29 mtkws01p1 kernel: o2net: no longer connected to<br>node<br>>>> mtkdb02p2 (num 2) at 10.1.3.51:7777<br>>>> Apr 22 14:30:18 mtkws01p1 kernel:<br>(5268,6):ocfs2_broadcast_vote:731<br>>>> ERROR: status = -107<br>>>> Apr 22 14:30:18 mtkws01p1 kernel:<br>(5268,6):ocfs2_do_request_vote:804<br>>>> ERROR: status = -107<br>>>> Apr 22 14:30:18 mtkws01p1 kernel: (5268,6):ocfs2_unlink:843 ERROR:<br>>>> status = -107<br>>>> Apr 22 14:34:23 mtkws01p1 syslogd 1.4.1:
restart.<br>>>><br>----------------------------------------------------------------------<br>>>> ----<br>>>><br>>>> Things that I have tried...<br>>>><br>>>> I've tried setting up the bond with both miimon and ARP<br>monitoring (at<br>>>> different times of course) because when the switch comes back up<br>the<br>>>> link detect goes up and down several times while the switch<br>>>> initialises and I hoped that ARP monitoring might be more reliable<br>-<br>>>> it made no difference at all.<br>>>><br>>>> I've increased the heartbeat timeout to 61 from 31 but, as<br>yet, I<br>>>> haven't played with any of the other cluster configuration<br>variables.<br>>>><br>>>> Has anyone with a similar configuration experienced problems like<br>this<br>>>> and found a solution?<br>>>><br>>>>
Regards,<br>>>><br>>>> Mick.<br>>>><br>>>><br>----------------------------------------------------------------------<br>>>> --<br>>>><br>>>> Mick Waters<br>>>> Senior Systems Developer<br>>>><br>>>> w: +44 (0)208 335 2011<br>>>> m: +44 (0)7849 887 277<br>>>> e: mick.waters@motortrak.com<br><mailto:mick.waters@motortrak.com><br>>>><br>>>> www.motortrak.com <http://www.motortrak.com/> digital media<br>solutions<br>>>><br>>>> Motortrak Ltd, AC Court, High St, Thames Ditton, Surrey, KT7 0SR,<br>>>> United Kingdom<br>>>><br>>>> The information contained in this message is for the intended<br>>>> addressee only and may contain confidential and/or privileged<br>>>> information. If you are not the intended addressee, please delete<br>this<br>>>> message
and notify the sender; do not copy or distribute this<br>message<br>>>> or disclose its contents to anyone. Any views or opinions<br>expressed in<br>>>> this message are those of the author and do not necessarily<br>represent<br>>>> those of Motortrak Limited or of any of its associated companies.<br>No<br>>>> reliance may be placed on this message without written<br>confirmation<br>>>> from an authorised representative of the company.<br>>>><br>>>> Registered in England 3098391 V.A.T. Registered No. 667463890<br>>>><br>>>><br>>>><br>----------------------------------------------------------------------<br>>>> --<br>>>><br>>>> _______________________________________________<br>>>> Ocfs2-users mailing list<br>>>> Ocfs2-users@oss.oracle.com<br>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users<br>>>>
<br>>>> <br>>> <br>>> <br>><br>><br>> _______________________________________________<br>> Ocfs2-users mailing list<br>> Ocfs2-users@oss.oracle.com<br>> http://oss.oracle.com/mailman/listinfo/ocfs2-users<br>> <br><br><br>_______________________________________________<br>Ocfs2-users mailing list<br>Ocfs2-users@oss.oracle.com<br>http://oss.oracle.com/mailman/listinfo/ocfs2-users</pre></blockquote></td></tr></table><br>
<hr size=1>Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. <a href="http://us.rd.yahoo.com/evt=51733/*http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ "> Try it now.</a>