<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Arial","sans-serif";
color:#1F497D;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1559975361;
mso-list-type:hybrid;
mso-list-template-ids:-838835470 1428701038 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Arial","sans-serif";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l1
{mso-list-id:2098480078;
mso-list-template-ids:-33416242;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'>Here are some hints:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'>Starting with versions 1.2.5 and later, the ocfs2 network timeout may be configured. <o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'>If using network bonding, you should set the network idle timeout to at least 30 seconds.<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'>Set O2CB_IDLE_TIMEOUT_MS to at least 30000. If the problem persists, set it to 60000.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> ocfs2-users-bounces@oss.oracle.com [mailto:ocfs2-users-bounces@oss.oracle.com] <b>On Behalf Of </b>Kolstee, Ronald A (Tony)<br><b>Sent:</b> Monday, November 22, 2010 11:57 PM<br><b>To:</b> ocfs2-users@oss.oracle.com<br><b>Subject:</b> [Ocfs2-users] RAC and OCFS2 timeout interaction issues<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal> <o:p></o:p></p><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>We have a RAC cluster as follows:</span><o:p></o:p></p></div><ul type=disc><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>3 nodes with RHEL5, Oracle 11.1.0.7, and OCFS2 1.4.2</span> <o:p></o:p></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Voting and OCR are on OCFS2, all other shared storage is on ASM</span> <o:p></o:p></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Storage hardware is provided by a fibre-channel SAN fabric</span> <o:p></o:p></li><li class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l1 level1 lfo1'><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Interconnect uses two bonded NICs per server, connected to different blades on a single switch</span><o:p></o:p></li></ul><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Previously we had an issue where all three nodes would reboot if one node had problems. This could be caused by one node crashing completely (OS crash), or losing interconnect. For testing purposes, we've been simulating OS crashes by suddenly resetting the server without a graceful shutdown, and simulating loss of interconnect by using ifconfig down on both interfaces in the bond.</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>From what we've seen, it appears that the issue is an interaction between the O2CB and CRS timeout values - namely that CRS is self-fencing on the surviving nodes before O2CB has a chance to time out and recover the dead node.</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>By adjusting O2CB's timeout (O2CB_HEARTBEAT_THRESHOLD x 2 seconds) lower than the CRS disktimeout value, we were able to configure the cluster so that the "OS Crash" scenario is properly handled. The other two nodes will survive when we completely reset the third. </span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>However, we haven't solved the loss of interconnect scenario and believe the problem to be similar. I'd rather not get bogged down in the specifics, logfiles, etc. at this point in time, as we have to apply these concepts to other environments as well, and there is still some tweaking of these configurations to be done. </span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Can anyone please provide a generic conceptual overview on how the CRS and O2CB timeout values interact in this scenario? Does O2CB_HEARTBEAT_THRESHOLD come into play at all when dealing with loss of interconnect, and how does it interact with the value of O2CB_IDLE_TIMEOUT_MS? How does this correlate with the timeouts on the CRS side of the equation?</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>I appreciate any help that anyone can offer.</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Thanks in advance,</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Tony Kolstee</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Sr. Systems Engineer</span><o:p></o:p></p></div><div><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Aetna</span><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><p class=MsoNormal>This e-mail may contain confidential or privileged information. If you think you have received this e-mail in error, please advise the sender by reply e-mail and then delete this e-mail immediately. Thank you. Aetna <o:p></o:p></p></div></body></html>