<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1129973508;
mso-list-type:hybrid;
mso-list-template-ids:-1506885904 -1944439890 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F02D;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>Hi John,<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>When multiple systems/nodes have access to
data via shared storage, the integrity of the data depends on inter-node
communication ensuring that each node is aware when other nodes are writing
data. When the coordination between the nodes fails, it results in a
“split brain” condition; A situation in which two servers try to
independently control the storage, potentially resulting in application failure
or even corruption of critical data.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>I/O fencing is a method of choice (used by
vendors cluster frameworks, including OCFS2) for ensuring the integrity of
critical information by preventing data corruption, allowing a set of systems
to have temporary registrations with the disk and coordinate a write-exclusive
reservation with the disk containing the data. With I/O fencing, the cluster
system ensures that errant nodes are “fenced” and do not have
access to the shared storage, while the eligible node(s) continue to have
access to the data, <u>virtually</u> eliminating the risk of data corruption.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>The quorum is the group of nodes in a cluster
that is allowed to operate on the shared storage. When there is a failure in
the cluster, nodes may be split into groups that can communicate in their
groups and with the shared storage but not between groups.<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>O2QUO determines which group is allowed to
continue and initiates fencing of the other group(s).<o:p></o:p></span></p>
<p class=MsoNormal style='text-autospace:none'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>Fencing is the act of forcefully removing a
node from a cluster. A node with OCFS2 mounted will fence itself when it
realizes that it does not have quorum in a degraded cluster. It does this so
that other nodes won’t be stuck trying to access its resources. <u>However,
the resources do NOT get released<o:p></o:p></u></span></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>O2CB
uses a node reset mechanism to fence; this however, is causing the machine(s)
to hang instead of seamless handover. In OCFS2 1.4, Oracle has introduced a new
fencing mechanism which no longer uses “panic” for fencing.
Instead, by default, it uses "machine restart".<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>In
your case, taking the network down the way you’ve done is causing the
servers to hang, including the mounted file system which becomes locked until
the OCFS cluster services is restarted.<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>RAC
handover fails due to exactly this problem: the file system is locked by
another node which was kicked out of the cluster, but still occupying the file
system<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>The
healthy node will try to continue to work, but the databases hosted on the
occupied file system will hang, and possibly the machine. At this time there is
no solution but to <o:p></o:p></span></p>
<p class=MsoPlainText style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='font-size:10.0pt;font-family:Symbol'><span style='mso-list:Ignore'>-<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Force
shutdown the troublesome node(s)<o:p></o:p></span></p>
<p class=MsoPlainText style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='font-size:10.0pt;font-family:Symbol'><span style='mso-list:Ignore'>-<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Shutdown
the databases processes<o:p></o:p></span></p>
<p class=MsoPlainText style='margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='font-size:10.0pt;font-family:Symbol'><span style='mso-list:Ignore'>-<span
style='font:7.0pt "Times New Roman"'>
</span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Restart
the OCFS2 services<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'> <o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Network
failure resolution can be applied in a situation where you have setup a net bonding
for the interconnects, which is highly recommended.<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p> </o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Best
regards,<o:p></o:p></span></p>
<p class=MsoPlainText><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Karim
Alkhayer<o:p></o:p></span></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>-----Original Message-----<br>
From: ocfs2-users-bounces@oss.oracle.com
[mailto:ocfs2-users-bounces@oss.oracle.com] On Behalf Of John Murphy<br>
Sent: Thursday, June 04, 2009 10:15 PM<br>
To: ocfs2-users@oss.oracle.com<br>
Subject: [Ocfs2-users] OCFS2 v1.4 hangs<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>I have four database servers in a high-availability,
load-balancing<o:p></o:p></p>
<p class=MsoPlainText>configuration. Each machine has a mount to a common data
source which is<o:p></o:p></p>
<p class=MsoPlainText>an OCFS2 v1.4 file-system. While working on three of the
servers, I<o:p></o:p></p>
<p class=MsoPlainText>restarted the IP network and found after-wards the fourth
machine hung.<o:p></o:p></p>
<p class=MsoPlainText>I could not reboot and could not unmount the ocfs2
partitions. I am<o:p></o:p></p>
<p class=MsoPlainText>pretty sure this was all caused by my taking down the
network on all<o:p></o:p></p>
<p class=MsoPlainText>three of the remaining machines, can anyone shed some
light on this for.<o:p></o:p></p>
<p class=MsoPlainText>Ironically, I have four machines in order to ensure
reliability.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>TIA<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>John<o:p></o:p></p>
<p class=MsoPlainText>-- <o:p></o:p></p>
<p class=MsoPlainText>John Murphy<o:p></o:p></p>
<p class=MsoPlainText>Technical And Managing Director<o:p></o:p></p>
<p class=MsoPlainText>MANDAC Ltd<o:p></o:p></p>
<p class=MsoPlainText>Kandoy House<o:p></o:p></p>
<p class=MsoPlainText>2 Fairview Strand<o:p></o:p></p>
<p class=MsoPlainText>Dublin 3<o:p></o:p></p>
<p class=MsoPlainText>p: +353 1 5143001<o:p></o:p></p>
<p class=MsoPlainText>m: +353 85 711 6844<o:p></o:p></p>
<p class=MsoPlainText>e: john.murphy@mandac.eu<o:p></o:p></p>
<p class=MsoPlainText>w: www.mandac.eu<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>_______________________________________________<o:p></o:p></p>
<p class=MsoPlainText>Ocfs2-users mailing list<o:p></o:p></p>
<p class=MsoPlainText>Ocfs2-users@oss.oracle.com<o:p></o:p></p>
<p class=MsoPlainText>http://oss.oracle.com/mailman/listinfo/ocfs2-users<o:p></o:p></p>
</div>
</body>
</html>