<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 01/20/2014 11:05 PM, Thiruselvam Velayutham wrote:<br>
<blockquote
cite="mid:947FA8586C84374D9E800452ED41A1D2102D940A@422117-rhs-exg.Bedford.local"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<style>
<!--
@font-face
{font-family:Calibri}
@font-face
{font-family:Tahoma}
@font-face
{font-family:Verdana}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif"}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif"}
span.hoenzb
{}
span.EmailStyle18
{font-family:"Calibri","sans-serif";
color:#1F497D}
span.BalloonTextChar
{font-family:"Tahoma","sans-serif"}
.MsoChpDefault
{font-family:"Calibri","sans-serif"}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Dear Experts,</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Our DBA team is facing following problem.
</span></p>
<div style="border:none; border-bottom:solid windowtext 1.0pt;
padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none; padding:0in"><span
style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">We did high availability testing and when we
crash DB node 1, DB Node 2 also went down, and from the
errors, i could see ocfs2 service has shutdown DB02</span></p>
</div>
</blockquote>
It appears the filesystem didn't go through a clean umount. In that
case if the network resource goes down before the storage, then the
other node thinks the network connection between the nodes broke.
When that happens, they think it's a split brain and in a 2 node
case, node with higher node number reboots.<br>
<br>
To fix this, make sure you unmount ocfs2 volumes before doing a
reboot.<br>
<br>
PS: If the cluster setup has more than 2 nodes, this wouldn't be an
issue, but still not a good idea to reboot the box without a clean
umount.<br>
<br>
<blockquote
cite="mid:947FA8586C84374D9E800452ED41A1D2102D940A@422117-rhs-exg.Bedford.local"
type="cite">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">here is the issue in detail.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">DB01</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">DB02</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Ap01</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">AP02</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">when i crash DB01 server, DB02 server also
goes down and total oracle is collapsed.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">when doing vice versa, crash DB02, DB01
survives . and oracle continues to work without any issues</span></p>
<div style="border:none; border-bottom:solid windowtext 1.0pt;
padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none; padding:0in"><span
style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">messages_DB02.txt</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">===================</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:15:52 kbmmoppdb02
avahi-daemon[8824]: Registering new address record for
172.20.1.9 on eth0.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:16:13 kbmmoppdb02 kernel: o2dlm:
Node 0 leaves domain 8155F09482C94D3AB99D0669B91C0B1E</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:16:13 kbmmoppdb02 kernel: o2dlm:
Nodes in domain 8155F09482C94D3AB99D0669B91C0B1E: 1
</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:27 kbmmoppdb02 kernel: o2net:
connection to node kbmmoppdb01 (num 0) at 10.255.255.3:7777
has been idle for 30.0 seconds, shutting it down.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:27 kbmmoppdb02 kernel:
(swapper,0,11):o2net_idle_timer:1515 here are some times
that might help debug the situation: (tmr 1390245417.409760
now 1390245447.410787 dr 1390245417.409740 adv
1390245417.409769:1390245417.409770 func (d9d367e5:505)
1390245414.653885:1390245414.653892)</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:27 kbmmoppdb02 kernel: o2net: no
longer connected to node kbmmoppdb01 (num 0) at
10.255.255.3:7777</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:27 kbmmoppdb02 kernel:
(kswapd0,576,10):dlm_send_remote_unlock_request:360 ERROR:
Error -112 when sending message 506 (key 0x60f827ee) to node
0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:48 kbmmoppdb02 kernel: o2net:
connection to node kbmmoppdb01 (num 0) at 10.255.255.3:7777
shutdown, state 7</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:57 kbmmoppdb02 kernel:
(o2net,6123,11):o2net_connect_expired:1676 ERROR: no
connection established with node 0 after 30.0 seconds,
giving up and returning errors.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:57 kbmmoppdb02 kernel:
(dlm_thread,6161,8):dlm_drop_lockres_ref:2191 ERROR: Error
-107 when sending message 507 (key 0x60f827ee) to node 0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:57 kbmmoppdb02 kernel:
(kswapd0,576,10):dlm_send_remote_unlock_request:360 ERROR:
Error -107 when sending message 506 (key 0x60f827ee) to node
0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:57 kbmmoppdb02 last message
repeated 73 times</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:57 kbmmoppdb02 kernel:
(dlm_thread,6161,8):dlm_purge_lockres:193 ERROR:
C5F98815D0BF43578B48C12C21114311: deref
O000000000000000124facd00000000 failed -107</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:17:57 kbmmoppdb02 kernel: o2net:
connection to node kbmmoppdb01 (num 0) at 10.255.255.3:7777
shutdown, state 7</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:25 kbmmoppdb02 last message
repeated 9 times</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(o2net,6123,11):o2net_connect_expired:1676 ERROR: no
connection established with node 0 after 30.0 seconds,
giving up and returning errors.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(dlm_thread,6161,8):dlm_drop_lockres_ref:2191 ERROR: Error
-107 when sending message 507 (key 0x60f827ee) to node 0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(kswapd0,576,10):dlm_send_remote_unlock_request:360 ERROR:
Error -107 when sending message 506 (key 0x60f827ee) to node
0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 last message
repeated 180 times</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(dlm_thread,6161,8):dlm_purge_lockres:193 ERROR:
C5F98815D0BF43578B48C12C21114311: deref
M000000000000000124facd00000000 failed -107</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(dlm_thread,6161,10):dlm_drop_lockres_ref:2191 ERROR: Error
-107 when sending message 507 (key 0x60f827ee) to node 0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(dlm_thread,6161,10):dlm_purge_lockres:193 ERROR:
C5F98815D0BF43578B48C12C21114311: deref
O000000000000000124facc00000000 failed -107</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(dlm_thread,6161,4):dlm_drop_lockres_ref:2191 ERROR: Error
-107 when sending message 507 (key 0x60f827ee) to node 0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:27 kbmmoppdb02 kernel:
(dlm_thread,6161,4):dlm_purge_lockres:193 ERROR:
C5F98815D0BF43578B48C12C21114311: deref
O000000000000000124fa8e00000000 failed -107</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:28 kbmmoppdb02 kernel: o2net:
connection to node kbmmoppdb01 (num 0) at 10.255.255.3:7777
shutdown, state 7</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:31 kbmmoppdb02 kernel: o2net:
connection to node kbmmoppdb01 (num 0) at 10.255.255.3:7777
shutdown, state 7</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:33 kbmmoppdb02 kernel:
(events/11,49,11):o2quo_make_decision:158 ERROR: fencing
this node because it is connected to a half-quorum of 1 out
of 2 nodes which doesn't include the lowest active node 0</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:18:33 kbmmoppdb02 kernel:
(events/11,49,11):o2hb_stop_all_regions:2026 ERROR: stopping
heartbeat on all active regions.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:23:10 kbmmoppdb02 syslogd 1.4.1:
restart.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D">Jan 20 13:23:10 kbmmoppdb02 kernel: klogd
1.4.1, log source = /proc/kmsg started</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"> </span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;
font-family:"Verdana","sans-serif";
color:gray" lang="EN-IN">Regards,</span><span
style="font-size:8.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-IN"></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;
font-family:"Verdana","sans-serif";
color:gray" lang="EN-IN">Thiruselvam V</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-IN"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D"><img id="Picture_x0020_2"
src="cid:part1.02070500.06020907@oracle.com"
alt="Description: Description: Description:
cid:image001.png@01CE5876.CAF7DD70" height="62" width="59"></span><span
style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-IN"></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;
font-family:"Verdana","sans-serif";
color:gray" lang="EN-IN">VOIP : 603 521 6544 | Mobile :+91
9986150593│Fax:+91 80 41122605 | Skype :
<a moz-do-not-send="true"
href="mailto:vthiruselvam@gmail.com">vthiruselvam@gmail.com</a>
AOL : <a moz-do-not-send="true"
href="mailto:tvelayutham@kbace.com">
tvelayutham@kbace.com</a></span><span
style="font-size:8.0pt;
font-family:"Verdana","sans-serif";
color:#7F7F7F"></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;
font-family:"Verdana","sans-serif";
color:#7F7F7F">KBACE Technologies
</span><span style="font-size:8.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-IN"></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-IN"><a moz-do-not-send="true"
href="http://www.kbace.com/"><span style="font-size:8.0pt;
font-family:"Verdana","sans-serif""
lang="EN-US">www.kbace.com</span></a></span><span
style="font-size:8.0pt;
font-family:"Calibri","sans-serif";
color:#1F497D" lang="EN-IN"></span></p>
</div>
<p style="font-family:Calibri; font-size:8pt; color:#333333;
padding-top:4px; border-top-style:solid; border-top-width:1px;
border-top-color:#666666">
Privileged/Confidential Information may be contained in this
message. If you are not the addressee indicated in this message
(or responsible for delivery of the message to such person), you
may not copy or deliver this message to anyone. In such case,
you should destroy this message, and notify the sender
immediately. If you or your employer does not consent to e-mail
messages of this kind, please advise the sender immediately.
Opinions, conclusions and other information expressed in this
message are not given or endorsed by KBACE unless otherwise
indicated by an authorized representative independent of this
message.</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Ocfs2-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a>
<a class="moz-txt-link-freetext" href="https://oss.oracle.com/mailman/listinfo/ocfs2-users">https://oss.oracle.com/mailman/listinfo/ocfs2-users</a></pre>
</blockquote>
<br>
</body>
</html>