<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
What do you mean by loss of file system? Are you referring to the time
when file system is hung? If so, that is unavoidable. They have to talk
to each other whenever a file system metadata is changing. When network
went down, they were waiting for it to come back. Since it was taking
longer, quorum logic kicked in and took down one of the node(node with
higher node number). If network was backup before the quorum kicked in,
they would have been fine.<br>
<br>
May I ask what your expectations are when the network goes down.<br>
<br>
thanks,<br>
--Srini<br>
<br>
Thompson, Mark wrote:
<blockquote
cite="mid5514E26926513E49B8C19E822B2904A5149C76@EXCH-NOT-MSG01.uk.experian.local"
type="cite">
<meta http-equiv="Content-Type" content="text/html; ">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>2 node OCFS2 clusters</title>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="Section1">
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">I
have done some more tests today, and I observed the following:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Test
1:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
0 - ifdown eth2<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
0 - OCFS2 filesystem stalls on both nodes<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
1 - Decides to reboot<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
0 - Resumes OCFS2 service (while still off the network) OCFS2
filesystem back online<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
1 - Cannot re-join cluster as node 0 is off the network and
has the fs lock (Transport endpoint error)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
0 - ifup eth2<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
1 - Re-joins the clusters and re-mounts OCFS2 filesystem.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Test
2:<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
1 - ifdown eth2<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
0 - OCFS2 filesystem stalls on both nodes<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
1 - Decides to reboot<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
0 - Resumes OCFS2 service, OCFS2 filesystem back online<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">node
1 – Boots up, re-joins cluster and re-mounts OCFS2
filesystem.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Is
this the expected behaviour? And if it is, is there anything
we can do avoid the loss of the OCFS2 filesystems?<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Here’s
the messages file outputs.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Test
1 - Node 0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:00:26 my_node0 kernel: ocfs2: Unmounting device
(253,9) on (node 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:21 my_node0 modprobe: FATAL: Module ocfs2_stackglue
not found.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:21 my_node0 kernel: OCFS2 Node Manager 1.4.4 Tue
Sep 8 11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:21 my_node0 kernel: OCFS2 DLM 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:21 my_node0 kernel: OCFS2 DLMFS 1.4.4 Tue Sep
8 11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:21 my_node0 kernel: OCFS2 User DLM kernel interface
loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:46 my_node0 kernel: OCFS2 1.4.4 Tue Sep 8
11:56:43 PDT 2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:46 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:46 my_node0 kernel: ocfs2: Mounting device (253,9)
on (node 0, slot 0) with ordered data mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:59 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:59 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:07:51 my_node0 kernel:
(15,1):dlm_do_master_request:1334 ERROR: link to 1 went down!<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:07:51 my_node0 kernel:
(15,1):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:917">resource:917</a> ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:34 my_node0 kernel:
(22108,1):ocfs2_dlm_eviction_cb:98 device (253,9): dlm has evicted node
1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:34 my_node0 kernel:
(29443,1):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:844">resource:844</a>
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one
node (1) to recover before lock mastery can begin<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:35 my_node0 kernel:
(29443,1):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:898">resource:898</a>
21751145F96E45649324C9EEF5485248:M000000000000000000001f96e7b609: at
least one
node (1) to recover before lock mastery can begin<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:36 my_node0 kernel:
(15,1):dlm_restart_lock_mastery:1223 ERROR: node down! 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:36 my_node0 kernel:
(15,1):dlm_wait_for_lock_mastery:1040 ERROR: status = -11<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:36 my_node0 kernel:
(22167,0):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:844">resource:844</a>
21751145F96E45649324C9EEF5485248:$RECOVERY:
at least one node (1) to recover before lock mastery can begin<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:36 my_node0 kernel:
(22167,0):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:878">resource:878</a> 21751145F96E45649324C9EEF5485248:
recovery
map is not empty, but must master $RECOVERY lock now<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:36 my_node0 kernel: (22167,0):dlm_do_recovery:524
(22167) Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:09:46 my_node0 kernel:
(29443,1):ocfs2_replay_journal:1183 Recovering node 1 from slot 1 on
device
(253,9)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:27 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:27 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Test
1 – Node 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:00:26 my_node1 kernel: ocfs2_dlm: Node 0 leaves domain
21751145F96E45649324C9EEF5485248<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:00:26 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:00:46 my_node1 kernel: ocfs2: Unmounting device
(253,9) on (node 1)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:30 my_node1 modprobe: FATAL: Module ocfs2_stackglue
not found.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:30 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue
Sep 8 11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:30 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:30 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep
8 11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:30 my_node1 kernel: OCFS2 User DLM kernel interface
loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:59 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8
11:56:43 PDT 2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:59 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:02:59 my_node1 kernel: ocfs2: Mounting device (253,9)
on (node 1, slot 1) with ordered data mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:07:27 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -112<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:07:27 my_node1 kernel:
(7351,3):dlm_wait_for_node_death:370 21751145F96E45649324C9EEF5485248:
waiting
5000ms for notification of death of node 0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:07:57 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395
ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:07:57 my_node1 kernel:
(7351,3):dlm_wait_for_node_death:370 21751145F96E45649324C9EEF5485248:
waiting
5000ms for notification of death of node 0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:08:27 my_node1 kernel: (15,1):dlm_do_master_request:1334
ERROR: link to 0 went down!<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:08:27 my_node1 kernel:
(7351,3):dlm_send_remote_convert_request:395 ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:08:27 my_node1 kernel:
(7351,3):dlm_wait_for_node_death:370 21751145F96E45649324C9EEF5485248:
waiting
5000ms for notification of death of node 0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:08:27 my_node1 kernel:
(15,1):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:917">resource:917</a> ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:11:31 my_node1 modprobe: FATAL: Module ocfs2_stackglue
not found.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:11:32 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue
Sep 8 11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:11:32 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:11:32 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep
8 11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:11:32 my_node1 kernel: OCFS2 User DLM kernel interface
loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:11:40 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8
11:56:43 PDT 2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:06 my_node1 kernel: (6282,0):dlm_request_join:1036
ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:06 my_node1 kernel: (6282,0):dlm_try_to_join_domain:1210
ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:06 my_node1 kernel: (6282,0):dlm_join_domain:1488
ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:06 my_node1 kernel:
(6282,0):dlm_register_domain:1754 ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:06 my_node1 kernel: (6282,0):ocfs2_dlm_init:2723
ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:06 my_node1 kernel:
(6282,0):ocfs2_mount_volume:1437 ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:06 my_node1 kernel: ocfs2: Unmounting device
(253,9) on (node 1)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:27 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:12:27 my_node1 kernel: ocfs2: Mounting device (253,9)
on (node 1, slot 1) with ordered data mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Test
2 – Node 0<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:16:37 my_node0 kernel:
(22166,3):dlm_send_proxy_ast_msg:458 ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:16:37 my_node0 kernel: (22166,3):dlm_flush_asts:600
ERROR: status = -107<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:17:35 my_node0 kernel:
(22108,1):ocfs2_dlm_eviction_cb:98 device (253,9): dlm has evicted node
1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:17:35 my_node0 kernel:
(6515,1):ocfs2_replay_journal:1183 Recovering node 1 from slot 1 on
device
(253,9)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:17:36 my_node0 kernel:
(22167,0):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:844">resource:844</a>
21751145F96E45649324C9EEF5485248:$RECOVERY:
at least one node (1) to recover before lock mastery can begin<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:17:36 my_node0 kernel:
(22167,0):dlm_get_lock_<a class="moz-txt-link-freetext" href="resource:878">resource:878</a> 21751145F96E45649324C9EEF5485248:
recovery
map is not empty, but must master $RECOVERY lock now<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:17:36 my_node0 kernel: (22167,0):dlm_do_recovery:524
(22167) Node 0 is the Recovery Master for the Dead Node 1 for Domain
21751145F96E45649324C9EEF5485248<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:31 my_node0 kernel: ocfs2_dlm: Node 1 joins domain
21751145F96E45649324C9EEF5485248<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:31 my_node0 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Test2
– Node 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:22 my_node1 modprobe: FATAL: Module ocfs2_stackglue
not found.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:23 my_node1 kernel: OCFS2 Node Manager 1.4.4 Tue
Sep 8 11:56:46 PDT 2009 (build 18a3a72794aaca6c0334f456bca873cd)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:23 my_node1 kernel: OCFS2 DLM 1.4.4 Tue Sep 8
11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:23 my_node1 kernel: OCFS2 DLMFS 1.4.4 Tue Sep
8 11:56:46 PDT 2009 (build e6e41b84c785deeea891e5873dbf19ab)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:23 my_node1 kernel: OCFS2 User DLM kernel interface
loaded<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:31 my_node1 kernel: OCFS2 1.4.4 Tue Sep 8
11:56:43 PDT 2009 (build 3a5bffa75b910d5bcdd5c607c4394b1e)<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:31 my_node1 kernel: ocfs2_dlm: Nodes in domain
("21751145F96E45649324C9EEF5485248"): 0 1<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Nov
17 11:19:31 my_node1 kernel: ocfs2: Mounting device (253,8)
on (node 1, slot 1) with ordered data mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Mark<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<div>
<div
style="border-style: solid none none; border-color: rgb(181, 196, 223) -moz-use-text-color -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;"
lang="EN-US">From:</span></b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;"
lang="EN-US">
Srinivas Eeda [<a class="moz-txt-link-freetext" href="mailto:srinivas.eeda@oracle.com">mailto:srinivas.eeda@oracle.com</a>] <br>
<b>Sent:</b> 16 November 2009 16:05<br>
<b>To:</b> Thompson, Mark<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br>
<b>Subject:</b> Re: [Ocfs2-users] 2 node OCFS2 clusters<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thompson, Mark wrote: <o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Hi
Srini,</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Thanks
for the response.</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">So
are the following statements correct:</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">If
I stop the networking on node 1, node 0 will continue to
allow OCFS2 filesystems to work and not reboot itself. </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">If
I stop the networking on node 0, node 1 (now being the lowest
node?) will continue to allow OCFS2 filesystems to work and not reboot
itself.</span><o:p></o:p></p>
<p class="MsoNormal">In both the cases node 0 will survive, because
that's the
node that has lowest node number (defined in cluster.conf). This
applies to the
scenario where interconnect went down but nodes are healthy and are
heartbeating
to the disk.<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">I
guess I just need to know if it’s possible to have a 2
node OCFS2 cluster that will cope with either one of the nodes dying,
and have
the remaining node still provide service.</span><o:p></o:p></p>
<p class="MsoNormal">If node 0 itself panics, reboots then node 1
will survive.<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);">Mark
</span><o:p></o:p></p>
<p class="MsoNormal"><span
style="font-size: 11pt; font-family: "Calibri","sans-serif"; color: rgb(31, 73, 125);"> </span><o:p></o:p></p>
<div>
<div
style="border-style: solid none none; border-color: -moz-use-text-color; border-width: 1pt medium medium; padding: 3pt 0cm 0cm;">
<p class="MsoNormal"><b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;"
lang="EN-US">From:</span></b><span
style="font-size: 10pt; font-family: "Tahoma","sans-serif"; color: windowtext;"
lang="EN-US">
Srinivas Eeda [<a href="mailto:srinivas.eeda@oracle.com">mailto:srinivas.eeda@oracle.com</a>]
<br>
<b>Sent:</b> 16 November 2009 14:57<br>
<b>To:</b> Thompson, Mark<br>
<b>Cc:</b> <a href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br>
<b>Subject:</b> Re: [Ocfs2-users] 2 node OCFS2 clusters</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">In a cluster with more than 2 nodes, if a
network on one
node goes down, that node will evict itself but other nodes will
survive. But
in a two node cluster, the node with lowest node number will survive no
mater
on which node network went down.<br>
<br>
thanks,<br>
--Srini<br>
<br>
Thompson, Mark wrote: <o:p></o:p></p>
<p><span style="font-family: "Calibri","sans-serif";">Hi,</span><o:p></o:p></p>
<p><span style="font-family: "Calibri","sans-serif";">This is my
first post here
so</span> <span style="font-family: "Calibri","sans-serif";">please be
gentle
with me.</span><o:p></o:p></p>
<p><span style="font-family: "Calibri","sans-serif";">My question is,
can you have
a 2 node OCFS2 cluster, disconnect one node from the network, and have
the
remaining node continue to function normally? Currently we have a 2
node
cluster and if we stop the NIC that has the OCFS2 o2cb net connection
running
on it, the</span> <span style="font-family: "Calibri","sans-serif";">other
node
will reboot itself.</span> <span
style="font-family: "Calibri","sans-serif";">I
have researched having a 2 node OCFS2 cluster but so far I have been
unable to
find a clear solution. I have looked at the FAQ regarding quorum,</span>
<span style="font-family: "Calibri","sans-serif";">and my OCFS2 init
scripts are
enabled etc.</span><o:p></o:p></p>
<p><span style="font-family: "Calibri","sans-serif";">Is this
possible, or should
we look at alternative solutions?</span><o:p></o:p></p>
<p><span style="font-family: "Calibri","sans-serif";">Regards,</span><o:p></o:p></p>
<p><span style="font-family: "Calibri","sans-serif";">Mark</span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">This
e-mail
has come from Experian, the only business to have been twice named the
UK's
'Business of the Year’ </span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";"> </span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">===================================================================================</span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">Information
in this e-mail and any attachments is confidential, and may not be
copied or
used by anyone other than the addressee, nor disclosed to any third
party
without our permission. There is no intention to create any legally
binding
contract or other binding commitment through the use of this electronic
communication unless it is issued in accordance with the Experian
Limited
standard terms and conditions of purchase or other express written
agreement
between Experian Limited and the recipient. </span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">Although
Experian
has taken reasonable steps to ensure that this communication and any
attachments are free from computer virus, you are advised to take your
own
steps to ensure that they are actually virus free. </span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">Companies
Act
information:</span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">Registered
name: Experian Limited </span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">Registered
office: Landmark House, Experian Way, NG2 Business Park, Nottingham,
NG80 1ZZ,
United Kingdom</span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">Place
of
registration: England and Wales </span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";">Registered
number: 653331</span><o:p></o:p></p>
<p><span style="font-size: 8pt; font-family: "Arial","sans-serif";"> </span><o:p></o:p></p>
<pre> <o:p></o:p></pre>
<pre style="text-align: center;"><o:p> </o:p></pre>
<pre style="text-align: center;">
<hr align="center" size="4" width="90%">
</pre>
<pre style="text-align: center;"><o:p> </o:p></pre>
<pre style="text-align: center;"><o:p> </o:p></pre>
<pre style="text-align: center;"> <o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Ocfs2-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a><o:p></o:p></pre>
<pre><a href="http://oss.oracle.com/mailman/listinfo/ocfs2-users">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><o:p></o:p></pre>
</div>
</blockquote>
</body>
</html>