<table cellspacing='0' cellpadding='0' border='0' background='none' style='font-family:arial;font-size:10pt;color:rgb(51, 51, 51);background-color:rgb(255, 255, 255);width:100%;'><tr><td valign='top' style='font: inherit;'>Martin,<br><br> Sunil and Mark are Oracle employees and involved on the development of OCFS2, I am just a user, :-).<br><br>Regards,<br>Luis<br><br>--- On <b>Fri, 6/6/08, Schmitter, Martin <i><Martin.Schmitter@opitz-consulting.de></i></b> wrote:<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px;">From: Schmitter, Martin <Martin.Schmitter@opitz-consulting.de><br>Subject: AW: [Ocfs2-users] CRS/CSS and OCFS2<br>To: "lfreitas34@yahoo.com" <lfreitas34@yahoo.com>, "alexandra.strauss@bayerbbs.com" <alexandra.strauss@bayerbbs.com><br>Cc: "ocfs2-users@oss.oracle.com" <ocfs2-users@oss.oracle.com><br>Date: Friday, June 6, 2008, 4:31
AM<br><br><div id="yiv2147115153">
<style>#yiv2147115153 P {
MARGIN-TOP:0px;MARGIN-BOTTOM:0px;}
</style>
<div dir="ltr"><font color="#000000" face="Tahoma" size="2">
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style="color: black; font-family: Tahoma;">Hi Luis,</span><span style="font-size: 12pt; font-family: 'Times New Roman';">
</span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style="font-size: 12pt; font-family: 'Times New Roman';"> </span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style="font-family: Tahoma;">now I am a bit confused, because I asked a few months ago this question. How must be the timing setting from OCFS2 and CRS?</span><span style="font-size: 12pt; font-family: 'Times New Roman';"></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style="font-size: 12pt; font-family: 'Times New Roman';"> </span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style="font-family: Tahoma;">Sunil and Mark stated that OCFS2 must be the leading system!</span><span style="font-size: 12pt; font-family: 'Times New Roman';"></span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style="font-size: 12pt; font-family: 'Times New Roman';"> </span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><span style="font-size: 12pt; font-family: 'Times New Roman';"> </span></p>
<p class="MsoNormal" style="margin: 0cm 0cm 0pt;"><font face="Arial">If I get it right, the SAN fail over comes first, then OCFS2 und least but not last CRS.</font></p>
</font><font face="tahoma" size="2"></font></div>
<div dir="ltr"><font face="tahoma" size="2"></font> </div>
<div dir="ltr"><font face="tahoma" size="2">BR</font></div>
<div dir="ltr"><font face="tahoma" size="2"></font><font face="tahoma" size="2"></font> </div>
<div>
<div><font face="Tahoma" size="2">
<div><font face="Arial" size="2">Martin Schmitter</font></div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">-- </font></div>
<div><font size="2"><font size="2"><span style="font-size: 10pt;"><font color="black" face="Arial" size="2"><span style="font-size: 10pt; color: black;"></span></font></span></font></font> </div>
<div><font size="2"><font size="2"><span style="font-size: 10pt;"><font color="black" face="arial" size="2"><span style="font-size: 10pt; color: black;"></span></font> </span></font></font></div>
<font size="2"><font size="2"></font><font size="2"><span style="font-size: 10pt;"></span></font></font><font size="2"></font>
<div><font face="Arial" size="2">OPITZ CONSULTING Gummersbach GmbH</font></div>
<div><font face="Arial" size="2">Martin Schmitter - Fachinformatiker</font></div>
<div><font face="Arial" size="2">Kirchstr. 6 - 51647 Gummersbach</font></div>
<div><font face="Arial" size="2">Telefon: +49 2261 6001-0</font></div>
<div><font face="Arial" size="2">Mobil: +49 173 2808193</font></div>
<div><font face="Arial" size="2"><a rel="nofollow" target="_blank" href="http://www.opitz-consulting.de">http://www.opitz-consulting.de</a></font></div>
<div><font face="C" size="2">
<p class="MsoNormal"><font color="black" size="2"><span style="font-size: 10pt; color: black; font-family: Arial;">Geschäftsführer: Bernhard Opitz, Dr. Jürgen Abel, Ulrich Kramer<br>
HRB-Nr. 39163 Amtsgericht Köln</span></font></p>
<p class="MsoNormal"><font color="black" face="arial" size="2"><span style="font-size: 10pt; color: black; font-family: Arial;"></span></font> </p>
</font></div>
</font></div>
</div>
<div id="divRpF50619" style="direction: ltr;">
<hr tabindex="-1">
<font face="Tahoma" size="2"><b>Von:</b> ocfs2-users-bounces@oss.oracle.com [ocfs2-users-bounces@oss.oracle.com] im Auftrag von Luis Freitas [lfreitas34@yahoo.com]<br>
<b>Gesendet:</b> Donnerstag, 5. Juni 2008 18:32<br>
<b>An:</b> alexandra.strauss@bayerbbs.com<br>
<b>Cc:</b> ocfs2-users@oss.oracle.com<br>
<b>Betreff:</b> Re: [Ocfs2-users] CRS/CSS and OCFS2<br>
</font><br>
</div>
<div></div>
<div>
<table style="font-size: 10pt; width: 100%; color: rgb(51, 51, 51); font-family: arial; background-color: rgb(255, 255, 255);" border="0" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<td valign="top">Alexandra,<br>
<br>
I usually make sure that one of the timeouts is large enought so that the other node death is detected before the other node "self-fence".<br>
<br>
To solve the problem you could configure the OCFS timeouts to be larger than the CRS timeouts, so that the CRS fences the node and OCFS detects the other node as dead before it takes any action.<br>
<br>
Maybe Sunil has a better solution that I am not aware of.<br>
<br>
This is particular of OCFS2 and CRS, which is kind of funny since both are developed by Oracle. With vendor clusterware (Sun cluster, Veritas, etc) CRS is integrated with the vendor clusterware stack so that this kind of situation does not occur.<br>
<br>
Btw, CRS is kind of picky about its interfaces, if it detects a link down on the interface, it will shutdown the services on the node. This is why I asked about the crossover cable, when using a crossover cable and one node goes down, the inteface goes down
on the other node and things does not work as expected. <br>
<br>
Regards,<br>
Luis<br>
<br>
--- On <b>Thu, 6/5/08, alexandra.strauss@bayerbbs.com <i><alexandra.strauss@bayerbbs.com></i></b> wrote:<br>
<blockquote style="border-left: 2px solid rgb(16, 16, 255); padding-left: 5px; margin-left: 5px;">
From: alexandra.strauss@bayerbbs.com <alexandra.strauss@bayerbbs.com><br>
Subject: [Ocfs2-users] CRS/CSS and OCFS2<br>
To: ocfs2-users@oss.oracle.com<br>
Date: Thursday, June 5, 2008, 12:38 PM<br>
<br>
<div id="yiv457573958"><br>
<font face="sans-serif" size="2">Hi Sunil,</font> <br>
<br>
<font face="sans-serif" size="2">sorry for the delay but I was ill the last 10 days.</font>
<br>
<br>
<font face="sans-serif" size="2">a. We do not use a crossover cable between the two nodes. The two systems are seated in two SANs in different building with redundant switches and HBA's inbetween.</font>
<br>
<br>
<font face="sans-serif" size="2">b.ocfs2-node numbers: [oracle@byaz05 etc]$ cat /etc/ocfs2/cluster.conf</font>
<br>
<font face="sans-serif" size="2">node:</font> <br>
<font face="sans-serif" size="2"> ip_port = 7777</font> <br>
<font face="sans-serif" size="2"> ip_address = 10.190.59.5</font> <br>
<font face="sans-serif" size="2"> number = 0</font> <br>
<font face="sans-serif" size="2"> name = byaz05.bayer-ag.com</font> <br>
<font face="sans-serif" size="2"> cluster = ocfs2</font> <br>
<br>
<font face="sans-serif" size="2">node:</font> <br>
<font face="sans-serif" size="2"> ip_port = 7777</font> <br>
<font face="sans-serif" size="2"> ip_address = 10.190.59.6</font> <br>
<font face="sans-serif" size="2"> number = 1</font> <br>
<font face="sans-serif" size="2"> name = byaz10.bayer-ag.com</font> <br>
<font face="sans-serif" size="2"> cluster = ocfs2</font> <br>
<br>
<font face="sans-serif" size="2">cluster:</font> <br>
<font face="sans-serif" size="2"> node_count = 2</font> <br>
<font face="sans-serif" size="2"> name = ocfs2</font> <br>
<br>
<font face="sans-serif" size="2">Clusterconfiguration css/crs:</font> <br>
<font face="sans-serif" size="2">/u01/app/oracle/product/crs/log/byaz05/crsd</font>
<br>
<font color="red" face="Arial" size="2">2008-04-25 09:29:01.855: [ OCRMAS][1210108256]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2</font>
<br>
<font color="red" face="Arial" size="2">2008-04-25 09:29:01.862: [ OCRRAW][1210108256]proprioo: for disk 0 (/dev/raw/raw101), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)</font> <br>
<font color="red" face="Arial" size="2">2008-04-25 09:29:01.862: [ OCRRAW][1210108256]proprioo: for disk 1 (/dev/raw/raw201), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)</font> <br>
<br>
<font face="sans-serif" size="2">/u01/app/oracle/product/crs/log/byaz10/crsd</font>
<br>
<font color="red" face="Arial" size="2">2008-04-25 10:21:28.781: [ OCRMAS][1210108256]th_master:13: I AM THE NEW OCR MASTER at incar 4. Node Number 1</font>
<br>
<font color="red" face="Arial" size="2">2008-04-25 10:21:28.781: [ OCRMSG][1505941856]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.</font>
<br>
<font color="red" face="Arial" size="2">2008-04-25 10:21:29.324: [ OCRRAW][1210108256]proprioo: for disk 0 (/dev/raw/raw101), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)</font> <br>
<font color="red" face="Arial" size="2">2008-04-25 10:21:29.324: [ OCRRAW][1210108256]proprioo: for disk 1 (/dev/raw/raw201), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)</font> <br>
<font color="red" face="Arial" size="2">2008-04-25 10:21:29.351: [ OCRMAS][1210108256]th_master: Deleted ver keys from cache (master)</font>
<br>
<br>
<font face="sans-serif" size="2">So the two nodes have the following nodenumbers:</font>
<br>
<br>
<img src="cid:1.615202034@web51410.mail.re2.yahoo.com"> <br>
<br>
<font face="sans-serif" size="2">Fencing the node with the higher node number ocfs2 would have fenced byaz10 and crs/css would have fenced byaz05. This is exactly the behaviour we watched. But how to solve this? Oracle says it's certified to use ocfs2 with
RAC. Then the used software combination is nearly the same as we use it. How can the combination of the two systems (ocfs2/css) fencing different nodes avoided then?</font>
<br>
<br>
<br>
<font face="sans-serif" size="2">Greets,</font> <br>
<font face="sans-serif" size="2">Alex</font> <br>
<br>
<br>
<tt><font size="3">>In such a situation, ocfs2 fences the higher node number. afaik,<br>
>css does the same. What are the css node numbers for the two nodes?<br>
<br>
</font></tt><a rel="nofollow" target="_blank" href="http://oss.oracle.com/mailman/listinfo/ocfs2-users"><tt><font color="blue" size="3"><u>>alexandra.strauss at bayerbbs.com</u></font></tt></a><tt><font size="3"> wrote:<br>
>><i><br>
</i>>><i> Hello,<br>
</i>>><i><br>
</i>>><i> I refer to you hoping you may help me with my problem... We have got <br>
</i>>><i> an issur here and opened a SR at Metalink but until now, we got no <br>
</i>>><i> useful information in solving our problem. SR-Number is 6855815.994...<br>
</i>>><i><br>
</i>>><i> We wanted to protect 9i Single-Instance Databases with 10g Clusterware <br>
</i>>><i> following the third-party-tool approach. There are no RAC-databases <br>
</i>>><i> involved. But we want to achieve high availability as the databases <br>
</i>>><i> are business critical systems. We want to make the systems able to<br>
</i>>><i> relocate to another machine in case of failure to keep downtimes <br>
</i>>><i> low... To achieve this we want to use OCFS2 for the filesystem. <br>
</i>>><i> Relocate is done by script with help of CRS.<br>
</i>>><i><br>
</i>>><i> So we took two systems (byaz05 and byaz10) and installed the following <br>
</i>>><i> software: 10g CRS (10.2.0.3) and Oracle Software 9.2.0.8 and OCFS2 1.2.8<br>
</i>>><i><br>
</i>>><i> We found the following Metalinknotes and adjusted the heartbeat and <br>
</i>>><i> timeouts for OCFS2: Metalink Note 395878.1: Heartbeat/Voting/Quorum <br>
</i>>><i> Related Timeout Configuration for Linux, OCFS2, RAC Stack to avoid <br>
</i>>><i> unnessary node fencing, panic and reboot<br>
</i>>><i> Metalink Note 391771.1: OCFS2 - FREQUENTLY ASKED QUESTIONS (hier <br>
</i>>><i> insbesondere der Abschnitt zu Fencing und Quorum)<br>
</i>>><i> Metalink Note 434255.1: Common reasons for OCFS2 Kernel Panic or <br>
</i>>><i> Reboot Issues<br>
</i>>><i> Metalink Note 457423.1: OCFS2 Fencing, Network, and Disk Heartbeat <br>
</i>>><i> Timeout Configuration<br>
</i>>><i><br>
</i>>><i> We did no changes to the CRS/CSS default settings until now.<br>
</i>>><i><br>
</i>>><i> During HA-testing we watched unexpected behaviour of the system. We <br>
</i>>><i> deactivated the bond for private interconnect and expected only one <br>
</i>>><i> node to go down. But we faced both nodes going down. As it seems to me <br>
</i>>><i> one node was rebooted from OCFS2 and the other one from CRS/CSS.<br>
</i>>><i><br>
</i>>><i> Timestamp<br>
</i>>><i> --------------------------------------------------------------------------------------------------------------
<br>
</i>>><i><br>
</i>>><i> 10:21:06 bond1 disabled (eth1)<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:06 byaz05 kernel: bonding: bond1: link status definitely <br>
</i>>><i> down for interface eth1, disabling it<br>
</i>>><i> Apr 25 10:21:06 byaz05 kernel: bonding: bond1: making interface eth5 <br>
</i>>><i> the new active one.<br>
</i>>><i><br>
</i>>><i> 10:21:09 bond1 disabled (eth5)<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:09 byaz05 kernel: bonding: bond1: link status definitely <br>
</i>>><i> down for interface eth5, disabling it<br>
</i>>><i> Apr 25 10:21:09 byaz05 kernel: bonding: bond1: now running without any <br>
</i>>><i> active interface !<br>
</i>>><i><br>
</i>>><i> 10:21:23 o2net – no longer connected<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:23 byaz05 kernel: o2net: no longer connected to node <br>
</i>>><i> byaz10.bayer-ag.com (num 1) at 10.190.59.6:7777<br>
</i>>><i> */var/log/messages byaz10*<br>
</i>>><i> Apr 25 10:21:23 byaz10 kernel: o2net: no longer connected to node <br>
</i>>><i> byaz05.bayer-ag.com (num 0) at 10.190.59.5:7777<br>
</i>>><i><br>
</i>>><i> 10:21:27 CSSD failure 134<br>
</i>>><i> 10:21:29 Reboot initiated by CRS<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:27 byaz05 logger: Oracle clsomon failed with fatal status <br>
</i>>><i> 12.<br>
</i>>><i> Apr 25 10:21:27 byaz05 logger: Oracle CSSD failure 134.<br>
</i>>><i> Apr 25 10:21:27 byaz05 su(pam_unix)[25839]: session closed for user <br>
</i>>><i> oracle<br>
</i>>><i> Apr 25 10:21:27 byaz05 logger: Oracle CRS failure. Rebooting for <br>
</i>>><i> cluster integrity.<br>
</i>>><i> Apr 25 10:21:27 byaz05 kernel: md: stopping all md devices.<br>
</i>>><i> Apr 25 10:21:27 byaz05 kernel: md: md0 switched to read-only mode.<br>
</i>>><i> Apr 25 10:21:29 byaz05 logger: Oracle CRS failure. Rebooting for <br>
</i>>><i> cluster integrity.<br>
</i>>><i> Apr 25 10:21:29 byaz05 kernel: e1000: eth2: e1000_watchdog_task: NIC <br>
</i>>><i> Link is Up 1000 Mbps Full Duplex<br>
</i>>><i> Apr 25 10:21:29 byaz05 logger: Oracle init script ceding reboot to <br>
</i>>><i> sibling 27383.<br>
</i>><i>><br>
</i>><i>> 10:21:58 Reboot initiated by OCFS2(?)<br>
</i>><i>> */var/log/messages byaz10*<br>
</i>><i>> Apr 25 10:21:58 byaz10 su(pam_unix)[4595]: session opened for user <br>
</i>><i>> oracle by (uid=0)<br>
</i>><i>> Apr 25 10:21:58 byaz10 su(pam_unix)[4595]: session closed for user oracle<br>
</i>><i>> Apr 25 10:25:58 byaz10 syslogd 1.4.1: restart.<br>
</i>><i>> Apr 25 10:25:58 byaz10 syslog: syslogd startup succeeded<br>
</i>><i>> Apr 25 10:25:58 byaz10 kernel: klogd 1.4.1, log source = /proc/kmsg <br>
</i>><i>> started.<br>
</i>><i>> Apr 25 10:25:58 byaz10 kernel: Bootdata ok (command line is ro <br>
</i>><i>> root=/dev/vgroot/_)<br>
</i>><i>><br>
</i>><i>><br>
</i>><i>> We supposed all the time this is a timing problem. But we don't know <br>
</i>>><i> which settings raise the problem and which steps to do to to correct <br>
</i>><i>> them. Otherwise we'll have to work over the complete concept for the <br>
</i>>><i> business critical systems.<br>
</i>>><i> Can anyone help me?<br>
</i>>><i><br>
</i><br>
>><i> Regards,<br>
</i>>><i> Alexandra<br>
</i></font></tt></div>
<pre>_______________________________________________<br>Ocfs2-users mailing list<br>Ocfs2-users@oss.oracle.com<br>http://oss.oracle.com/mailman/listinfo/ocfs2-users</pre>
</blockquote>
</td>
</tr>
</tbody>
</table>
<br>
</div>
</div></blockquote></td></tr></table><br>