<br><font size=2 face="sans-serif">Hi Sunil,</font>
<br>
<br><font size=2 face="sans-serif">sorry for the delay but I was ill the
last 10 days.</font>
<br>
<br><font size=2 face="sans-serif">a. We do not use a crossover cable between
the two nodes. The two systems are seated in two SANs in different building
with redundant switches and HBA's inbetween.</font>
<br>
<br><font size=2 face="sans-serif">b.ocfs2-node numbers: [oracle@byaz05
etc]$ cat /etc/ocfs2/cluster.conf</font>
<br><font size=2 face="sans-serif">node:</font>
<br><font size=2 face="sans-serif"> ip_port
= 7777</font>
<br><font size=2 face="sans-serif"> ip_address
= 10.190.59.5</font>
<br><font size=2 face="sans-serif"> number =
0</font>
<br><font size=2 face="sans-serif"> name = byaz05.bayer-ag.com</font>
<br><font size=2 face="sans-serif"> cluster
= ocfs2</font>
<br>
<br><font size=2 face="sans-serif">node:</font>
<br><font size=2 face="sans-serif"> ip_port
= 7777</font>
<br><font size=2 face="sans-serif"> ip_address
= 10.190.59.6</font>
<br><font size=2 face="sans-serif"> number =
1</font>
<br><font size=2 face="sans-serif"> name = byaz10.bayer-ag.com</font>
<br><font size=2 face="sans-serif"> cluster
= ocfs2</font>
<br>
<br><font size=2 face="sans-serif">cluster:</font>
<br><font size=2 face="sans-serif"> node_count
= 2</font>
<br><font size=2 face="sans-serif"> name = ocfs2</font>
<br>
<br><font size=2 face="sans-serif">Clusterconfiguration css/crs:</font>
<br><font size=2 face="sans-serif">/u01/app/oracle/product/crs/log/byaz05/crsd</font>
<br><font size=2 color=red face="Arial">2008-04-25 09:29:01.855: [ OCRMAS][1210108256]th_master:12:
I AM THE NEW OCR MASTER at incar 1. Node Number 2</font>
<br><font size=2 color=red face="Arial">2008-04-25 09:29:01.862: [ OCRRAW][1210108256]proprioo:
for disk 0 (/dev/raw/raw101), id match (1), my id set (1723799148,1710759834)
total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes
(1), total votes (2)</font>
<br><font size=2 color=red face="Arial">2008-04-25 09:29:01.862: [ OCRRAW][1210108256]proprioo:
for disk 1 (/dev/raw/raw201), id match (1), my id set (1723799148,1710759834)
total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes
(1), total votes (2)</font>
<br>
<br><font size=2 face="sans-serif">/u01/app/oracle/product/crs/log/byaz10/crsd</font>
<br><font size=2 color=red face="Arial">2008-04-25 10:21:28.781: [ OCRMAS][1210108256]th_master:13:
I AM THE NEW OCR MASTER at incar 4. Node Number 1</font>
<br><font size=2 color=red face="Arial">2008-04-25 10:21:28.781: [ OCRMSG][1505941856]prom_rpc:1:
NULL con. Probably got disconnected due to a remote server failure.</font>
<br><font size=2 color=red face="Arial">2008-04-25 10:21:29.324: [ OCRRAW][1210108256]proprioo:
for disk 0 (/dev/raw/raw101), id match (1), my id set (1723799148,1710759834)
total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes
(1), total votes (2)</font>
<br><font size=2 color=red face="Arial">2008-04-25 10:21:29.324: [ OCRRAW][1210108256]proprioo:
for disk 1 (/dev/raw/raw201), id match (1), my id set (1723799148,1710759834)
total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes
(1), total votes (2)</font>
<br><font size=2 color=red face="Arial">2008-04-25 10:21:29.351: [ OCRMAS][1210108256]th_master:
Deleted ver keys from cache (master)</font>
<br>
<br><font size=2 face="sans-serif">So the two nodes have the following
nodenumbers:</font>
<br>
<br><img src=cid:_1_068F9CEC068F9A740055F337C125745F>
<br>
<br><font size=2 face="sans-serif">Fencing the node with the higher node
number ocfs2 would have fenced byaz10 and crs/css would have fenced byaz05.
This is exactly the behaviour we watched. But how to solve this? Oracle
says it's certified to use ocfs2 with RAC. Then the used software combination
is nearly the same as we use it. How can the combination of the two systems
(ocfs2/css) fencing different nodes avoided then?</font>
<br>
<br>
<br><font size=2 face="sans-serif">Greets,</font>
<br><font size=2 face="sans-serif">Alex</font>
<br>
<br>
<br><tt><font size=3>>In such a situation, ocfs2 fences the higher node
number. afaik,<br>
>css does the same. What are the css node numbers for the two nodes?<br>
<br>
</font></tt><a href="http://oss.oracle.com/mailman/listinfo/ocfs2-users"><tt><font size=3 color=blue><u>>alexandra.strauss
at bayerbbs.com</u></font></tt></a><tt><font size=3> wrote:<br>
>><i><br>
</i>>><i> Hello,<br>
</i>>><i><br>
</i>>><i> I refer to you hoping you may help me with my problem...
We have got <br>
</i>>><i> an issur here and opened a SR at Metalink but until now,
we got no <br>
</i>>><i> useful information in solving our problem. SR-Number is
6855815.994...<br>
</i>>><i><br>
</i>>><i> We wanted to protect 9i Single-Instance Databases with
10g Clusterware <br>
</i>>><i> following the third-party-tool approach. There are no RAC-databases
<br>
</i>>><i> involved. But we want to achieve high availability as the
databases <br>
</i>>><i> are business critical systems. We want to make the systems
able to<br>
</i>>><i> relocate to another machine in case of failure to keep
downtimes <br>
</i>>><i> low... To achieve this we want to use OCFS2 for the filesystem.
<br>
</i>>><i> Relocate is done by script with help of CRS.<br>
</i>>><i><br>
</i>>><i> So we took two systems (byaz05 and byaz10) and installed
the following <br>
</i>>><i> software: 10g CRS (10.2.0.3) and Oracle Software 9.2.0.8
and OCFS2 1.2.8<br>
</i>>><i><br>
</i>>><i> We found the following Metalinknotes and adjusted the heartbeat
and <br>
</i>>><i> timeouts for OCFS2: Metalink Note 395878.1: Heartbeat/Voting/Quorum
<br>
</i>>><i> Related Timeout Configuration for Linux, OCFS2, RAC Stack
to avoid <br>
</i>>><i> unnessary node fencing, panic and reboot<br>
</i>>><i> Metalink Note 391771.1: OCFS2 - FREQUENTLY ASKED QUESTIONS
(hier <br>
</i>>><i> insbesondere der Abschnitt zu Fencing und Quorum)<br>
</i>>><i> Metalink Note 434255.1: Common reasons for OCFS2 Kernel
Panic or <br>
</i>>><i> Reboot Issues<br>
</i>>><i> Metalink Note 457423.1: OCFS2 Fencing, Network, and Disk
Heartbeat <br>
</i>>><i> Timeout Configuration<br>
</i>>><i><br>
</i>>><i> We did no changes to the CRS/CSS default settings until
now.<br>
</i>>><i><br>
</i>>><i> During HA-testing we watched unexpected behaviour of the
system. We <br>
</i>>><i> deactivated the bond for private interconnect and expected
only one <br>
</i>>><i> node to go down. But we faced both nodes going down. As
it seems to me <br>
</i>>><i> one node was rebooted from OCFS2 and the other one from
CRS/CSS.<br>
</i>>><i><br>
</i>>><i> Timestamp<br>
</i>>><i> --------------------------------------------------------------------------------------------------------------
<br>
</i>>><i><br>
</i>>><i> 10:21:06 bond1 disabled (eth1)<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:06 byaz05 kernel: bonding: bond1: link status
definitely <br>
</i>>><i> down for interface eth1, disabling it<br>
</i>>><i> Apr 25 10:21:06 byaz05 kernel: bonding: bond1: making interface
eth5 <br>
</i>>><i> the new active one.<br>
</i>>><i><br>
</i>>><i> 10:21:09 bond1 disabled (eth5)<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:09 byaz05 kernel: bonding: bond1: link status
definitely <br>
</i>>><i> down for interface eth5, disabling it<br>
</i>>><i> Apr 25 10:21:09 byaz05 kernel: bonding: bond1: now running
without any <br>
</i>>><i> active interface !<br>
</i>>><i><br>
</i>>><i> 10:21:23 o2net – no longer connected<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:23 byaz05 kernel: o2net: no longer connected
to node <br>
</i>>><i> byaz10.bayer-ag.com (num 1) at 10.190.59.6:7777<br>
</i>>><i> */var/log/messages byaz10*<br>
</i>>><i> Apr 25 10:21:23 byaz10 kernel: o2net: no longer connected
to node <br>
</i>>><i> byaz05.bayer-ag.com (num 0) at 10.190.59.5:7777<br>
</i>>><i><br>
</i>>><i> 10:21:27 CSSD failure 134<br>
</i>>><i> 10:21:29 Reboot initiated by CRS<br>
</i>>><i> */var/log/messages byaz05*<br>
</i>>><i> Apr 25 10:21:27 byaz05 logger: Oracle clsomon failed with
fatal status <br>
</i>>><i> 12.<br>
</i>>><i> Apr 25 10:21:27 byaz05 logger: Oracle CSSD failure 134.<br>
</i>>><i> Apr 25 10:21:27 byaz05 su(pam_unix)[25839]: session closed
for user <br>
</i>>><i> oracle<br>
</i>>><i> Apr 25 10:21:27 byaz05 logger: Oracle CRS failure. Rebooting
for <br>
</i>>><i> cluster integrity.<br>
</i>>><i> Apr 25 10:21:27 byaz05 kernel: md: stopping all md devices.<br>
</i>>><i> Apr 25 10:21:27 byaz05 kernel: md: md0 switched to read-only
mode.<br>
</i>>><i> Apr 25 10:21:29 byaz05 logger: Oracle CRS failure. Rebooting
for <br>
</i>>><i> cluster integrity.<br>
</i>>><i> Apr 25 10:21:29 byaz05 kernel: e1000: eth2: e1000_watchdog_task:
NIC <br>
</i>>><i> Link is Up 1000 Mbps Full Duplex<br>
</i>>><i> Apr 25 10:21:29 byaz05 logger: Oracle init script ceding
reboot to <br>
</i>>><i> sibling 27383.<br>
</i>><i>><br>
</i>><i>> 10:21:58 Reboot initiated by OCFS2(?)<br>
</i>><i>> */var/log/messages byaz10*<br>
</i>><i>> Apr 25 10:21:58 byaz10 su(pam_unix)[4595]: session opened
for user <br>
</i>><i>> oracle by (uid=0)<br>
</i>><i>> Apr 25 10:21:58 byaz10 su(pam_unix)[4595]: session closed
for user oracle<br>
</i>><i>> Apr 25 10:25:58 byaz10 syslogd 1.4.1: restart.<br>
</i>><i>> Apr 25 10:25:58 byaz10 syslog: syslogd startup succeeded<br>
</i>><i>> Apr 25 10:25:58 byaz10 kernel: klogd 1.4.1, log source
= /proc/kmsg <br>
</i>><i>> started.<br>
</i>><i>> Apr 25 10:25:58 byaz10 kernel: Bootdata ok (command line
is ro <br>
</i>><i>> root=/dev/vgroot/_)<br>
</i>><i>><br>
</i>><i>><br>
</i>><i>> We supposed all the time this is a timing problem. But
we don't know <br>
</i>>><i> which settings raise the problem and which steps to do
to to correct <br>
</i>><i>> them. Otherwise we'll have to work over the complete concept
for the <br>
</i>>><i> business critical systems.<br>
</i>>><i> Can anyone help me?<br>
</i>>><i><br>
</i><br>
>><i> Regards,<br>
</i>>><i> Alexandra<br>
</i></font></tt>