[Ocfs2-users] CRS/CSS and OCFS2
Luis Freitas
lfreitas34 at yahoo.com
Fri Jun 6 08:23:37 PDT 2008
Martin,
Sunil and Mark are Oracle employees and involved on the development of OCFS2, I am just a user, :-).
Regards,
Luis
--- On Fri, 6/6/08, Schmitter, Martin <Martin.Schmitter at opitz-consulting.de> wrote:
From: Schmitter, Martin <Martin.Schmitter at opitz-consulting.de>
Subject: AW: [Ocfs2-users] CRS/CSS and OCFS2
To: "lfreitas34 at yahoo.com" <lfreitas34 at yahoo.com>, "alexandra.strauss at bayerbbs.com" <alexandra.strauss at bayerbbs.com>
Cc: "ocfs2-users at oss.oracle.com" <ocfs2-users at oss.oracle.com>
Date: Friday, June 6, 2008, 4:31 AM
#yiv2147115153 P {
MARGIN-TOP:0px;MARGIN-BOTTOM:0px;}
Hi Luis,
now I am a bit confused, because I asked a few months ago this question. How must be the timing setting from OCFS2 and CRS?
Sunil and Mark stated that OCFS2 must be the leading system!
If I get it right, the SAN fail over comes first, then OCFS2 und least but not last CRS.
BR
Martin Schmitter
--
OPITZ CONSULTING Gummersbach GmbH
Martin Schmitter - Fachinformatiker
Kirchstr. 6 - 51647 Gummersbach
Telefon: +49 2261 6001-0
Mobil: +49 173 2808193
http://www.opitz-consulting.de
Geschäftsführer: Bernhard Opitz, Dr. Jürgen Abel, Ulrich Kramer
HRB-Nr. 39163 Amtsgericht Köln
Von: ocfs2-users-bounces at oss.oracle.com [ocfs2-users-bounces at oss.oracle.com] im Auftrag von Luis Freitas [lfreitas34 at yahoo.com]
Gesendet: Donnerstag, 5. Juni 2008 18:32
An: alexandra.strauss at bayerbbs.com
Cc: ocfs2-users at oss.oracle.com
Betreff: Re: [Ocfs2-users] CRS/CSS and OCFS2
Alexandra,
I usually make sure that one of the timeouts is large enought so that the other node death is detected before the other node "self-fence".
To solve the problem you could configure the OCFS timeouts to be larger than the CRS timeouts, so that the CRS fences the node and OCFS detects the other node as dead before it takes any action.
Maybe Sunil has a better solution that I am not aware of.
This is particular of OCFS2 and CRS, which is kind of funny since both are developed by Oracle. With vendor clusterware (Sun cluster, Veritas, etc) CRS is integrated with the vendor clusterware stack so that this kind of situation does not occur.
Btw, CRS is kind of picky about its interfaces, if it detects a link down on the interface, it will shutdown the services on the node. This is why I asked about the crossover cable, when using a crossover cable and one node goes down, the inteface goes down
on the other node and things does not work as expected.
Regards,
Luis
--- On Thu, 6/5/08, alexandra.strauss at bayerbbs.com <alexandra.strauss at bayerbbs.com> wrote:
From: alexandra.strauss at bayerbbs.com <alexandra.strauss at bayerbbs.com>
Subject: [Ocfs2-users] CRS/CSS and OCFS2
To: ocfs2-users at oss.oracle.com
Date: Thursday, June 5, 2008, 12:38 PM
Hi Sunil,
sorry for the delay but I was ill the last 10 days.
a. We do not use a crossover cable between the two nodes. The two systems are seated in two SANs in different building with redundant switches and HBA's inbetween.
b.ocfs2-node numbers: [oracle at byaz05 etc]$ cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 10.190.59.5
number = 0
name = byaz05.bayer-ag.com
cluster = ocfs2
node:
ip_port = 7777
ip_address = 10.190.59.6
number = 1
name = byaz10.bayer-ag.com
cluster = ocfs2
cluster:
node_count = 2
name = ocfs2
Clusterconfiguration css/crs:
/u01/app/oracle/product/crs/log/byaz05/crsd
2008-04-25 09:29:01.855: [ OCRMAS][1210108256]th_master:12: I AM THE NEW OCR MASTER at incar 1. Node Number 2
2008-04-25 09:29:01.862: [ OCRRAW][1210108256]proprioo: for disk 0 (/dev/raw/raw101), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)
2008-04-25 09:29:01.862: [ OCRRAW][1210108256]proprioo: for disk 1 (/dev/raw/raw201), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)
/u01/app/oracle/product/crs/log/byaz10/crsd
2008-04-25 10:21:28.781: [ OCRMAS][1210108256]th_master:13: I AM THE NEW OCR MASTER at incar 4. Node Number 1
2008-04-25 10:21:28.781: [ OCRMSG][1505941856]prom_rpc:1: NULL con. Probably got disconnected due to a remote server failure.
2008-04-25 10:21:29.324: [ OCRRAW][1210108256]proprioo: for disk 0 (/dev/raw/raw101), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)
2008-04-25 10:21:29.324: [ OCRRAW][1210108256]proprioo: for disk 1 (/dev/raw/raw201), id match (1), my id set (1723799148,1710759834) total id sets (1), 1st set (1723799148,1710759834), 2nd set (0,0) my votes (1), total
votes (2)
2008-04-25 10:21:29.351: [ OCRMAS][1210108256]th_master: Deleted ver keys from cache (master)
So the two nodes have the following nodenumbers:
Fencing the node with the higher node number ocfs2 would have fenced byaz10 and crs/css would have fenced byaz05. This is exactly the behaviour we watched. But how to solve this? Oracle says it's certified to use ocfs2 with
RAC. Then the used software combination is nearly the same as we use it. How can the combination of the two systems (ocfs2/css) fencing different nodes avoided then?
Greets,
Alex
>In such a situation, ocfs2 fences the higher node number. afaik,
>css does the same. What are the css node numbers for the two nodes?
>alexandra.strauss at bayerbbs.com wrote:
>>
>> Hello,
>>
>> I refer to you hoping you may help me with my problem... We have got
>> an issur here and opened a SR at Metalink but until now, we got no
>> useful information in solving our problem. SR-Number is 6855815.994...
>>
>> We wanted to protect 9i Single-Instance Databases with 10g Clusterware
>> following the third-party-tool approach. There are no RAC-databases
>> involved. But we want to achieve high availability as the databases
>> are business critical systems. We want to make the systems able to
>> relocate to another machine in case of failure to keep downtimes
>> low... To achieve this we want to use OCFS2 for the filesystem.
>> Relocate is done by script with help of CRS.
>>
>> So we took two systems (byaz05 and byaz10) and installed the following
>> software: 10g CRS (10.2.0.3) and Oracle Software 9.2.0.8 and OCFS2 1.2.8
>>
>> We found the following Metalinknotes and adjusted the heartbeat and
>> timeouts for OCFS2: Metalink Note 395878.1: Heartbeat/Voting/Quorum
>> Related Timeout Configuration for Linux, OCFS2, RAC Stack to avoid
>> unnessary node fencing, panic and reboot
>> Metalink Note 391771.1: OCFS2 - FREQUENTLY ASKED QUESTIONS (hier
>> insbesondere der Abschnitt zu Fencing und Quorum)
>> Metalink Note 434255.1: Common reasons for OCFS2 Kernel Panic or
>> Reboot Issues
>> Metalink Note 457423.1: OCFS2 Fencing, Network, and Disk Heartbeat
>> Timeout Configuration
>>
>> We did no changes to the CRS/CSS default settings until now.
>>
>> During HA-testing we watched unexpected behaviour of the system. We
>> deactivated the bond for private interconnect and expected only one
>> node to go down. But we faced both nodes going down. As it seems to me
>> one node was rebooted from OCFS2 and the other one from CRS/CSS.
>>
>> Timestamp
>> --------------------------------------------------------------------------------------------------------------
>>
>> 10:21:06 bond1 disabled (eth1)
>> */var/log/messages byaz05*
>> Apr 25 10:21:06 byaz05 kernel: bonding: bond1: link status definitely
>> down for interface eth1, disabling it
>> Apr 25 10:21:06 byaz05 kernel: bonding: bond1: making interface eth5
>> the new active one.
>>
>> 10:21:09 bond1 disabled (eth5)
>> */var/log/messages byaz05*
>> Apr 25 10:21:09 byaz05 kernel: bonding: bond1: link status definitely
>> down for interface eth5, disabling it
>> Apr 25 10:21:09 byaz05 kernel: bonding: bond1: now running without any
>> active interface !
>>
>> 10:21:23 o2net – no longer connected
>> */var/log/messages byaz05*
>> Apr 25 10:21:23 byaz05 kernel: o2net: no longer connected to node
>> byaz10.bayer-ag.com (num 1) at 10.190.59.6:7777
>> */var/log/messages byaz10*
>> Apr 25 10:21:23 byaz10 kernel: o2net: no longer connected to node
>> byaz05.bayer-ag.com (num 0) at 10.190.59.5:7777
>>
>> 10:21:27 CSSD failure 134
>> 10:21:29 Reboot initiated by CRS
>> */var/log/messages byaz05*
>> Apr 25 10:21:27 byaz05 logger: Oracle clsomon failed with fatal status
>> 12.
>> Apr 25 10:21:27 byaz05 logger: Oracle CSSD failure 134.
>> Apr 25 10:21:27 byaz05 su(pam_unix)[25839]: session closed for user
>> oracle
>> Apr 25 10:21:27 byaz05 logger: Oracle CRS failure. Rebooting for
>> cluster integrity.
>> Apr 25 10:21:27 byaz05 kernel: md: stopping all md devices.
>> Apr 25 10:21:27 byaz05 kernel: md: md0 switched to read-only mode.
>> Apr 25 10:21:29 byaz05 logger: Oracle CRS failure. Rebooting for
>> cluster integrity.
>> Apr 25 10:21:29 byaz05 kernel: e1000: eth2: e1000_watchdog_task: NIC
>> Link is Up 1000 Mbps Full Duplex
>> Apr 25 10:21:29 byaz05 logger: Oracle init script ceding reboot to
>> sibling 27383.
>>
>> 10:21:58 Reboot initiated by OCFS2(?)
>> */var/log/messages byaz10*
>> Apr 25 10:21:58 byaz10 su(pam_unix)[4595]: session opened for user
>> oracle by (uid=0)
>> Apr 25 10:21:58 byaz10 su(pam_unix)[4595]: session closed for user oracle
>> Apr 25 10:25:58 byaz10 syslogd 1.4.1: restart.
>> Apr 25 10:25:58 byaz10 syslog: syslogd startup succeeded
>> Apr 25 10:25:58 byaz10 kernel: klogd 1.4.1, log source = /proc/kmsg
>> started.
>> Apr 25 10:25:58 byaz10 kernel: Bootdata ok (command line is ro
>> root=/dev/vgroot/_)
>>
>>
>> We supposed all the time this is a timing problem. But we don't know
>> which settings raise the problem and which steps to do to to correct
>> them. Otherwise we'll have to work over the complete concept for the
>> business critical systems.
>> Can anyone help me?
>>
>> Regards,
>> Alexandra
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20080606/84bb912e/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ATT00001.gif
Type: image/gif
Size: 2206 bytes
Desc: ATT00001.gif
Url : http://oss.oracle.com/pipermail/ocfs2-users/attachments/20080606/84bb912e/attachment-0001.gif
More information about the Ocfs2-users
mailing list