[Ocfs2-users] Aditional interconnect interface

Thu Oct 4 17:06:55 PDT 2007

This is pretty simple:

(1) Every server have, in most cases, more then 1 physical interface. In 
many cases, these interfaces are already
connected to a different switches or even uses different medias. Even if 
bonded, we have still a few of them.
(2) Cluster is extremely sensitive to the lost of interconnection, because 
it cause fencing or self-fencing
(reboot or panic, in simple words) on some nodes, and delays on anothers.
(3) If we use heartbeat over few independent IP interfaces, then we don't 
depend of switch timeouts, bonding timeouts, bonding rules
and other things that affects all low level bondings, trunkings and so on. 
These timeouts are not small - for example, STP reconfiguration in Ethernet 
takes 40 seconds BY DEFAULT.
(4) WE DO HAVE THEE INTERFACES ALREADY.
(5) Implementing few heartbeats costs almost 0 for developers.

It explains why most clusters supports few heartbeats. I always configure 
heartbeat over ALL available medias. If I have NFS cluster, I use 3 media - 
eth0 (public), eth1 (SAN), eth2 (management), plus serial. The reason is 
simple - it is very important to keep cluster interconnection in case of 
medial failure - it allows nodes to switchover in a few seconds because they 
can communicate and know each other's status.

Now let's image that I can not use multi interface for heartbeat (or VCS, it 
don't make much difference) cluster. Switch fail so that node1 lost 
ethernet1.
What shoudl node2 do:
- reboot? But what if node1 is dead.
- take control? But it is not sure if node1 is dead.
- kill node1? But what if node1 lost ethernet and node2 is the only alive 
member.

No way to resolve it,. If I use bonding (aside of need in 2 more 
interfaces) - node still must wait 30 - 40 seconds to eliminate switch 
timeouts, bonding timeouts and so on.

Now notice - nodes can talk, they are interconnected. By eth0, eth2 and 
serial. So what's the problem? heartbeat is pretty simple thing with OR 
rule - if
any channel respond, node is alive. No complexity to implement it.

Now I run thru a few implementations. Every one which have not ORACLE in 
name allows me to use many heartbeats. 5 examples - Cisco PIX,
Veritas VCS, Linux heartbeat1, Linux heartbeat2, NetApp cluster (2 
independent inter-connections). And Oracle is the only exception, Why?

----- Original Message ----- 
From: "Bob Murching" <bob.murching at muveka.com>

I'm still trying to understand your earlier post.  I understand why Veritas
wants multiple heartbeats.  That said, I thought that RAC mitigates the risk
with voting disk, etc.  I guess I'm not understanding why "single heartbeat,
use bonding" is necessarily a bad thing...?

-----Original Message-----
From: Alexei_Roudnev [mailto:Alexei_Roudnev at exigengroup.com]
Sent: Thursday, October 04, 2007 5:20 PM
To: bob.murching at muveka.com
Subject: Re: [Ocfs2-users] Aditional interconnect interface

It did not changed, as I can see:
- <<IT is recommenede to configure at leat 2 huigh priority links and 1 low
priority links>>
(link == interface)

When configuring heartbeat, I configure 3 links:
- 1 over access network
- 1 over SAN network
- 1 over Serial or over management interface
(all on different switches).

----- Original Message ----- 
From: "Bob Murching" <bob.murching at muveka.com>
To: "'Alexei_Roudnev'" <Alexei_Roudnev at exigengroup.com>
Sent: Thursday, October 04, 2007 1:43 PM
Subject: RE: [Ocfs2-users] Aditional interconnect interface

nevermind!
http://seer.entsupport.symantec.com/docs/283846.htm

-----Original Message-----
From: ocfs2-users-bounces at oss.oracle.com
[mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of Alexei_Roudnev
Sent: Thursday, October 04, 2007 3:31 PM
To: Sunil Mushran; Pedro Figueira
Cc: ocfs2-users at oss.oracle.com
Subject: Re: [Ocfs2-users] Aditional interconnect interface

It's old relogious discussion. All people who works in data centers knows,
that cluster MUST use more then 1 heartbeat (better 3 or 4, not bonded)
interface for stability, but Oracle developers (both RAC and OCFSv2) believe

in bonding.

(Bonding DO NOT solve the problem! It have sugnificant switchover time and
it is not reliable enough.  Open Veritas cluster documentation and READ IT -

they have a pretty good explanation why you must configure 2 independent
heartbeats using 2 independent switches or hubs. Veritas use multi
interface. Cisco PIX uses multi interface. Heartbeat1 uses multi interface.
Heartbeat2 uses multi interfaces. And only Oracle* developers use 1
interface and trust into the bonding!)

Using additional loopback interface  and routing (via OSPF which is pretty
fast) can solve problem too. (for OCFS bot not for RAC cluster).

----- Original Message ----- 
From: "Sunil Mushran" <Sunil.Mushran at oracle.com>
To: "Pedro Figueira" <Pedro.Figueira at SEF.pt>
Cc: <ocfs2-users at oss.oracle.com>
Sent: Wednesday, October 03, 2007 9:26 AM
Subject: Re: [Ocfs2-users] Aditional interconnect interface

Use network bonding.

Pedro Figueira wrote:
> Hello
>
> Is there a way of using more than one network interface for ocfs2
> interconnect? I've searched for this in the documentation but I cannot
> find any information about it.
>
> Best regards
>
> Pedro Figueira Serviço de Estrangeiros e Fronteiras Direcção Central de
> Informática
> Departamento de Produção Telefone: + 351 217 115 153
>
> CONFIDENCIAL NOTICE:
> This message, as well as any existing attached files, is confidential and
> intended exclusively for the individual(s) named as addressees. If you are

> not the intended recipient, you are kindly requested not to make any use
> whatsoever of its contents and to proceed to the destruction of the
> message, thereby notifying the sender. DISCLAIMER:
> The sender of this message can NOT ensure the security of its electronic
> transmission and consequently does not accept liability for any fact,
> which may interfere with the integrity of its content.
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>

_______________________________________________
Ocfs2-users mailing list
Ocfs2-users at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

_______________________________________________
Ocfs2-users mailing list
Ocfs2-users at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users