[Ocfs2-users] re: how should ocfs2 react to nic hardware issue

Peter Santos psantos at cheetahmail.com
Fri Dec 1 20:31:27 PST 2006 

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Scott,
thanks for your remarks .. interesting stuff.

I have to say that when I took down the interconnect on my node2 (3 node cluster),
the observed behavior is pretty much what you described.. node1 and node3
both showed o2cb messages saying the node2 had been evicted from the cluster.
The vip did failover in this case and node2 eventually rebooted.

The behavior was much different when I took down eth0. I'll have
to repeat this test to really get a handle on what happened, but I do
believe the vip never failed over to another node and the instance on that machine
stayed up but could not be accessed.  I waited about 15 minutes then via SRVCTL
did a shutdown abort on the instance.. only then did my connection failover to another
instance.

I'll test further, but thanks for your feedback.
- -peter


SCOTT, Gavin wrote:
> I've recently gone through this, however I addressed it as a RAC/CRS
> issue, as it didn't really relate to OCFS2. But at the risk of being
> slightly off topic, I'll cover it briefly here anyway.
> 
> In a 2 node RAC cluster, a failure of the interconnect NIC on node 2
> causes node 2 to get evicted from the cluster and it reboots. This is
> all fine.
> 
> I would have thought that a hardware failure of the interconnect NIC on
> node 1 would cause node 1 to evict and fence, leaving node 2 to take
> over. However this is not the case. Node 2 still evicts and fences,
> leaving the faulty node 1 to continue. After some back and forth to
> Oracle, they eventually confirmed that this is the correct behaviour,
> and that down time on the cluster is necessary to return to full
> functionality. Not ideal.
> 
> The next problem is that the instance on Node 1 would crash about 5
> minutes after NIC failure, reporting at the database level that it could
> not locate the NIC (this is on RHEL 4). According to the RAC/CRS people,
> I now have to raise this as a TAR with the RDBMS team, however I have
> not done this yet.
> 
> As for users hanging on node failure when the NIC fails, this could be
> attributable to the TCP/IP settings that determine when a connection is
> deemed to have failed. You need to tune your IP settings on the server
> and client side to ensure that the failure is detected in a timely
> fashion. Depending on the OS, there are various settings (on VMS:
> tcp_keepalive, tcp_keepcnt, tcp_keepintvl, tcp_keepinit), the defaults
> on which will not result in failover in this situation for several
> hours.
> 
> Note that you will not see these delays on an instance or node failure,
> that is were the CRS MISSCOUNT parameter determines when to evict a node
> from a cluster and comes into play.
> 
> Note also that tuning IP settings will affect the entire network and
> that this needs to be balanced to ensure short network glitches don't
> trigger a failover.
> 
> There is some good info on Metalink on this, but as Peter implied, this
> looks more like a CRS and TCP/IP issue than an OCFS2 one.
> 
> This info is a quick rehash from some time ago, so sorry if it's got
> some holes. Feel free to contact me offline, as I did go through quite a
> lot of digging to resolve this.
> 
> Regards,
> Gavin
> 
> -----Original Message-----
> From: ocfs2-users-bounces at oss.oracle.com
> [mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of Adam Kenger
> Sent: Friday, 1 December 2006 07:34
> To: Peter Santos
> Cc: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] re: how should ocfs2 react to nic hardware
> issue
> 
> Peter - depending on how you have your RAC cluster setup, the hang on  
> the front end is not that unexpected.  It depends on how the user was  
> connected and how the TAF policy was set up.  Eth0 is your public  
> interface I assume.  Was the user connecting to the IP on that  
> interface or to the VIP set up by RAC?  When you down eth0 the VIP on  
> that interface should get pushed over onto one of the other 2 nodes  
> in the cluster.  If you're connecting to a "service" versus an actual  
> "instance" there should be no hang on the front end.  If you're  
> actually connected directly to the instance on the node, then you'll  
> be out of luck if you disconnect that instance.  As an example, this  
> is what the corresponding tnsnames.ora file looks like :
> 
> MYDBSERVICE =
>    (DESCRIPTION =
>      (ADDRESS_LIST =
>        (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
>        (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
>        (ADDRESS = (PROTOCOL = TCP)(HOST = node3-vip)(PORT = 1521))
>      )
>      (CONNECT_DATA =
>        (SERVICE_NAME = mydbservice.db.mydomain.com)
>      )
>    )
> 
> MYDB1 =
>    (DESCRIPTION =
>      (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 15
> 21))
>      (CONNECT_DATA =
>        (SERVER = DEDICATED)
>        (SERVICE_NAME = mydb.db.mydomain.com)
>        (INSTANCE_NAME = mydb1)
>      )
>    )
> 
> If you connected to the service "MYDBSERVICE" you could survive the  
> failure of any given node.  You'd seamlessly fail-over onto one of  
> the other nodes.  If you connect directly to the "MYDB1" instance,  
> you'll be out of luck if you drop the connection to it.
> 
> As far as o2cb goes, you are right I believe.  Eventually, it will be  
> determined that the node is no longer heartbeating and will either  
> panic or reboot.
> 
> For your testing, just be careful you're not confusing the OCFS2  
> layer with the Oracle CRS/RAC layers.
> 
> Comments welcome....
> 
> Hope that helps
> 
> Adam
> 
> 
> 
> 
> On Nov 30, 2006, at 3:30 PM, Peter Santos wrote:
> 
> guys,
> 	I'm trying to test how my 10gR2 oracle cluster (3 nodes) on SuSe
> 
> reacts to a network card hardware failure.
> 	I have eth0 and eth1 as my network cards, I took down eth0
>> (ifdown  
> eth0) to see what would happen and
> 	I didn't get any reaction from the o2cb service. This is
>> probably  
> the correct behavior since my
>  	/etc/ocfs2/cluster.conf uses eth1 as the connection channel?
> 
> 	If I take down eth1 I suspect o2cb will eventually reboot the  
> machine right? I'm not using any bonding.
> 
> 	My concern is that when I took down eth0, I had a user logged
>> into  
> the instance and everything just "hung" for
> 	that user, until I manually took down the instance with  
> "SRVCTL"... then the user connection failed over to
> 	a working instance.
> 
> 	Anyway, just trying to get some general knowledge of the
>> behavior  
> of o2cb in order to understand
> 	my testing.
> 
> -peter
> 
> 
>>
_______________________________________________
Ocfs2-users mailing list
Ocfs2-users at oss.oracle.com
http://oss.oracle.com/mailman/listinfo/ocfs2-users

> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users

> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFcQGfoyy5QBCjoT0RAjpJAJ9A64ci0JZPFAW9LOlJf8utRsCEMgCgkelC
O8GuMDkAjVhLBcPTArAGt7s=
=Uinj
-----END PGP SIGNATURE-----

More information about the Ocfs2-users mailing list