[Ocfs2-users] OCFS2 v1.4 hangs

[Karim Alkhayer]

Hi John,

 

When multiple systems/nodes have access to data via shared storage, the
integrity of the data depends on inter-node communication ensuring that each
node is aware when other nodes are writing data. When the coordination
between the nodes fails, it results in a "split brain" condition; A
situation in which two servers try to independently control the storage,
potentially resulting in application failure or even corruption of critical
data.

 

I/O fencing is a method of choice (used by vendors cluster frameworks,
including OCFS2) for ensuring the integrity of critical information by
preventing data corruption, allowing a set of systems to have temporary
registrations with the disk and coordinate a write-exclusive reservation
with the disk containing the data. With I/O fencing, the cluster system
ensures that errant nodes are "fenced" and do not have access to the shared
storage, while the eligible node(s) continue to have access to the data,
virtually eliminating the risk of data corruption.

 

The quorum is the group of nodes in a cluster that is allowed to operate on
the shared storage. When there is a failure in the cluster, nodes may be
split into groups that can communicate in their groups and with the shared
storage but not between groups.

 

O2QUO determines which group is allowed to continue and initiates fencing of
the other group(s).

Fencing is the act of forcefully removing a node from a cluster. A node with
OCFS2 mounted will fence itself when it realizes that it does not have
quorum in a degraded cluster. It does this so that other nodes won't be
stuck trying to access its resources. However, the resources do NOT get
released

 

O2CB uses a node reset mechanism to fence; this however, is causing the
machine(s) to hang instead of seamless handover. In OCFS2 1.4, Oracle has
introduced a new fencing mechanism which no longer uses "panic" for fencing.
Instead, by default, it uses "machine restart".

 

In your case, taking the network down the way you've done is causing the
servers to hang, including the mounted file system which becomes locked
until the OCFS cluster services is restarted.

 

RAC handover fails due to exactly this problem: the file system is locked by
another node which was kicked out of the cluster, but still occupying the
file system

The healthy node will try to continue to work, but the databases hosted on
the occupied file system will hang, and possibly the machine. At this time
there is no solution but to 

-         Force shutdown the troublesome node(s)

-         Shutdown the databases processes

-         Restart the OCFS2 services

 

Network failure resolution can be applied in a situation where you have
setup a net bonding for the interconnects, which is highly recommended.

 

Best regards,

Karim Alkhayer

 

 

-----Original Message-----
From: ocfs2-users-bounces at oss.oracle.com
[mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of John Murphy
Sent: Thursday, June 04, 2009 10:15 PM
To: ocfs2-users at oss.oracle.com
Subject: [Ocfs2-users] OCFS2 v1.4 hangs

 

I have four database servers in a high-availability, load-balancing

configuration. Each machine has a mount to a common data source which is

an OCFS2 v1.4 file-system. While working on three of the servers, I

restarted the IP network and found after-wards the fourth machine hung.

I could not reboot and could not unmount the ocfs2 partitions. I am

pretty sure this was all caused by my taking down the network on all

three of the remaining machines, can anyone shed some light on this for.

Ironically, I have four machines in order to ensure reliability.

 

TIA

 

John

-- 

John Murphy

Technical And Managing Director

MANDAC Ltd

Kandoy House

2 Fairview Strand

Dublin 3

p: +353 1 5143001

m: +353 85 711 6844

e: john.murphy at mandac.eu

w: www.mandac.eu

 

 

 

_______________________________________________

Ocfs2-users mailing list

Ocfs2-users at oss.oracle.com

http://oss.oracle.com/mailman/listinfo/ocfs2-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090605/d7590c00/attachment.html