[Ocfs2-users] OCFS2 Fencing, then panic

[Alexei_Roudnev]

It's noty an issue; it is really OCFSv2 killer:
- in 99% cases, it is not split brain condition but just a short (20 - 30
seconds) network interruption. Systems can (in most cases) see each other by
network or thru the voting disk, so they can communicate by one or another
way;
- in 90% cases system have not any pending IO activity, so it have not any
reason to fence itself at least until some IO happen on OCFSv2 file system.
For example, if OCFSv2 is used for backups, it is active 3 hours at night +
at the time of restoring only, and server can remount it without any fencing
if it lost consensus.
- timeouts and other fencing parameters are badly designed, and it makes a
problem worst. IT can't work out of the box on the most SAN networks (with
recoinfiguration timeouts all about 30 seconds - 1 minute by default). For
example, NetApp cluster takepooevr takes about 20 seconds, and giveback
about 40 seconds - which kills OCFSv2 for 100% sure (with default settings).
STP timeout (in classical mode) is 40 seconds, which kills OCFSv2 for 100%
sure. Network switch remoot time is about 1 minute for most switches, which
kills OCFSv2 for 100% sure. Result - if I reboot staging network switch, I
have all stand alone servers working, all RAC clusters working, all other
servers working, and all OCFSv2 cluster fenced themself.

For me, I baned OCFSv2 from any usage except backup and archive logs, and
only with using cross connection cable for heartbeat.
All other scenarios are catastrofic (cause overall cluster failure in many
cases). And all because of this fencing behavior.

PS> SLES9 SP3 build 283 have a very stable OCFSv2, with one well known
problem in buffer use - it don't release small buffers after file is
created/deleted (so if you run create file / remove file loop for a long
time, you will deplete system memory in apporox a few days). It is not a
case if files are big enough (Oracle backups, oracle archive logs,
application home) but must be taken into account if you have more than
100,000 - 1,000,000 files on OCFSv2 file system(s).

But fencing problem exists in all versions (little better in modern ones,
because developers added configurable network timeout). If you add _one
heartbeat interface only_ design and _no serial heartbeat_ design, it really
became a problem, ad it's why I was thinking about testing OCFSv2 in SLES10
with heartbeat2 (heartbeat2 have a very reliable heartbeat and have external
fencing, but unfortunately SLES10 is not production ready yet for us, de
facto).



----- Original Message ----- 
From: "Jeff Mahoney" <jeffm at suse.com>
To: "enohi ibekwe" <enohiaghe at hotmail.com>
Cc: <ocfs2-users at oss.oracle.com>
Sent: Saturday, April 07, 2007 12:06 PM
Subject: Re: [Ocfs2-users] OCFS2 Fencing, then panic


> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> enohi ibekwe wrote:
> > Is this also an issue on SLES9?
> >
> > I see this exact issue on my SLES9 + ocfs 1.2.1-4.2 RAC cluster. I see
> > the error on the same box on the cluster.
>
> I'm not sure what you mean by "issue." This is designed behavior. When
> the cluster ends up in a split condition, one or more nodes will fence
> themselves.
>
> - -Jeff
>
> - --
> Jeff Mahoney
> SUSE Labs
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org
>
> iD8DBQFGF+vDLPWxlyuTD7IRAuNPAJ9lZPLSaH7nOCNammYyW3bwC2Wj5wCgomUp
> zcRzcaedVAmk+AaJ/OFeddE=
> =8e6c
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>