[Ocfs-users] Major RAC slowdown

[Brian M. Diehl]

Hi All.  I had a very very similar problem, loads of 70 during our
"peak" times of the day, the end result was nothing to do with OCFS, but
lack of proper indexes.  Doing a Top SQL report, and then getting the
execution plan for "heavy" SQL statements, will show if you are doing
full table scans, and such that will busy out even the fastest of
arrays.  Just a little thought, as it was what fixed my problems, my
2-node cluster now runs with a combined load of just over 1.2 :-)

HTH,

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Brian M. Diehl              Sr. Network Admin
A-1 Limousine Inc.               609-919-2019
 "Our greatest glory is not in never falling,
       but in rising every time we fall"
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-


-----Original Message-----
From: Derek Suzuki [mailto:DSuzuki at ziprealty.com] 
Sent: Tuesday, June 08, 2004 4:23 AM
To: ocfs-users at oss.oracle.com
Subject: [Ocfs-users] Major RAC slowdown


Hello again.  Our production cluster has begun experiencing some vicious
slowdowns that may (or may not) be related to the filesystems.  When the
problem occurs, the load average on the servers jumps up to 30 or
higher.  Usually one node will climb while the other drops, then they
will switch places a few minutes later.  At one point, we had one node's
load average up over 300.  Our site activity has been on the rise, and
the problems usually occur during peak mid-day hours.

Under normal conditions, "top" shows the CPUs spending most of their
time waiting on the very busy fibre channel.  During the slowdowns, the
processors are mostly busy with system calls.  Traffic over both the
fibre channel and gigabit interconnect seems to drop off considerably at
the same time.

I've got a TAR open, but the support people are still in the very
preliminary stages (for example, we just installed a switch between the
two nodes because a crossover cable is apparently not supported).  There
doesn't seem to be any good indication of what's going on.  We suspected
the interconnect, but the private interfaces seem to behave normally
while Oracle is grinding to a halt.

After 10-30 minutes, the problem will fade away on its own.  I'm
inclined to blame something in the RAC inter-node communications code,
but I was wondering if this situation resembled any kind of OCFS problem
anyone has seen.  These servers are still on 1.0.9-12, with plans to go
to 1.0.12 soon after this issue is resolved.