[Ocfs2-users] Extremely high memory usage and iowait times
Dan Lark
dan.lark at htc.hargray.com
Tue Sep 7 12:43:48 PDT 2010
Sure. We're running a four node CommuniGate cluster with two front-end
nodes and two backend nodes. Each node is a Dell M610 blade with a dual
core Xeon running @ 1.87 Ghz, 8Gb RAM, boots iSCSI diskless to two Dell
EqualLogic SANs. The main storage portion of the for the CG cluster, aka
backends, have the shared portions of the CG stuff mounted OCFS2
partition. This partition is running multipath'd (each node had two
Ethernet adapters) back into the same EL SANs.
Anything else needed?
-dan
-----Original Message-----
From: Sunil Mushran [mailto:sunil.mushran at oracle.com]
Sent: Tuesday, September 07, 2010 3:00 PM
To: Dan Lark
Cc: Ocfs2-users at oss.oracle.com
Subject: Re: [Ocfs2-users] Extremely high memory usage and iowait times
Can you describe your usage and your setup a bit?
On 09/07/2010 11:55 AM, Dan Lark wrote:
> I know this has been discussed before, but I am seeing high iowait
times and the occasional deadlock between my two node OCFS2 cluster. I
will be turning on "noatime" for the mount on both nodes in a
maintenance window, which should help. What I am not sure of, whether
that will be of much help.
>
> For example, sar is reporting the following from one of the nodes over
the last little bit:
>
> 10:00:01 AM CPU %user %nice %system %iowait
%steal %idle
> 10:10:01 AM all 2.45 0.00 5.11 23.94
0.00 68.51
> 10:20:01 AM all 2.61 0.00 5.56 23.50
0.00 68.33
> 10:30:01 AM all 2.22 0.00 5.31 23.27
0.00 69.20
> 10:40:01 AM all 2.48 0.00 5.27 23.83
0.00 68.42
> 10:50:01 AM all 2.53 0.00 5.29 22.02
0.00 70.16
> 11:00:01 AM all 2.39 0.00 5.66 24.42
0.00 67.53
> 11:10:01 AM all 2.78 0.00 5.75 30.11
0.00 61.37
> 11:20:01 AM all 2.90 0.00 5.74 25.83
0.00 65.53
> 11:30:01 AM all 2.78 0.00 5.61 24.49
0.00 67.12
> 11:40:01 AM all 2.78 0.00 5.37 24.43
0.00 67.41
> 11:50:01 AM all 2.64 0.00 5.00 21.90
0.00 70.47
> 12:00:01 PM all 2.60 0.00 5.41 24.22
0.00 67.78
> 12:10:01 PM all 2.51 0.00 5.20 23.89
0.00 68.39
> 12:20:01 PM all 2.50 0.00 5.02 25.70
0.00 66.78
> 12:30:01 PM all 2.49 0.00 5.24 24.61
0.00 67.67
> 12:40:01 PM all 2.53 0.00 4.90 24.36
0.00 68.21
> 12:50:01 PM all 2.32 0.00 4.81 23.18
0.00 69.69
> 01:00:01 PM all 2.38 0.00 4.92 23.07
0.00 69.64
> 01:10:01 PM all 2.09 0.00 5.17 23.45
0.00 69.29
> 01:20:01 PM all 2.63 0.00 5.35 23.24
0.00 68.77
> 01:30:01 PM all 2.28 0.00 4.63 25.15
0.00 67.93
> 01:40:01 PM all 2.39 0.00 4.76 20.93
0.00 71.92
> 01:50:01 PM all 2.24 0.00 4.95 24.81
0.00 68.01
> 02:00:01 PM all 2.36 0.00 4.99 27.45
0.00 65.19
> 02:10:01 PM all 3.31 0.00 6.65 43.20
0.00 46.85
> 02:20:01 PM all 2.88 0.00 6.47 42.47
0.00 48.18
> 02:30:01 PM all 2.66 0.00 6.61 44.67
0.00 46.06
> 02:40:01 PM all 2.67 0.00 5.97 43.46
0.00 47.90
>
> The iowait percentage is evening itself back down, but a 44.67%
iowait? WOW! Any suggestions?
>
> Specs:
> OEL 5.5 - Kernel 2.6.18-194.8.1.0.1.el5
> ocfs2-tools-1.4.4-1.el5
> ocfs2-2.6.18-194.8.1.0.1.el5-1.4.7-1.el5
> ocfs2console-1.4.4-1.el5
>
> Help or suggestions appreciated.
>
> Thanks!
>
> -dan
> ______________________
> Dan Lark, RHCE
> Systems Engineer, Hargray Communications
> dan.lark at htc.hargray.com
> Office: 843-706-1819 | Cell: 843-227-2263
> 7 Arley Way, Suite 200 (2nd Floor)
> P.O. Box 3380
> Bluffton, SC 29910
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>
More information about the Ocfs2-users
mailing list