[Ocfs2-users] Extremely high memory usage and iowait times

[Dan Lark]

I know this has been discussed before, but I am seeing high iowait times and the occasional deadlock between my two node OCFS2 cluster. I will be turning on "noatime" for the mount on both nodes in a maintenance window, which should help. What I am not sure of, whether that will be of much help.

For example, sar is reporting the following from one of the nodes over the last little bit:

10:00:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
10:10:01 AM       all      2.45      0.00      5.11     23.94      0.00     68.51
10:20:01 AM       all      2.61      0.00      5.56     23.50      0.00     68.33
10:30:01 AM       all      2.22      0.00      5.31     23.27      0.00     69.20
10:40:01 AM       all      2.48      0.00      5.27     23.83      0.00     68.42
10:50:01 AM       all      2.53      0.00      5.29     22.02      0.00     70.16
11:00:01 AM       all      2.39      0.00      5.66     24.42      0.00     67.53
11:10:01 AM       all      2.78      0.00      5.75     30.11      0.00     61.37
11:20:01 AM       all      2.90      0.00      5.74     25.83      0.00     65.53
11:30:01 AM       all      2.78      0.00      5.61     24.49      0.00     67.12
11:40:01 AM       all      2.78      0.00      5.37     24.43      0.00     67.41
11:50:01 AM       all      2.64      0.00      5.00     21.90      0.00     70.47
12:00:01 PM       all      2.60      0.00      5.41     24.22      0.00     67.78
12:10:01 PM       all      2.51      0.00      5.20     23.89      0.00     68.39
12:20:01 PM       all      2.50      0.00      5.02     25.70      0.00     66.78
12:30:01 PM       all      2.49      0.00      5.24     24.61      0.00     67.67
12:40:01 PM       all      2.53      0.00      4.90     24.36      0.00     68.21
12:50:01 PM       all      2.32      0.00      4.81     23.18      0.00     69.69
01:00:01 PM       all      2.38      0.00      4.92     23.07      0.00     69.64
01:10:01 PM       all      2.09      0.00      5.17     23.45      0.00     69.29
01:20:01 PM       all      2.63      0.00      5.35     23.24      0.00     68.77
01:30:01 PM       all      2.28      0.00      4.63     25.15      0.00     67.93
01:40:01 PM       all      2.39      0.00      4.76     20.93      0.00     71.92
01:50:01 PM       all      2.24      0.00      4.95     24.81      0.00     68.01
02:00:01 PM       all      2.36      0.00      4.99     27.45      0.00     65.19
02:10:01 PM       all      3.31      0.00      6.65     43.20      0.00     46.85
02:20:01 PM       all      2.88      0.00      6.47     42.47      0.00     48.18
02:30:01 PM       all      2.66      0.00      6.61     44.67      0.00     46.06
02:40:01 PM       all      2.67      0.00      5.97     43.46      0.00     47.90

The iowait percentage is evening itself back down, but a 44.67% iowait? WOW! Any suggestions?

Specs:
OEL 5.5 - Kernel 2.6.18-194.8.1.0.1.el5
ocfs2-tools-1.4.4-1.el5
ocfs2-2.6.18-194.8.1.0.1.el5-1.4.7-1.el5
ocfs2console-1.4.4-1.el5

Help or suggestions appreciated.

Thanks!

-dan
______________________
Dan Lark, RHCE
Systems Engineer, Hargray Communications
dan.lark at htc.hargray.com
Office: 843-706-1819 | Cell: 843-227-2263
7 Arley Way, Suite 200 (2nd Floor)
P.O. Box 3380
Bluffton, SC 29910