[Ocfs2-users] Extremely high memory usage and iowait times

[Sunil Mushran]

How are you using the fs? Including approx rate of creation/deletion
of files, avg size of files, etc.

Do you have any one file that highlights the problem?

On 09/07/2010 12:43 PM, Dan Lark wrote:
> Sure. We're running a four node CommuniGate cluster with two front-end
> nodes and two backend nodes. Each node is a Dell M610 blade with a dual
> core Xeon running @ 1.87 Ghz, 8Gb RAM, boots iSCSI diskless to two Dell
> EqualLogic SANs. The main storage portion of the for the CG cluster, aka
> backends, have the shared portions of the CG stuff mounted OCFS2
> partition. This partition is running multipath'd (each node had two
> Ethernet adapters) back into the same EL SANs.
>
> Anything else needed?
>
> -dan
>
>
>
> -----Original Message-----
> From: Sunil Mushran [mailto:sunil.mushran at oracle.com]
> Sent: Tuesday, September 07, 2010 3:00 PM
> To: Dan Lark
> Cc: Ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] Extremely high memory usage and iowait times
>
> Can you describe your usage and your setup a bit?
>
> On 09/07/2010 11:55 AM, Dan Lark wrote:
>    
>> I know this has been discussed before, but I am seeing high iowait
>>      
> times and the occasional deadlock between my two node OCFS2 cluster. I
> will be turning on "noatime" for the mount on both nodes in a
> maintenance window, which should help. What I am not sure of, whether
> that will be of much help.
>    
>> For example, sar is reporting the following from one of the nodes over
>>      
> the last little bit:
>    
>> 10:00:01 AM       CPU     %user     %nice   %system   %iowait
>>      
> %steal     %idle
>    
>> 10:10:01 AM       all      2.45      0.00      5.11     23.94
>>      
> 0.00     68.51
>    
>> 10:20:01 AM       all      2.61      0.00      5.56     23.50
>>      
> 0.00     68.33
>    
>> 10:30:01 AM       all      2.22      0.00      5.31     23.27
>>      
> 0.00     69.20
>    
>> 10:40:01 AM       all      2.48      0.00      5.27     23.83
>>      
> 0.00     68.42
>    
>> 10:50:01 AM       all      2.53      0.00      5.29     22.02
>>      
> 0.00     70.16
>    
>> 11:00:01 AM       all      2.39      0.00      5.66     24.42
>>      
> 0.00     67.53
>    
>> 11:10:01 AM       all      2.78      0.00      5.75     30.11
>>      
> 0.00     61.37
>    
>> 11:20:01 AM       all      2.90      0.00      5.74     25.83
>>      
> 0.00     65.53
>    
>> 11:30:01 AM       all      2.78      0.00      5.61     24.49
>>      
> 0.00     67.12
>    
>> 11:40:01 AM       all      2.78      0.00      5.37     24.43
>>      
> 0.00     67.41
>    
>> 11:50:01 AM       all      2.64      0.00      5.00     21.90
>>      
> 0.00     70.47
>    
>> 12:00:01 PM       all      2.60      0.00      5.41     24.22
>>      
> 0.00     67.78
>    
>> 12:10:01 PM       all      2.51      0.00      5.20     23.89
>>      
> 0.00     68.39
>    
>> 12:20:01 PM       all      2.50      0.00      5.02     25.70
>>      
> 0.00     66.78
>    
>> 12:30:01 PM       all      2.49      0.00      5.24     24.61
>>      
> 0.00     67.67
>    
>> 12:40:01 PM       all      2.53      0.00      4.90     24.36
>>      
> 0.00     68.21
>    
>> 12:50:01 PM       all      2.32      0.00      4.81     23.18
>>      
> 0.00     69.69
>    
>> 01:00:01 PM       all      2.38      0.00      4.92     23.07
>>      
> 0.00     69.64
>    
>> 01:10:01 PM       all      2.09      0.00      5.17     23.45
>>      
> 0.00     69.29
>    
>> 01:20:01 PM       all      2.63      0.00      5.35     23.24
>>      
> 0.00     68.77
>    
>> 01:30:01 PM       all      2.28      0.00      4.63     25.15
>>      
> 0.00     67.93
>    
>> 01:40:01 PM       all      2.39      0.00      4.76     20.93
>>      
> 0.00     71.92
>    
>> 01:50:01 PM       all      2.24      0.00      4.95     24.81
>>      
> 0.00     68.01
>    
>> 02:00:01 PM       all      2.36      0.00      4.99     27.45
>>      
> 0.00     65.19
>    
>> 02:10:01 PM       all      3.31      0.00      6.65     43.20
>>      
> 0.00     46.85
>    
>> 02:20:01 PM       all      2.88      0.00      6.47     42.47
>>      
> 0.00     48.18
>    
>> 02:30:01 PM       all      2.66      0.00      6.61     44.67
>>      
> 0.00     46.06
>    
>> 02:40:01 PM       all      2.67      0.00      5.97     43.46
>>      
> 0.00     47.90
>    
>> The iowait percentage is evening itself back down, but a 44.67%
>>      
> iowait? WOW! Any suggestions?
>    
>> Specs:
>> OEL 5.5 - Kernel 2.6.18-194.8.1.0.1.el5
>> ocfs2-tools-1.4.4-1.el5
>> ocfs2-2.6.18-194.8.1.0.1.el5-1.4.7-1.el5
>> ocfs2console-1.4.4-1.el5
>>
>> Help or suggestions appreciated.
>>
>> Thanks!
>>
>> -dan
>> ______________________
>> Dan Lark, RHCE
>> Systems Engineer, Hargray Communications
>> dan.lark at htc.hargray.com
>> Office: 843-706-1819 | Cell: 843-227-2263
>> 7 Arley Way, Suite 200 (2nd Floor)
>> P.O. Box 3380
>> Bluffton, SC 29910
>>
>>
>> _______________________________________________
>> Ocfs2-users mailing list
>> Ocfs2-users at oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>
>>      
>