[Ocfs2-users] out of memory?

[Paul Jimenez]

oom-killer shows up and then the box locks up. I'll try and get a  
dmesg with the oom-killer message in it.

   --pj

On Jun 29, 2006, at 5:02 PM, Sunil Mushran wrote:

> HighFree:     11877028 kB
> LowFree:        391020 kB
> HighFree:     11761892 kB
> LowFree:        342380 kB
> HighFree:     11654316 kB
> LowFree:        315860 kB
> HighFree:     11578756 kB
> LowFree:        291928 kB
> HighFree:     11490936 kB
> LowFree:        264788 kB
>
> That's at the end. I fail to see the enomem. Plenty of lowfree and  
> highfree.
> Some of the slabs do have high counts, but this is a big box.
>
> What is crashing? Is the server oopsing? oom-kill?
> Or, is the user-space process erroring out?
>
> Paul Jimenez wrote:
>> I have that complete file - from before rsync to the crash (~ 4MB)  
>> at  http://www.rgmadvisors.com/~pj/memslabinfo.
>>
>> Kernel is 2.6.16.7 vanilla, and the version of ocfs2 it came with.
>>
>>    --pj
>>
>>
>> On Jun 29, 2006, at 2:10 PM, Sunil Mushran wrote:
>>
>>
>>> I would like the entire /proc/meminfo and /proc/slabinfo.
>>> Dump it to a file every 1 min or so.
>>>
>>> What version of the kernel/ocfs2?
>>>
>>> Paul Jimenez wrote:
>>>
>>>> On Jun 29, 2006, at 8:22 AM, Brian Long wrote:
>>>>
>>>>
>>>>
>>>>> On Wed, 2006-06-28 at 17:03 -0500, Paul Jimenez wrote:
>>>>>
>>>>>
>>>>>> I'm getting out of memory errors trying to do 'rsync -av /foo / 
>>>>>> bar'
>>>>>> where /foo is a local dir and /bar is an ocfs2 filesystem   
>>>>>> running on
>>>>>> an ~ 6T ATA-over-Ethernet box.
>>>>>>
>>>>>>
>>>>> Paul,
>>>>>
>>>>> Can you also include some information about your /foo   
>>>>> partition?   It is
>>>>> millions of little files or hundreds of large files?  What is   
>>>>> the  RSS of
>>>>> rsync when you run out of memory?
>>>>>
>>>>> http://samba.anu.edu.au/rsync/FAQ.html#5
>>>>> http://lists.samba.org/archive/rsync/2002-July/003160.html
>>>>>
>>>>>
>>>>>
>>>> /foo is ~ 4600 files each about 60GB for a total of ~259GB.
>>>>
>>>> Some output after or slightly-before it crashed:
>>>>
>>>>
>>>> Every 2s: cat /proc/slabinfo | sort -rnk 2 |    
>>>> head                           Thu Jun 29 11:58:01 2006
>>>>
>>>> buffer_head       754620 754632     52   72    1 : tunables    
>>>> 120    60    8 : slabdata  10481  10481
>>>>       0
>>>> bio               225600 225600    128   30    1 : tunables    
>>>> 120    60    8 : slabdata   7520   7520
>>>>       0
>>>> biovec-1          225593 225736     16  203    1 : tunables    
>>>> 120    60    8 : slabdata   1112   1112
>>>>       0
>>>> journal_head      175548 182448     52   72    1 : tunables    
>>>> 120    60    8 : slabdata   2530   2534
>>>>       0
>>>> aoe_bufs          112536 112554     48   78    1 : tunables    
>>>> 120    60    8 : slabdata   1443   1443
>>>>       0
>>>> radix_tree_node    41510  41510    276   14    1 : tunables     
>>>> 54    27    8 : slabdata   2965   2965
>>>>       0
>>>> sysfs_dir_cache     3644   3772     40   92    1 : tunables    
>>>> 120    60    8 : slabdata     41     41
>>>>       0
>>>> size-32             2938   4407     32  113    1 : tunables    
>>>> 120    60    8 : slabdata     39     39
>>>>       0
>>>> size-64             2354   2596     64   59    1 : tunables    
>>>> 120    60    8 : slabdata     44     44
>>>>       0
>>>> dentry_cache        2086   3090    128   30    1 : tunables    
>>>> 120    60    8 : slabdata    103    103
>>>>       0
>>>>
>>>>
>>>> Free swap: 16779608kB
>>>> 4718592 pages of RAM
>>>> 4489216 pages of HIGHMEM
>>>> 562809 reserved pages
>>>> 530215 pages shared
>>>> 0 pages swap cached
>>>> 136994 pages dirty
>>>> 61878 pages writeback
>>>> 142502 pages mapped
>>>> 29403 pages slab
>>>> 480 pages pagetables
>>>>
>>>> 4718592 pages of RAM
>>>> 4489216 pages of HIGHMEM
>>>> 562809 reserved pages
>>>> 530215 pages shared
>>>> 0 pages swap cached
>>>> 136994 pages dirty
>>>> 61876 pages writeback
>>>> 142502 pages mapped
>>>> 29425 pages slab
>>>> 480 pages pagetables
>>>>
>>>> I don't think it's rsync running things oom; its memory   
>>>> consumption  is filecount based and 4600 files just isn't that  
>>>> many.
>>>>
>>>> The tunables that I had in place from the AoE faq (http://   
>>>> www.coraid.com/support/linux/EtherDrive-2.6-HOWTO.html#toc5.18)
>>>> this time were:
>>>>
>>>> vm.overcommit_memory=2
>>>> vm.dirty_ratio=3
>>>> vm.dirty_background_ratio=3
>>>> vm.min_free_kbytes=5120
>>>>
>>>> Any help appreciated.
>>>>
>>>>    --pj
>>>>
>>>> _______________________________________________
>>>> Ocfs2-users mailing list
>>>> Ocfs2-users at oss.oracle.com
>>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>>>
>>>>
>>
>>
>> _______________________________________________
>> Ocfs2-users mailing list
>> Ocfs2-users at oss.oracle.com
>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>>