[Ocfs2-users] No space left on the device

[Tao Ma]

Hi Aravind,

Aravind Divakaran wrote:
> Hi Tao,
>> Hi Aravind,
>>
>> Aravind Divakaran wrote:
>>>> Hi Aravind,
>>>>
>>>> Aravind Divakaran wrote:
>>>>> Hi Tao,
>>>>>
>>>>>> Hi Aravind,
>>>>>>
>>>>>> Aravind Divakaran wrote:
>>>>>>> Hi All,
>>>>>>>
>>>>>>> I have already sent one mail regarding the space issue i am facing
>>>>>>> with
>>>>>>> my
>>>>>>> ocfs filesystem. As mentioned in the below link it is an issue
>>>>>>> related
>>>>>>> to
>>>>>>> free space fragmentation.
>>>>>>>
>>>>>>> http://oss.oracle.com/bugzilla/show_bug.cgi?id=1189
>>>>>>>
>>>>>>> I have seen a patch for stealing extent allocation which was there
>>>>>>> is
>>>>>>> 2.6.34-rc1 kernel. So i compiled my new kernel and installed on my
>>>>>>> system.
>>>>>>>
>>>>>>> Below is my ocfs details on my system
>>>>>>>
>>>>>>> #modinfo ocfs2
>>>>>>>
>>>>>>> filename:       /lib/modules/2.6.34-rc1/kernel/fs/ocfs2/ocfs2.ko
>>>>>>> license:        GPL
>>>>>>> author:         Oracle
>>>>>>> version:        1.5.0
>>>>>>> description:    OCFS2 1.5.0
>>>>>>> srcversion:     A8B69947E8FF56D74858993
>>>>>>> depends:        jbd2,ocfs2_stackglue,quota_tree,ocfs2_nodemanager
>>>>>>> vermagic:       2.6.34-rc1 SMP mod_unload modversions
>>>>>>>
>>>>>>> This is my stat_sysdir.sh output
>>>>>>>
>>>>>>> http://pastebin.com/RZH9DkTk
>>>>>>>
>>>>>>> Can anyone help me how to resolve this, please as the problem occurs
>>>>>>> on
>>>>>>> production mail server with 3000 emailid.
>>>>>> I just checked your stat_sysdir output. It isn't caused by extent
>>>>>> block
>>>>>> alloc actually. So the patch doesn't work for you. Yes, the problem
>>>>>> you
>>>>>> meet is fragmentation issue, but the root cause is that inode_alloc
>>>>>> can't allocate any more inodes(a little different from 1189).
>>>>>>
>>>>>> I am now working on discontiguous block group. It will resolve your
>>>>>> issue I think. Hope it can be get into mainline in 2.6.35.
>>>>>>
>>>>>> Regards,
>>>>>> Tao
>>>>>>
>>>>> For my previous mail i got reply from you
>>>>>
>>>>> "Another way is that you can cp the file to another volume, remove it
>>>>> and
>>>>> then cp back. It should be contiguous enough."
>>>>>
>>>>> As mentioned in the 1189
>>>>>
>>>>> "However, reducing the slot count by 1 (to 4) may not be enough as it
>>>>> does
>>>>> not
>>>>> have much contiguous space. It may work. But reducing it by 2 will
>>>>> definitely work.
>>>>>
>>>>> Umount the volume on all nodes and run:
>>>>> # tunefs.ocfs2 -N 3 /dev/sda1
>>>>>
>>>>> Run fsck.ocfs2 for sanity checking."
>>>>>
>>>>> Will anyone of the above solution will temporary solve my problem.
>>>> Yes, it works. I just replied you in another e-mail.
>>>>
>>>> Regards,
>>>> Tao
>>>>
>>> I am running tunefs.ocfs2 on my 500gb harddisk which contain 215gb of
>>> data, in order to reduce the slots. I had used the below command.
>>>
>>> tunefs.ocfs2  -N 3 /dev/mapper/store
>>>
>>> Now almost 7hours is over still it didnt finished the execution. Below
>>> is
>>> the output i am getting.
>>>
>>> node01:~# tunefs.ocfs2 -N 3 /dev/mapper/store
>>> tunefs.ocfs2 1.4.1
>>>
>>> How much time it will take to reduce the slots. Whether it will be
>>> finished within 10hours. Can anyone help me.
>> It shouldn't cost so much time. I guess it get blocked in some case. So
>> is this volume umounted in all the nodes? If yes, could you please
>> strace it to see what's wrong?
>>
>> Regards,
>> Tao
>>
> 
> We have ocfs running with two cluster node. Before doing tunefs.ocfs2  i
> have stopped my ocfs2 service in both the systems which will unmount my
> ocfs filesystem. I have crossverified with mounted.ocfs2, which gives all
> unmounted. I tried to strace also but my strace is also not giving any
> output for that process id.
Oh, so is the program still running? If yes, 'top' should show its state.
If it is in 'D', please run
ps -e -o pid,stat,comm,wchan=WIDE-WCHAN-COLUMN to see what's wrong.
If it is in 'R', I guess there is a dead loop or somehow.
anyway, could you please o2image/compress your disk image and then send 
to us? We can test it locally.
See man o2image.

Regards,
Tao