[Ocfs2-users] how do I remove ocfs2 orphaned files that are taking up diskspace ?

Tue Mar 10 10:06:20 PDT 2009

Tim,

yes, it would be great if you can capture and provide me the following.
a)run stat_sysdir.sh (http://oss.oracle.com/~seeda/misc/stat_sysdir.sh)
b) lsof on all nodes
c)enable ocfs2 tracing on all nodes
debugfs.ocfs2 -l ENTRY EXIT DLM DLM_MASTER DLM_RECOVERY SUPER JOURNAL 
DLM_GLUE INODE VOTE DCACHE NAMEI allow
d)rm the file
e)make sure it is in the orphan dir
f)run stat_sysdir.sh again
g)lsof on all nodes
h)disable tracing

please upload outputs of stat_sysdir, lsof and messages files from all nodes

thanks,
--Srini.

Tim Hughes wrote:
> Sorry about the delay on this. I managed to get some downtime on the 
> system and the "debugfs.ocfs2 -f" command did the trick. I have taken 
> a snapshot of the lun while it was in the bad state so if I get some 
> time I will try and investigate it further.
>
> Srinivas, Is there anything you would like me to run on this snapshot 
> to get some diagnostics for you or will i need to have those traces 
> running when the issue initially occures ? In its current state 
> anything I delete from the snapshot causes orphans. Would that be 
> enough to get some traces out of it?
> Tim Hughes
>
> www.tradefair.com
>
> Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11 4AN
>
> The information in this e-mail and any attachment is confidential and 
> is intended only for the named recipient(s). The e-mail may not be 
> disclosed or used by any person other than the addressee, nor may it 
> be copied in any way. If you are not a named recipient please notify 
> the sender immediately and delete any copies of this message. Any 
> unauthorized copying, disclosure or distribution of the material in 
> this e-mail is strictly forbidden. Any view or opinions presented are 
> solely those of the author and do not necessarily represent those of 
> the company.
>
> ----- Original Message -----
> From: "Srinivas Eeda" <srinivas.eeda at oracle.com>
> To: "Tim Hughes" <tim.hughes at tradefair.com>
> Cc: ocfs2-users at oss.oracle.com
> Sent: Friday, 27 February, 2009 10:21:50 PM GMT +00:00 GMT Britain, 
> Ireland, Portugal
> Subject: Re: [Ocfs2-users] how do I remove ocfs2 orphaned files that 
> are taking up diskspace ?
>
> please run fsck.ocfs2 -f to clean the orphans.
>
> If you can reproduce it at will, can you please enable tracing
> debugfs.ocfs2 -l JOURNAL, SUPER, INODE, NAMEI, DLM_GLUE ENTRY EXIT allow
>
> and send us the messages files from all nodes.
>
> thanks,
> --Srini
>
> Tim Hughes wrote:
> > I am looking for a little help with some orphaned files that are 
> taking up diskspace.
> >
> > I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from a 
> three node ocfs2 cluster. The files appeared removed from the 
> filesystem but the results of a `df -h /var/lib/mysql` showed that no 
> disk space has been cleared. A `du -sh /var/lib/mysql` on the other 
> hand says that ~ 30GB was removed.
> >
> > <snip>
> > [root at host1 mysql]# du -sh /var/lib/mysql
> > 50G /var/lib/mysql
> > [root at host1 mysql]# df -h /var/lib/mysql/
> > Filesystem Size Used Avail Use% Mounted on
> > /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql
> > [root at host1 mysql]#
> > <snip>
> >
> > First thoughts were that mysql was holding the file descriptors open 
> but `lsof` showed nothing. After trying a few other things we 
> discovered that the files were orphans of ocfs2.
> >
> > <snip>
> > [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001" /dev/xvdb1
> > 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 .
> > 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 ..
> > 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14 0000000000103eba
> > 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb
> > 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14 0000000000103eb9
> > 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33 0000000000103ebd
> > 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37 0000000000103ebc
> > 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18 
> 00000000001236d9
> > 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10 
> 00000000001236da
> > 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29 
> 00000000001236db
> > 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37 
> 00000000001236dc
> > 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59 
> 00000000001236dd
> > 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36 
> 00000000001236de
> > 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05 
> 00000000001236df
> > 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18 
> 00000000001236e0
> > 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46 
> 00000000001236e1
> > 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06 
> 00000000001236e2
> > <snip>
> >
> > We have shutdown mysql and tried unmounting and mounting the ocfs2 
> file system from the node where we deleted the files. This made no 
> difference so we decided to take down the mysql cluster and unmounting 
> the ocfs2 filesystem from all nodes just incase something was for some 
> reason holding these files open that we couldn't find. This didn't 
> clear the space and the orphans were still there.
> >
> > Next we unmounted it all again and ran `fsck.ocfs2 /dev/xvdb1` on it 
> with the following results which indicated there was nothing wrong.
> >
> > <snip>
> > Checking OCFS2 filesystem in /dev/xvdb1:
> > label: db-store01
> > uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb
> > number of blocks: 26214047
> > bytes per block: 4096
> > number of clusters: 26214047
> > bytes per cluster: 4096
> > max slots: 16
> >
> > /dev/xvdb1 is clean. It will be checked after 20 additional mounts.
> > <snip>
> >
> > We are running :
> > ocfs2-tools-1.4.1-1.el5
> > ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5
> >
> > with kernel 2.6.18-92.1.10.el5xen on RHEL5
> >
> >
> > How do we reclaim this space ?
> >
> >
> >
> > Tim Hughes
> >
> > www.tradefair.com
> >
> > Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | 
> W11 4AN
> >
> > The information in this e-mail and any attachment is confidential 
> and is intended only for the named recipient(s). The e-mail may not be 
> disclosed or used by any person other than the addressee, nor may it 
> be copied in any way. If you are not a named recipient please notify 
> the sender immediately and delete any copies of this message. Any 
> unauthorized copying, disclosure or distribution of the material in 
> this e-mail is strictly forbidden. Any view or opinions presented are 
> solely those of the author and do not necessarily represent those of 
> the company.
> >
> > _______________________________________________
> > Ocfs2-users mailing list
> > Ocfs2-users at oss.oracle.com
> > http://oss.oracle.com/mailman/listinfo/ocfs2-users
> >  
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090310/bb5146b4/attachment-0001.html 