<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
Tim,<br>
<br>
yes, it would be great if you can capture and provide me the following.<br>
a)run stat_sysdir.sh (<a class="moz-txt-link-freetext" href="http://oss.oracle.com/~seeda/misc/stat_sysdir.sh">http://oss.oracle.com/~seeda/misc/stat_sysdir.sh</a>)<br>
b) lsof on all nodes<br>
c)enable ocfs2 tracing on all nodes<br>
debugfs.ocfs2 -l ENTRY EXIT DLM DLM_MASTER DLM_RECOVERY SUPER JOURNAL
DLM_GLUE INODE VOTE DCACHE NAMEI allow<br>
d)rm the file<br>
e)make sure it is in the orphan dir<br>
f)run stat_sysdir.sh again<br>
g)lsof on all nodes<br>
h)disable tracing<br>
<br>
please upload outputs of stat_sysdir, lsof and messages files from all
nodes<br>
<br>
thanks,<br>
--Srini.<br>
<br>
Tim Hughes wrote:
<blockquote
cite="mid919271318.335561236702451859.JavaMail.root@t2mail01.t2.tradefair"
type="cite">
<style type="text/css">p { margin: 0; }</style>
<div
style="font-family: Times New Roman; font-size: 10pt; color: rgb(0, 0, 0);">Sorry
about the delay on this. I managed to get some downtime on the system
and the "debugfs.ocfs2 -f" command did the trick. I have taken a
snapshot of the lun while it was in the bad state so if I get some time
I will try and investigate it further. <br>
<br>
Srinivas, Is there anything you would like me to run on this snapshot
to get some diagnostics for you or will i need to have those traces
running when the issue initially occures ? In its current state
anything I delete from the snapshot causes orphans. Would that be
enough to get some traces out of it?<br>
Tim Hughes<br>
<br>
<a class="moz-txt-link-abbreviated" href="http://www.tradefair.com">www.tradefair.com</a><br>
<br>
Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London | W11
4AN<br>
<br>
The information in this e-mail and any attachment is confidential and
is intended only for the named recipient(s). The e-mail may not be
disclosed or used by any person other than the addressee, nor may it be
copied in any way. If you are not a named recipient please notify the
sender immediately and delete any copies of this message. Any
unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. Any view or opinions presented are
solely those of the author and do not necessarily represent those of
the company.<br>
<br>
----- Original Message -----<br>
From: "Srinivas Eeda" <a class="moz-txt-link-rfc2396E" href="mailto:srinivas.eeda@oracle.com"><srinivas.eeda@oracle.com></a><br>
To: "Tim Hughes" <a class="moz-txt-link-rfc2396E" href="mailto:tim.hughes@tradefair.com"><tim.hughes@tradefair.com></a><br>
Cc: <a class="moz-txt-link-abbreviated" href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br>
Sent: Friday, 27 February, 2009 10:21:50 PM GMT +00:00 GMT Britain,
Ireland, Portugal<br>
Subject: Re: [Ocfs2-users] how do I remove ocfs2 orphaned files that
are taking up diskspace ?<br>
<br>
please run fsck.ocfs2 -f to clean the orphans.<br>
<br>
If you can reproduce it at will, can you please enable tracing<br>
debugfs.ocfs2 -l JOURNAL, SUPER, INODE, NAMEI, DLM_GLUE ENTRY EXIT allow<br>
<br>
and send us the messages files from all nodes.<br>
<br>
thanks,<br>
--Srini<br>
<br>
Tim Hughes wrote:<br>
> I am looking for a little help with some orphaned files that are
taking up diskspace. <br>
><br>
> I deleted a approximately 30 x 1GB mysql-bin-XXXXX.log files from
a three node ocfs2 cluster. The files appeared removed from the
filesystem but the results of a `df -h /var/lib/mysql` showed that no
disk space has been cleared. A `du -sh /var/lib/mysql` on the other
hand says that ~ 30GB was removed. <br>
><br>
> <snip> <br>
> [root@host1 mysql]# du -sh /var/lib/mysql <br>
> 50G /var/lib/mysql <br>
> [root@host1 mysql]# df -h /var/lib/mysql/ <br>
> Filesystem Size Used Avail Use% Mounted on <br>
> /dev/xvdb1 100G 83G 18G 83% /var/lib/mysql <br>
> [root@host1 mysql]# <br>
> <snip> <br>
><br>
> First thoughts were that mysql was holding the file descriptors
open but `lsof` showed nothing. After trying a few other things we
discovered that the files were orphans of ocfs2. <br>
><br>
> <snip> <br>
> [root@ host1 ~]# debugfs.ocfs2 -R "ls -l //orphan_dir:0001"
/dev/xvdb1 <br>
> 13 drwxr-xr-x 2 0 0 4096 23-Feb-2009 21:54 . <br>
> 6 drwxr-xr-x 18 0 0 4096 20-Aug-2008 15:54 .. <br>
> 1064634 -rw-rw---- 0 100 101 1024 5-Sep-2008 14:14
0000000000103eba <br>
> 1064635 -rw-rw---- 0 100 101 0 5-Sep-2008 14:14 0000000000103ebb <br>
> 1064633 -rw-rw---- 0 100 101 8554 5-Sep-2008 14:14
0000000000103eb9 <br>
> 1064637 -rw-rw---- 0 100 101 114688 1-Oct-2008 12:33
0000000000103ebd <br>
> 1064636 -rw-rw---- 0 100 101 8844 1-Oct-2008 11:37
0000000000103ebc <br>
> 1193689 -rw-rw---- 0 100 101 1073742258 21-Jan-2009 15:18
00000000001236d9 <br>
> 1193690 -rw-rw---- 0 100 101 1073741942 21-Jan-2009 20:10
00000000001236da <br>
> 1193691 -rw-rw---- 0 100 101 1073743434 22-Jan-2009 14:29
00000000001236db <br>
> 1193692 -rw-rw---- 0 100 101 1073742303 22-Jan-2009 18:37
00000000001236dc <br>
> 1193693 -rw-rw---- 0 100 101 1073741875 23-Jan-2009 12:59
00000000001236dd <br>
> 1193694 -rw-rw---- 0 100 101 1073741973 23-Jan-2009 18:36
00000000001236de <br>
> 1193695 -rw-rw---- 0 100 101 1073742198 26-Jan-2009 14:05
00000000001236df <br>
> 1193696 -rw-rw---- 0 100 101 1073742221 26-Jan-2009 20:18
00000000001236e0 <br>
> 1193697 -rw-rw---- 0 100 101 1073742068 27-Jan-2009 14:46
00000000001236e1 <br>
> 1193698 -rw-rw---- 0 100 101 1005225645 28-Jan-2009 00:06
00000000001236e2 <br>
> <snip> <br>
><br>
> We have shutdown mysql and tried unmounting and mounting the ocfs2
file system from the node where we deleted the files. This made no
difference so we decided to take down the mysql cluster and unmounting
the ocfs2 filesystem from all nodes just incase something was for some
reason holding these files open that we couldn't find. This didn't
clear the space and the orphans were still there. <br>
><br>
> Next we unmounted it all again and ran `fsck.ocfs2 /dev/xvdb1` on
it with the following results which indicated there was nothing wrong. <br>
><br>
> <snip> <br>
> Checking OCFS2 filesystem in /dev/xvdb1: <br>
> label: db-store01 <br>
> uuid: 74 c1 14 6c af ee 4e 29 84 e4 c1 7a a8 cc 96 eb <br>
> number of blocks: 26214047 <br>
> bytes per block: 4096 <br>
> number of clusters: 26214047 <br>
> bytes per cluster: 4096 <br>
> max slots: 16 <br>
><br>
> /dev/xvdb1 is clean. It will be checked after 20 additional
mounts. <br>
> <snip> <br>
><br>
> We are running : <br>
> ocfs2-tools-1.4.1-1.el5 <br>
> ocfs2-2.6.18-92.1.10.el5xen-1.4.1-1.el5 <br>
><br>
> with kernel 2.6.18-92.1.10.el5xen on RHEL5 <br>
><br>
><br>
> How do we reclaim this space ? <br>
><br>
><br>
><br>
> Tim Hughes <br>
><br>
> <a class="moz-txt-link-abbreviated" href="http://www.tradefair.com">www.tradefair.com</a> <br>
><br>
> Tradefair | Level 2, Yellow Building | 1 Nicholas Road | London |
W11 4AN <br>
><br>
> The information in this e-mail and any attachment is confidential
and is intended only for the named recipient(s). The e-mail may not be
disclosed or used by any person other than the addressee, nor may it be
copied in any way. If you are not a named recipient please notify the
sender immediately and delete any copies of this message. Any
unauthorized copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. Any view or opinions presented are
solely those of the author and do not necessarily represent those of
the company. <br>
><br>
> _______________________________________________<br>
> Ocfs2-users mailing list<br>
> <a class="moz-txt-link-abbreviated" href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a><br>
> <a class="moz-txt-link-freetext" href="http://oss.oracle.com/mailman/listinfo/ocfs2-users">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a><br>
> <br>
<br>
</div>
</blockquote>
</body>
</html>