[Ocfs2-users] OCFS2 and Apache Problem

[Mark Fasheh]

On Tue, Nov 13, 2007 at 06:18:25PM -0800, Michael Moody wrote:
> It's a fairly sizable filesystem, with 10's of thousands of little files, 
> and thousands of directories.
>
> Also, given as it's an apache server node, the ls -l test would likely be 
> inconclusive.
>
> However, I forgot to mention it, my stupid mistake, I had this happen in 
> some of my directories:
>
> pba1 profilepic # ls -l 6*
> ls: cannot access 67705.jpg: Permission denied
> ls: cannot access 67705a.jpg: Permission denied
> ls: cannot access 67705orig.jpg: Permission denied
> ls: cannot access 67705t.jpg: Permission denied
> -rw-r--r-- 1 apache apache  17897 May  9  2007 60001.jpg
> -rw-r--r-- 1 apache apache  15096 May  9  2007 60001a.jpg
> -rw-r--r-- 1 apache apache  36963 May  9  2007 60001orig.jpg
>
> if I just do an ls -l, they show up like this:
>
> -rw-r--r-- 1 apache apache  14174 May 25 11:52 65963t.jpg
> -????????? ? ?      ?           ?            ? 67705.jpg
> -????????? ? ?      ?           ?            ? 67705a.jpg
> -????????? ? ?      ?           ?            ? 67705orig.jpg
> -????????? ? ?      ?           ?            ? 67705t.jpg
> -rw-r--r-- 1 apache apache  78527 Jun  5 12:00 70023.jpg
>
> So I think I actually found it:
> (and there may be a few other files like it)

Ok, so we found them then, great.


> I went into that directory, issued this command in one ssh session:
>
> watch -d -n .3 'dmesg | tail -n 20'
>
> Then in the other:
>
> ls -l
>
> I saw the new entries get added to dmesg/syslog everytime I ls -l'ed.
>
> Any ideas what causes/caused this, and how I can fix it without fsck? I 
> don't really care about these files, losing 4 jpgs isn't really so bad.

I can't think of anything that would cause it off the top of my head. Some
more information would help though.

Would you mind filing a bugzilla with the following info?

Other than the inode messages, did you get anything else ocfs2 related in
your logs? Possibly even from days ago? Is there anything in particular
about the usage of that directory which you can tell me? Do things get
renamed in it a lot, are there many unlinks, etc.

Run stat_sysdir.sh against the device:

http://oss.oracle.com/~smushran/.debug/scripts/stat_sysdir.sh

also, you can dump the corrupted directory. 1st, find it's inode number -
the command 'stat' can give you that from the shell (it's in the "Inode: "
field). Also, "ls -li" in the parent will print it.

Once you have the inode number, use debugfs.ocfs2 to dump the inode info via
stat:

echo "stat <inode #>" | debugfs.ocfs2 /dev/XXXX

then get a raw dump of the directory data:

echo "dump <inode #> /tmp/dirdata" | debugfs.ocfs2 /dev/XXXX

You might want to gzip up that dir data before uploading it.

Thanks,
	--Mark

--
Mark Fasheh
Senior Software Developer, Oracle
mark.fasheh at oracle.com