[Ocfs2-users] OCFS2 and Apache Problem

[Michael Moody]

I have opened a bug, yesterday, 
http://oss.oracle.com/bugzilla/show_bug.cgi?id=928

We were using ocfs2-tools-1.2.2 with kernel 2.6.20-gentoo-r8

I would like to do the backported fixes, I might be able to do that, but 
at this point, it's not something I can do immediately (next few days).
I have been given (via the bugzilla post) some tools, like scanlocks, 
etc, and I will use those the next time this behavior occurs, and see if 
I can get a little more information as to what's going on.

I mean "live" by that apache is running, on all nodes, and serving 
pages. Sometimes, rarely, writes occur to the same file, mostly logging 
files, but that is not what seems to cause the problem. Large uploads 
(video files), and changed include files (say, a php database config 
file) cause the apache processes to go to uninterruptible sleep states. 
I'd like to trigger it, and have time for information gathering, but as 
this is a very popular site, doing so is sometimes not an option.

When the behavior occurs, no processes are eating cpu at all, but are in 
uninterruptible sleep (D) states.

Output from  (on first node)

echo "stats -h" | debugfs.ocfs2 /dev/sdb2

       Revision: 0.90
        Mount Count: 0   Max Mount Count: 20
        State: 0   Errors: 0
        Check Interval: 0   Last Check: Fri Feb 23 02:43:43 2007
        Creator OS: 0
        Feature Compat: 0 None
        Feature Incompat: 0 None
        Feature RO compat: 0 None
        Root Blknum: 17   System Dir Blknum: 18
        First Cluster Group Blknum: 8
        Block Size Bits: 12   Cluster Size Bits: 15
        Max Node Slots: 10
        Label: www
        UUID: 0F98901B58E64D57B8A5556E8DC6DDC1

debugfs.ocfs2 1.2.6

I'm using ocfs2-tools 1.2.6 from ocfs2-tools project page.

The storage network a fiber channel SAN, with each machine using qlogic 
cards (qla2xxx drivers, ql2400 firmware latest from qlogic site), to a 
qlogic sanbox 5602 switch, and a proware (whitebox, same as jetstor) 16 
disk 12tb raid6 array.



I'll post this information in the bugzilla post.

Mark Fasheh wrote:
> Have you tried any of the backported fixes for 2.6.22? You can find them at:
>
> http://www.kernel.org/pub/linux/kernel/people/mfasheh/ocfs2/backports/
>
> If at all possible, I'd upgrade to the latest 2.6.22 stable kernel
> (2.6.22.10 at the moment), and apply the patches at:
>
> http://www.kernel.org/pub/linux/kernel/people/mfasheh/ocfs2/backports/2.6.22.6/
>
> That'd get at least those known issues out of the way.
>
> Btw, which kernel / tools were you using previous to the upgrade?
>
>   
> I'm not 100% clear on what you mean by "live"... Are the nodes all doing
> writes to the same file? That'd certainly incur a high locking overhead. It
> shouldn't hang unrelated processes on the nodes though.
>
> Could you file a bugzilla with the following information please:
>
> - What processes are eating the cpu (I guess some info from "top" on all the
>   nodes would do)
> - Attach your kernel config
> - File system options (the output from "echo stats -h | debugfs.ocfs2 /dev/XXX")
> - Describe your storage and network
> - Exact ocfs2-tools version
>
> Btw, feel free to put my e-mail address as CC in the bugzilla.
>
> Thanks,
> 	--Mark
>
> --
> Mark Fasheh
> Senior Software Developer, Oracle
> mark.fasheh at oracle.com
>   

-- 

Michael S. Moody
Systems Engineer
Global Systems Consulting
Direct: (650) 265-4154
Web: http://www.GlobalSystemsConsulting.com

Engineering Support: support at gsc.cc
Billing Support: billing at gsc.cc
Customer Support Portal:  http://my.gsc.cc


NOTICE - This message contains privileged and confidential information intended only for the use of the addressee named above. If you are not the intended recipient of this message, you are hereby notified that you must not disseminate, copy or take any action in reliance on it. If you have received this message in error, please immediately notify Global Systems Consulting, its subsidiaries or associates. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the view of Global Systems Consulting, its subsidiaries and associates.