[Ocfs2-users] OCFS2 performance and debug help

[Jay V]

On Mon, Apr 9, 2012 at 9:38 AM, Joel Becker <jlbec at evilplan.org> wrote:
> On Sun, Apr 08, 2012 at 10:54:52AM -0700, Jay V wrote:
>> I have a similar blocking/hanging/stall issue. On Oracle 6.2/x64. We
>> are running OCFS2 on 3 partitions about 14-15TB each. One of the
>> partitions has been running extremely slowly too. They are running on
>> the same hardware-- LSI HW Raid Cards and Enterprise Drives. I am
>> running over drbd and nfsd.
>
>        How do drbd and nfsd fit in your scheme?  nfsd is on top of
> ocfs2, right?  You're exporting ocfs2 volumes via NFS?  Can you think of
> any differences between the slow partition and the fast ones?
>

Oh I miss wrote. It's a basic setup. I'm exporting NFS to the clients.
There are no differences I can think of of this partition that the
other 2, except it is more full at 83% vs the others at 76% and 74%.

>> I am getting really slow performance in writes. The process
>> [jbd2-drbd-18] seems to be stuck for a long length of time (about 2
>> minutes) before it finally commits. This jbd2-drbd2 prevents any other
>> writes from happening which stalls the system.
>
>        I'm worried that drbd is doing things slowly and getting in your
> way.  But given the slowdown you are replying to, that might not be the
> only issue.
>

It's not DRBD. Even when I disconnect DRBD, so it's running on just 1
node, it has the performance issues.

>> This would be from "ps auxr"
>> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
>> root      6374  0.0  0.0      0     0 ?        D    Apr03   0:28 [jbd2/drbd2-18]
>> root      6876  0.0  0.0      0     0 ?        D    Apr06   0:11 [nfsd]
>> root      6884  0.0  0.0      0     0 ?        D    Apr06   0:09 [nfsd]
>> root      6999  0.0  0.0      0     0 ?        D    Apr06   0:45 [nfsd]
>> root      7046  0.0  0.0      0     0 ?        D    Apr06   0:09 [nfsd]
>> root      7053  0.0  0.0      0     0 ?        D    Apr06   0:09 [nfsd]
>> root      7054  0.0  0.0      0     0 ?        D    Apr06   0:08 [nfsd]
>>
>> Running "scan_locks2" shows nothing. Nothing is held up locking wise.
>> It seems to happen more with files copying about 1MB or larger. It
>> only happens for me on my second partition, but not the other 2. It
>> seems to super slow in writes. Reads are fast.
>>
>> I hope to find a solution quickly too. I wonder it is because we have
>> very large partitions.
>
>        It shouldn't be because of large partitions.  How long have your
> partitions been around?  How full are they?
>

The partitions have been around about 1 1/2 years. It has been running
well for so long. It's not the hard drives too. Everything is optimal,
just slow on writes. Reads are fast.

I opened up bug 1349, it gives more information:
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1349

Thanks for your help,
Jay


> Joel
>
> --
>
>  One look at the From:
>  understanding has blossomed
>  .procmailrc grows
>        - Alexander Viro
>
>                        http://www.jlbec.org/
>                        jlbec at evilplan.org