[Ocfs2-users] Linux kernel crash due to ocfs2

[Betzos Giorgos]

# rpm -q ocfs2-tools
ocfs2-tools-1.4.4-1.el5.ppc

On Wed, 2011-09-07 at 09:13 -0700, Sunil Mushran wrote:
> version of ocfs2-tools?
> 
> On 09/07/2011 09:10 AM, Betzos Giorgos wrote:
> > Hello,
> >
> > I tried what you suggested but here is what I got:
> >
> > # o2image /dev/mapper/mpath0 /files_shared/u02.o2image
> > *** glibc detected *** o2image: corrupted double-linked list: 0x10045000 ***
> > ======= Backtrace: =========
> > /lib/libc.so.6[0xfeb1ab4]
> > /lib/libc.so.6(cfree+0xc8)[0xfeb5b68]
> > o2image[0x10007bb0]
> > o2image[0x10002748]
> > o2image[0x10001f50]
> > o2image[0x10002334]
> > o2image[0x100026a0]
> > o2image[0x10001f50]
> > o2image[0x10002334]
> > o2image[0x100026a0]
> > o2image[0x1000358c]
> > o2image[0x10003e28]
> > /lib/libc.so.6[0xfe4dc60]
> > /lib/libc.so.6[0xfe4dea0]
> > ======= Memory map: ========
> > 00100000-00120000 r-xp 00100000 00:00 0                                  [vdso]
> > 0f550000-0f560000 r-xp 00000000 08:13 2881590                            /lib/libcom_err.so.2.1
> > 0f560000-0f570000 rw-p 00000000 08:13 2881590                            /lib/libcom_err.so.2.1
> > 0f900000-0f9c0000 r-xp 00000000 08:13 2881576                            /lib/libglib-2.0.so.0.1200.3
> > 0f9c0000-0f9d0000 rw-p 000b0000 08:13 2881576                            /lib/libglib-2.0.so.0.1200.3
> > 0fa40000-0fa50000 r-xp 00000000 08:13 2881575                            /lib/librt-2.5.so
> > 0fa50000-0fa60000 r--p 00000000 08:13 2881575                            /lib/librt-2.5.so
> > 0fa60000-0fa70000 rw-p 00010000 08:13 2881575                            /lib/librt-2.5.so
> > 0fce0000-0fd00000 r-xp 00000000 08:13 2881574                            /lib/libpthread-2.5.so
> > 0fd00000-0fd10000 r--p 00010000 08:13 2881574                            /lib/libpthread-2.5.so
> > 0fd10000-0fd20000 rw-p 00020000 08:13 2881574                            /lib/libpthread-2.5.so
> > 0fe30000-0ffa0000 r-xp 00000000 08:13 2881571                            /lib/libc-2.5.so
> > 0ffa0000-0ffb0000 r--p 00160000 08:13 2881571                            /lib/libc-2.5.so
> > 0ffb0000-0ffc0000 rw-p 00170000 08:13 2881571                            /lib/libc-2.5.so
> > 0ffc0000-0ffe0000 r-xp 00000000 08:13 2881570                            /lib/ld-2.5.so
> > 0ffe0000-0fff0000 r--p 00010000 08:13 2881570                            /lib/ld-2.5.so
> > 0fff0000-10000000 rw-p 00020000 08:13 2881570                            /lib/ld-2.5.so
> > 10000000-10020000 r-xp 00000000 08:13 15058799                           /sbin/o2image
> > 10020000-10030000 rw-p 00010000 08:13 15058799                           /sbin/o2image
> > 10030000-10060000 rwxp 10030000 00:00 0                                  [heap]
> > f7680000-f7ff0000 rw-p f7680000 00:00 0
> > ffc60000-ffdb0000 rw-p ffc60000 00:00 0                                  [stack]
> > Aborted (core dumped)
> >
> > I have the core file, if you need it.
> >
> > Here is some information about the fs in question.
> > It is used to store Oracle Archive Logs and also to store the rman backup of the DB
> > In the last crash case the fs became full while rman was running. Maybe we can estimate from
> > this the size of the write in that particular case. Oracle DB rman backup files are from 7 to 11Gb.
> > Maybe Oracle DataGuard was also using on the same fs.
> > After the crash, when we rebooted the servers, they would crash again. We then noticed that
> > the fs was full and we removed some unneeded files.
> >
> > The system has crashed a couple more times when the above conditions may not have been the same.
> >
> > Thanks,
> >
> > George
> >
> > ________________________________________
> > From: Sunil Mushran
> > Sent: Friday, September 02, 2011 8:24 PM
> > To: Betzos Giorgos
> > Cc: ocfs2-users at oss.oracle.com
> > Subject: Re: [Ocfs2-users] Linux kernel crash due to ocfs2
> >
> > Can you provide me with the o2image. It includes the entire fs metadata.
> > The size of the image file depends on the number of files/dirs.
> >
> > # o2image /dev/sdX  /path/to/image/file
> >
> > So the error is clear. We have underestimated the amount of credits
> > (num of blocks that need to be dirtied in that transaction). This is the most
> > common write path in the fs and thus hit heavily. So I am surprised by this.
> >
> > One way to fix it is by reproducing it inhouse. And having the image will allow
> > us to mount the fs and reproduce the issue. Do you know the size of the write?
> >
> > On 09/02/2011 07:23 AM, Betzos Giorgos wrote:
> >> Hello,
> >>
> >> we have a pair of IBM P570 servers running RHEL5.2
> >> kernel 2.6.18-92.el5.ppc64
> >> We have Oracle RAC on ocfs2 storage
> >> ocfs2 is 1.4.7-1 for the above kernel (downloaded from oracle oss site)
> >>
> >> Recently both servers have been crashing with the following error:
> >>
> >> Assertion failure in journal_dirty_metadata() at
> >> fs/jbd/transaction.c:1130: "handle->h_buffer_credits>   0"
> >> kernel BUG in journal_dirty_metadata at fs/jbd/transaction.c:1130!
> >>
> >> We get some kind of kernel debug prompt.
> >>
> >> the stack is as follows:
> >>
> >> .ocfs2_journal_dirty+0x78/0x13c [ocfs2]
> >> .ocfs2_search_chain+0x131c/0x165c [ocfs2]
> >> .ocfs2_claim_suballoc_bits+0xadc/0xd94 [ocfs2]
> >> .__ocfs2_claim_clusters+0x1b0/0x348 [ocfs2]
> >> .ocf2_do_extend_allocation+0x1f8/0x5b4 [ocfs2]
> >> .ocfs2_write_cluster_by_desc+0x128/0x850 [ocfs2]
> >> .ocfs2_write_begin_nolock+0xdc0/0xfbc [ocfs2]
> >> .ocfs2_write_begin+0x124/0x224 [ocfs2]
> >> .ocfs2_file_aio_write+0x6a4/0xb40 [ocfs2]
> >> .aio_pwrite+0x50/0xb4
> >> .aio_run_iocb+0x140/0x214
> >> .io_submit_one+0x2fc/0x3a8
> >> .sys_io_submit+0xd0/0x17c
> >> syscall_exit+0x0/0x40
> >>
> >> In the last crash case, the file system was full.
> >>
> >> Any clues?
> >>
> >> There seems to be a ocfs2 kernel patch some time ago for the 2.6.20.2
> >> kernel that fixed some journal credits updates.
> >>
> >> Is this another bug?
> >>
> >> Any help will be greatly appreciated, because this is a production
> >> system.
> >>
> >> Thanks,
> >>
> >> George
>