[Ocfs2-users] ocfs2 - Kernel panic on many write/read from both servers
Sérgio Surkamp
sergio at gruposinternet.com.br
Wed Dec 7 04:31:58 PST 2011
Hi,
I have seen a similar problem, can you confirm if you have the same
symptoms?
The performance suddenly drop from the normal performance (in our
cluster, about 30Mbytes/s), reading/writing, to a few Kbytes/s (about
200Kbytes/s), reading only, for a while, and as sudden as it started, it
backs to the normal read/write performance, cycling randomly. When the
"read only" occurs on one node, the other shows only the heartbeat
activity (about 2 io's per 2 seconds) until the first back to normal
and vice-versa.
The servers are running e-mail application (IMAP/POP/SMTP -- Maildir
format) with more than 20.000 user, so they are constantly creating,
removing and moving files.
Dumping the processes in D state when the server is in "constant few
kbytes read only state", they look like:
node#0:
10739 D imapd ocfs2_lookup_lock_orphan_dir
11658 D imapd ocfs2_reserve_suballoc_bits
12326 D imapd ocfs2_lookup_lock_orphan_dir
12330 D pop3d lock_rename
12351 D imapd ocfs2_lookup_lock_orphan_dir
12357 D imapd ocfs2_lookup_lock_orphan_dir
12359 D imapd unlinkat
12381 D imapd ocfs2_lookup_lock_orphan_dir
12498 D deliverquota ocfs2_wait_for_mask
12710 D pop3d ocfs2_reserve_suballoc_bits
12712 D imapd unlinkat
12726 D imapd ocfs2_reserve_suballoc_bits
12730 D imapd unlinkat
12736 D imapd ocfs2_reserve_suballoc_bits
12738 D imapd unlinkat
12749 D pop3d lock_rename
12891 D pop3d ocfs2_reserve_suballoc_bits
12971 D pop3d mutex_fastpath_lock_retval
12985 D pop3d lock_rename
13006 D deliverquota ocfs2_reserve_suballoc_bits
13061 D pop3d lock_rename
13117 D pop3d lock_rename
[-- suppressed --]
100+ processes in D state
node#1:
24428 D deliverquota ocfs2_wait_for_mask
Some stacktraces from the processes:
Call Trace:
[<ffffffff81437e31>] __mutex_lock_common+0x12f/0x1a1
[<ffffffff81437ef2>] __mutex_lock_slowpath+0x19/0x1b
[<ffffffff81437f5b>] mutex_lock+0x23/0x3a
[<ffffffffa065ba1f>] ocfs2_lookup_lock_orphan_dir+0xb8/0x18a [ocfs2]
[<ffffffffa065c7d5>] ocfs2_prepare_orphan_dir+0x3f/0x229 [ocfs2]
[<ffffffffa0660bab>] ocfs2_unlink+0x523/0xa81 [ocfs2]
[<ffffffff810425b3>] ? need_resched+0x23/0x2d
[<ffffffff810425cb>] ? should_resched+0xe/0x2f
[<ffffffff810425cb>] ? should_resched+0xe/0x2f
[<ffffffff8116324d>] ? dquot_initialize+0x126/0x13d
[<ffffffff810425b3>] ? need_resched+0x23/0x2d
[<ffffffff81122c0c>] vfs_unlink+0x82/0xd1
[<ffffffff81124bcc>] do_unlinkat+0xc6/0x178
[<ffffffff8112186b>] ? path_put+0x22/0x27
[<ffffffff810a7d03>] ? audit_syscall_entry+0x103/0x12f
[<ffffffff81124c94>] sys_unlink+0x16/0x18
[<ffffffff81011db2>] system_call_fastpath+0x16/0x1b
Call Trace:
[<ffffffff81437e31>] __mutex_lock_common+0x12f/0x1a1
[<ffffffffa0633682>] ? ocfs2_match+0x2c/0x3a [ocfs2]
[<ffffffff81437ef2>] __mutex_lock_slowpath+0x19/0x1b
[<ffffffff81437f5b>] mutex_lock+0x23/0x3a
[<ffffffffa0676a82>] ocfs2_reserve_suballoc_bits+0x11a/0x499 [ocfs2]
[<ffffffffa0678b4c>] ocfs2_reserve_new_inode+0x134/0x37a [ocfs2]
[<ffffffffa065d409>] ocfs2_mknod+0x2d4/0xf26 [ocfs2]
[<ffffffffa063d02c>] ? ocfs2_should_refresh_lock_res+0x8f/0x1ad [ocfs2]
[<ffffffffa0653cf6>] ? ocfs2_wait_for_recovery+0x1a/0x8f [ocfs2]
[<ffffffff81437f4e>] ? mutex_lock+0x16/0x3a
[<ffffffffa065e0fd>] ocfs2_create+0xa2/0x10a [ocfs2]
[<ffffffff8112268f>] vfs_create+0x7e/0x9d
[<ffffffff81125794>] do_filp_open+0x302/0x92d
[<ffffffff810425cb>] ? should_resched+0xe/0x2f
[<ffffffff81437731>] ? _cond_resched+0xe/0x22
[<ffffffff81238109>] ? might_fault+0xe/0x10
[<ffffffff812381f3>] ? __strncpy_from_user+0x20/0x4a
[<ffffffff81114bc8>] do_sys_open+0x62/0x109
[<ffffffff81114ca2>] sys_open+0x20/0x22
[<ffffffff81011db2>] system_call_fastpath+0x16/0x1b
Checking the bz, these two bugs seems to have similar behavior:
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1281
http://oss.oracle.com/bugzilla/show_bug.cgi?id=1300
on the mailing list archive, this thread also shows similar behavior:
http://www.mail-archive.com/ocfs2-users@oss.oracle.com/msg02509.html
The cluster is formed by two Dell PE 1950 with 8G ram, attached via
2Gbit FC to a Dell EMC AX/100 storage. The network between them is
running at 1Gbit.
Running CenOS 5.7, OCFS2 1.6.4 and ULEK 2.6.32-100.0.19.el5.
Tests so far:
* We have changed mount option data from ordered to writeback -- no
success;
* We have added mount option localalloc=16 -- no success;
* We have turned off group and user quota support -- no success;
* Rebooted the servers (to test with everything fresh) -- no success;
* Mounted the filesystem only in one node -- success;
The problem does not show when the filesystem is mounted on only one
node, so we are currently working around by exporting the filesystem
via NFS, leading me to conclude that the problem is inside the cluster
stack.
We have checked logs, debugs, traces trying to pinpoint the problem,
but with no success.
In our case, node#0 has heavier I/O load than node#1, could it trigger
something?
The filesystem is about 94% full (751G of 803G).
Regards,
Sérgio
Em Wed, 7 Dec 2011 12:02:33 +0100
Eduardo Diaz - Gmail <ediazrod at gmail.com> escreveu:
> I make the same test that you...
>
> and.. error too :-(
>
> 75120.532071] INFO: task o2quot/0:3714 blocked for more than 120
> seconds. [75120.532091] "echo 0
> > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [75120.532108] o2quot/0 D f5939dc8 0 3714 2 0x00000000
> [75120.532113] f6bc4c80 00000046 c1441e20 f5939dc8 f6bc4e3c c1441e20
> c1441e20 f9bbdbf2
> [75120.532121] f6bc4e3c c3a08e20 00000000 c33d1f43 0000442f f9a7c716
> f6bc4c80 f6bc4c80
> [75120.532128] eedd5b7c f6bc4e3c f6bd6414 f6bd643c 00000004 00000000
> 011cd9ce f6bd6630
> [75120.532136] Call Trace:
> [75120.532154] [<f9bbdbf2>] ?
> ocfs2_metadata_cache_io_unlock+0x11/0x12 [ocfs2] [75120.532163]
> [<f9a7c716>] ? start_this_handle+0x2fb/0x37e [jbd2] [75120.532171]
> [<c127f4d7>] ? __mutex_lock_common+0xe8/0x13b [75120.532176]
> [<c127f539>] ? __mutex_lock_slowpath+0xf/0x11 [75120.532180]
> [<c127f5ca>] ? mutex_lock+0x17/0x24 [75120.532184] [<c127f5ca>] ?
> mutex_lock+0x17/0x24 [75120.532198] [<f9bc3e91>] ?
> ocfs2_sync_dquot_helper+0x166/0x2c5 [ocfs2] [75120.532204]
> [<c10ec21d>] ? dquot_scan_active+0x63/0xab [75120.532217]
> [<f9bc3d2b>] ? ocfs2_sync_dquot_helper+0x0/0x2c5 [ocfs2]
> [75120.532231] [<f9bc325b>] ? qsync_work_fn+0x23/0x3b [ocfs2]
> [75120.532236] [<c1047917>] ? worker_thread+0x141/0x1bd
> [75120.532249] [<f9bc3238>] ? qsync_work_fn+0x0/0x3b [ocfs2]
> [75120.532254] [<c104a65a>] ? autoremove_wake_function+0x0/0x2d
> [75120.532258] [<c10477d6>] ? worker_thread+0x0/0x1bd
> [75120.532262] [<c104a428>] ? kthread+0x61/0x66 [75120.532266]
> [<c104a3c7>] ? kthread+0x0/0x66 [75120.532271] [<c1008d87>] ?
> kernel_thread_helper+0x7/0x10 [75120.532276] INFO: task
> jbd2/drbd0-21:3723 blocked for more than 120 seconds.
>
> On Wed, Dec 7, 2011 at 11:42 AM, Eduardo Diaz - Gmail
> <ediazrod at gmail.com> wrote:
> > You can make a fsck.ocfs2 to see if the filesystem are broken?
> >
> > On Wed, Dec 7, 2011 at 11:30 AM, Marek Krolikowski
> > <admin at wset.edu.pl> wrote:
> >> Hello
> >> I use: sys-fs/ocfs2-tools-1.6.4 and create file system with all
> >> features: mkfs.ocfs2 -N 2 -L MAIL
> >> --fs-feature-level=max-features /dev/dm-0 and after this got
> >> kernel panic :(
> >>
> >>
> >>
> >> -----Oryginalna wiadomość----- From: Eduardo Diaz - Gmail
> >> Sent: Wednesday, December 07, 2011 11:08 AM
> >> To: Marek Krolikowski
> >>
> >> Cc: ocfs2-users at oss.oracle.com
> >> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many write/read
> >> from both servers
> >>
> >> Try to use other filesystem for example xfs and make a full test.
> >>
> >> Try create a one node cluster and filesystem and make tests..
> >> (fileindex can be the problem)
> >>
> >> make a fsck to the filesystem
> >>
> >> Try upgrade ocfs2 to last version and use the max features, only
> >> has two nodes?..
> >>
> >> I will do. make a backup, create a new filesystem with all features
> >> that you need and make mkfs. the cluster only with the number of
> >> the nodes that you will use.
> >>
> >> Restore de data.
> >>
> >> Make extensive test for a week before put in production :-)..
> >>
> >> On Tue, Dec 6, 2011 at 2:04 PM, Marek Krolikowski
> >> <admin at wset.edu.pl> wrote:
> >>>
> >>> hey m8
> >>> Like i say i am not expert too but when i use ext3 and write/read
> >>> working with np.
> >>>
> >>>
> >>> -----Oryginalna wiadomość----- From: Eduardo Diaz - Gmail
> >>> Sent: Tuesday, December 06, 2011 3:06 AM
> >>> To: Marek Królikowski
> >>> Cc: ocfs2-users at oss.oracle.com
> >>> Subject: Re: [Ocfs2-users] ocfs2 - Kernel panic on many
> >>> write/read from both
> >>> servers
> >>>
> >>>
> >>> I am not a expert, but you have a problem in you EMC system
> >>> (multipath system) or drivers.
> >>>
> >>> Did you test this before put in production? or test this NAS with
> >>> other filesystem, xfs for example?.
> >>>
> >>> As I read I see" that hung_task_timeout_secs" some task wait more
> >>> that 120 seg may be a problem of you EMC/Fiber/cable problem...
> >>>
> >>> 2011/12/4 Marek Królikowski <admin at wset.edu.pl>:
> >>>>
> >>>>
> >>>> I do for all night tests with write/read files from ocfs2 on
> >>>> both servers something like this:
> >>>> On MAIL1 server:
> >>>> #!/bin/bash
> >>>> while true
> >>>> do
> >>>> rm -rf /mnt/EMC/MAIL1
> >>>> mkdir /mnt/EMC/MAIL1
> >>>> cp -r /usr /mnt/EMC/MAIL1
> >>>> rm -rf /mnt/EMC/MAIL1
> >>>> done;
> >>>> On MAIL2 server:
> >>>> #!/bin/bash
> >>>> while true
> >>>> do
> >>>> rm -rf /mnt/EMC/MAIL2
> >>>> mkdir /mnt/EMC/MAIL2
> >>>> cp -r /usr /mnt/EMC/MAIL2
> >>>> rm -rf /mnt/EMC/MAIL2
> >>>> done;
> >>>>
> >>>> Today i check logs and see:
> >>>> o2dlm: Node 1 joins domain EAC7942B71964050AE2046D3F0CDD7B2
> >>>> o2dlm: Nodes in domain EAC7942B71964050AE2046D3F0CDD7B2: 0 1
> >>>> (rm,26136,0):ocfs2_unlink:953 ERROR: status = -2
> >>>> (touch,26137,0):ocfs2_check_dir_for_entry:2120 ERROR: status =
> >>>> -17 (touch,26137,0):ocfs2_mknod:461 ERROR: status = -17
> >>>> (touch,26137,0):ocfs2_create:631 ERROR: status = -17
> >>>> (rm,26142,0):ocfs2_unlink:953 ERROR: status = -2
> >>>> INFO: task kworker/u:2:20246 blocked for more than 120 seconds.
> >>>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
> >>>> message. kworker/u:2 D ffff88107f4525c0 0 20246 2
> >>>> 0x00000000 ffff880b730b57d0 0000000000000046 ffff8810201297d0
> >>>> 00000000000125c0 ffff880f5a399fd8 00000000000125c0
> >>>> 00000000000125c0 00000000000125c0 ffff880f5a398000
> >>>> 00000000000125c0 ffff880f5a399fd8 00000000000125c0 Call Trace:
> >>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> >>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> >>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> >>>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
> >>>> [ocfs2] [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
> >>>> [ocfs2] [<ffffffffa0919a10>] ?
> >>>> ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> >>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> >>>> [<ffffffff81145900>] ? evict+0x70/0x140 [<ffffffffa0919322>] ?
> >>>> __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
> >>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> >>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> >>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0 [<ffffffff8148bb24>] ?
> >>>> kernel_thread_helper+0x4/0x10 [<ffffffff81073f10>] ?
> >>>> kthread_worker_fn+0x1a0/0x1a0 [<ffffffff8148bb20>] ?
> >>>> gs_change+0x13/0x13 INFO: task rm:5192 blocked for more than 120
> >>>> seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>>> disables this message. rm D ffff88107f2725c0 0
> >>>> 5192 16338 0x00000000 ffff881014ccb040 0000000000000082
> >>>> ffff8810206b8040 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 00000000000125c0 00000000000125c0
> >>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 Call Trace: [<ffffffff8148148d>] ?
> >>>> schedule_timeout+0x1ed/0x2e0 [<ffffffffa0886162>] ?
> >>>> dlmconvert_master+0xe2/0x190 [ocfs2_dlm] [<ffffffffa08878bf>] ?
> >>>> dlmlock+0x7f/0xb70 [ocfs2_dlm] [<ffffffff81480e0a>] ?
> >>>> wait_for_common+0x13a/0x190 [<ffffffff8104bc50>] ?
> >>>> try_to_wake_up+0x280/0x280 [<ffffffffa0928a38>] ?
> >>>> __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
> >>>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
> >>>> [ocfs2] [<ffffffffa0943c1b>] ?
> >>>> ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
> >>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
> >>>> [ocfs2] [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> >>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> >>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> >>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> >>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> >>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> >>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> >>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> >>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b INFO: task
> >>>> kworker/u:2:20246 blocked for more than 120 seconds. "echo 0
> >>>> > /proc/sys/kernel/hung_task_timeout_secs" disables this
> >>>> > message. kworker/u:2 D ffff88107f4525c0 0 20246 2
> >>>> > 0x00000000 ffff880b730b57d0 0000000000000046 ffff8810201297d0
> >>>> > 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
> >>>> 00000000000125c0 ffff880f5a398000 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 Call Trace:
> >>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> >>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> >>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> >>>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
> >>>> [ocfs2] [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
> >>>> [ocfs2] [<ffffffffa0919a10>] ?
> >>>> ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> >>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> >>>> [<ffffffff81145900>] ? evict+0x70/0x140 [<ffffffffa0919322>] ?
> >>>> __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
> >>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> >>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> >>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0 [<ffffffff8148bb24>] ?
> >>>> kernel_thread_helper+0x4/0x10 [<ffffffff81073f10>] ?
> >>>> kthread_worker_fn+0x1a0/0x1a0 [<ffffffff8148bb20>] ?
> >>>> gs_change+0x13/0x13 INFO: task rm:5192 blocked for more than 120
> >>>> seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>>> disables this message. rm D ffff88107f2725c0 0
> >>>> 5192 16338 0x00000000 ffff881014ccb040 0000000000000082
> >>>> ffff8810206b8040 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 00000000000125c0 00000000000125c0
> >>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 Call Trace: [<ffffffff8148148d>] ?
> >>>> schedule_timeout+0x1ed/0x2e0 [<ffffffffa0886162>] ?
> >>>> dlmconvert_master+0xe2/0x190 [ocfs2_dlm] [<ffffffffa08878bf>] ?
> >>>> dlmlock+0x7f/0xb70 [ocfs2_dlm] [<ffffffff81480e0a>] ?
> >>>> wait_for_common+0x13a/0x190 [<ffffffff8104bc50>] ?
> >>>> try_to_wake_up+0x280/0x280 [<ffffffffa0928a38>] ?
> >>>> __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
> >>>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
> >>>> [ocfs2] [<ffffffffa0943c1b>] ?
> >>>> ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
> >>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
> >>>> [ocfs2] [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> >>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> >>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> >>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> >>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> >>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> >>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> >>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> >>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b INFO: task
> >>>> kworker/u:2:20246 blocked for more than 120 seconds. "echo 0
> >>>> > /proc/sys/kernel/hung_task_timeout_secs" disables this
> >>>> > message. kworker/u:2 D ffff88107f4525c0 0 20246 2
> >>>> > 0x00000000 ffff880b730b57d0 0000000000000046 ffff8810201297d0
> >>>> > 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
> >>>> 00000000000125c0 ffff880f5a398000 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 Call Trace:
> >>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> >>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> >>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> >>>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
> >>>> [ocfs2] [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
> >>>> [ocfs2] [<ffffffffa0919a10>] ?
> >>>> ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> >>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> >>>> [<ffffffff81145900>] ? evict+0x70/0x140 [<ffffffffa0919322>] ?
> >>>> __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
> >>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> >>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> >>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0 [<ffffffff8148bb24>] ?
> >>>> kernel_thread_helper+0x4/0x10 [<ffffffff81073f10>] ?
> >>>> kthread_worker_fn+0x1a0/0x1a0 [<ffffffff8148bb20>] ?
> >>>> gs_change+0x13/0x13 INFO: task rm:5192 blocked for more than 120
> >>>> seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>>> disables this message. rm D ffff88107f2725c0 0
> >>>> 5192 16338 0x00000000 ffff881014ccb040 0000000000000082
> >>>> ffff8810206b8040 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 00000000000125c0 00000000000125c0
> >>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 Call Trace: [<ffffffff8148148d>] ?
> >>>> schedule_timeout+0x1ed/0x2e0 [<ffffffffa0886162>] ?
> >>>> dlmconvert_master+0xe2/0x190 [ocfs2_dlm] [<ffffffffa08878bf>] ?
> >>>> dlmlock+0x7f/0xb70 [ocfs2_dlm] [<ffffffff81480e0a>] ?
> >>>> wait_for_common+0x13a/0x190 [<ffffffff8104bc50>] ?
> >>>> try_to_wake_up+0x280/0x280 [<ffffffffa0928a38>] ?
> >>>> __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
> >>>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
> >>>> [ocfs2] [<ffffffffa0943c1b>] ?
> >>>> ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
> >>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
> >>>> [ocfs2] [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> >>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> >>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> >>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> >>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> >>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> >>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> >>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> >>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b INFO: task
> >>>> kworker/u:2:20246 blocked for more than 120 seconds. "echo 0
> >>>> > /proc/sys/kernel/hung_task_timeout_secs" disables this
> >>>> > message. kworker/u:2 D ffff88107f4525c0 0 20246 2
> >>>> > 0x00000000 ffff880b730b57d0 0000000000000046 ffff8810201297d0
> >>>> > 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
> >>>> 00000000000125c0 ffff880f5a398000 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 Call Trace:
> >>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> >>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> >>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> >>>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
> >>>> [ocfs2] [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
> >>>> [ocfs2] [<ffffffffa0919a10>] ?
> >>>> ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> >>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> >>>> [<ffffffff81145900>] ? evict+0x70/0x140 [<ffffffffa0919322>] ?
> >>>> __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
> >>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> >>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> >>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0 [<ffffffff8148bb24>] ?
> >>>> kernel_thread_helper+0x4/0x10 [<ffffffff81073f10>] ?
> >>>> kthread_worker_fn+0x1a0/0x1a0 [<ffffffff8148bb20>] ?
> >>>> gs_change+0x13/0x13 INFO: task rm:5192 blocked for more than 120
> >>>> seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>>> disables this message. rm D ffff88107f2725c0 0
> >>>> 5192 16338 0x00000000 ffff881014ccb040 0000000000000082
> >>>> ffff8810206b8040 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 00000000000125c0 00000000000125c0
> >>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 Call Trace: [<ffffffff8148148d>] ?
> >>>> schedule_timeout+0x1ed/0x2e0 [<ffffffffa0886162>] ?
> >>>> dlmconvert_master+0xe2/0x190 [ocfs2_dlm] [<ffffffffa08878bf>] ?
> >>>> dlmlock+0x7f/0xb70 [ocfs2_dlm] [<ffffffff81480e0a>] ?
> >>>> wait_for_common+0x13a/0x190 [<ffffffff8104bc50>] ?
> >>>> try_to_wake_up+0x280/0x280 [<ffffffffa0928a38>] ?
> >>>> __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
> >>>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
> >>>> [ocfs2] [<ffffffffa0943c1b>] ?
> >>>> ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
> >>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
> >>>> [ocfs2] [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> >>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> >>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> >>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> >>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> >>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> >>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> >>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> >>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b INFO: task
> >>>> kworker/u:2:20246 blocked for more than 120 seconds. "echo 0
> >>>> > /proc/sys/kernel/hung_task_timeout_secs" disables this
> >>>> > message. kworker/u:2 D ffff88107f4525c0 0 20246 2
> >>>> > 0x00000000 ffff880b730b57d0 0000000000000046 ffff8810201297d0
> >>>> > 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 00000000000125c0
> >>>> 00000000000125c0 ffff880f5a398000 00000000000125c0
> >>>> ffff880f5a399fd8 00000000000125c0 Call Trace:
> >>>> [<ffffffff81481b71>] ? __mutex_lock_slowpath+0xd1/0x140
> >>>> [<ffffffff814818d3>] ? mutex_lock+0x23/0x40
> >>>> [<ffffffffa0937d95>] ? ocfs2_wipe_inode+0x105/0x690 [ocfs2]
> >>>> [<ffffffffa0935cfb>] ? ocfs2_query_inode_wipe.clone.9+0xcb/0x370
> >>>> [ocfs2] [<ffffffffa09385a4>] ? ocfs2_delete_inode+0x284/0x3f0
> >>>> [ocfs2] [<ffffffffa0919a10>] ?
> >>>> ocfs2_dentry_attach_lock+0x5a0/0x5a0 [ocfs2]
> >>>> [<ffffffffa093872e>] ? ocfs2_evict_inode+0x1e/0x50 [ocfs2]
> >>>> [<ffffffff81145900>] ? evict+0x70/0x140 [<ffffffffa0919322>] ?
> >>>> __ocfs2_drop_dl_inodes.clone.2+0x32/0x60 [ocfs2]
> >>>> [<ffffffffa0919a39>] ? ocfs2_drop_dl_inodes+0x29/0x90 [ocfs2]
> >>>> [<ffffffff8106e56f>] ? process_one_work+0x11f/0x440
> >>>> [<ffffffff8106f279>] ? worker_thread+0x159/0x330
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff8106f120>] ? manage_workers.clone.21+0x120/0x120
> >>>> [<ffffffff81073fa6>] ? kthread+0x96/0xa0 [<ffffffff8148bb24>] ?
> >>>> kernel_thread_helper+0x4/0x10 [<ffffffff81073f10>] ?
> >>>> kthread_worker_fn+0x1a0/0x1a0 [<ffffffff8148bb20>] ?
> >>>> gs_change+0x13/0x13 INFO: task rm:5192 blocked for more than 120
> >>>> seconds. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>>> disables this message. rm D ffff88107f2725c0 0
> >>>> 5192 16338 0x00000000 ffff881014ccb040 0000000000000082
> >>>> ffff8810206b8040 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 00000000000125c0 00000000000125c0
> >>>> ffff8804d7696000 00000000000125c0 ffff8804d7697fd8
> >>>> 00000000000125c0 Call Trace: [<ffffffff8148148d>] ?
> >>>> schedule_timeout+0x1ed/0x2e0 [<ffffffffa0886162>] ?
> >>>> dlmconvert_master+0xe2/0x190 [ocfs2_dlm] [<ffffffffa08878bf>] ?
> >>>> dlmlock+0x7f/0xb70 [ocfs2_dlm] [<ffffffff81480e0a>] ?
> >>>> wait_for_common+0x13a/0x190 [<ffffffff8104bc50>] ?
> >>>> try_to_wake_up+0x280/0x280 [<ffffffffa0928a38>] ?
> >>>> __ocfs2_cluster_lock.clone.21+0x1d8/0x6b0 [ocfs2]
> >>>> [<ffffffffa0928fcc>] ? ocfs2_inode_lock_full_nested+0xbc/0x490
> >>>> [ocfs2] [<ffffffffa0943c1b>] ?
> >>>> ocfs2_lookup_lock_orphan_dir+0x6b/0x1b0 [ocfs2]
> >>>> [<ffffffffa09454ba>] ? ocfs2_prepare_orphan_dir+0x4a/0x280
> >>>> [ocfs2] [<ffffffffa094616f>] ? ocfs2_unlink+0x6ef/0xb90 [ocfs2]
> >>>> [<ffffffff811b35a9>] ? may_link.clone.22+0xd9/0x170
> >>>> [<ffffffff8113aa58>] ? vfs_unlink+0x98/0x100
> >>>> [<ffffffff8113ac41>] ? do_unlinkat+0x181/0x1b0
> >>>> [<ffffffff8113e7cd>] ? vfs_readdir+0x9d/0xe0
> >>>> [<ffffffff811653d8>] ? fsnotify_find_inode_mark+0x28/0x40
> >>>> [<ffffffff81166324>] ? dnotify_flush+0x54/0x110
> >>>> [<ffffffff8112b07f>] ? filp_close+0x5f/0x90
> >>>> [<ffffffff8148aa12>] ? system_call_fastpath+0x16/0x1b
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> Ocfs2-users mailing list
> >>>> Ocfs2-users at oss.oracle.com
> >>>> http://oss.oracle.com/mailman/listinfo/ocfs2-users
> >>>
> >>>
> >>>
> >>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
--
.:''''':.
.:' ` Sérgio Surkamp | Administrador de Redes
:: ........ sergio at gruposinternet.com.br
`:. .:'
`:, ,.:' *Grupos Internet S.A.*
`: :' R. Lauro Linhares, 2123 Torre B - Sala 201
: : Trindade - Florianópolis - SC
:.'
:: +55 48 3234-4109
:
' http://www.gruposinternet.com.br
More information about the Ocfs2-users
mailing list