[Ocfs2-users] Non-responsive OCFS2 filesystems

[Isaac Clerencia]

Our OCFS2 filesystems are having real troubles and we're getting these kernel 
traces ... any idea about what can be causing this?

------------[ cut here ]------------
kernel BUG at fs/ocfs2/dlmglue.c:536!
invalid opcode: 0000 [#1]
SMP
Modules linked in: sg iptable_filter ip_tables x_tables nfs lockd nfs_acl 
sunrpc ipv6 dm_snapshot dm_mirror ide_disk e752x_edac psmouse floppy iTCO_wdt 
edac_mc serio_raw pcspkr shpchp pci_hotplug evdev joydev tsdev dm_round_robin 
dm_emc dm_multipath dm_mod ext3 mbcache sd_mod ide_generic ide_cd cdrom 
ata_generic libata cciss qla2xxx firmware_class uhci_hcd scsi_transport_fc 
ehci_hcd tg3 scsi_mod piix generic ide_core usbcore thermal processor fan 
ocfs2_dlmfs ocfs2 jbd ocfs2_dlm ocfs2_nodemanager configfs
CPU:    2
EIP:    0060:[<f8943902>]    Not tainted VLI
EFLAGS: 00210046   (2.6.21-1-686 #1)
EIP is at ocfs2_cluster_unlock+0xd2/0x2ae [ocfs2]
eax: 00000000   ebx: f54cfa48   ecx: 00200246   edx: f54cfa50
esi: f761a000   edi: 00000005   ebp: f54cfa50   esp: f48b9cf0
ds: 007b   es: 007b   fs: 00d8  gs: 0033  ss: 0068
Process maildrop (pid: 5514, ti=f48b8000 task=f5ca0540 task.ti=f48b8000)
Stack: f54cfc34 f761a000 f6183f28 f592e8b4 f71935b8 00200246 f761a000 00000005
       f54cfc34 f54cf980 f8943c16 f761a000 f54cfc34 00000000 ffffffe4 f761a000
       f48b9e08 00000000 f8969b3a f761a000 f54cfc34 f39ba540 f48b9e08 00000000
Call Trace:
 [<f8943c16>] ocfs2_meta_unlock+0x138/0x18f [ocfs2]
 [<f8969b3a>] ocfs2_reserve_cluster_bitmap_bits+0x23/0xde [ocfs2]
 [<f8968a60>] ocfs2_free_alloc_context+0x1c/0x40 [ocfs2]
 [<f8969f1c>] ocfs2_reserve_clusters+0x327/0x396 [ocfs2]
 [<f89459d1>] ocfs2_data_lock_full+0x1db/0x2eb [ocfs2]
 [<f894cf71>] ocfs2_extend_file+0x589/0x12ee [ocfs2]
 [<f8946b81>] ocfs2_meta_lock_full+0x450/0x721 [ocfs2]
 [<c0179271>] mntput_no_expire+0x11/0x6a
 [<f894e506>] ocfs2_prepare_inode_for_write+0x830/0x913 [ocfs2]
 [<c0165ddf>] get_unused_fd+0x4a/0xaa
 [<f894eed2>] ocfs2_file_aio_write+0x1a5/0x31d [ocfs2]
 [<c016f619>] do_path_lookup+0x16e/0x189
 [<c016711c>] do_sync_write+0xc7/0x10a
 [<c01700f0>] open_namei+0x6e/0x56a
 [<c0132969>] autoremove_wake_function+0x0/0x35
 [<c0135544>] hrtimer_start+0xf7/0x101
 [<c0167055>] do_sync_write+0x0/0x10a
 [<c0167900>] vfs_write+0xa8/0x12a
 [<c0167e8b>] sys_write+0x41/0x67
 [<c0103d88>] syscall_call+0x7/0xb
 [<c0290000>] __find_acq_core+0x2a8/0x30d
 =======================
Code: 00 00 c7 04 24 8e 6c 97 f8 89 44 24 04 e8 1a ed 7d c7 85 db 75 04 0f 0b 
eb fe 83 ff 03 74 16 83 ff 05 75 22 8b 43 4c 85 c0 75 04 <0f> 0b eb fe 48 89 
43 4c eb 15 8b 43 48 85 c0 75 04 0f 0b eb fe
EIP: [<f8943902>] ocfs2_cluster_unlock+0xd2/0x2ae [ocfs2] SS:ESP 0068:f48b9cf0
(166,3):ocfs2_lock_res_free:497 ERROR: bug expression: res->l_ex_holders
(166,3):ocfs2_lock_res_free:497 ERROR: Lockres W000000000000000b37790d0805ff41 
has 1 ex holders
------------[ cut here ]------------
kernel BUG at fs/ocfs2/dlmglue.c:497!
invalid opcode: 0000 [#2]
SMP
Modules linked in: tcp_diag inet_diag sg iptable_filter ip_tables x_tables nfs 
lockd nfs_acl sunrpc ipv6 dm_snapshot dm_mirror ide_disk e752x_edac psmouse 
floppy iTCO_wdt edac_mc serio_raw pcspkr shpchp pci_hotplug evdev joydev 
tsdev dm_round_robin dm_emc dm_multipath dm_mod ext3 mbcache sd_mod 
ide_generic ide_cd cdrom ata_generic libata cciss qla2xxx firmware_class 
uhci_hcd scsi_transport_fc ehci_hcd tg3 scsi_mod piix generic ide_core 
usbcore thermal processor fan ocfs2_dlmfs ocfs2 jbd ocfs2_dlm 
ocfs2_nodemanager configfs
CPU:    3
EIP:    0060:[<f8944342>]    Not tainted VLI
EFLAGS: 00010292   (2.6.21-1-686 #1)
EIP is at ocfs2_lock_res_free+0x50e/0x578 [ocfs2]
eax: 00000063   ebx: 00000003   ecx: 00000046   edx: 00000000
esi: f761a000   edi: 0000006a   ebp: 00000000   esp: c21cfe5c
ds: 007b   es: 007b   fs: 00d8  gs: 0000  ss: 0068
Process kswapd0 (pid: 166, ti=c21ce000 task=c2194580 task.ti=c21ce000)
Stack: f8977233 000000a6 00000003 f8973a43 000001f1 f0cef6ac 00000001 f7164940
       0000006a f8953db2 c011af62 c21cfed4 00000003 c2001890 00000001 c2001890
       c21cfed4 00000001 f0cef934 f0cef680 f8e326e3 dc5e0d94 dc5e0e2c f55ff200
Call Trace:
 [<f8953db2>] ocfs2_clear_inode+0x50c/0xbdc [ocfs2]
 [<c011af62>] __wake_up_common+0x32/0x55
 [<f8e326e3>] nfs3_forget_cached_acls+0x44/0x4e [nfs]
 [<c017755c>] clear_inode+0xa6/0xf4
 [<c0177825>] dispose_list+0x46/0xc4
 [<c0177a3d>] shrink_icache_memory+0x19a/0x1c2
 [<c01548c5>] shrink_slab+0xd9/0x13e
 [<c0154c83>] kswapd+0x2c2/0x3cf
 [<c0132969>] autoremove_wake_function+0x0/0x35
 [<c01549c1>] kswapd+0x0/0x3cf
 [<c013289e>] kthread+0xb2/0xdc
 [<c01327ec>] kthread+0x0/0xdc
 [<c01049a3>] kernel_thread_helper+0x7/0x10
 =======================
Code: 54 24 18 89 4c 24 14 c7 44 24 10 f1 01 00 00 c7 44 24 0c 43 3a 97 f8 89 
5c 24 08 89 44 24 04 c7 04 24 33 72 97 f8 e8 c1 e2 7d c7 <0f> 0b eb fe 8d 7b 
50 b9 13 00 00 00 f3 ab c7 43 20 00 00 00 00
EIP: [<f8944342>] ocfs2_lock_res_free+0x50e/0x578 [ocfs2] SS:ESP 0068:c21cfe5c