[Ocfs2-users] Soft lockup problem

[Sunil Mushran]

Please file a bugzilla on oss.oracle.com/bugzilla. It's easier
to keep track of issues that-a-way.

Attach the messages file from all nodes in the cluster. While
the logs you have provided should be enough, having the complete
logs is better as it provides a fuller picture.

Daniel wrote:
> Hello
>
> This appeared in my messages log:
>
> Jul 24 07:27:41 tilesrv2 kernel: BUG: soft lockup detected on CPU#0!
> Jul 24 07:27:41 tilesrv2 kernel:
> Jul 24 07:27:41 tilesrv2 kernel: Call Trace:
> Jul 24 07:27:41 tilesrv2 kernel:  <IRQ>  [<ffffffff800b2ca3>] 
> softlockup_tick+0xdb/0xed
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80093424>] 
> update_process_times+0x42/0x68
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80073d99>] 
> smp_local_timer_interrupt+0x23/0x47
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8007445b>] 
> smp_apic_timer_interrupt+0x41/0x47
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8005bcc2>] 
> apic_timer_interrupt+0x66/0x6c
> Jul 24 07:27:41 tilesrv2 kernel:  <EOI>  [<ffffffff8006270f>] 
> .text.lock.spinlock+0x5/0x30
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff884da534>] 
> :ocfs2_dlm:dlm_assert_master_handler+0x93d/0xd3a
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8001c239>] 
> __mod_timer+0xb0/0xbe
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8849f3a7>] 
> :ocfs2_nodemanager:o2net_rx_until_empty+0x0/0x9ca
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8849dfcf>] 
> :ocfs2_nodemanager:o2net_process_message+0x3ef/0x58b
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8849fbf6>] 
> :ocfs2_nodemanager:o2net_rx_until_empty+0x84f/0x9ca
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8849f3a7>] 
> :ocfs2_nodemanager:o2net_rx_until_empty+0x0/0x9ca
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8004b2cf>] 
> run_workqueue+0x94/0xe5
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80047c2e>] 
> worker_thread+0x0/0x122
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8009b4f6>] 
> keventd_create_kthread+0x0/0x61
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80047d1e>] 
> worker_thread+0xf0/0x122
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80086c6f>] 
> default_wake_function+0x0/0xe
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8009b4f6>] 
> keventd_create_kthread+0x0/0x61
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8009b4f6>] 
> keventd_create_kthread+0x0/0x61
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80032189>] kthread+0xfe/0x132
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8005bfe5>] child_rip+0xa/0x11
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8009b4f6>] 
> keventd_create_kthread+0x0/0x61
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8003208b>] kthread+0x0/0x132
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8005bfdb>] child_rip+0x0/0x11
> Jul 24 07:27:41 tilesrv2 kernel:
> Jul 24 07:27:41 tilesrv2 kernel: BUG: soft lockup detected on CPU#1!
> Jul 24 07:27:41 tilesrv2 kernel:
> Jul 24 07:27:41 tilesrv2 kernel: Call Trace:
> Jul 24 07:27:41 tilesrv2 kernel:  <IRQ>  [<ffffffff800b2ca3>] 
> softlockup_tick+0xdb/0xed
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80093424>] 
> update_process_times+0x42/0x68
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80073d99>] 
> smp_local_timer_interrupt+0x23/0x47
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8007445b>] 
> smp_apic_timer_interrupt+0x41/0x47
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8005bcc2>] 
> apic_timer_interrupt+0x66/0x6c
> Jul 24 07:27:41 tilesrv2 kernel:  <EOI>  [<ffffffff884d2e44>] 
> :ocfs2_dlm:__dlm_lookup_lockres_full+0xbe/0x108
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff884d2e55>] 
> :ocfs2_dlm:__dlm_lookup_lockres_full+0xcf/0x108
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff884dadaf>] 
> :ocfs2_dlm:dlm_get_lock_resource+0xcb/0x18e4
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80061552>] 
> __wait_on_bit+0x60/0x6f
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff80014bf0>] 
> sync_buffer+0x0/0x3f
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff884e29d8>] 
> :ocfs2_dlm:dlm_in_recovery+0xd/0x20
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff884e6ec1>] 
> :ocfs2_dlm:dlm_wait_for_recovery+0xa1/0x116
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885ecfdb>] 
> :ocfs2:ocfs2_inode_ast_func+0x0/0x6da
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff884e0424>] 
> :ocfs2_dlm:dlmlock+0x751/0x1220
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885f677d>] 
> :ocfs2:ocfs2_populate_inode+0x4d3/0x558
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8002cc23>] 
> wake_up_bit+0x11/0x22
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885e915a>] 
> :ocfs2:ocfs2_cluster_unlock+0x65/0x2cb
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885e9509>] 
> :ocfs2:ocfs2_meta_unlock+0x121/0x180
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885ea34a>] 
> :ocfs2:ocfs2_lock_create+0x137/0x346
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885ed6b5>] 
> :ocfs2:ocfs2_inode_bast_func+0x0/0x15b
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885ea943>] 
> :ocfs2:ocfs2_cluster_lock+0x205/0x898
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885e8a3e>] 
> :ocfs2:ocfs2_status_completion_cb+0x0/0xb
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885eeaf1>] 
> :ocfs2:ocfs2_meta_lock_full+0x216/0xd35
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885f7f7f>] 
> :ocfs2:ocfs2_inode_revalidate+0x14f/0x228
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885f2fc6>] 
> :ocfs2:ocfs2_getattr+0x79/0x159
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8003e713>] 
> vfs_lstat_fd+0x2f/0x47
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff885e68c1>] 
> :ocfs2:ocfs2_readdir+0x40e/0x426
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff800252e1>] filldir+0x0/0xb7
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8002a4da>] 
> sys_newlstat+0x19/0x31
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8005b261>] tracesys+0x71/0xdc
> Jul 24 07:27:41 tilesrv2 kernel:  [<ffffffff8005b2c1>] tracesys+0xd1/0xdc
>
> Dell 1959 2xQuadcore, EMC 3-20, CentOS 5 2.6.18-8.1.8.el5 OCFS2 1.2.6-1
>
> What can cause this? Where do I start looking?
>
> Daniel
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users