[Ocfs2-users] Kernel Panics every day
Adam Kenger
akenger at gmail.com
Wed Mar 15 16:04:49 CST 2006
Hi all - I'm having a strange issue and I'm trying to figure out what
the problem is. We have a 3 node RHEL 4 AS Oracle 10gR2 RAC cluster
which utilizes an iscsi SAN. We have 2 disks mounted through OCFS2
(most current release) for the ORACLE_HOME and for the CRS Voting
Disk. The rest of the disks are mounted via ASM (also the most
current release). It seems that for some reason, one node keeps
crashing (Watson) and when it goes, one of the other nodes is soon to
follow. I have attached the rather lengthy messages from /var/log/
messages. Has anyone seen any behavior like this? The weird thing
to note is that when nodes 1 and 2 go down, the third one keeps
chugging without any apparent issues.
Here are the logs - first from Node 2 (watson) which goes down first
I think - then Node 1 (sherlock).
If anyone has any ideas as to where to look for the source of these
troubles, I'd greatly appreciate it.
Thanks
Adam
Mar 12 05:26:15 watson kernel: Unable to handle kernel paging request
at virtual
address 003b0222
Mar 12 05:26:15 watson kernel: printing eip:
Mar 12 05:26:15 watson kernel: c01c074e
Mar 12 05:26:15 watson kernel: *pde = 359b1001
Mar 12 05:26:15 watson kernel: Oops: 0002 [#1]
Mar 12 05:26:15 watson kernel: SMP
Mar 12 05:26:15 watson kernel: Modules linked in: ocfs2(U) debugfs(U)
ipv6 parpo
rt_pc lp parport crc32c libcrc32c md5 iscsi_sfnet
scsi_transport_iscsi oracleasm
(U) autofs4 i2c_dev i2c_core nfs lockd nfs_acl ocfs2_dlmfs(U)
ocfs2_dlm(U) ocfs2
_nodemanager(U) configfs(U) sunrpc dm_mirror dm_mod button battery ac
uhci_hcd e
hci_hcd hw_random shpchp e1000 bond2(U) bond1(U) bond0(U) floppy sg
ext3 jbd meg
araid_mbox megaraid_mm sd_mod scsi_mod
Mar 12 05:26:15 watson kernel: CPU: 1
Mar 12 05:26:15 watson kernel: EIP: 0060:[<c01c074e>] Not
tainted VLI
Mar 12 05:26:15 watson kernel: EFLAGS: 00010206 (2.6.9-34.ELsmp)
Mar 12 05:26:15 watson kernel: EIP is at __rb_rotate_left+0x12/0x36
Mar 12 05:26:15 watson kernel: eax: 003b0222 ebx: eef832c8 ecx:
dd29fe00 e
dx: d200ae08
Mar 12 05:26:15 watson kernel: esi: eef832c8 edi: f6429e5c ebp:
dd29fdc0 e
sp: f6429e28
Mar 12 05:26:15 watson kernel: ds: 007b es: 007b ss: 0068
Mar 12 05:26:15 watson kernel: Process ocfs2vote-1 (pid: 3978,
threadinfo=f64290
00 task=c3a8eeb0)
Mar 12 05:26:15 watson kernel: Stack: dd29fdc0 c01c0989 dd29fd80
00000000 f91b30
22 eef832c4 00000000 00000000
Mar 12 05:26:15 watson kernel: eef83324 eef832ac eef83100
f91b30a3 f6429e
60 dd29fd40 f7c43480 f7c43480
Mar 12 05:26:15 watson kernel: eef83100 00000001 f91b9d14
f5faf800 eef833
24 c3a8eeb0 c38a2200 00000002
Mar 12 05:26:15 watson kernel: Call Trace:
Mar 12 05:26:15 watson kernel: [<c01c0989>] __rb_erase_color
+0x120/0x16b
Mar 12 05:26:15 watson kernel: [<f91b3022>] __ocfs2_extent_map_drop
+0x4b/0x86 [
ocfs2]
Mar 12 05:26:15 watson kernel: [<f91b30a3>] ocfs2_extent_map_drop
+0x2c/0x69 [oc
fs2]
Mar 12 05:26:15 watson kernel: [<f91b9d14>] ocfs2_clear_inode
+0x404/0xc0a [ocfs
2]
Mar 12 05:26:15 watson kernel: [<c01766b4>] __sync_single_inode
+0x1b7/0x1c1
Mar 12 05:26:15 watson kernel: [<c0170008>] clear_inode+0xcc/0x102
Mar 12 05:26:15 watson kernel: [<f91b98c9>] ocfs2_delete_inode
+0x404/0x44b [ocf
s2]
Mar 12 05:26:15 watson kernel: [<f91b94c5>] ocfs2_delete_inode
+0x0/0x44b [ocfs2
]
Mar 12 05:26:15 watson kernel: [<c0170be4>] generic_delete_inode
+0xa2/0x104
Mar 12 05:26:15 watson kernel: [<f91ba600>] ocfs2_drop_inode
+0xe6/0x12a [ocfs2]
Mar 12 05:26:15 watson kernel: [<c0170dc0>] iput+0x5f/0x61
Mar 12 05:26:15 watson kernel: [<f91d5a47>] ocfs2_process_vote+0x71f/
0x727 [ocf
s2]
Mar 12 05:26:15 watson kernel: [<c02d0589>] schedule+0x83d/0x8d3
Mar 12 05:26:15 watson kernel: [<f91d5c46>] ocfs2_vote_thread
+0x0/0x121 [ocfs2]
Mar 12 05:26:15 watson kernel: [<f91d5b68>] ocfs2_vote_thread_do_work
+0x119/0x1
7a [ocfs2]
Mar 12 05:26:15 watson kernel: [<f91d5c46>] ocfs2_vote_thread
+0x0/0x121 [ocfs2]
Mar 12 05:26:15 watson kernel: [<f91d5d39>] ocfs2_vote_thread
+0xf3/0x121 [ocfs2
]
Mar 12 05:26:15 watson kernel: [<c0120291>] autoremove_wake_function
+0x0/0x2d
Mar 12 05:26:15 watson kernel: [<c0120291>] autoremove_wake_function
+0x0/0x2d
Mar 12 05:26:15 watson kernel: [<c0133ecd>] kthread+0x73/0x9b
Mar 12 05:26:15 watson kernel: [<c0133e5a>] kthread+0x0/0x9b
Mar 12 05:26:15 watson kernel: [<c01041f5>] kernel_thread_helper
+0x5/0xb
Mar 12 05:26:15 watson kernel: Code: f9 01 76 ed 31 c0 5b c3 57 b9 45
00 00 00 8
9 c7 31 c0 f3 ab 5f c3 90 90 90 53 89 d3 8b 50 08 89 c1 8b 42 0c 85
c0 89 41 08
74 02 <89> 08 89 4a 0c 8b 01 85 c0 89 02 74 11 8b 01 3b 48 0c 75 05 89
Mar 12 05:26:15 watson kernel: <0>Fatal exception: panic in 5 seconds
Mar 12 05:25:09 sherlock kernel: (0,1):o2net_idle_timer:1310
connection to node
watson (num 1) at 10.3.3.11:7777 has been idle for 10 seconds,
shutting it down.
Mar 12 05:25:09 sherlock kernel: (0,1):o2net_idle_timer:1321 here are
some times
that might help debug the situation: (tmr 1142159099.775825 now
1142159109.7738
47 dr 1142159099.775812 adv 1142159099.775829:1142159099.775831 func
(1167074b:5
05) 1142159099.198517:1142159099.198535)
Mar 12 05:25:09 sherlock kernel: (2903,1):o2net_set_nn_state:411 no
longer conne
cted to node watson (num 1) at 10.3.3.11:7777
Mar 12 05:25:09 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -112
Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:09 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:09 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:09 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:10 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:10 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:11 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:11 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:12 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:12 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,0):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,0):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:13 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:13 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel: (2903,1):ocfs2_dlm_eviction_cb:118
device (8,49
): dlm has evicted node 1
Mar 12 05:25:14 sherlock kernel:
(8109,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (8109,1):dlm_wait_for_node_death:285
BD08C3AD58
994411981AAA8C5938F5E0: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel:
(7982,1):dlm_send_remote_convert_request:393 ER
ROR: status = -107
Mar 12 05:25:14 sherlock kernel: (7982,1):dlm_wait_for_node_death:285
9ED43B6CE8
DE48FEBB35FA1D3AF46661: waiting 5000ms for notification of death of
node 1
Mar 12 05:25:14 sherlock kernel: eip: f8ceeba2
Mar 12 05:25:14 sherlock kernel: ------------[ cut here ]------------
Mar 12 05:25:14 sherlock kernel: kernel BUG at include/asm/spinlock.h:
133!
Mar 12 05:25:14 sherlock kernel: invalid operand: 0000 [#1]
Mar 12 05:25:14 sherlock kernel: SMP
Mar 12 05:25:14 sherlock kernel: Modules linked in: ocfs2(U) debugfs
(U) ipv6 par
port_pc lp parport crc32c libcrc32c md5 iscsi_sfnet
scsi_transport_iscsi oraclea
sm(U) autofs4 i2c_dev i2c_core nfs lockd nfs_acl ocfs2_dlmfs(U)
ocfs2_dlm(U) ocf
s2_nodemanager(U) configfs(U) sunrpc dm_mirror dm_mod button battery
ac uhci_hcd
ehci_hcd hw_random shpchp e1000 bond2(U) bond1(U) bond0(U) floppy
sg ext3 jbd m
egaraid_mbox megaraid_mm sd_mod scsi_mod
Mar 12 05:25:14 sherlock kernel: CPU: 2
Mar 12 05:25:14 sherlock kernel: EIP: 0060:[<c02d11e8>] Not
tainted VLI
Mar 12 05:25:14 sherlock kernel: EFLAGS: 00010216 (2.6.9-34.ELsmp)
Mar 12 05:25:14 sherlock kernel: EIP is at _spin_lock+0x1c/0x34
Mar 12 05:25:14 sherlock kernel: eax: c02e4ca6 ebx: e7726c94 ecx:
f648de50
edx: f8ceeba2
Mar 12 05:25:14 sherlock kernel: esi: e7726c80 edi: 00000001 ebp:
00000000
esp: f648de54
Mar 12 05:25:14 sherlock kernel: ds: 007b es: 007b ss: 0068
Mar 12 05:25:14 sherlock kernel: Process o2hb-BD08C3AD58 (pid: 3862,
threadinfo=
f648d000 task=f5c800b0)
Mar 12 05:25:14 sherlock kernel: Stack: 00000001 f8ceeba2 e7726c88
f5992000 f8ce
eb88 00000001 00000001 f5992000
Mar 12 05:25:14 sherlock kernel: 00000001 00000001 f8cfe684
f5992030 f599
2000 f8cfe76a f599215c f5992158
Mar 12 05:25:14 sherlock kernel: f8dd0920 f8dba8f7 c3897c80
00000000 f648
dedc f648dedc f8dce8a0 f8dbaa27
Mar 12 05:25:14 sherlock kernel: Call Trace:
Mar 12 05:25:14 sherlock kernel: [<f8ceeba2>] dlm_mle_node_down
+0x10/0x73 [ocfs
2_dlm]
Mar 12 05:25:14 sherlock kernel: [<f8ceeb88>]
dlm_hb_event_notify_attached+0x6e
/0x78 [ocfs2_dlm]
Mar 12 05:25:14 sherlock kernel: [<f8cfe684>] __dlm_hb_node_down
+0x1a6/0x267 [o
cfs2_dlm]
Mar 12 05:25:14 sherlock kernel: [<f8cfe76a>] dlm_hb_node_down_cb
+0x25/0x3a [oc
fs2_dlm]
Mar 12 05:25:14 sherlock kernel: [<f8dba8f7>] o2hb_fire_callbacks
+0x62/0x6c [oc
fs2_nodemanager]
Mar 12 05:25:14 sherlock kernel: [<f8dbaa27>] o2hb_run_event_list
+0x126/0x162 [
ocfs2_nodemanager]
Mar 12 05:25:14 sherlock kernel: [<f8dbb0f9>] o2hb_check_slot
+0x4d2/0x4e7 [ocfs
2_nodemanager]
Mar 12 05:25:14 sherlock kernel: [<c02243c2>] submit_bio+0xca/0xd2
Mar 12 05:25:14 sherlock kernel: [<f8dbb3ed>] o2hb_do_disk_heartbeat
+0x2b4/0x32
5 [ocfs2_nodemanager]
Mar 12 05:25:14 sherlock kernel: [<f8dbb4e2>] o2hb_thread+0x0/0x291
[ocfs2_node
manager]
Mar 12 05:25:14 sherlock kernel: [<f8dbb56b>] o2hb_thread+0x89/0x291
[ocfs2_nod
emanager]
Mar 12 05:25:14 sherlock kernel: [<f8dbb4e2>] o2hb_thread+0x0/0x291
[ocfs2_node
manager]
Mar 12 05:25:14 sherlock kernel: [<c0133ecd>] kthread+0x73/0x9b
Mar 12 05:25:14 sherlock kernel: [<c0133e5a>] kthread+0x0/0x9b
Mar 12 05:25:14 sherlock kernel: [<c01041f5>] kernel_thread_helper
+0x5/0xb
Mar 12 05:25:14 sherlock kernel: Code: 00 75 09 f0 81 02 00 00 00 01
30 c9 89 c8
c3 53 89 c3 81 78 04 ad 4e ad de 74 18 ff 74 24 04 68 a6 4c 2e c0
e8 54 14 e5 f
f 58 5a <0f> 0b 85 00 60 3d 2e c0 f0 fe 0b 79 09 f3 90 80 3b 00 7e f9 eb
Mar 12 05:25:14 sherlock kernel: <0>Fatal exception: panic in 5 seconds
More information about the Ocfs2-users
mailing list