We have been experiencing unplanned outages on a subset of the clustered systems we have deployed to support SAP. The following captured information came from one of three ocfs2 clusters which handle SAP SEM/BW functionality. Each of those has experienced multiple kernel panics which get reported as dlm_drop_lockres_ref and dlm_defer_lockres_handler errors.<br>
<br>This particular environment is deployed on two identical HP DL580-G5 servers running RHEL5.2 with kernel 2.6.18-92.1.1.el5. It should be noted that we have encountered similar issues with similar frequency on kernel 2.6.18-92.1.22.el5. We are running ocfs2 1.2.9. These two servers support Oracle RAC (10.2.0.2) and they also sharing through NFS two ocfs2 file systems to associated SAP application servers. Database devices and RMAN backups are supported by ocfs2 file systems with "datavolume,nointr" options. The additional cluster file systems which support Oracle and SAP do not contain those options.<br>
<br>We have encountered recurring unplanned reboots which appear to be initiated by the crashing of either node in the cluster. This particular cluster had experienced a crashed of the other cluster member one week previous to this most recent crash. Both crashes had occurred at times with very minimal activity.<br>
<br>Support for netconsole is implemented and we are redirecting the kernel's console output to a central syslog server. The logs below originated from netconsole. Though not yet in place at the time of this crash, kexec/kdump functionality was also introduced.<br>
<br>Thanks,<br>Ward<br><br>################################################################################<br>netconsole capture of rebooted server:<br>################################################################################<br>
<br>Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_drop_lockres_ref:2298 ERROR: while dropping ref on F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26 (master=1) got -22.<br>Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_one_lock_resource:461 lockres: M00000000000000003c1c868d5afc26, owner=1, state=64<br>
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:476 lockres: M00000000000000003c1c868d5afc26, owner=1, state=64<br>Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:478 last used: 4728024828, on purge list: yes<br>
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_lockres_refmap:444 refmap nodes: [<br>Mar 7 07:45:25 sapbdb-uat-d2 ], inflight=0<br>Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:480 granted queue:<br>
Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:495 converting queue:<br>Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):__dlm_print_one_lock_resource:510 blocked queue:<br>Mar 7 07:45:25 sapbdb-uat-d2 ----------- [cut here ] --------- [please bite here ] ---------<br>
Mar 7 07:45:25 sapbdb-uat-d2 Kernel BUG at ...mushran/BUILD/ocfs2-1.2.9/fs/ocfs2/dlm/dlmmaster.c:2300<br>Mar 7 07:45:25 sapbdb-uat-d2 invalid opcode: 0000 [1]<br>Mar 7 07:45:25 sapbdb-uat-d2 SMP<br>Mar 7 07:45:25 sapbdb-uat-d2<br>
Mar 7 07:45:25 sapbdb-uat-d2 last sysfs file: /devices/pci0000:00/0000:00:00.0/irq<br>Mar 7 07:45:25 sapbdb-uat-d2 CPU 2<br>Mar 7 07:45:25 sapbdb-uat-d2<br>Mar 7 07:45:25 sapbdb-uat-d2 Modules linked in:<br>Mar 7 07:45:25 sapbdb-uat-d2 iptable_filter<br>
Mar 7 07:45:26 sapbdb-uat-d2 ip_tables<br>Mar 7 07:45:26 sapbdb-uat-d2 x_tables<br>Mar 7 07:45:26 sapbdb-uat-d2 netconsole<br>Mar 7 07:45:26 sapbdb-uat-d2 nfsd<br>Mar 7 07:45:26 sapbdb-uat-d2 exportfs<br>Mar 7 07:45:26 sapbdb-uat-d2 auth_rpcgss<br>
Mar 7 07:45:26 sapbdb-uat-d2 autofs4<br>Mar 7 07:45:26 sapbdb-uat-d2 hidp<br>Mar 7 07:45:26 sapbdb-uat-d2 ocfs2(U)<br>Mar 7 07:45:26 sapbdb-uat-d2 nfs<br>Mar 7 07:45:26 sapbdb-uat-d2 lockd<br>Mar 7 07:45:26 sapbdb-uat-d2 fscache<br>
Mar 7 07:45:26 sapbdb-uat-d2 nfs_acl<br>Mar 7 07:45:26 sapbdb-uat-d2 rfcomm<br>Mar 7 07:45:26 sapbdb-uat-d2 l2cap<br>Mar 7 07:45:26 sapbdb-uat-d2 bluetooth<br>Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlmfs(U)<br>Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlm(U)<br>
Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_nodemanager(U)<br>Mar 7 07:45:26 sapbdb-uat-d2 lock_dlm(U)<br>Mar 7 07:45:26 sapbdb-uat-d2 gfs2(U)<br>Mar 7 07:45:26 sapbdb-uat-d2 dlm<br>Mar 7 07:45:26 sapbdb-uat-d2 configfs<br>
Mar 7 07:45:26 sapbdb-uat-d2 sunrpc<br>Mar 7 07:45:26 sapbdb-uat-d2 bonding<br>Mar 7 07:45:26 sapbdb-uat-d2 ipv6<br>Mar 7 07:45:26 sapbdb-uat-d2 xfrm_nalgo<br>Mar 7 07:45:26 sapbdb-uat-d2 crypto_api<br>Mar 7 07:45:26 sapbdb-uat-d2 dm_round_robin<br>
Mar 7 07:45:26 sapbdb-uat-d2 dm_emc<br>Mar 7 07:45:26 sapbdb-uat-d2 dm_multipath<br>Mar 7 07:45:26 sapbdb-uat-d2 video<br>Mar 7 07:45:26 sapbdb-uat-d2 sbs<br>Mar 7 07:45:26 sapbdb-uat-d2 backlight<br>Mar 7 07:45:26 sapbdb-uat-d2 i2c_ec<br>
Mar 7 07:45:26 sapbdb-uat-d2 i2c_core<br>Mar 7 07:45:26 sapbdb-uat-d2 button<br>Mar 7 07:45:26 sapbdb-uat-d2 battery<br>Mar 7 07:45:26 sapbdb-uat-d2 asus_acpi<br>Mar 7 07:45:26 sapbdb-uat-d2 acpi_memhotplug<br>Mar 7 07:45:26 sapbdb-uat-d2 ac<br>
Mar 7 07:45:26 sapbdb-uat-d2 parport_pc<br>Mar 7 07:45:26 sapbdb-uat-d2 lp<br>Mar 7 07:45:26 sapbdb-uat-d2 parport<br>Mar 7 07:45:26 sapbdb-uat-d2 joydev<br>Mar 7 07:45:26 sapbdb-uat-d2 ata_piix<br>Mar 7 07:45:26 sapbdb-uat-d2 libata<br>
Mar 7 07:45:26 sapbdb-uat-d2 sg<br>Mar 7 07:45:26 sapbdb-uat-d2 ide_cd<br>Mar 7 07:45:26 sapbdb-uat-d2 shpchp<br>Mar 7 07:45:27 sapbdb-uat-d2 cdrom<br>Mar 7 07:45:27 sapbdb-uat-d2 bnx2<br>Mar 7 07:45:27 sapbdb-uat-d2 pcspkr<br>
Mar 7 07:45:27 sapbdb-uat-d2 serio_raw<br>Mar 7 07:45:27 sapbdb-uat-d2 dm_snapshot<br>Mar 7 07:45:27 sapbdb-uat-d2 dm_zero<br>Mar 7 07:45:27 sapbdb-uat-d2 dm_mirror<br>Mar 7 07:45:27 sapbdb-uat-d2 dm_mod<br>Mar 7 07:45:27 sapbdb-uat-d2 qla2xxx<br>
Mar 7 07:45:27 sapbdb-uat-d2 scsi_transport_fc<br>Mar 7 07:45:27 sapbdb-uat-d2 cciss<br>Mar 7 07:45:27 sapbdb-uat-d2 sd_mod<br>Mar 7 07:45:27 sapbdb-uat-d2 scsi_mod<br>Mar 7 07:45:27 sapbdb-uat-d2 ext3<br>Mar 7 07:45:27 sapbdb-uat-d2 jbd<br>
Mar 7 07:45:27 sapbdb-uat-d2 uhci_hcd<br>Mar 7 07:45:27 sapbdb-uat-d2 ohci_hcd<br>Mar 7 07:45:27 sapbdb-uat-d2 ehci_hcd<br>Mar 7 07:45:27 sapbdb-uat-d2 <br>Mar 7 07:45:27 sapbdb-uat-d2 Pid: 11241, comm: dlm_thread Tainted: G 2.6.18-92.1.1.el5 #1<br>
Mar 7 07:45:27 sapbdb-uat-d2 RIP: 0010:[<ffffffff885ba6a0>]<br>Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885ba6a0>] :ocfs2_dlm:dlm_drop_lockres_ref+0x1dc/0x1f5<br>Mar 7 07:45:27 sapbdb-uat-d2 RSP: 0018:ffff8105e1395de0 EFLAGS: 00010246<br>
Mar 7 07:45:27 sapbdb-uat-d2 RAX: ffff81005ef3fe38 RBX: 0000000000000000 RCX: ffffffff802ed9a8<br>Mar 7 07:45:27 sapbdb-uat-d2 RDX: ffffffff802ed9a8 RSI: 0000000000000000 RDI: ffffffff802ed9a0<br>Mar 7 07:45:27 sapbdb-uat-d2 RBP: 000000000000001f R08: ffffffff802ed9a8 R09: 0000000000000046<br>
Mar 7 07:45:27 sapbdb-uat-d2 R10: 0000000000000000 R11: 0000000000000280 R12: ffff81005ef3fe00<br>Mar 7 07:45:27 sapbdb-uat-d2 R13: ffff8105e227fc00 R14: ffff81002aec2d40 R15: ffffffff8009dced<br>Mar 7 07:45:27 sapbdb-uat-d2 FS: 0000000000000000(0000) GS:ffff81102ff1ce40(0000) knlGS:0000000000000000<br>
Mar 7 07:45:27 sapbdb-uat-d2 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b<br>Mar 7 07:45:27 sapbdb-uat-d2 CR2: 00000000006bdc5c CR3: 0000000594e3c000 CR4: 00000000000006e0<br>Mar 7 07:45:27 sapbdb-uat-d2 Process dlm_thread (pid: 11241, threadinfo ffff8105e1394000, task ffff8105e30a2820)<br>
Mar 7 07:45:27 sapbdb-uat-d2 Stack:<br>Mar 7 07:45:27 sapbdb-uat-d2 1f02000000000000<br>Mar 7 07:45:27 sapbdb-uat-d2 303030303030304d<br>Mar 7 07:45:27 sapbdb-uat-d2 3030303030303030<br>Mar 7 07:45:27 sapbdb-uat-d2 3836386331633330<br>
Mar 7 07:45:27 sapbdb-uat-d2 <br>Mar 7 07:45:27 sapbdb-uat-d2 0036326366613564<br>Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000<br>Mar 7 07:45:27 sapbdb-uat-d2 last message repeated 2 times<br>Mar 7 07:45:27 sapbdb-uat-d2 <br>
Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000<br>Mar 7 07:45:27 sapbdb-uat-d2 ffffffea00100100<br>Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe48<br>Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe00<br>Mar 7 07:45:27 sapbdb-uat-d2 <br>
Mar 7 07:45:27 sapbdb-uat-d2 Call Trace:<br>Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cd72b>] :ocfs2_dlm:dlm_purge_lockres+0x175/0x34f<br>Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdb98>] :ocfs2_dlm:dlm_thread+0xd7/0x579<br>
Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff8009df05>] autoremove_wake_function+0x0/0x2e<br>Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdac1>] :ocfs2_dlm:dlm_thread+0x0/0x579<br>Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003253d>] kthread+0xfe/0x132<br>
Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfb1>] child_rip+0xa/0x11<br>Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8009dced>] keventd_create_kthread+0x0/0xc4<br>Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003243f>] kthread+0x0/0x132<br>
Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfa7>] child_rip+0x0/0x11<br>Mar 7 07:45:28 sapbdb-uat-d2 <br>Mar 7 07:45:28 sapbdb-uat-d2 <br>Mar 7 07:45:28 sapbdb-uat-d2 Code:<br>Mar 7 07:45:28 sapbdb-uat-d2 0f<br>
Mar 7 07:45:28 sapbdb-uat-d2 0b<br>Mar 7 07:45:28 sapbdb-uat-d2 68<br>Mar 7 07:45:28 sapbdb-uat-d2 2d<br>Mar 7 07:45:28 sapbdb-uat-d2 10<br>Mar 7 07:45:28 sapbdb-uat-d2 5d<br>Mar 7 07:45:28 sapbdb-uat-d2 88<br>Mar 7 07:45:28 sapbdb-uat-d2 c2<br>
Mar 7 07:45:28 sapbdb-uat-d2 fc<br><br><br><br>################################################################################<br>netconsole capture for remaining server:<br>################################################################################<br>
<br>Mar 7 07:45:25 sapbdb-uat-d1 (9917,0):dlm_deref_lockres_handler:2340 ERROR: F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26: bad lockres name<br>Mar 7 07:45:40 sapbdb-uat-d1 dlm: closing connection to node 2<br>
Mar 7 07:45:55 sapbdb-uat-d1 o2net: connection to node sapbdb-uat-d2 (num 2) at <a href="http://192.168.1.2:7777">192.168.1.2:7777</a> has been idle for 30.0 seconds, shutting it down.<br>Mar 7 07:45:55 sapbdb-uat-d1 (0,0):o2net_idle_timer:1426 here are some times that might help debug the situation: (tmr 1236429925.620147 now 1236429955.618766 dr 1236429925.620142 adv 1236429925.622211:1236429925.622212 func (90648c68:507) 1236429925.620148:1236429925.622206)<br>
Mar 7 07:45:55 sapbdb-uat-d1 o2net: no longer connected to node sapbdb-uat-d2 (num 2) at <a href="http://192.168.1.2:7777">192.168.1.2:7777</a><br>Mar 7 07:45:55 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293 ERROR: status = -112<br>
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -112<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -112<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_do_master_request:1418 ERROR: link to 2 went down!<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:995 ERROR: status = -112<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR: status = -112<br>Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293 ERROR: status = -107<br>
Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR: status = -107<br>Mar 7 07:46:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 ERROR: no connection established with node 2 after 30.0 seconds, giving up and returning errors.<br>
Mar 7 07:46:55 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 ERROR: no connection established with node 2 after 30.0 seconds, giving up and returning errors.<br>Mar 7 07:47:01 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107<br>
Mar 7 07:47:01 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death of node 2<br>Mar 7 07:47:06 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107<br>
Mar 7 07:47:06 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death of node 2<br>Mar 7 07:47:11 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107<br>
Mar 7 07:47:11 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death of node 2<br>Mar 7 07:47:17 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107<br>
Mar 7 07:47:17 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death of node 2<br>Mar 7 07:47:22 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107<br>
Mar 7 07:47:22 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death of node 2<br>Mar 7 07:47:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585 ERROR: no connection established with node 2 after 30.0 seconds, giving up and returning errors.<br>
Mar 7 07:47:27 sapbdb-uat-d1 (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:27 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32060,5):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32060,5):dlm_wait_for_node_death:365 EF2FF243631540E0A9418F491C21D661: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32062,11):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32062,11):dlm_wait_for_node_death:365 02C0423D592746DD89EAC08217172364: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32063,11):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32063,11):dlm_wait_for_node_death:365 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32064,11):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32064,11):dlm_wait_for_node_death:365 D82D1AFEC06F4A5591C051A8AD55F6B8: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32065,11):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32065,11):dlm_wait_for_node_death:365 F1EC618443464E5295BAEC6CB6A0289A: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32066,11):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32066,11):dlm_wait_for_node_death:365 F96CA86A377E4562A237A572D5AAE337: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32068,11):dlm_send_remote_convert_request:398 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32068,11):dlm_wait_for_node_death:365 0715F21DDB174D4FA4F3AC717751BB8A: waiting 5000ms for notification of death of node 2<br>
Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_do_master_request:1418 ERROR: link to 2 went down!<br>Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_get_lock_resource:995 ERROR: status = -107<br>Mar 7 07:47:28 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:921 3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:921 D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_do_master_request:1418 ERROR: link to 2 went down!<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_get_lock_resource:995 ERROR: status = -107<br>Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:921 87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:921 549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (32053,1):dlm_get_lock_resource:921 3DF97CEDF6C04E9AAECE7665905F655C:M000000000000000000008fd0deae0b: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:921 FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_do_master_request:1418 ERROR: link to 2 went down!<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_get_lock_resource:995 ERROR: status = -107<br>Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:921 C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):dlm_get_lock_resource:921 EB739A1B7BCE4948A408D8ACBAC2FCE4:M000000000000000000008f9047acff: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (32059,1):dlm_get_lock_resource:921 87868F0E137342B0A1C5C525717DA24B:M000000000000000000008fef7ad4cb: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:921 EB739A1B7BCE4948A408D8ACBAC2FCE4:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:955 EB739A1B7BCE4948A408D8ACBAC2FCE4: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_do_recovery:519 (10569) Node 1 is the Recovery Master for the Dead Node 2 for Domain EB739A1B7BCE4948A408D8ACBAC2FCE4<br>Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:921 87868F0E137342B0A1C5C525717DA24B:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:955 87868F0E137342B0A1C5C525717DA24B: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_do_recovery:519 (10726) Node 1 is the Recovery Master for the Dead Node 2 for Domain 87868F0E137342B0A1C5C525717DA24B<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:921 D82D1AFEC06F4A5591C051A8AD55F6B8:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:955 D82D1AFEC06F4A5591C051A8AD55F6B8: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_do_recovery:519 (10883) Node 1 is the Recovery Master for the Dead Node 2 for Domain D82D1AFEC06F4A5591C051A8AD55F6B8<br>Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:921 17D5F5A73FA94B59A2C3CD91E32DCA99:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:955 17D5F5A73FA94B59A2C3CD91E32DCA99: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_do_recovery:519 (11040) Node 1 is the Recovery Master for the Dead Node 2 for Domain 17D5F5A73FA94B59A2C3CD91E32DCA99<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:921 3DF97CEDF6C04E9AAECE7665905F655C:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:955 3DF97CEDF6C04E9AAECE7665905F655C: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_do_recovery:519 (10480) Node 1 is the Recovery Master for the Dead Node 2 for Domain 3DF97CEDF6C04E9AAECE7665905F655C<br>Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:976 D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:976 87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:976 549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32053,8):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,42)<br>Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:976 FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:976 C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,43)<br>
Mar 7 07:47:29 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:976 3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (32059,8):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,49)<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:921 C2FD0D8F366742D4A9F399FDFF914A78:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:955 C2FD0D8F366742D4A9F399FDFF914A78: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_do_recovery:519 (10637) Node 1 is the Recovery Master for the Dead Node 2 for Domain C2FD0D8F366742D4A9F399FDFF914A78<br>Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:921 988B694C74FC44FBB22F5D59CAA5686F:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:955 988B694C74FC44FBB22F5D59CAA5686F: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_do_recovery:519 (10794) Node 1 is the Recovery Master for the Dead Node 2 for Domain 988B694C74FC44FBB22F5D59CAA5686F<br>
Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:921 0715F21DDB174D4FA4F3AC717751BB8A:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:955 0715F21DDB174D4FA4F3AC717751BB8A: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_do_recovery:519 (10951) Node 1 is the Recovery Master for the Dead Node 2 for Domain 0715F21DDB174D4FA4F3AC717751BB8A<br>Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_restart_lock_mastery:1301 ERROR: node down! 2<br>
Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_wait_for_lock_mastery:1118 ERROR: status = -11<br>Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:921 FD5BCEBA9AB041E2AC257C529FEAF692:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:955 FD5BCEBA9AB041E2AC257C529FEAF692: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_do_recovery:519 (10463) Node 1 is the Recovery Master for the Dead Node 2 for Domain FD5BCEBA9AB041E2AC257C529FEAF692<br>
Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:921 549A03816D5A4BEA912BC04E7F118822:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:955 549A03816D5A4BEA912BC04E7F118822: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_do_recovery:519 (10620) Node 1 is the Recovery Master for the Dead Node 2 for Domain 549A03816D5A4BEA912BC04E7F118822<br>Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:921 02C0423D592746DD89EAC08217172364:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:955 02C0423D592746DD89EAC08217172364: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_do_recovery:519 (10777) Node 1 is the Recovery Master for the Dead Node 2 for Domain 02C0423D592746DD89EAC08217172364<br>
Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:921 B0A423E24D5A47DEB93D3448C0B43073:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:955 B0A423E24D5A47DEB93D3448C0B43073: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_do_recovery:519 (10934) Node 1 is the Recovery Master for the Dead Node 2 for Domain B0A423E24D5A47DEB93D3448C0B43073<br>Mar 7 07:47:31 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:976 EF2FF243631540E0A9418F491C21D661:D000000000000000002060e659b79cc: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:921 87038E39DCFC4111A179DE517E1B71FC:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:955 87038E39DCFC4111A179DE517E1B71FC: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_do_recovery:519 (10603) Node 1 is the Recovery Master for the Dead Node 2 for Domain 87038E39DCFC4111A179DE517E1B71FC<br>Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:921 D213C5F3386B474FA09DA88744CB0B3F:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:955 D213C5F3386B474FA09DA88744CB0B3F: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_do_recovery:519 (10760) Node 1 is the Recovery Master for the Dead Node 2 for Domain D213C5F3386B474FA09DA88744CB0B3F<br>
Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:921 F96CA86A377E4562A237A572D5AAE337:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:955 F96CA86A377E4562A237A572D5AAE337: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_do_recovery:519 (10917) Node 1 is the Recovery Master for the Dead Node 2 for Domain F96CA86A377E4562A237A572D5AAE337<br>Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:921 3269F5BC97A64188821CB931CAA5F005:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:955 3269F5BC97A64188821CB931CAA5F005: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_do_recovery:519 (10586) Node 1 is the Recovery Master for the Dead Node 2 for Domain 3269F5BC97A64188821CB931CAA5F005<br>
Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:921 EF2FF243631540E0A9418F491C21D661:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:955 EF2FF243631540E0A9418F491C21D661: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_do_recovery:519 (10743) Node 1 is the Recovery Master for the Dead Node 2 for Domain EF2FF243631540E0A9418F491C21D661<br>Mar 7 07:47:33 sapbdb-uat-d1 (32060,8):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,32)<br>
Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:921 F1EC618443464E5295BAEC6CB6A0289A:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:955 F1EC618443464E5295BAEC6CB6A0289A: recovery map is not empty, but must master $RECOVERY lock now<br>
Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_do_recovery:519 (10900) Node 1 is the Recovery Master for the Dead Node 2 for Domain F1EC618443464E5295BAEC6CB6A0289A<br>Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:921 2950FD80D24445598E284B386255DEA2:$RECOVERY: at least one node (2) torecover before lock mastery can begin<br>
Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:955 2950FD80D24445598E284B386255DEA2: recovery map is not empty, but must master $RECOVERY lock now<br>Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_do_recovery:519 (11057) Node 1 is the Recovery Master for the Dead Node 2 for Domain 2950FD80D24445598E284B386255DEA2<br>
Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_restart_lock_mastery:1301 ERROR: node down! 2<br>Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_wait_for_lock_mastery:1118 ERROR: status = -11<br>Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_restart_lock_mastery:1301 ERROR: node down! 2<br>
Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_wait_for_lock_mastery:1118 ERROR: status = -11<br>Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_restart_lock_mastery:1301 ERROR: node down! 2<br>Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_wait_for_lock_mastery:1118 ERROR: status = -11<br>
Mar 7 07:47:33 sapbdb-uat-d1 (32068,11):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,37)<br>Mar 7 07:47:33 sapbdb-uat-d1 (32065,2):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,39)<br>
Mar 7 07:47:33 sapbdb-uat-d1 (32062,11):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,33)<br>Mar 7 07:47:33 sapbdb-uat-d1 (32066,11):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,35)<br>
Mar 7 07:47:33 sapbdb-uat-d1 (32063,8):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,40)<br>Mar 7 07:47:33 sapbdb-uat-d1 (32064,11):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,38)<br>
Mar 7 07:47:34 sapbdb-uat-d1 kjournald starting. Commit interval 5 seconds<br>Mar 7 07:47:34 sapbdb-uat-d1 last message repeated 2 times<br>Mar 7 07:47:34 sapbdb-uat-d1 (32070,11):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,41)<br>
Mar 7 07:47:34 sapbdb-uat-d1 (32069,11):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,34)<br>Mar 7 07:47:34 sapbdb-uat-d1 (32067,11):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,36)<br>
Mar 7 07:47:36 sapbdb-uat-d1 kjournald starting. Commit interval 5 seconds<br>Mar 7 07:47:40 sapbdb-uat-d1 last message repeated 8 times<br>Mar 7 07:47:40 sapbdb-uat-d1 (32056,6):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,46)<br>
Mar 7 07:47:40 sapbdb-uat-d1 (32057,15):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,47)<br>Mar 7 07:47:40 sapbdb-uat-d1 (32052,6):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,44)<br>
Mar 7 07:47:40 sapbdb-uat-d1 (32058,2):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,48)<br>Mar 7 07:47:40 sapbdb-uat-d1 (32061,5):ocfs2_replay_journal:1238 Recovering node 2 from slot 0 on device (253,31)<br>
Mar 7 07:47:40 sapbdb-uat-d1 (32055,4):ocfs2_replay_journal:1238 Recovering node 2 from slot 1 on device (253,45)<br>Mar 7 07:47:44 sapbdb-uat-d1 kjournald starting. Commit interval 5 seconds<br>Mar 7 07:47:47 sapbdb-uat-d1 last message repeated 6 times<br>
Mar 7 07:49:49 sapbdb-uat-d1 dlm: connecting to 2<br>Mar 7 07:49:49 sapbdb-uat-d1 dlm: got connection from 2<br>Mar 7 07:49:59 sapbdb-uat-d1 o2net: accepted connection from node sapbdb-uat-d2 (num 2) at <a href="http://192.168.1.2:7777">192.168.1.2:7777</a><br>
Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain FD5BCEBA9AB041E2AC257C529FEAF692<br>Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("FD5BCEBA9AB041E2AC257C529FEAF692"):<br>Mar 7 07:50:03 sapbdb-uat-d1 1<br>
Mar 7 07:50:04 sapbdb-uat-d1 2<br>Mar 7 07:50:04 sapbdb-uat-d1<br>Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 3DF97CEDF6C04E9AAECE7665905F655C<br>Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("3DF97CEDF6C04E9AAECE7665905F655C"):<br>
Mar 7 07:50:08 sapbdb-uat-d1 1<br>Mar 7 07:50:08 sapbdb-uat-d1 2<br>Mar 7 07:50:08 sapbdb-uat-d1<br>Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain EB739A1B7BCE4948A408D8ACBAC2FCE4<br>Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("EB739A1B7BCE4948A408D8ACBAC2FCE4"):<br>
Mar 7 07:50:12 sapbdb-uat-d1 1<br>Mar 7 07:50:12 sapbdb-uat-d1 2<br>Mar 7 07:50:12 sapbdb-uat-d1<br>Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 3269F5BC97A64188821CB931CAA5F005<br>Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("3269F5BC97A64188821CB931CAA5F005"):<br>
Mar 7 07:50:16 sapbdb-uat-d1 1<br>Mar 7 07:50:16 sapbdb-uat-d1 2<br>Mar 7 07:50:16 sapbdb-uat-d1<br>Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 87038E39DCFC4111A179DE517E1B71FC<br>Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("87038E39DCFC4111A179DE517E1B71FC"):<br>
Mar 7 07:50:20 sapbdb-uat-d1 1<br>Mar 7 07:50:20 sapbdb-uat-d1 2<br>Mar 7 07:50:20 sapbdb-uat-d1<br>Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 549A03816D5A4BEA912BC04E7F118822<br>Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("549A03816D5A4BEA912BC04E7F118822"):<br>
Mar 7 07:50:24 sapbdb-uat-d1 1<br>Mar 7 07:50:24 sapbdb-uat-d1 2<br>Mar 7 07:50:24 sapbdb-uat-d1<br>Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain C2FD0D8F366742D4A9F399FDFF914A78<br>Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("C2FD0D8F366742D4A9F399FDFF914A78"):<br>
Mar 7 07:50:28 sapbdb-uat-d1 1<br>Mar 7 07:50:28 sapbdb-uat-d1 2<br>Mar 7 07:50:28 sapbdb-uat-d1<br>Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 87868F0E137342B0A1C5C525717DA24B<br>Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("87868F0E137342B0A1C5C525717DA24B"):<br>
Mar 7 07:50:32 sapbdb-uat-d1 1<br>Mar 7 07:50:32 sapbdb-uat-d1 2<br>Mar 7 07:50:32 sapbdb-uat-d1<br>Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain EF2FF243631540E0A9418F491C21D661<br>Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("EF2FF243631540E0A9418F491C21D661"):<br>
Mar 7 07:50:36 sapbdb-uat-d1 1<br>Mar 7 07:50:36 sapbdb-uat-d1 2<br>Mar 7 07:50:36 sapbdb-uat-d1<br>Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain D213C5F3386B474FA09DA88744CB0B3F<br>Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("D213C5F3386B474FA09DA88744CB0B3F"):<br>
Mar 7 07:50:40 sapbdb-uat-d1 1<br>Mar 7 07:50:40 sapbdb-uat-d1 2<br>Mar 7 07:50:40 sapbdb-uat-d1<br>Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 02C0423D592746DD89EAC08217172364<br>Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("02C0423D592746DD89EAC08217172364"):<br>
Mar 7 07:50:44 sapbdb-uat-d1 1<br>Mar 7 07:50:44 sapbdb-uat-d1 2<br>Mar 7 07:50:44 sapbdb-uat-d1<br>Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 988B694C74FC44FBB22F5D59CAA5686F<br>Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("988B694C74FC44FBB22F5D59CAA5686F"):<br>
Mar 7 07:50:48 sapbdb-uat-d1 1<br>Mar 7 07:50:48 sapbdb-uat-d1 2<br>Mar 7 07:50:48 sapbdb-uat-d1<br>Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain D82D1AFEC06F4A5591C051A8AD55F6B8<br>Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("D82D1AFEC06F4A5591C051A8AD55F6B8"):<br>
Mar 7 07:50:52 sapbdb-uat-d1 1<br>Mar 7 07:50:52 sapbdb-uat-d1 2<br>Mar 7 07:50:52 sapbdb-uat-d1<br>Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain F1EC618443464E5295BAEC6CB6A0289A<br>Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("F1EC618443464E5295BAEC6CB6A0289A"):<br>
Mar 7 07:50:56 sapbdb-uat-d1 1<br>Mar 7 07:50:56 sapbdb-uat-d1 2<br>Mar 7 07:50:56 sapbdb-uat-d1<br>Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain F96CA86A377E4562A237A572D5AAE337<br>Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("F96CA86A377E4562A237A572D5AAE337"):<br>
Mar 7 07:51:00 sapbdb-uat-d1 1<br>Mar 7 07:51:00 sapbdb-uat-d1 2<br>Mar 7 07:51:00 sapbdb-uat-d1<br>Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain B0A423E24D5A47DEB93D3448C0B43073<br>Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("B0A423E24D5A47DEB93D3448C0B43073"):<br>
Mar 7 07:51:04 sapbdb-uat-d1 1<br>Mar 7 07:51:04 sapbdb-uat-d1 2<br>Mar 7 07:51:04 sapbdb-uat-d1<br>Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 0715F21DDB174D4FA4F3AC717751BB8A<br>Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("0715F21DDB174D4FA4F3AC717751BB8A"):<br>
Mar 7 07:51:09 sapbdb-uat-d1 1<br>Mar 7 07:51:09 sapbdb-uat-d1 2<br>Mar 7 07:51:09 sapbdb-uat-d1<br>Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 17D5F5A73FA94B59A2C3CD91E32DCA99<br>Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("17D5F5A73FA94B59A2C3CD91E32DCA99"):<br>
Mar 7 07:51:13 sapbdb-uat-d1 1<br>Mar 7 07:51:13 sapbdb-uat-d1 2<br>Mar 7 07:51:13 sapbdb-uat-d1<br>Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain 2950FD80D24445598E284B386255DEA2<br>Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain ("2950FD80D24445598E284B386255DEA2"):<br>
Mar 7 07:51:17 sapbdb-uat-d1 1<br>Mar 7 07:51:17 sapbdb-uat-d1 2<br>Mar 7 07:51:17 sapbdb-uat-d1<br>