[Ocfs2-users] Experiencing occasional system crashes with RHEL5 and ocfs2 1.2.9
Sunil Mushran
sunil.mushran at oracle.com
Mon Mar 9 18:29:15 PDT 2009
Known issue. We have a potential fix for it. It is in testing.
How often do you hit this?
On Mon, Mar 09, 2009 at 06:52:57PM -0400, Ward Fenton wrote:
> We have been experiencing unplanned outages on a subset of the
> clustered systems we have deployed to support SAP. The following
> captured information came from one of three ocfs2 clusters which handle
> SAP SEM/BW functionality. Each of those has experienced multiple kernel
> panics which get reported as dlm_drop_lockres_ref and
> dlm_defer_lockres_handler errors.
> This particular environment is deployed on two identical HP DL580-G5
> servers running RHEL5.2 with kernel 2.6.18-92.1.1.el5. It should be
> noted that we have encountered similar issues with similar frequency on
> kernel 2.6.18-92.1.22.el5. We are running ocfs2 1.2.9. These two
> servers support Oracle RAC (10.2.0.2) and they also sharing through NFS
> two ocfs2 file systems to associated SAP application servers. Database
> devices and RMAN backups are supported by ocfs2 file systems with
> "datavolume,nointr" options. The additional cluster file systems which
> support Oracle and SAP do not contain those options.
> We have encountered recurring unplanned reboots which appear to be
> initiated by the crashing of either node in the cluster. This
> particular cluster had experienced a crashed of the other cluster
> member one week previous to this most recent crash. Both crashes had
> occurred at times with very minimal activity.
> Support for netconsole is implemented and we are redirecting the
> kernel's console output to a central syslog server. The logs below
> originated from netconsole. Though not yet in place at the time of this
> crash, kexec/kdump functionality was also introduced.
> Thanks,
> Ward
> #######################################################################
> #########
> netconsole capture of rebooted server:
> #######################################################################
> #########
> Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_drop_lockres_ref:2298
> ERROR: while dropping ref on
> F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26
> (master=1) got -22.
> Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_one_lock_resource:461
> lockres: M00000000000000003c1c868d5afc26, owner=1, state=64
> Mar 7 07:45:25 sapbdb-uat-d2
> (11241,2):__dlm_print_one_lock_resource:476 lockres:
> M00000000000000003c1c868d5afc26, owner=1, state=64
> Mar 7 07:45:25 sapbdb-uat-d2
> (11241,2):__dlm_print_one_lock_resource:478 last used: 4728024828, on
> purge list: yes
> Mar 7 07:45:25 sapbdb-uat-d2 (11241,2):dlm_print_lockres_refmap:444
> refmap nodes: [
> Mar 7 07:45:25 sapbdb-uat-d2 ], inflight=0
> Mar 7 07:45:25 sapbdb-uat-d2
> (11241,2):__dlm_print_one_lock_resource:480 granted queue:
> Mar 7 07:45:25 sapbdb-uat-d2
> (11241,2):__dlm_print_one_lock_resource:495 converting queue:
> Mar 7 07:45:25 sapbdb-uat-d2
> (11241,2):__dlm_print_one_lock_resource:510 blocked queue:
> Mar 7 07:45:25 sapbdb-uat-d2 ----------- [cut here ] --------- [please
> bite here ] ---------
> Mar 7 07:45:25 sapbdb-uat-d2 Kernel BUG at
> ...mushran/BUILD/ocfs2-1.2.9/fs/ocfs2/dlm/dlmmaster.c:2300
> Mar 7 07:45:25 sapbdb-uat-d2 invalid opcode: 0000 [1]
> Mar 7 07:45:25 sapbdb-uat-d2 SMP
> Mar 7 07:45:25 sapbdb-uat-d2
> Mar 7 07:45:25 sapbdb-uat-d2 last sysfs file:
> /devices/pci0000:00/0000:00:00.0/irq
> Mar 7 07:45:25 sapbdb-uat-d2 CPU 2
> Mar 7 07:45:25 sapbdb-uat-d2
> Mar 7 07:45:25 sapbdb-uat-d2 Modules linked in:
> Mar 7 07:45:25 sapbdb-uat-d2 iptable_filter
> Mar 7 07:45:26 sapbdb-uat-d2 ip_tables
> Mar 7 07:45:26 sapbdb-uat-d2 x_tables
> Mar 7 07:45:26 sapbdb-uat-d2 netconsole
> Mar 7 07:45:26 sapbdb-uat-d2 nfsd
> Mar 7 07:45:26 sapbdb-uat-d2 exportfs
> Mar 7 07:45:26 sapbdb-uat-d2 auth_rpcgss
> Mar 7 07:45:26 sapbdb-uat-d2 autofs4
> Mar 7 07:45:26 sapbdb-uat-d2 hidp
> Mar 7 07:45:26 sapbdb-uat-d2 ocfs2(U)
> Mar 7 07:45:26 sapbdb-uat-d2 nfs
> Mar 7 07:45:26 sapbdb-uat-d2 lockd
> Mar 7 07:45:26 sapbdb-uat-d2 fscache
> Mar 7 07:45:26 sapbdb-uat-d2 nfs_acl
> Mar 7 07:45:26 sapbdb-uat-d2 rfcomm
> Mar 7 07:45:26 sapbdb-uat-d2 l2cap
> Mar 7 07:45:26 sapbdb-uat-d2 bluetooth
> Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlmfs(U)
> Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_dlm(U)
> Mar 7 07:45:26 sapbdb-uat-d2 ocfs2_nodemanager(U)
> Mar 7 07:45:26 sapbdb-uat-d2 lock_dlm(U)
> Mar 7 07:45:26 sapbdb-uat-d2 gfs2(U)
> Mar 7 07:45:26 sapbdb-uat-d2 dlm
> Mar 7 07:45:26 sapbdb-uat-d2 configfs
> Mar 7 07:45:26 sapbdb-uat-d2 sunrpc
> Mar 7 07:45:26 sapbdb-uat-d2 bonding
> Mar 7 07:45:26 sapbdb-uat-d2 ipv6
> Mar 7 07:45:26 sapbdb-uat-d2 xfrm_nalgo
> Mar 7 07:45:26 sapbdb-uat-d2 crypto_api
> Mar 7 07:45:26 sapbdb-uat-d2 dm_round_robin
> Mar 7 07:45:26 sapbdb-uat-d2 dm_emc
> Mar 7 07:45:26 sapbdb-uat-d2 dm_multipath
> Mar 7 07:45:26 sapbdb-uat-d2 video
> Mar 7 07:45:26 sapbdb-uat-d2 sbs
> Mar 7 07:45:26 sapbdb-uat-d2 backlight
> Mar 7 07:45:26 sapbdb-uat-d2 i2c_ec
> Mar 7 07:45:26 sapbdb-uat-d2 i2c_core
> Mar 7 07:45:26 sapbdb-uat-d2 button
> Mar 7 07:45:26 sapbdb-uat-d2 battery
> Mar 7 07:45:26 sapbdb-uat-d2 asus_acpi
> Mar 7 07:45:26 sapbdb-uat-d2 acpi_memhotplug
> Mar 7 07:45:26 sapbdb-uat-d2 ac
> Mar 7 07:45:26 sapbdb-uat-d2 parport_pc
> Mar 7 07:45:26 sapbdb-uat-d2 lp
> Mar 7 07:45:26 sapbdb-uat-d2 parport
> Mar 7 07:45:26 sapbdb-uat-d2 joydev
> Mar 7 07:45:26 sapbdb-uat-d2 ata_piix
> Mar 7 07:45:26 sapbdb-uat-d2 libata
> Mar 7 07:45:26 sapbdb-uat-d2 sg
> Mar 7 07:45:26 sapbdb-uat-d2 ide_cd
> Mar 7 07:45:26 sapbdb-uat-d2 shpchp
> Mar 7 07:45:27 sapbdb-uat-d2 cdrom
> Mar 7 07:45:27 sapbdb-uat-d2 bnx2
> Mar 7 07:45:27 sapbdb-uat-d2 pcspkr
> Mar 7 07:45:27 sapbdb-uat-d2 serio_raw
> Mar 7 07:45:27 sapbdb-uat-d2 dm_snapshot
> Mar 7 07:45:27 sapbdb-uat-d2 dm_zero
> Mar 7 07:45:27 sapbdb-uat-d2 dm_mirror
> Mar 7 07:45:27 sapbdb-uat-d2 dm_mod
> Mar 7 07:45:27 sapbdb-uat-d2 qla2xxx
> Mar 7 07:45:27 sapbdb-uat-d2 scsi_transport_fc
> Mar 7 07:45:27 sapbdb-uat-d2 cciss
> Mar 7 07:45:27 sapbdb-uat-d2 sd_mod
> Mar 7 07:45:27 sapbdb-uat-d2 scsi_mod
> Mar 7 07:45:27 sapbdb-uat-d2 ext3
> Mar 7 07:45:27 sapbdb-uat-d2 jbd
> Mar 7 07:45:27 sapbdb-uat-d2 uhci_hcd
> Mar 7 07:45:27 sapbdb-uat-d2 ohci_hcd
> Mar 7 07:45:27 sapbdb-uat-d2 ehci_hcd
> Mar 7 07:45:27 sapbdb-uat-d2
> Mar 7 07:45:27 sapbdb-uat-d2 Pid: 11241, comm: dlm_thread Tainted:
> G 2.6.18-92.1.1.el5 #1
> Mar 7 07:45:27 sapbdb-uat-d2 RIP: 0010:[<ffffffff885ba6a0>]
> Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885ba6a0>]
> :ocfs2_dlm:dlm_drop_lockres_ref+0x1dc/0x1f5
> Mar 7 07:45:27 sapbdb-uat-d2 RSP: 0018:ffff8105e1395de0 EFLAGS:
> 00010246
> Mar 7 07:45:27 sapbdb-uat-d2 RAX: ffff81005ef3fe38 RBX:
> 0000000000000000 RCX: ffffffff802ed9a8
> Mar 7 07:45:27 sapbdb-uat-d2 RDX: ffffffff802ed9a8 RSI:
> 0000000000000000 RDI: ffffffff802ed9a0
> Mar 7 07:45:27 sapbdb-uat-d2 RBP: 000000000000001f R08:
> ffffffff802ed9a8 R09: 0000000000000046
> Mar 7 07:45:27 sapbdb-uat-d2 R10: 0000000000000000 R11:
> 0000000000000280 R12: ffff81005ef3fe00
> Mar 7 07:45:27 sapbdb-uat-d2 R13: ffff8105e227fc00 R14:
> ffff81002aec2d40 R15: ffffffff8009dced
> Mar 7 07:45:27 sapbdb-uat-d2 FS: 0000000000000000(0000)
> GS:ffff81102ff1ce40(0000) knlGS:0000000000000000
> Mar 7 07:45:27 sapbdb-uat-d2 CS: 0010 DS: 0018 ES: 0018 CR0:
> 000000008005003b
> Mar 7 07:45:27 sapbdb-uat-d2 CR2: 00000000006bdc5c CR3:
> 0000000594e3c000 CR4: 00000000000006e0
> Mar 7 07:45:27 sapbdb-uat-d2 Process dlm_thread (pid: 11241,
> threadinfo ffff8105e1394000, task ffff8105e30a2820)
> Mar 7 07:45:27 sapbdb-uat-d2 Stack:
> Mar 7 07:45:27 sapbdb-uat-d2 1f02000000000000
> Mar 7 07:45:27 sapbdb-uat-d2 303030303030304d
> Mar 7 07:45:27 sapbdb-uat-d2 3030303030303030
> Mar 7 07:45:27 sapbdb-uat-d2 3836386331633330
> Mar 7 07:45:27 sapbdb-uat-d2
> Mar 7 07:45:27 sapbdb-uat-d2 0036326366613564
> Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000
> Mar 7 07:45:27 sapbdb-uat-d2 last message repeated 2 times
> Mar 7 07:45:27 sapbdb-uat-d2
> Mar 7 07:45:27 sapbdb-uat-d2 0000000000000000
> Mar 7 07:45:27 sapbdb-uat-d2 ffffffea00100100
> Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe48
> Mar 7 07:45:27 sapbdb-uat-d2 ffff81005ef3fe00
> Mar 7 07:45:27 sapbdb-uat-d2
> Mar 7 07:45:27 sapbdb-uat-d2 Call Trace:
> Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cd72b>]
> :ocfs2_dlm:dlm_purge_lockres+0x175/0x34f
> Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdb98>]
> :ocfs2_dlm:dlm_thread+0xd7/0x579
> Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff8009df05>]
> autoremove_wake_function+0x0/0x2e
> Mar 7 07:45:27 sapbdb-uat-d2 [<ffffffff885cdac1>]
> :ocfs2_dlm:dlm_thread+0x0/0x579
> Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003253d>] kthread+0xfe/0x132
> Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfb1>] child_rip+0xa/0x11
> Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8009dced>]
> keventd_create_kthread+0x0/0xc4
> Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8003243f>] kthread+0x0/0x132
> Mar 7 07:45:28 sapbdb-uat-d2 [<ffffffff8005dfa7>] child_rip+0x0/0x11
> Mar 7 07:45:28 sapbdb-uat-d2
> Mar 7 07:45:28 sapbdb-uat-d2
> Mar 7 07:45:28 sapbdb-uat-d2 Code:
> Mar 7 07:45:28 sapbdb-uat-d2 0f
> Mar 7 07:45:28 sapbdb-uat-d2 0b
> Mar 7 07:45:28 sapbdb-uat-d2 68
> Mar 7 07:45:28 sapbdb-uat-d2 2d
> Mar 7 07:45:28 sapbdb-uat-d2 10
> Mar 7 07:45:28 sapbdb-uat-d2 5d
> Mar 7 07:45:28 sapbdb-uat-d2 88
> Mar 7 07:45:28 sapbdb-uat-d2 c2
> Mar 7 07:45:28 sapbdb-uat-d2 fc
> #######################################################################
> #########
> netconsole capture for remaining server:
> #######################################################################
> #########
> Mar 7 07:45:25 sapbdb-uat-d1 (9917,0):dlm_deref_lockres_handler:2340
> ERROR:
> F96CA86A377E4562A237A572D5AAE337:M00000000000000003c1c868d5afc26: bad
> lockres name
> Mar 7 07:45:40 sapbdb-uat-d1 dlm: closing connection to node 2
> Mar 7 07:45:55 sapbdb-uat-d1 o2net: connection to node sapbdb-uat-d2
> (num 2) at [1]192.168.1.2:7777 has been idle for 30.0 seconds, shutting
> it down.
> Mar 7 07:45:55 sapbdb-uat-d1 (0,0):o2net_idle_timer:1426 here are some
> times that might help debug the situation: (tmr 1236429925.620147 now
> 1236429955.618766 dr 1236429925.620142 adv
> 1236429925.622211:1236429925.622212 func (90648c68:507)
> 1236429925.620148:1236429925.622206)
> Mar 7 07:45:55 sapbdb-uat-d1 o2net: no longer connected to node
> sapbdb-uat-d2 (num 2) at [2]192.168.1.2:7777
> Mar 7 07:45:55 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293
> ERROR: status = -112
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -112
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -112
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_do_master_request:1418
> ERROR: link to 2 went down!
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:995 ERROR:
> status = -112
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:55 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:56 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10759,0):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR:
> status = -112
> Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_drop_lockres_ref:2293
> ERROR: status = -107
> Mar 7 07:45:57 sapbdb-uat-d1 (10950,9):dlm_purge_lockres:191 ERROR:
> status = -107
> Mar 7 07:46:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585
> ERROR: no connection established with node 2 after 30.0 seconds, giving
> up and returning errors.
> Mar 7 07:46:55 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585
> ERROR: no connection established with node 2 after 30.0 seconds, giving
> up and returning errors.
> Mar 7 07:47:01 sapbdb-uat-d1
> (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:01 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
> 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:06 sapbdb-uat-d1
> (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:06 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
> 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:11 sapbdb-uat-d1
> (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:11 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
> 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:17 sapbdb-uat-d1
> (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:17 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
> 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:22 sapbdb-uat-d1
> (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:22 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
> 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:25 sapbdb-uat-d1 (9917,0):o2net_connect_expired:1585
> ERROR: no connection established with node 2 after 30.0 seconds, giving
> up and returning errors.
> Mar 7 07:47:27 sapbdb-uat-d1
> (31270,15):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:27 sapbdb-uat-d1 (31270,15):dlm_wait_for_node_death:365
> 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1
> (32060,5):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32060,5):dlm_wait_for_node_death:365
> EF2FF243631540E0A9418F491C21D661: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1
> (32062,11):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32062,11):dlm_wait_for_node_death:365
> 02C0423D592746DD89EAC08217172364: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1
> (32063,11):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32063,11):dlm_wait_for_node_death:365
> 988B694C74FC44FBB22F5D59CAA5686F: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1
> (32064,11):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32064,11):dlm_wait_for_node_death:365
> D82D1AFEC06F4A5591C051A8AD55F6B8: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1
> (32065,11):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32065,11):dlm_wait_for_node_death:365
> F1EC618443464E5295BAEC6CB6A0289A: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1
> (32066,11):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32066,11):dlm_wait_for_node_death:365
> F96CA86A377E4562A237A572D5AAE337: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1
> (32068,11):dlm_send_remote_convert_request:398 ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32068,11):dlm_wait_for_node_death:365
> 0715F21DDB174D4FA4F3AC717751BB8A: waiting 5000ms for notification of
> death of node 2
> Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_do_master_request:1418
> ERROR: link to 2 went down!
> Mar 7 07:47:28 sapbdb-uat-d1 (32070,11):dlm_get_lock_resource:995
> ERROR: status = -107
> Mar 7 07:47:28 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:921
> 3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:921
> D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_do_master_request:1418
> ERROR: link to 2 went down!
> Mar 7 07:47:29 sapbdb-uat-d1 (32069,11):dlm_get_lock_resource:995
> ERROR: status = -107
> Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:921
> 87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:921
> 549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32053,1):dlm_get_lock_resource:921
> 3DF97CEDF6C04E9AAECE7665905F655C:M000000000000000000008fd0deae0b: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:921
> FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_do_master_request:1418
> ERROR: link to 2 went down!
> Mar 7 07:47:29 sapbdb-uat-d1 (32067,11):dlm_get_lock_resource:995
> ERROR: status = -107
> Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:921
> C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):dlm_get_lock_resource:921
> EB739A1B7BCE4948A408D8ACBAC2FCE4:M000000000000000000008f9047acff: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32059,1):dlm_get_lock_resource:921
> 87868F0E137342B0A1C5C525717DA24B:M000000000000000000008fef7ad4cb: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:921
> EB739A1B7BCE4948A408D8ACBAC2FCE4:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_get_lock_resource:955
> EB739A1B7BCE4948A408D8ACBAC2FCE4: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:29 sapbdb-uat-d1 (10569,12):dlm_do_recovery:519 (10569)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> EB739A1B7BCE4948A408D8ACBAC2FCE4
> Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:921
> 87868F0E137342B0A1C5C525717DA24B:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_get_lock_resource:955
> 87868F0E137342B0A1C5C525717DA24B: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:29 sapbdb-uat-d1 (10726,10):dlm_do_recovery:519 (10726)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 87868F0E137342B0A1C5C525717DA24B
> Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:921
> D82D1AFEC06F4A5591C051A8AD55F6B8:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_get_lock_resource:955
> D82D1AFEC06F4A5591C051A8AD55F6B8: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:29 sapbdb-uat-d1 (10883,3):dlm_do_recovery:519 (10883)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> D82D1AFEC06F4A5591C051A8AD55F6B8
> Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:921
> 17D5F5A73FA94B59A2C3CD91E32DCA99:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_get_lock_resource:955
> 17D5F5A73FA94B59A2C3CD91E32DCA99: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:29 sapbdb-uat-d1 (11040,6):dlm_do_recovery:519 (11040)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 17D5F5A73FA94B59A2C3CD91E32DCA99
> Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:921
> 3DF97CEDF6C04E9AAECE7665905F655C:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_get_lock_resource:955
> 3DF97CEDF6C04E9AAECE7665905F655C: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:29 sapbdb-uat-d1 (10480,1):dlm_do_recovery:519 (10480)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 3DF97CEDF6C04E9AAECE7665905F655C
> Mar 7 07:47:29 sapbdb-uat-d1 (32061,11):dlm_get_lock_resource:976
> D213C5F3386B474FA09DA88744CB0B3F:M000000000000000000002e6b72ff40: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32056,1):dlm_get_lock_resource:976
> 87038E39DCFC4111A179DE517E1B71FC:M000000000000000000008f63a37ee5: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32057,15):dlm_get_lock_resource:976
> 549A03816D5A4BEA912BC04E7F118822:M000000000000000000008f32b3eb36: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32053,8):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,42)
> Mar 7 07:47:29 sapbdb-uat-d1 (32052,6):dlm_get_lock_resource:976
> FD5BCEBA9AB041E2AC257C529FEAF692:M000000000000000000008f3c46ad60: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32058,2):dlm_get_lock_resource:976
> C2FD0D8F366742D4A9F399FDFF914A78:M000000000000000000008f09f23f76: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32054,0):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,43)
> Mar 7 07:47:29 sapbdb-uat-d1 (32055,4):dlm_get_lock_resource:976
> 3269F5BC97A64188821CB931CAA5F005:M000000000000000000008f86f22490: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (32059,8):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,49)
> Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:921
> C2FD0D8F366742D4A9F399FDFF914A78:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_get_lock_resource:955
> C2FD0D8F366742D4A9F399FDFF914A78: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:29 sapbdb-uat-d1 (10637,10):dlm_do_recovery:519 (10637)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> C2FD0D8F366742D4A9F399FDFF914A78
> Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:921
> 988B694C74FC44FBB22F5D59CAA5686F:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_get_lock_resource:955
> 988B694C74FC44FBB22F5D59CAA5686F: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:30 sapbdb-uat-d1 (10794,5):dlm_do_recovery:519 (10794)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 988B694C74FC44FBB22F5D59CAA5686F
> Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:921
> 0715F21DDB174D4FA4F3AC717751BB8A:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_get_lock_resource:955
> 0715F21DDB174D4FA4F3AC717751BB8A: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:30 sapbdb-uat-d1 (10951,3):dlm_do_recovery:519 (10951)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 0715F21DDB174D4FA4F3AC717751BB8A
> Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_restart_lock_mastery:1301
> ERROR: node down! 2
> Mar 7 07:47:30 sapbdb-uat-d1 (8703,3):dlm_wait_for_lock_mastery:1118
> ERROR: status = -11
> Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:921
> FD5BCEBA9AB041E2AC257C529FEAF692:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_get_lock_resource:955
> FD5BCEBA9AB041E2AC257C529FEAF692: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:30 sapbdb-uat-d1 (10463,11):dlm_do_recovery:519 (10463)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> FD5BCEBA9AB041E2AC257C529FEAF692
> Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:921
> 549A03816D5A4BEA912BC04E7F118822:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_get_lock_resource:955
> 549A03816D5A4BEA912BC04E7F118822: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:30 sapbdb-uat-d1 (10620,3):dlm_do_recovery:519 (10620)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 549A03816D5A4BEA912BC04E7F118822
> Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:921
> 02C0423D592746DD89EAC08217172364:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_get_lock_resource:955
> 02C0423D592746DD89EAC08217172364: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:31 sapbdb-uat-d1 (10777,15):dlm_do_recovery:519 (10777)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 02C0423D592746DD89EAC08217172364
> Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:921
> B0A423E24D5A47DEB93D3448C0B43073:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_get_lock_resource:955
> B0A423E24D5A47DEB93D3448C0B43073: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:31 sapbdb-uat-d1 (10934,7):dlm_do_recovery:519 (10934)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> B0A423E24D5A47DEB93D3448C0B43073
> Mar 7 07:47:31 sapbdb-uat-d1 (8703,3):dlm_get_lock_resource:976
> EF2FF243631540E0A9418F491C21D661:D000000000000000002060e659b79cc: at
> least one node (2) torecover before lock mastery can begin
> Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:921
> 87038E39DCFC4111A179DE517E1B71FC:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_get_lock_resource:955
> 87038E39DCFC4111A179DE517E1B71FC: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:31 sapbdb-uat-d1 (10603,4):dlm_do_recovery:519 (10603)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 87038E39DCFC4111A179DE517E1B71FC
> Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:921
> D213C5F3386B474FA09DA88744CB0B3F:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_get_lock_resource:955
> D213C5F3386B474FA09DA88744CB0B3F: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:31 sapbdb-uat-d1 (10760,3):dlm_do_recovery:519 (10760)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> D213C5F3386B474FA09DA88744CB0B3F
> Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:921
> F96CA86A377E4562A237A572D5AAE337:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_get_lock_resource:955
> F96CA86A377E4562A237A572D5AAE337: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:32 sapbdb-uat-d1 (10917,5):dlm_do_recovery:519 (10917)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> F96CA86A377E4562A237A572D5AAE337
> Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:921
> 3269F5BC97A64188821CB931CAA5F005:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_get_lock_resource:955
> 3269F5BC97A64188821CB931CAA5F005: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:32 sapbdb-uat-d1 (10586,12):dlm_do_recovery:519 (10586)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 3269F5BC97A64188821CB931CAA5F005
> Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:921
> EF2FF243631540E0A9418F491C21D661:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_get_lock_resource:955
> EF2FF243631540E0A9418F491C21D661: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:32 sapbdb-uat-d1 (10743,9):dlm_do_recovery:519 (10743)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> EF2FF243631540E0A9418F491C21D661
> Mar 7 07:47:33 sapbdb-uat-d1 (32060,8):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,32)
> Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:921
> F1EC618443464E5295BAEC6CB6A0289A:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_get_lock_resource:955
> F1EC618443464E5295BAEC6CB6A0289A: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:33 sapbdb-uat-d1 (10900,6):dlm_do_recovery:519 (10900)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> F1EC618443464E5295BAEC6CB6A0289A
> Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:921
> 2950FD80D24445598E284B386255DEA2:$RECOVERY: at least one node (2)
> torecover before lock mastery can begin
> Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_get_lock_resource:955
> 2950FD80D24445598E284B386255DEA2: recovery map is not empty, but must
> master $RECOVERY lock now
> Mar 7 07:47:33 sapbdb-uat-d1 (11057,11):dlm_do_recovery:519 (11057)
> Node 1 is the Recovery Master for the Dead Node 2 for Domain
> 2950FD80D24445598E284B386255DEA2
> Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_restart_lock_mastery:1301
> ERROR: node down! 2
> Mar 7 07:47:33 sapbdb-uat-d1 (32070,11):dlm_wait_for_lock_mastery:1118
> ERROR: status = -11
> Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_restart_lock_mastery:1301
> ERROR: node down! 2
> Mar 7 07:47:33 sapbdb-uat-d1 (32069,11):dlm_wait_for_lock_mastery:1118
> ERROR: status = -11
> Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_restart_lock_mastery:1301
> ERROR: node down! 2
> Mar 7 07:47:33 sapbdb-uat-d1 (32067,11):dlm_wait_for_lock_mastery:1118
> ERROR: status = -11
> Mar 7 07:47:33 sapbdb-uat-d1 (32068,11):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,37)
> Mar 7 07:47:33 sapbdb-uat-d1 (32065,2):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,39)
> Mar 7 07:47:33 sapbdb-uat-d1 (32062,11):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,33)
> Mar 7 07:47:33 sapbdb-uat-d1 (32066,11):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,35)
> Mar 7 07:47:33 sapbdb-uat-d1 (32063,8):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,40)
> Mar 7 07:47:33 sapbdb-uat-d1 (32064,11):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,38)
> Mar 7 07:47:34 sapbdb-uat-d1 kjournald starting. Commit interval 5
> seconds
> Mar 7 07:47:34 sapbdb-uat-d1 last message repeated 2 times
> Mar 7 07:47:34 sapbdb-uat-d1 (32070,11):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,41)
> Mar 7 07:47:34 sapbdb-uat-d1 (32069,11):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,34)
> Mar 7 07:47:34 sapbdb-uat-d1 (32067,11):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,36)
> Mar 7 07:47:36 sapbdb-uat-d1 kjournald starting. Commit interval 5
> seconds
> Mar 7 07:47:40 sapbdb-uat-d1 last message repeated 8 times
> Mar 7 07:47:40 sapbdb-uat-d1 (32056,6):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,46)
> Mar 7 07:47:40 sapbdb-uat-d1 (32057,15):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,47)
> Mar 7 07:47:40 sapbdb-uat-d1 (32052,6):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,44)
> Mar 7 07:47:40 sapbdb-uat-d1 (32058,2):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,48)
> Mar 7 07:47:40 sapbdb-uat-d1 (32061,5):ocfs2_replay_journal:1238
> Recovering node 2 from slot 0 on device (253,31)
> Mar 7 07:47:40 sapbdb-uat-d1 (32055,4):ocfs2_replay_journal:1238
> Recovering node 2 from slot 1 on device (253,45)
> Mar 7 07:47:44 sapbdb-uat-d1 kjournald starting. Commit interval 5
> seconds
> Mar 7 07:47:47 sapbdb-uat-d1 last message repeated 6 times
> Mar 7 07:49:49 sapbdb-uat-d1 dlm: connecting to 2
> Mar 7 07:49:49 sapbdb-uat-d1 dlm: got connection from 2
> Mar 7 07:49:59 sapbdb-uat-d1 o2net: accepted connection from node
> sapbdb-uat-d2 (num 2) at [3]192.168.1.2:7777
> Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> FD5BCEBA9AB041E2AC257C529FEAF692
> Mar 7 07:50:03 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("FD5BCEBA9AB041E2AC257C529FEAF692"):
> Mar 7 07:50:03 sapbdb-uat-d1 1
> Mar 7 07:50:04 sapbdb-uat-d1 2
> Mar 7 07:50:04 sapbdb-uat-d1
> Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 3DF97CEDF6C04E9AAECE7665905F655C
> Mar 7 07:50:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("3DF97CEDF6C04E9AAECE7665905F655C"):
> Mar 7 07:50:08 sapbdb-uat-d1 1
> Mar 7 07:50:08 sapbdb-uat-d1 2
> Mar 7 07:50:08 sapbdb-uat-d1
> Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> EB739A1B7BCE4948A408D8ACBAC2FCE4
> Mar 7 07:50:12 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("EB739A1B7BCE4948A408D8ACBAC2FCE4"):
> Mar 7 07:50:12 sapbdb-uat-d1 1
> Mar 7 07:50:12 sapbdb-uat-d1 2
> Mar 7 07:50:12 sapbdb-uat-d1
> Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 3269F5BC97A64188821CB931CAA5F005
> Mar 7 07:50:16 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("3269F5BC97A64188821CB931CAA5F005"):
> Mar 7 07:50:16 sapbdb-uat-d1 1
> Mar 7 07:50:16 sapbdb-uat-d1 2
> Mar 7 07:50:16 sapbdb-uat-d1
> Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 87038E39DCFC4111A179DE517E1B71FC
> Mar 7 07:50:20 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("87038E39DCFC4111A179DE517E1B71FC"):
> Mar 7 07:50:20 sapbdb-uat-d1 1
> Mar 7 07:50:20 sapbdb-uat-d1 2
> Mar 7 07:50:20 sapbdb-uat-d1
> Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 549A03816D5A4BEA912BC04E7F118822
> Mar 7 07:50:24 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("549A03816D5A4BEA912BC04E7F118822"):
> Mar 7 07:50:24 sapbdb-uat-d1 1
> Mar 7 07:50:24 sapbdb-uat-d1 2
> Mar 7 07:50:24 sapbdb-uat-d1
> Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> C2FD0D8F366742D4A9F399FDFF914A78
> Mar 7 07:50:28 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("C2FD0D8F366742D4A9F399FDFF914A78"):
> Mar 7 07:50:28 sapbdb-uat-d1 1
> Mar 7 07:50:28 sapbdb-uat-d1 2
> Mar 7 07:50:28 sapbdb-uat-d1
> Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 87868F0E137342B0A1C5C525717DA24B
> Mar 7 07:50:32 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("87868F0E137342B0A1C5C525717DA24B"):
> Mar 7 07:50:32 sapbdb-uat-d1 1
> Mar 7 07:50:32 sapbdb-uat-d1 2
> Mar 7 07:50:32 sapbdb-uat-d1
> Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> EF2FF243631540E0A9418F491C21D661
> Mar 7 07:50:36 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("EF2FF243631540E0A9418F491C21D661"):
> Mar 7 07:50:36 sapbdb-uat-d1 1
> Mar 7 07:50:36 sapbdb-uat-d1 2
> Mar 7 07:50:36 sapbdb-uat-d1
> Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> D213C5F3386B474FA09DA88744CB0B3F
> Mar 7 07:50:40 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("D213C5F3386B474FA09DA88744CB0B3F"):
> Mar 7 07:50:40 sapbdb-uat-d1 1
> Mar 7 07:50:40 sapbdb-uat-d1 2
> Mar 7 07:50:40 sapbdb-uat-d1
> Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 02C0423D592746DD89EAC08217172364
> Mar 7 07:50:44 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("02C0423D592746DD89EAC08217172364"):
> Mar 7 07:50:44 sapbdb-uat-d1 1
> Mar 7 07:50:44 sapbdb-uat-d1 2
> Mar 7 07:50:44 sapbdb-uat-d1
> Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 988B694C74FC44FBB22F5D59CAA5686F
> Mar 7 07:50:48 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("988B694C74FC44FBB22F5D59CAA5686F"):
> Mar 7 07:50:48 sapbdb-uat-d1 1
> Mar 7 07:50:48 sapbdb-uat-d1 2
> Mar 7 07:50:48 sapbdb-uat-d1
> Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> D82D1AFEC06F4A5591C051A8AD55F6B8
> Mar 7 07:50:52 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("D82D1AFEC06F4A5591C051A8AD55F6B8"):
> Mar 7 07:50:52 sapbdb-uat-d1 1
> Mar 7 07:50:52 sapbdb-uat-d1 2
> Mar 7 07:50:52 sapbdb-uat-d1
> Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> F1EC618443464E5295BAEC6CB6A0289A
> Mar 7 07:50:56 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("F1EC618443464E5295BAEC6CB6A0289A"):
> Mar 7 07:50:56 sapbdb-uat-d1 1
> Mar 7 07:50:56 sapbdb-uat-d1 2
> Mar 7 07:50:56 sapbdb-uat-d1
> Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> F96CA86A377E4562A237A572D5AAE337
> Mar 7 07:51:00 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("F96CA86A377E4562A237A572D5AAE337"):
> Mar 7 07:51:00 sapbdb-uat-d1 1
> Mar 7 07:51:00 sapbdb-uat-d1 2
> Mar 7 07:51:00 sapbdb-uat-d1
> Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> B0A423E24D5A47DEB93D3448C0B43073
> Mar 7 07:51:04 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("B0A423E24D5A47DEB93D3448C0B43073"):
> Mar 7 07:51:04 sapbdb-uat-d1 1
> Mar 7 07:51:04 sapbdb-uat-d1 2
> Mar 7 07:51:04 sapbdb-uat-d1
> Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 0715F21DDB174D4FA4F3AC717751BB8A
> Mar 7 07:51:08 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("0715F21DDB174D4FA4F3AC717751BB8A"):
> Mar 7 07:51:09 sapbdb-uat-d1 1
> Mar 7 07:51:09 sapbdb-uat-d1 2
> Mar 7 07:51:09 sapbdb-uat-d1
> Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 17D5F5A73FA94B59A2C3CD91E32DCA99
> Mar 7 07:51:13 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("17D5F5A73FA94B59A2C3CD91E32DCA99"):
> Mar 7 07:51:13 sapbdb-uat-d1 1
> Mar 7 07:51:13 sapbdb-uat-d1 2
> Mar 7 07:51:13 sapbdb-uat-d1
> Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Node 2 joins domain
> 2950FD80D24445598E284B386255DEA2
> Mar 7 07:51:17 sapbdb-uat-d1 ocfs2_dlm: Nodes in domain
> ("2950FD80D24445598E284B386255DEA2"):
> Mar 7 07:51:17 sapbdb-uat-d1 1
> Mar 7 07:51:17 sapbdb-uat-d1 2
> Mar 7 07:51:17 sapbdb-uat-d1
>
> References
>
> 1. http://192.168.1.2:7777/
> 2. http://192.168.1.2:7777/
> 3. http://192.168.1.2:7777/
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
More information about the Ocfs2-users
mailing list