[Ocfs2-users] HELP - Nodes (A and B) hang mounting ocfs2 filesystem
Shaffin Bhanji
shaffin.bhanji at gmail.com
Fri Jun 5 04:29:11 PDT 2009
Hello,
I am trying to setup a 2 node cluster using SLES11 + OpenAIS +
Pacemaker + OCFS2. Node A is running the iSCSI Target (partition
/dev/sda4) as well as the iSCSI Initiator (partition /sdb1), and Node
B the iSCSI Initiator (partition /dev/sdb1). I have setup DLM and O2CB
on my cluster and running on both nodes. The problem is when I mount
(mount /dev/sdb1 /shared) on Node A all work well but when I try the
same on Node B - it hangs. I checked my /var/log/messages and there is
no error. I am attaching my /var/log/messages and notice at the end of
the file that there are no errors though. The /var/log/messages
fllows:
Jun 3 08:40:56 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 08:40:56 srv-dell cib: [13828]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-14.raw
Jun 3 08:40:57 srv-dell cib: [13828]: info: write_cib_contents: Wrote
version 0.14.0 of the CIB to disk (digest:
23778c5192581531ff34caa8a77dff0a)
Jun 3 08:40:57 srv-dell cib: [13828]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.cI0AbI (digest:
/var/lib/heartbeat/crm/cib.WCjq25)
Jun 3 08:40:57 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 08:41:06 srv-dell openais[13697]: [TOTEM] entering GATHER state from 11.
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] Saving state aru 12
high seq received 12
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] Storing new sequence
id for ring b8
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] entering COMMIT state.
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] entering RECOVERY state.
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] previous ring seq 180
rep 192.168.2.120
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] aru a high delivered
a received flag 1
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] previous ring seq 180
rep 192.168.2.220
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] aru 12 high delivered
12 received flag 1
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] New Configuration:
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] Members Left:
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] Members Joined:
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 184: memb=1,
new=0, lost=0
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] New Configuration:
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] Members Left:
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] Members Joined:
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 184: memb=2, new=1,
lost=0
Jun 3 08:41:07 srv-dell openais[13697]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: member
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: NEW: srv-lenovo 2013440192
Jun 3 08:41:07 srv-dell crmd: [13708]: notice: ais_dispatch:
Membership 184: quorum aquired
Jun 3 08:41:07 srv-dell crmd: [13708]: info: ais_status_callback:
status: srv-lenovo is now member (was lost)
Jun 3 08:41:07 srv-dell crmd: [13708]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=176 seen=184
proc=00000000000000000000000000000002
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: MEMB: srv-lenovo 2013440192
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] info:
send_member_notification: Sending membership update 184 to 2 children
Jun 3 08:41:07 srv-dell openais[13697]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 08:41:07 srv-dell openais[13697]: [TOTEM] entering OPERATIONAL state.
Jun 3 08:41:07 srv-dell openais[13697]: [MAIN ] info: update_member:
0x7fcaa40349b0 Node 2013440192 (srv-lenovo) born on: 184
Jun 3 08:41:07 srv-dell openais[13697]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 3 08:41:07 srv-dell crmd: [13708]: info: crm_update_quorum:
Updating quorum status to true (call=82)
Jun 3 08:41:07 srv-dell openais[13697]: [crm ] info:
send_member_notification: Sending membership update 184 to 2 children
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 08:41:07 srv-dell openais[13697]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 08:41:07 srv-dell cib: [13704]: notice: ais_dispatch:
Membership 184: quorum aquired
Jun 3 08:41:07 srv-dell cib: [13704]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=176 seen=184
proc=00000000000000000000000000000002
Jun 3 08:41:07 srv-dell cib: [13704]: info: ais_dispatch: Membership
184: quorum retained
Jun 3 08:41:07 srv-dell cib: [13704]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=184 seen=184 proc=00000000000000000000000000053312 (new)
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/78,
version=0.14.2): ok (rc=0)
Jun 3 08:41:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=local/crmd/79, version=0.14.3): ok (rc=0)
Jun 3 08:41:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/80, version=0.14.3): ok (rc=0)
Jun 3 08:41:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:07 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib have-quorum="0" admin_epoch="0" epoch="14"
num_updates="4" />
Jun 3 08:41:07 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib have-quorum="1" admin_epoch="0" epoch="15"
num_updates="1" />
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/82, version=0.15.1): ok (rc=0)
Jun 3 08:41:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_sync_one for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.15.1): ok (rc=0)
Jun 3 08:41:07 srv-dell crmd: [13708]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=cluster_ip_address_monitor_0,
magic=0:7;6:4:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2) : Resource op
removal
Jun 3 08:41:07 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/84, version=0.15.1): ok (rc=0)
Jun 3 08:41:07 srv-dell crmd: [13708]: info: abort_transition_graph:
te_update_diff:157 - Triggered transition abort (complete=1,
tag=transient_attributes, id=srv-lenovo, magic=NA) : Transient
attribute: removal
Jun 3 08:41:07 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/transient_attributes":
ok (rc=0)
Jun 3 08:41:07 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 08:41:07 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to have-quorum
Jun 3 08:41:07 srv-dell crmd: [13708]: info: ais_dispatch: Membership
184: quorum retained
Jun 3 08:41:07 srv-dell crmd: [13708]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=184 seen=184 proc=00000000000000000000000000053312 (new)
Jun 3 08:41:07 srv-dell cib: [13829]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-15.raw
Jun 3 08:41:07 srv-dell cib: [13829]: info: write_cib_contents: Wrote
version 0.15.0 of the CIB to disk (digest:
d2105f8dd2178886f72432d2c3920f59)
Jun 3 08:41:07 srv-dell cib: [13829]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.e33W5w (digest:
/var/lib/heartbeat/crm/cib.v0m2Vh)
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/85, version=0.15.1): ok (rc=0)
Jun 3 08:41:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:07 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 08:41:07 srv-dell crmd: [13708]: info: do_state_transition:
Membership changed: 176 -> 184 - join restart
Jun 3 08:41:07 srv-dell crmd: [13708]: info: do_pe_invoke: Query 89:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 08:41:07 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=do_state_transition ]
Jun 3 08:41:07 srv-dell crmd: [13708]: info: update_dc: Unset DC srv-dell
Jun 3 08:41:07 srv-dell crmd: [13708]: info: join_make_offer: Making
join offers based on membership 184
Jun 3 08:41:07 srv-dell crmd: [13708]: info: do_dc_join_offer_all:
join-4: Waiting on 2 outstanding join acks
Jun 3 08:41:07 srv-dell crmd: [13708]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 08:41:07 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/88, version=0.15.2): ok (rc=0)
Jun 3 08:41:07 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 08:41:08 srv-dell crmd: [13708]: info: update_dc: Unset DC srv-dell
Jun 3 08:41:08 srv-dell crmd: [13708]: info: do_dc_join_offer_all: A
new node joined the cluster
Jun 3 08:41:08 srv-dell crmd: [13708]: info: do_dc_join_offer_all:
join-5: Waiting on 2 outstanding join acks
Jun 3 08:41:08 srv-dell crmd: [13708]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 08:41:08 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 08:41:08 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes responded to the join offer.
Jun 3 08:41:08 srv-dell crmd: [13708]: info: do_dc_join_finalize:
join-5: Syncing the CIB from srv-dell to the rest of the cluster
Jun 3 08:41:08 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/92, version=0.15.2): ok (rc=0)
Jun 3 08:41:08 srv-dell crmd: [13708]: info: do_dc_join_ack: join-5:
Updating node state to member for srv-lenovo
Jun 3 08:41:08 srv-dell crmd: [13708]: info: do_dc_join_ack: join-5:
Updating node state to member for srv-dell
Jun 3 08:41:08 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/93, version=0.15.2): ok (rc=0)
Jun 3 08:41:08 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/94, version=0.15.2): ok (rc=0)
Jun 3 08:41:09 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/95,
version=0.15.2): ok (rc=0)
Jun 3 08:41:09 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 3 08:41:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:09 srv-dell crmd: [13708]: info: crm_update_quorum:
Updating quorum status to true (call=101)
Jun 3 08:41:09 srv-dell crmd: [13708]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_pe_invoke: Query 102:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 08:41:09 srv-dell attrd: [13706]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 08:41:09 srv-dell attrd: [13706]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 08:41:09 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/97,
version=0.15.4): ok (rc=0)
Jun 3 08:41:09 srv-dell crmd: [13708]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=cluster_ip_address_monitor_0,
magic=0:7;4:4:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2) : Resource op
removal
Jun 3 08:41:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:09 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_pe_invoke: Query 103:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 08:41:09 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=srv-lenovo/crmd/6, version=0.15.4): ok (rc=0)
Jun 3 08:41:09 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=srv-lenovo/crmd/7,
version=0.15.5): ok (rc=0)
Jun 3 08:41:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:09 srv-dell attrd: [13706]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 08:41:09 srv-dell crmd: [13708]: info: process_graph_event:
Detected action cluster_ip_address_monitor_0 from a different
transition: 4 vs. 6
Jun 3 08:41:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:09 srv-dell crmd: [13708]: info: abort_transition_graph:
process_graph_event:472 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=cluster_ip_address_monitor_0,
magic=0:7;4:4:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2) : Old event
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_pe_invoke: Query 104:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 08:41:09 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/99, version=0.15.6): ok (rc=0)
Jun 3 08:41:09 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/101, version=0.15.6): ok (rc=0)
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244032869-47, seq=184, quorate=1
Jun 3 08:41:09 srv-dell pengine: [13707]: WARN: unpack_resources: No
STONITH resources have been defined
Jun 3 08:41:09 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 08:41:09 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 08:41:09 srv-dell pengine: [13707]: notice: native_print:
cluster_ip_address (ocf::heartbeat:IPaddr): Stopped
Jun 3 08:41:09 srv-dell pengine: [13707]: WARN: native_color:
Resource cluster_ip_address cannot run anywhere
Jun 3 08:41:09 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cluster_ip_address (Stopped)
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 08:41:09 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 7: 3 actions in 3 synapses
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 7 (ref=pe_calc-dc-1244032869-47) derived from
/var/lib/pengine/pe-warn-5.bz2
Jun 3 08:41:09 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 5: monitor cluster_ip_address_monitor_0 on
srv-lenovo
Jun 3 08:41:09 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 7: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-5.bz2
Jun 3 08:41:09 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 08:41:09 srv-dell crmd: [13708]: info: match_graph_event:
Action cluster_ip_address_monitor_0 (5) confirmed on srv-lenovo (rc=0)
Jun 3 08:41:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:41:09 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 4: probe_complete probe_complete on srv-lenovo - no
waiting
Jun 3 08:41:09 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 08:41:09 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 08:41:09 srv-dell crmd: [13708]: notice: run_graph: Transition
7 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-5.bz2): Complete
Jun 3 08:41:09 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 7 is now complete
Jun 3 08:41:09 srv-dell crmd: [13708]: info: notify_crmd: Transition
7 status: done - <null>
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 08:41:09 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 08:41:09 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 08:43:32 srv-dell cib: [13704]: info: cib_stats: Processed 65
operations (4153.00us average, 0% utilization) in the last 10min
Jun 3 08:46:30 srv-dell crm_shadow: [13875]: info: Invoked: crm_shadow
Jun 3 08:46:30 srv-dell crm_shadow: [13889]: info: Invoked: crm_shadow
Jun 3 08:46:30 srv-dell crm_shadow: [13891]: info: Invoked: crm_shadow
Jun 3 08:46:34 srv-dell cibadmin: [13892]: info: Invoked: cibadmin -Ql
Jun 3 08:48:20 srv-dell cibadmin: [13901]: info: Invoked: cibadmin -Ql
Jun 3 08:48:20 srv-dell cibadmin: [13902]: info: Invoked: cibadmin -p -U
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="15" num_updates="8" />
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="16" num_updates="1" >
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <configuration >
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <resources >
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <clone id="fencing" __crm_diff_marker__="added:top"
>
Jun 3 08:48:20 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <primitive class="stonith" id="st-null"
type="null" >
Jun 3 08:48:20 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <instance_attributes
id="st-null-instance_attributes" >
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair
id="st-null-instance_attributes-hostlist" name="hostlist"
value="srv-dell srv-lenovo" />
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </instance_attributes>
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </primitive>
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </clone>
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_pe_invoke: Query 105:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </resources>
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <rsc_defaults __crm_diff_marker__="added:top" />
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <op_defaults __crm_diff_marker__="added:top" />
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </configuration>
Jun 3 08:48:20 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </cib>
Jun 3 08:48:20 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section 'all'
(origin=local/cibadmin/2, version=0.16.1): ok (rc=0)
Jun 3 08:48:20 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244033300-50, seq=184, quorate=1
Jun 3 08:48:20 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 08:48:20 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 08:48:20 srv-dell pengine: [13707]: notice: native_print:
cluster_ip_address (ocf::heartbeat:IPaddr): Stopped
Jun 3 08:48:20 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: fencing
Jun 3 08:48:20 srv-dell pengine: [13707]: notice: print_list:
Stopped: [ st-null:0 st-null:1 ]
Jun 3 08:48:20 srv-dell pengine: [13707]: WARN: native_color:
Resource cluster_ip_address cannot run anywhere
Jun 3 08:48:20 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cluster_ip_address (Stopped)
Jun 3 08:48:20 srv-dell pengine: [13707]: notice: LogActions: Start
st-null:0 (srv-dell)
Jun 3 08:48:20 srv-dell pengine: [13707]: notice: LogActions: Start
st-null:1 (srv-lenovo)
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 08:48:20 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 8: 10 actions in 10 synapses
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 8 (ref=pe_calc-dc-1244033300-50) derived from
/var/lib/pengine/pe-warn-6.bz2
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 4: monitor st-null:0_monitor_0 on srv-dell (local)
Jun 3 08:48:20 srv-dell lrmd: [13705]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=4:8:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=st-null:0_monitor_0 )
Jun 3 08:48:20 srv-dell lrmd: [13705]: info: rsc:st-null:0: monitor
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 6: monitor st-null:1_monitor_0 on srv-lenovo
Jun 3 08:48:20 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 8: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-6.bz2
Jun 3 08:48:20 srv-dell cib: [13903]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-16.raw
Jun 3 08:48:20 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 08:48:20 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation st-null:0_monitor_0 (call=3, rc=7, cib-update=106,
confirmed=true) complete not running
Jun 3 08:48:20 srv-dell cib: [13903]: info: write_cib_contents: Wrote
version 0.16.0 of the CIB to disk (digest:
9de07e7cc38f17dc15cc27056e2a23c8)
Jun 3 08:48:20 srv-dell crmd: [13708]: info: match_graph_event:
Action st-null:0_monitor_0 (4) confirmed on srv-dell (rc=0)
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 3: probe_complete probe_complete on srv-dell (local)
- no waiting
Jun 3 08:48:20 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:48:20 srv-dell cibadmin: [13905]: info: Invoked: cibadmin -Ql
Jun 3 08:48:20 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:48:20 srv-dell crmd: [13708]: info: match_graph_event:
Action st-null:1_monitor_0 (6) confirmed on srv-lenovo (rc=0)
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 5: probe_complete probe_complete on srv-lenovo - no
waiting
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 11 fired and confirmed
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 9 fired and confirmed
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 7: start st-null:0_start_0 on srv-dell (local)
Jun 3 08:48:20 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=7:8:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=st-null:0_start_0 )
Jun 3 08:48:20 srv-dell lrmd: [13705]: info: rsc:st-null:0: start
Jun 3 08:48:20 srv-dell cib: [13903]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.rkAHyy (digest:
/var/lib/heartbeat/crm/cib.OrddUv)
Jun 3 08:48:20 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 8: start st-null:1_start_0 on srv-lenovo
Jun 3 08:48:20 srv-dell lrmd: [13907]: info: Try to start STONITH
resource <rsc_id=st-null:0> : Device=null
Jun 3 08:48:20 srv-dell stonithd: [13703]: info: st-null:0 stonith
resource started
Jun 3 08:48:20 srv-dell lrmd: [13705]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 08:48:20 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation st-null:0_start_0 (call=4, rc=0, cib-update=109,
confirmed=true) complete ok
Jun 3 08:48:20 srv-dell crmd: [13708]: info: match_graph_event:
Action st-null:0_start_0 (7) confirmed on srv-dell (rc=0)
Jun 3 08:48:20 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:48:20 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 08:48:21 srv-dell crmd: [13708]: info: match_graph_event:
Action st-null:1_start_0 (8) confirmed on srv-lenovo (rc=0)
Jun 3 08:48:21 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 10 fired and confirmed
Jun 3 08:48:21 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 08:48:21 srv-dell crmd: [13708]: notice: run_graph: Transition
8 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-6.bz2): Complete
Jun 3 08:48:21 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:48:21 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 8 is now complete
Jun 3 08:48:21 srv-dell crmd: [13708]: info: notify_crmd: Transition
8 status: done - <null>
Jun 3 08:48:21 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 08:48:21 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 08:48:21 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 08:50:16 srv-dell mgmtd: [13709]: info: CIB replace: resources
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="16" num_updates="5" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <resources >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <primitive class="ocf" id="cluster_ip_address"
provider="heartbeat" type="IPaddr" __crm_diff_marker__="removed:top" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <meta_attributes
id="cluster_ip_address-meta_attributes" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nvpair
id="nvpair-52dbd2d5-53cf-4911-afd0-2b82addf3a0e" name="target-role"
value="Stopped" />
Jun 3 08:50:16 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 08:50:16 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </meta_attributes>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <operations id="cluster_ip_address-operations" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <op id="cluster_ip_address-op-monitor-5s"
interval="5s" name="monitor" start-delay="1s" timeout="20s" />
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </operations>
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <instance_attributes
id="cluster_ip_address-instance_attributes" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nvpair
id="nvpair-59d8b5b6-85db-4e96-a73d-28436c29cefd" name="ip"
value="192.168.2.10" />
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_pe_invoke: Query 110:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </instance_attributes>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </primitive>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </resources>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </cib>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="17" num_updates="1" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <configuration >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <resources >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <primitive class="ocf" id="resource_clusterip"
provider="heartbeat" type="IPaddr" __crm_diff_marker__="added:top" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <meta_attributes
id="cluster_ip_address-meta_attributes" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair
id="nvpair-52dbd2d5-53cf-4911-afd0-2b82addf3a0e" name="target-role"
value="Stopped" />
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </meta_attributes>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <operations id="cluster_ip_address-operations" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <op id="cluster_ip_address-op-monitor-5s"
interval="5s" name="monitor" start-delay="1s" timeout="20s" />
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </operations>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <instance_attributes
id="cluster_ip_address-instance_attributes" >
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair
id="nvpair-59d8b5b6-85db-4e96-a73d-28436c29cefd" name="ip"
value="192.168.2.10" />
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </instance_attributes>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </primitive>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </resources>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </configuration>
Jun 3 08:50:16 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </cib>
Jun 3 08:50:16 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_replace for section resources
(origin=local/mgmtd/22, version=0.17.1): ok (rc=0)
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244033416-57, seq=184, quorate=1
Jun 3 08:50:16 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 08:50:16 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 08:50:16 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 08:50:16 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 08:50:16 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 08:50:16 srv-dell pengine: [13707]: notice: native_print:
resource_clusterip (ocf::heartbeat:IPaddr): Stopped
Jun 3 08:50:16 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: fencing
Jun 3 08:50:16 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 08:50:16 srv-dell pengine: [13707]: WARN: native_color:
Resource resource_clusterip cannot run anywhere
Jun 3 08:50:16 srv-dell pengine: [13707]: notice: LogActions: Leave
resource resource_clusterip (Stopped)
Jun 3 08:50:16 srv-dell pengine: [13707]: notice: LogActions: Leave
resource st-null:0 (Started srv-dell)
Jun 3 08:50:16 srv-dell pengine: [13707]: notice: LogActions: Leave
resource st-null:1 (Started srv-lenovo)
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 08:50:16 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 9: 5 actions in 5 synapses
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 9 (ref=pe_calc-dc-1244033416-57) derived from
/var/lib/pengine/pe-warn-7.bz2
Jun 3 08:50:16 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 4: monitor resource_clusterip_monitor_0 on srv-dell
(local)
Jun 3 08:50:16 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_sync_one for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.17.1): ok (rc=0)
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=4:9:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=resource_clusterip_monitor_0 )
Jun 3 08:50:16 srv-dell lrmd: [13705]: info: rsc:resource_clusterip: monitor
Jun 3 08:50:16 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 6: monitor resource_clusterip_monitor_0 on
srv-lenovo
Jun 3 08:50:16 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 9: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-7.bz2
Jun 3 08:50:16 srv-dell cib: [13909]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-17.raw
Jun 3 08:50:16 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 08:50:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:50:16 srv-dell cib: [13909]: info: write_cib_contents: Wrote
version 0.17.0 of the CIB to disk (digest:
fb26239c776fadfda5f14214b3fe83e2)
Jun 3 08:50:16 srv-dell cib: [13909]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.lOb7GD (digest:
/var/lib/heartbeat/crm/cib.Ood7EF)
Jun 3 08:50:16 srv-dell crmd: [13708]: info: match_graph_event:
Action resource_clusterip_monitor_0 (6) confirmed on srv-lenovo (rc=0)
Jun 3 08:50:16 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 5: probe_complete probe_complete on srv-lenovo - no
waiting
Jun 3 08:50:16 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation resource_clusterip_monitor_0 (call=5, rc=7, cib-update=111,
confirmed=true) complete not running
Jun 3 08:50:16 srv-dell crmd: [13708]: info: match_graph_event:
Action resource_clusterip_monitor_0 (4) confirmed on srv-dell (rc=0)
Jun 3 08:50:16 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 3: probe_complete probe_complete on srv-dell (local)
- no waiting
Jun 3 08:50:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:50:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:50:16 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 08:50:16 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 08:50:16 srv-dell crmd: [13708]: notice: run_graph: Transition
9 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-7.bz2): Complete
Jun 3 08:50:16 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 9 is now complete
Jun 3 08:50:16 srv-dell crmd: [13708]: info: notify_crmd: Transition
9 status: done - <null>
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 08:50:16 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 08:50:16 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 08:53:32 srv-dell cib: [13704]: info: cib_stats: Processed 29
operations (3103.00us average, 0% utilization) in the last 10min
Jun 3 08:54:18 srv-dell mgmtd: [13709]: info: CIB replace: resources
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="17" num_updates="3" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <resources >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <primitive class="ocf" id="resource_clusterip"
provider="heartbeat" type="IPaddr" __crm_diff_marker__="removed:top" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <meta_attributes
id="cluster_ip_address-meta_attributes" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nvpair
id="nvpair-52dbd2d5-53cf-4911-afd0-2b82addf3a0e" name="target-role"
value="Stopped" />
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </meta_attributes>
Jun 3 08:54:18 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <operations id="cluster_ip_address-operations" >
Jun 3 08:54:18 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <op id="cluster_ip_address-op-monitor-5s"
interval="5s" name="monitor" start-delay="1s" timeout="20s" />
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </operations>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <instance_attributes
id="cluster_ip_address-instance_attributes" >
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nvpair
id="nvpair-59d8b5b6-85db-4e96-a73d-28436c29cefd" name="ip"
value="192.168.2.10" />
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </instance_attributes>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </primitive>
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_pe_invoke: Query 114:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <clone id="fencing"
__crm_diff_marker__="removed:top" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <primitive class="stonith" id="st-null"
type="null" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <instance_attributes
id="st-null-instance_attributes" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nvpair
id="st-null-instance_attributes-hostlist" name="hostlist"
value="srv-dell srv-lenovo" />
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </instance_attributes>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </primitive>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </clone>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </resources>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </cib>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="18" num_updates="1" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <configuration >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <resources >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <primitive class="ocf"
id="primitiveresource_cluster_ip" provider="heartbeat" type="IPaddr"
__crm_diff_marker__="added:top" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <meta_attributes
id="cluster_ip_address-meta_attributes" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair
id="nvpair-52dbd2d5-53cf-4911-afd0-2b82addf3a0e" name="target-role"
value="Stopped" />
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </meta_attributes>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <operations id="cluster_ip_address-operations" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <op id="cluster_ip_address-op-monitor-5s"
interval="5s" name="monitor" start-delay="1s" timeout="20s" />
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </operations>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <instance_attributes
id="cluster_ip_address-instance_attributes" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair
id="nvpair-59d8b5b6-85db-4e96-a73d-28436c29cefd" name="ip"
value="192.168.2.10" />
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </instance_attributes>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </primitive>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <clone id="cloneresource_cluster_fencing"
__crm_diff_marker__="added:top" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <primitive class="stonith"
id="cloneresource_cluster_fencing_null" type="null" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <instance_attributes
id="cloneresource_cluster_fencing_null-instance_attributes" >
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair
id="cloneresource_cluster_fencing_null-instance_attributes-hostlist"
name="hostlist" value="srv-dell srv-lenovo" />
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </instance_attributes>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </primitive>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </clone>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </resources>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </configuration>
Jun 3 08:54:18 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </cib>
Jun 3 08:54:18 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_replace for section resources
(origin=local/mgmtd/24, version=0.18.1): ok (rc=0)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244033658-62, seq=184, quorate=1
Jun 3 08:54:18 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 08:54:18 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 08:54:18 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 08:54:18 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 08:54:18 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 08:54:18 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 08:54:18 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 08:54:18 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 08:54:18 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 08:54:18 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 08:54:18 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 08:54:18 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 08:54:18 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 08:54:18 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: native_print:
st-null:0 (stonith:null): ORPHANED Started srv-dell
Jun 3 08:54:18 srv-dell cib: [13928]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-18.raw
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: native_print:
st-null:1 (stonith:null): ORPHANED Started srv-lenovo
Jun 3 08:54:18 srv-dell pengine: [13707]: WARN: native_color:
Resource primitiveresource_cluster_ip cannot run anywhere
Jun 3 08:54:18 srv-dell pengine: [13707]: info: native_color:
Stopping orphan resource st-null:0
Jun 3 08:54:18 srv-dell pengine: [13707]: info: native_color:
Stopping orphan resource st-null:1
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 08:54:18 srv-dell cib: [13928]: info: write_cib_contents: Wrote
version 0.18.0 of the CIB to disk (digest:
ad36f57f4af4ab0d8a77c68468472ab6)
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: LogActions: Start
cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: LogActions: Start
cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: LogActions: Stop
resource st-null:0 (srv-dell)
Jun 3 08:54:18 srv-dell pengine: [13707]: notice: LogActions: Stop
resource st-null:1 (srv-lenovo)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 08:54:18 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 10: 15 actions in 15 synapses
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 10 (ref=pe_calc-dc-1244033658-62) derived from
/var/lib/pengine/pe-warn-8.bz2
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 4: monitor primitiveresource_cluster_ip_monitor_0 on
srv-dell (local)
Jun 3 08:54:18 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 10: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-8.bz2
Jun 3 08:54:18 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=4:10:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 08:54:18 srv-dell cib: [13928]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.s3aoIQ (digest:
/var/lib/heartbeat/crm/cib.KQQXcn)
Jun 3 08:54:18 srv-dell lrmd: [13705]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 7: monitor primitiveresource_cluster_ip_monitor_0 on
srv-lenovo
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 5: monitor
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell (local)
Jun 3 08:54:18 srv-dell lrmd: [13705]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=5:10:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=cloneresource_cluster_fencing_null:0_monitor_0 )
Jun 3 08:54:18 srv-dell lrmd: [13705]: info:
rsc:cloneresource_cluster_fencing_null:0: monitor
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 8: monitor
cloneresource_cluster_fencing_null:1_monitor_0 on srv-lenovo
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 1 fired and confirmed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 15: stop st-null:0_stop_0 on srv-dell (local)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=15:10:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=st-null:0_stop_0 )
Jun 3 08:54:18 srv-dell lrmd: [13705]: info: rsc:st-null:0: stop
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 16: stop st-null:1_stop_0 on srv-lenovo
Jun 3 08:54:18 srv-dell lrmd: [13931]: info: Try to stop STONITH
resource <rsc_id=st-null:0> : Device=null
Jun 3 08:54:18 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_monitor_0 (call=7,
rc=7, cib-update=115, confirmed=true) complete not running
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation st-null:0_stop_0 (call=8, rc=0, cib-update=116,
confirmed=true) complete ok
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:0_monitor_0 (5) confirmed on
srv-dell (rc=0)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action st-null:0_stop_0 (15) confirmed on srv-dell (rc=0)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:1_monitor_0 (8) confirmed on
srv-lenovo (rc=0)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=6, rc=7,
cib-update=117, confirmed=true) complete not running
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action st-null:1_stop_0 (16) confirmed on srv-lenovo (rc=0)
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_monitor_0 (7) confirmed on
srv-lenovo (rc=0)
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 6: probe_complete probe_complete on srv-lenovo - no
waiting
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_monitor_0 (4) confirmed on
srv-dell (rc=0)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 3: probe_complete probe_complete on srv-dell (local)
- no waiting
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 13 fired and confirmed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 11 fired and confirmed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 9: start
cloneresource_cluster_fencing_null:0_start_0 on srv-dell (local)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=9:10:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 08:54:18 srv-dell lrmd: [13705]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 10: start
cloneresource_cluster_fencing_null:1_start_0 on srv-lenovo
Jun 3 08:54:18 srv-dell lrmd: [13952]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 08:54:18 srv-dell stonithd: [13703]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 08:54:18 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=9, rc=0,
cib-update=120, confirmed=true) complete ok
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:0_start_0 (9) confirmed on
srv-dell (rc=0)
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:1_start_0 (10) confirmed on
srv-lenovo (rc=0)
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 12 fired and confirmed
Jun 3 08:54:18 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 08:54:18 srv-dell crmd: [13708]: notice: run_graph: Transition
10 (Complete=15, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-8.bz2): Complete
Jun 3 08:54:18 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 10 is now complete
Jun 3 08:54:18 srv-dell crmd: [13708]: info: notify_crmd: Transition
10 status: done - <null>
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 08:54:18 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 08:54:18 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 08:59:49 srv-dell lrmd: [13705]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 09:03:06 srv-dell mgmtd: [13709]: info: CIB replace: resources
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="18" num_updates="9" >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <resources >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <primitive id="primitiveresource_cluster_ip" >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <meta_attributes
id="cluster_ip_address-meta_attributes" >
Jun 3 09:03:06 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 09:03:06 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nvpair value="Stopped"
id="nvpair-52dbd2d5-53cf-4911-afd0-2b82addf3a0e" />
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </meta_attributes>
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </primitive>
Jun 3 09:03:06 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </resources>
Jun 3 09:03:06 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </cib>
Jun 3 09:03:06 srv-dell crmd: [13708]: info: do_pe_invoke: Query 121:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="19" num_updates="1" >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <configuration >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <resources >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <primitive id="primitiveresource_cluster_ip" >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <meta_attributes
id="cluster_ip_address-meta_attributes" >
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair value="Started"
id="nvpair-52dbd2d5-53cf-4911-afd0-2b82addf3a0e" />
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </meta_attributes>
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </primitive>
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </resources>
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </configuration>
Jun 3 09:03:06 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </cib>
Jun 3 09:03:06 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_replace for section resources
(origin=local/mgmtd/27, version=0.19.1): ok (rc=0)
Jun 3 09:03:06 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244034186-73, seq=184, quorate=1
Jun 3 09:03:06 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 09:03:06 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:03:06 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:03:06 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:03:06 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:03:06 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:03:06 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:03:06 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:03:06 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:03:06 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:03:06 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 09:03:06 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:03:06 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:03:06 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:03:06 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 09:03:06 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:03:06 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 09:03:06 srv-dell cib: [13982]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-19.raw
Jun 3 09:03:06 srv-dell pengine: [13707]: notice: RecurringOp: Start
recurring monitor (5s) for primitiveresource_cluster_ip on srv-dell
Jun 3 09:03:06 srv-dell pengine: [13707]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 3 09:03:06 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 09:03:06 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 09:03:06 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:03:06 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 11: 2 actions in 2 synapses
Jun 3 09:03:07 srv-dell pengine: [13707]: info: process_pe_message:
Transition 11: PEngine Input stored in:
/var/lib/pengine/pe-input-10.bz2
Jun 3 09:03:07 srv-dell cib: [13982]: info: write_cib_contents: Wrote
version 0.19.0 of the CIB to disk (digest:
3d0ab2034e7f74095cdd595e05c1a7b2)
Jun 3 09:03:07 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 11 (ref=pe_calc-dc-1244034186-73) derived from
/var/lib/pengine/pe-input-10.bz2
Jun 3 09:03:07 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:03:07 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 5: start primitiveresource_cluster_ip_start_0 on
srv-dell (local)
Jun 3 09:03:07 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=5:11:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_start_0 )
Jun 3 09:03:07 srv-dell lrmd: [13705]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 09:03:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:07 srv-dell cib: [13982]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.QcxUQq (digest:
/var/lib/heartbeat/crm/cib.S1AjXw)
Jun 3 09:03:07 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 09:03:07 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_09:03:07 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 09:03:07 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 09:03:07 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_09:03:07 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 09:03:07 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 09:03:07 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_09:03:07 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 09:03:07 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:03:07 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=10, rc=0,
cib-update=122, confirmed=true) complete ok
Jun 3 09:03:07 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_start_0 (5) confirmed on srv-dell
(rc=0)
Jun 3 09:03:07 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 6: monitor primitiveresource_cluster_ip_monitor_5000
on srv-dell (local)
Jun 3 09:03:07 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=6:11:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 09:03:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:07 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:03:08 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=11, rc=0,
cib-update=123, confirmed=false) complete ok
Jun 3 09:03:08 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_monitor_5000 (6) confirmed on
srv-dell (rc=0)
Jun 3 09:03:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:08 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:03:08 srv-dell crmd: [13708]: notice: run_graph: Transition
11 (Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-10.bz2): Complete
Jun 3 09:03:08 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 11 is now complete
Jun 3 09:03:08 srv-dell crmd: [13708]: info: notify_crmd: Transition
11 status: done - <null>
Jun 3 09:03:08 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:03:08 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:03:08 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:03:32 srv-dell cib: [13704]: info: cib_stats: Processed 25
operations (6000.00us average, 0% utilization) in the last 10min
Jun 3 09:03:37 srv-dell mgmtd: [13709]: info: CIB replace: resources
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="19" num_updates="3" />
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="20" num_updates="1" >
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <configuration >
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <resources >
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <primitive id="primitiveresource_cluster_ip" >
Jun 3 09:03:37 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <meta_attributes
id="cluster_ip_address-meta_attributes" >
Jun 3 09:03:37 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair
id="nvpair-563142fa-0290-4a4a-a627-ad13f8c0a185"
name="migration-threshold" value="2" __crm_diff_marker__="added:top"
/>
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </meta_attributes>
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </primitive>
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </resources>
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </configuration>
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_pe_invoke: Query 124:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:03:37 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </cib>
Jun 3 09:03:37 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_replace for section resources
(origin=local/mgmtd/32, version=0.20.1): ok (rc=0)
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244034217-76, seq=184, quorate=1
Jun 3 09:03:37 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 09:03:37 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:03:37 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:03:37 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:03:37 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:03:37 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:03:37 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:03:37 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:03:37 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:03:37 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:03:37 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 09:03:37 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:03:37 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:03:37 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:03:37 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 09:03:37 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:03:37 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 09:03:37 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 09:03:37 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 09:03:37 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:03:37 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 12: 0 actions in 0 synapses
Jun 3 09:03:37 srv-dell pengine: [13707]: info: process_pe_message:
Transition 12: PEngine Input stored in:
/var/lib/pengine/pe-input-11.bz2
Jun 3 09:03:37 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 12 (ref=pe_calc-dc-1244034217-76) derived from
/var/lib/pengine/pe-input-11.bz2
Jun 3 09:03:37 srv-dell cib: [14197]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-20.raw
Jun 3 09:03:37 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:03:37 srv-dell crmd: [13708]: notice: run_graph: Transition
12 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-11.bz2): Complete
Jun 3 09:03:37 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 12 is now complete
Jun 3 09:03:37 srv-dell crmd: [13708]: info: notify_crmd: Transition
12 status: done - <null>
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:03:37 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:03:37 srv-dell cib: [14197]: info: write_cib_contents: Wrote
version 0.20.0 of the CIB to disk (digest:
68fd5c47c45e87d7a7f75fe6e2c955db)
Jun 3 09:03:37 srv-dell cib: [14197]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.4tOIUN (digest:
/var/lib/heartbeat/crm/cib.XRgFdY)
Jun 3 09:03:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:37 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="20" num_updates="1" />
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="21" num_updates="1" >
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <configuration >
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nodes >
Jun 3 09:03:52 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <node id="srv-dell" >
Jun 3 09:03:52 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <instance_attributes id="nodes-srv-dell"
__crm_diff_marker__="added:top" >
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair id="standby-srv-dell" name="standby"
value="true" />
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </instance_attributes>
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </node>
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </nodes>
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_pe_invoke: Query 125:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </configuration>
Jun 3 09:03:52 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </cib>
Jun 3 09:03:52 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/mgmtd/37, version=0.21.1): ok (rc=0)
Jun 3 09:03:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244034232-77, seq=184, quorate=1
Jun 3 09:03:52 srv-dell pengine: [13707]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 09:03:52 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is standby
Jun 3 09:03:52 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:03:52 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:03:52 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:03:52 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:03:52 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:03:52 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:03:52 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:03:52 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:03:52 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:03:52 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 09:03:52 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:03:52 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:03:52 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:03:52 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 09:03:52 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:03:52 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 09:03:52 srv-dell pengine: [13707]: WARN: native_color:
Resource cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 09:03:52 srv-dell pengine: [13707]: notice: RecurringOp: Start
recurring monitor (5s) for primitiveresource_cluster_ip on srv-lenovo
Jun 3 09:03:52 srv-dell pengine: [13707]: notice: LogActions: Move
resource primitiveresource_cluster_ip (Started srv-dell -> srv-lenovo)
Jun 3 09:03:52 srv-dell pengine: [13707]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 09:03:52 srv-dell cib: [14252]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-21.raw
Jun 3 09:03:52 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:03:52 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 13: 7 actions in 7 synapses
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 13 (ref=pe_calc-dc-1244034232-77) derived from
/var/lib/pengine/pe-warn-9.bz2
Jun 3 09:03:52 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 6: stop primitiveresource_cluster_ip_stop_0 on
srv-dell (local)
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=6:13:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 09:03:52 srv-dell lrmd: [13705]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 09:03:52 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 09:03:52 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=11, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 09:03:52 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 13: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-9.bz2
Jun 3 09:03:52 srv-dell cib: [14252]: info: write_cib_contents: Wrote
version 0.21.0 of the CIB to disk (digest:
7852e0590f3114ae7613f3a868623128)
Jun 3 09:03:52 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:03:52 srv-dell cib: [14252]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ETw1dE (digest:
/var/lib/heartbeat/crm/cib.cSjKLl)
Jun 3 09:03:52 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 09:03:52 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 09:03:52 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 09:03:52 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_09:03:52 INFO:
ifconfig eth1:0 down
Jun 3 09:03:52 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=12, rc=0,
cib-update=126, confirmed=true) complete ok
Jun 3 09:03:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:52 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_stop_0 (6) confirmed on srv-dell
(rc=0)
Jun 3 09:03:52 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 7: start primitiveresource_cluster_ip_start_0 on
srv-lenovo
Jun 3 09:03:52 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 09:03:52 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 9: stop cloneresource_cluster_fencing_null:0_stop_0
on srv-dell (local)
Jun 3 09:03:52 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=9:13:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=cloneresource_cluster_fencing_null:0_stop_0 )
Jun 3 09:03:52 srv-dell lrmd: [13705]: info:
rsc:cloneresource_cluster_fencing_null:0: stop
Jun 3 09:03:52 srv-dell lrmd: [14279]: info: Try to stop STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 09:03:52 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_stop_0 (call=13, rc=0,
cib-update=127, confirmed=true) complete ok
Jun 3 09:03:52 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:0_stop_0 (9) confirmed on
srv-dell (rc=0)
Jun 3 09:03:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:52 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 15 fired and confirmed
Jun 3 09:03:52 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:03:53 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_start_0 (7) confirmed on
srv-lenovo (rc=0)
Jun 3 09:03:53 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 8: monitor primitiveresource_cluster_ip_monitor_5000
on srv-lenovo
Jun 3 09:03:53 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:53 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:03:54 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_monitor_5000 (8) confirmed on
srv-lenovo (rc=0)
Jun 3 09:03:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:03:54 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:03:54 srv-dell crmd: [13708]: notice: run_graph: Transition
13 (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-9.bz2): Complete
Jun 3 09:03:54 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 13 is now complete
Jun 3 09:03:54 srv-dell crmd: [13708]: info: notify_crmd: Transition
13 status: done - <null>
Jun 3 09:03:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:03:54 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:03:54 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="21" num_updates="5" >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nodes >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <node id="srv-dell" >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <instance_attributes id="nodes-srv-dell" >
Jun 3 09:04:13 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <nvpair value="true" id="standby-srv-dell" />
Jun 3 09:04:13 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </instance_attributes>
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </node>
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </nodes>
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - </cib>
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_pe_invoke: Query 128:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="22" num_updates="1" >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <configuration >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nodes >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <node id="srv-dell" >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <instance_attributes id="nodes-srv-dell" >
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <nvpair value="false" id="standby-srv-dell" />
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </instance_attributes>
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </node>
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </nodes>
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </configuration>
Jun 3 09:04:13 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + </cib>
Jun 3 09:04:13 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/mgmtd/45, version=0.22.1): ok (rc=0)
Jun 3 09:04:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244034253-82, seq=184, quorate=1
Jun 3 09:04:13 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 09:04:13 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:04:13 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:04:13 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:04:13 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:04:13 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:04:13 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:04:13 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:04:13 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:04:13 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:04:13 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 09:04:13 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:04:13 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:04:13 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started
srv-lenovo
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-lenovo ]
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0 ]
Jun 3 09:04:13 srv-dell cib: [14281]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-22.raw
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: RecurringOp: Start
recurring monitor (5s) for primitiveresource_cluster_ip on srv-dell
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: LogActions: Move
resource primitiveresource_cluster_ip (Started srv-lenovo -> srv-dell)
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: LogActions: Start
cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 09:04:13 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:04:13 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 14: 7 actions in 7 synapses
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 14 (ref=pe_calc-dc-1244034253-82) derived from
/var/lib/pengine/pe-input-12.bz2
Jun 3 09:04:13 srv-dell cib: [14281]: info: write_cib_contents: Wrote
version 0.22.0 of the CIB to disk (digest:
1d8daa297576885625be773a31d50903)
Jun 3 09:04:13 srv-dell pengine: [13707]: info: process_pe_message:
Transition 14: PEngine Input stored in:
/var/lib/pengine/pe-input-12.bz2
Jun 3 09:04:13 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:04:13 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 6: stop primitiveresource_cluster_ip_stop_0 on
srv-lenovo
Jun 3 09:04:13 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 12 fired and confirmed
Jun 3 09:04:13 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 9: start
cloneresource_cluster_fencing_null:0_start_0 on srv-dell (local)
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=9:14:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 09:04:13 srv-dell lrmd: [13705]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 09:04:13 srv-dell lrmd: [14282]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 09:04:13 srv-dell stonithd: [13703]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 09:04:13 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=14, rc=0,
cib-update=129, confirmed=true) complete ok
Jun 3 09:04:13 srv-dell cib: [14281]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Ux9OXV (digest:
/var/lib/heartbeat/crm/cib.7nDItm)
Jun 3 09:04:13 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:0_start_0 (9) confirmed on
srv-dell (rc=0)
Jun 3 09:04:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:04:13 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 13 fired and confirmed
Jun 3 09:04:13 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:04:13 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_stop_0 (6) confirmed on srv-lenovo
(rc=0)
Jun 3 09:04:13 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 7: start primitiveresource_cluster_ip_start_0 on
srv-dell (local)
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=7:14:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_start_0 )
Jun 3 09:04:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:04:13 srv-dell lrmd: [13705]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 09:04:13 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 09:04:13 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 09:04:13 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_09:04:13 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 09:04:13 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 09:04:13 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_09:04:13 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 09:04:13 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 09:04:13 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_09:04:13 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 09:04:13 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=15, rc=0,
cib-update=130, confirmed=true) complete ok
Jun 3 09:04:13 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_start_0 (7) confirmed on srv-dell
(rc=0)
Jun 3 09:04:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:04:13 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 8: monitor primitiveresource_cluster_ip_monitor_5000
on srv-dell (local)
Jun 3 09:04:13 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=8:14:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 09:04:14 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:04:14 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=16, rc=0,
cib-update=131, confirmed=false) complete ok
Jun 3 09:04:14 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_monitor_5000 (8) confirmed on
srv-dell (rc=0)
Jun 3 09:04:14 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:04:14 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:04:14 srv-dell crmd: [13708]: notice: run_graph: Transition
14 (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-12.bz2): Complete
Jun 3 09:04:14 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 14 is now complete
Jun 3 09:04:14 srv-dell crmd: [13708]: info: notify_crmd: Transition
14 status: done - <null>
Jun 3 09:04:14 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:04:14 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:04:15 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:05:22 srv-dell syslog-ng[3116]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=2529', processed='center(received)=2440',
processed='destination(newsnotice)=0',
processed='destination(acpid)=0',
processed='destination(firewall)=29', processed='destination(null)=0',
processed='destination(mail)=4', processed='destination(mailinfo)=4',
processed='destination(console)=5',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=2407',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=5', processed='destination(warn)=73',
processed='source(src)=2440'
Jun 3 09:10:43 srv-dell kernel: No iBFT detected.
Jun 3 09:10:45 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 3 09:10:45 srv-dell kernel: iscsi: registered transport (tcp)
Jun 3 09:10:45 srv-dell kernel: iscsi: registered transport (iser)
Jun 3 09:10:45 srv-dell iscsid: iSCSI logger with pid=16137 started!
Jun 3 09:10:46 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 3 09:10:46 srv-dell iscsid: iSCSI daemon with pid=16138 started!
Jun 3 09:13:32 srv-dell cib: [13704]: info: cib_stats: Processed 34
operations (6176.00us average, 0% utilization) in the last 10min
Jun 3 09:14:15 srv-dell kernel: st: Version 20080504, fixed bufsize
32768, s/g segs 256
Jun 3 09:19:14 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 09:19:14 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 09:19:14 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 09:19:14 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 09:19:14 srv-dell crmd: [13708]: info: do_pe_invoke: Query 132:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:19:15 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244035155-87, seq=184, quorate=1
Jun 3 09:19:15 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 09:19:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:19:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:19:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:19:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:19:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:19:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:19:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:19:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:19:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:19:15 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 09:19:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:19:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:19:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:19:15 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 09:19:15 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:19:15 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 09:19:15 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 09:19:15 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 09:19:15 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 09:19:15 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:19:15 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 15: 0 actions in 0 synapses
Jun 3 09:19:15 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 15 (ref=pe_calc-dc-1244035155-87) derived from
/var/lib/pengine/pe-input-13.bz2
Jun 3 09:19:15 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:19:15 srv-dell crmd: [13708]: notice: run_graph: Transition
15 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-13.bz2): Complete
Jun 3 09:19:15 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 15 is now complete
Jun 3 09:19:15 srv-dell crmd: [13708]: info: notify_crmd: Transition
15 status: done - <null>
Jun 3 09:19:15 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:19:15 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:19:15 srv-dell pengine: [13707]: info: process_pe_message:
Transition 15: PEngine Input stored in:
/var/lib/pengine/pe-input-13.bz2
Jun 3 09:19:15 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:21:20 srv-dell kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Jun 3 09:21:20 srv-dell kernel: ISO 9660 Extensions: RRIP_1991A
Jun 3 09:21:32 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 09:21:32 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 09:21:32 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 09:21:32 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 09:23:32 srv-dell cib: [13704]: info: cib_stats: Processed 1
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 09:23:47 srv-dell SuSEfirewall2: batch committing...
Jun 3 09:23:47 srv-dell SuSEfirewall2: Firewall rules unloaded.
Jun 3 09:23:47 srv-dell SuSEfirewall2: Setting up rules from
/etc/sysconfig/SuSEfirewall2 ...
Jun 3 09:23:47 srv-dell SuSEfirewall2: using default zone 'ext' for
interface eth0
Jun 3 09:23:47 srv-dell SuSEfirewall2: batch committing...
Jun 3 09:23:47 srv-dell SuSEfirewall2: Firewall rules successfully set
Jun 3 09:23:57 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 09:23:57 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 09:23:57 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 09:23:57 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 09:23:57 srv-dell ietd: Can't create a logical unit 2 1 0
Path=/devsdb1,Type=fileio
Jun 3 09:23:57 srv-dell kernel: iscsi_trgt: open_path(119) Can't open
/devsdb1 -2
Jun 3 09:23:57 srv-dell kernel: iscsi_trgt: fileio_attach(283) -2
Jun 3 09:25:03 srv-dell kernel: scsi3 : iSCSI Initiator over TCP/IP
Jun 3 09:25:04 srv-dell iscsid: connection1:0 is operational now
Jun 3 09:29:35 srv-dell SuSEfirewall2: batch committing...
Jun 3 09:29:36 srv-dell SuSEfirewall2: Firewall rules unloaded.
Jun 3 09:29:36 srv-dell SuSEfirewall2: Setting up rules from
/etc/sysconfig/SuSEfirewall2 ...
Jun 3 09:29:36 srv-dell SuSEfirewall2: using default zone 'ext' for
interface eth0
Jun 3 09:29:36 srv-dell SuSEfirewall2: batch committing...
Jun 3 09:29:36 srv-dell SuSEfirewall2: Firewall rules successfully set
Jun 3 09:29:39 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 09:29:39 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 09:29:39 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 09:29:39 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 09:29:39 srv-dell kernel: iscsi_trgt: open_path(119) Can't open
/devsdb1 -2
Jun 3 09:29:39 srv-dell kernel: iscsi_trgt: fileio_attach(283) -2
Jun 3 09:29:39 srv-dell ietd: Can't create a logical unit 2 1 0
Path=/devsdb1,Type=fileio
Jun 3 09:29:43 srv-dell kernel: connection1:0: detected conn error (1011)
Jun 3 09:29:44 srv-dell iscsid: Kernel reported iSCSI connection 1:0
error (1011) state (3)
Jun 3 09:29:47 srv-dell iscsid: connection1:0 is operational after
recovery (1 attempts)
Jun 3 09:30:16 srv-dell SuSEfirewall2: batch committing...
Jun 3 09:30:16 srv-dell SuSEfirewall2: Firewall rules unloaded.
Jun 3 09:34:15 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 09:34:15 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_pe_invoke: Query 133:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244036055-88, seq=184, quorate=1
Jun 3 09:34:15 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 09:34:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:34:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:34:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:34:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:34:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:34:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:34:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:34:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:34:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:34:15 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 09:34:15 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:34:15 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:34:15 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:34:15 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 09:34:15 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:34:15 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 09:34:15 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 09:34:15 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 09:34:15 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:34:15 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 16: 0 actions in 0 synapses
Jun 3 09:34:15 srv-dell pengine: [13707]: info: process_pe_message:
Transition 16: PEngine Input stored in:
/var/lib/pengine/pe-input-14.bz2
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 16 (ref=pe_calc-dc-1244036055-88) derived from
/var/lib/pengine/pe-input-14.bz2
Jun 3 09:34:15 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:34:15 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:34:15 srv-dell crmd: [13708]: notice: run_graph: Transition
16 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-14.bz2): Complete
Jun 3 09:34:15 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 16 is now complete
Jun 3 09:34:15 srv-dell crmd: [13708]: info: notify_crmd: Transition
16 status: done - <null>
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:34:15 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:35:51 srv-dell openais[13697]: [TOTEM] The token was lost in
the OPERATIONAL state.
Jun 3 09:35:51 srv-dell openais[13697]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 09:35:51 srv-dell openais[13697]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 09:35:51 srv-dell openais[13697]: [TOTEM] entering GATHER state from 2.
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] entering GATHER state from 0.
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] Saving state aru c6
high seq received c6
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] Storing new sequence
id for ring bc
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] entering COMMIT state.
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] entering RECOVERY state.
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] previous ring seq 184
rep 192.168.2.120
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] aru c6 high delivered
c6 received flag 1
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] Sending initial ORF token
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] New Configuration:
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] Members Left:
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] Members Joined:
Jun 3 09:35:54 srv-dell openais[13697]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 188: memb=1,
new=0, lost=1
Jun 3 09:35:54 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 09:35:54 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: lost: srv-lenovo 2013440192
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] New Configuration:
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] Members Left:
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] Members Joined:
Jun 3 09:35:54 srv-dell openais[13697]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 188: memb=1, new=0,
lost=0
Jun 3 09:35:54 srv-dell crmd: [13708]: notice: ais_dispatch:
Membership 188: quorum lost
Jun 3 09:35:54 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 09:35:54 srv-dell openais[13697]: [crm ] info:
ais_mark_unseen_peer_dead: Node srv-lenovo was not seen in the
previous transition
Jun 3 09:35:54 srv-dell openais[13697]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: lost
Jun 3 09:35:54 srv-dell openais[13697]: [crm ] info:
send_member_notification: Sending membership update 188 to 2 children
Jun 3 09:35:54 srv-dell openais[13697]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 09:35:54 srv-dell openais[13697]: [TOTEM] entering OPERATIONAL state.
Jun 3 09:35:54 srv-dell crmd: [13708]: info: ais_status_callback:
status: srv-lenovo is now lost (was member)
Jun 3 09:35:54 srv-dell openais[13697]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 09:35:54 srv-dell crmd: [13708]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=184 seen=184 proc=00000000000000000000000000053312
Jun 3 09:35:54 srv-dell crmd: [13708]: info: erase_node_from_join:
Removed node srv-lenovo from join calculations: welcomed=0 itegrated=0
finalized=0 confirmed=1
Jun 3 09:35:54 srv-dell cib: [13704]: notice: ais_dispatch:
Membership 188: quorum lost
Jun 3 09:35:54 srv-dell cib: [13704]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=184 seen=184 proc=00000000000000000000000000053312
Jun 3 09:35:54 srv-dell crmd: [13708]: info: crm_update_quorum:
Updating quorum status to false (call=136)
Jun 3 09:35:54 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/134, version=0.22.5): ok (rc=0)
Jun 3 09:35:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:35:54 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib have-quorum="1" admin_epoch="0" epoch="22"
num_updates="6" />
Jun 3 09:35:54 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib have-quorum="0" admin_epoch="0" epoch="23"
num_updates="1" />
Jun 3 09:35:54 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/136, version=0.23.1): ok (rc=0)
Jun 3 09:35:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:35:54 srv-dell crmd: [13708]: WARN: match_down_event: No
match for shutdown action on srv-lenovo
Jun 3 09:35:54 srv-dell crmd: [13708]: info: te_update_diff:
Stonith/shutdown of srv-lenovo not matched
Jun 3 09:35:54 srv-dell crmd: [13708]: info: abort_transition_graph:
te_update_diff:191 - Triggered transition abort (complete=1,
tag=node_state, id=srv-lenovo, magic=NA) : Node failure
Jun 3 09:35:54 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 09:35:54 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to have-quorum
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_pe_invoke: Query 139:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_pe_invoke: Query 140:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:35:54 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/138, version=0.23.1): ok (rc=0)
Jun 3 09:35:54 srv-dell cib: [24539]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-23.raw
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244036154-90, seq=188, quorate=0
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 09:35:54 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:35:54 srv-dell cib: [24539]: info: write_cib_contents: Wrote
version 0.23.0 of the CIB to disk (digest:
78da52f03573ee95801fac04bfa8ad66)
Jun 3 09:35:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:35:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:35:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:35:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:35:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:35:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 09:35:54 srv-dell pengine: [13707]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=member, expected=member
Jun 3 09:35:54 srv-dell cib: [24539]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.bdExGJ (digest:
/var/lib/heartbeat/crm/cib.KEYRRV)
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:35:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:35:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell ]
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:1 ]
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN: native_color:
Resource cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN: custom_action:
Marking node srv-lenovo unclean
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: stage6: Cannot
fence unclean nodes until quorum is attained (or no-quorum-policy is
set to ignore)
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: LogActions: Stop
resource primitiveresource_cluster_ip (srv-dell)
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 09:35:54 srv-dell pengine: [13707]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:35:54 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 17: 4 actions in 4 synapses
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 17 (ref=pe_calc-dc-1244036154-90) derived from
/var/lib/pengine/pe-warn-10.bz2
Jun 3 09:35:54 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 5: stop primitiveresource_cluster_ip_stop_0 on
srv-dell (local)
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=5:17:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 09:35:54 srv-dell lrmd: [13705]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 09:35:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 12 fired and confirmed
Jun 3 09:35:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 13 fired and confirmed
Jun 3 09:35:54 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=16, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 09:35:54 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 17: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-10.bz2
Jun 3 09:35:54 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:35:54 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 09:35:54 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 09:35:54 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 09:35:54 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_09:35:54 INFO:
ifconfig eth1:0 down
Jun 3 09:35:54 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=17, rc=0,
cib-update=141, confirmed=true) complete ok
Jun 3 09:35:54 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_stop_0 (5) confirmed on srv-dell
(rc=0)
Jun 3 09:35:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 09:35:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 09:35:54 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:35:54 srv-dell crmd: [13708]: notice: run_graph: Transition
17 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-10.bz2): Complete
Jun 3 09:35:54 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 17 is now complete
Jun 3 09:35:54 srv-dell crmd: [13708]: info: notify_crmd: Transition
17 status: done - <null>
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:35:54 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:35:54 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 09:43:32 srv-dell cib: [13704]: info: cib_stats: Processed 12
operations (6666.00us average, 0% utilization) in the last 10min
Jun 3 09:50:54 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 09:50:54 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_pe_invoke: Query 142:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244037054-92, seq=188, quorate=0
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 09:50:54 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 09:50:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:50:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 09:50:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 09:50:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 09:50:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 09:50:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 09:50:54 srv-dell pengine: [13707]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=member, expected=member
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 09:50:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 09:50:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell ]
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:1 ]
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN: native_color:
Resource cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN: custom_action:
Marking node srv-lenovo unclean
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: stage6: Cannot
fence unclean nodes until quorum is attained (or no-quorum-policy is
set to ignore)
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 09:50:54 srv-dell pengine: [13707]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 09:50:54 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 18: 3 actions in 3 synapses
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 18 (ref=pe_calc-dc-1244037054-92) derived from
/var/lib/pengine/pe-warn-11.bz2
Jun 3 09:50:54 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 18: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-11.bz2
Jun 3 09:50:54 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 09:50:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 11 fired and confirmed
Jun 3 09:50:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 12 fired and confirmed
Jun 3 09:50:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 1 fired and confirmed
Jun 3 09:50:54 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 09:50:54 srv-dell crmd: [13708]: notice: run_graph: Transition
18 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-11.bz2): Complete
Jun 3 09:50:54 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 18 is now complete
Jun 3 09:50:54 srv-dell crmd: [13708]: info: notify_crmd: Transition
18 status: done - <null>
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 09:50:54 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 09:53:32 srv-dell cib: [13704]: info: cib_stats: Processed 1
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 10:03:30 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 10:03:30 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 10:03:30 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 10:03:30 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 10:03:39 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 3 10:03:39 srv-dell kernel: iscsi: registered transport (tcp)
Jun 3 10:03:39 srv-dell kernel: iscsi: registered transport (iser)
Jun 3 10:03:39 srv-dell iscsid: iSCSI logger with pid=25124 started!
Jun 3 10:03:40 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 3 10:03:40 srv-dell iscsid: iSCSI daemon with pid=25125 started!
Jun 3 10:05:22 srv-dell syslog-ng[3116]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=2937', processed='center(received)=2771',
processed='destination(newsnotice)=0',
processed='destination(acpid)=0',
processed='destination(firewall)=54', processed='destination(null)=0',
processed='destination(mail)=6', processed='destination(mailinfo)=6',
processed='destination(console)=15',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=2711',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=15',
processed='destination(warn)=128', processed='source(src)=2771'
Jun 3 10:05:54 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 10:05:54 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_pe_invoke: Query 143:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244037954-93, seq=188, quorate=0
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 10:05:54 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 10:05:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:05:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 10:05:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 10:05:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 10:05:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:05:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 10:05:54 srv-dell pengine: [13707]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=member, expected=member
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:1
running on srv-lenovo
Jun 3 10:05:54 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:1"
type="null" class="stonith" />
Jun 3 10:05:54 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:1 is stopped
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell ]
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:1 ]
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN: native_color:
Resource cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN: custom_action:
Marking node srv-lenovo unclean
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: stage6: Cannot
fence unclean nodes until quorum is attained (or no-quorum-policy is
set to ignore)
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 10:05:54 srv-dell pengine: [13707]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 10:05:54 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 19: 3 actions in 3 synapses
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 19 (ref=pe_calc-dc-1244037954-93) derived from
/var/lib/pengine/pe-warn-12.bz2
Jun 3 10:05:54 srv-dell pengine: [13707]: WARN: process_pe_message:
Transition 19: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-12.bz2
Jun 3 10:05:54 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 10:05:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 11 fired and confirmed
Jun 3 10:05:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 12 fired and confirmed
Jun 3 10:05:54 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 1 fired and confirmed
Jun 3 10:05:54 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 10:05:54 srv-dell crmd: [13708]: notice: run_graph: Transition
19 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-12.bz2): Complete
Jun 3 10:05:54 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 19 is now complete
Jun 3 10:05:54 srv-dell crmd: [13708]: info: notify_crmd: Transition
19 status: done - <null>
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 10:05:54 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 10:06:04 srv-dell kernel: kjournald starting. Commit interval 5 seconds
Jun 3 10:06:04 srv-dell kernel: EXT3 FS on dm-1, internal journal
Jun 3 10:06:04 srv-dell kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Jun 3 10:06:57 srv-dell openais[13697]: [TOTEM] entering GATHER state from 11.
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] Saving state aru 13
high seq received 13
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] Storing new sequence
id for ring c0
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] entering COMMIT state.
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] entering RECOVERY state.
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] previous ring seq 188
rep 192.168.2.120
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] aru a high delivered
a received flag 1
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] previous ring seq 188
rep 192.168.2.220
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] aru 13 high delivered
13 received flag 1
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] New Configuration:
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] Members Left:
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] Members Joined:
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 192: memb=1,
new=0, lost=0
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] New Configuration:
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] Members Left:
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] Members Joined:
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 192: memb=2, new=1,
lost=0
Jun 3 10:06:59 srv-dell openais[13697]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: member
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: NEW: srv-lenovo 2013440192
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: MEMB: srv-lenovo 2013440192
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] info:
send_member_notification: Sending membership update 192 to 2 children
Jun 3 10:06:59 srv-dell openais[13697]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 10:06:59 srv-dell openais[13697]: [TOTEM] entering OPERATIONAL state.
Jun 3 10:06:59 srv-dell openais[13697]: [MAIN ] info: update_member:
0x7fcaa40349b0 Node 2013440192 (srv-lenovo) born on: 192
Jun 3 10:06:59 srv-dell openais[13697]: [crm ] info:
send_member_notification: Sending membership update 192 to 2 children
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 10:06:59 srv-dell openais[13697]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 10:06:59 srv-dell crmd: [13708]: notice: ais_dispatch:
Membership 192: quorum aquired
Jun 3 10:06:59 srv-dell crmd: [13708]: info: ais_status_callback:
status: srv-lenovo is now member (was lost)
Jun 3 10:06:59 srv-dell crmd: [13708]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=184 seen=192
proc=00000000000000000000000000053312
Jun 3 10:06:59 srv-dell cib: [13704]: notice: ais_dispatch:
Membership 192: quorum aquired
Jun 3 10:06:59 srv-dell cib: [13704]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=184 seen=192
proc=00000000000000000000000000053312
Jun 3 10:06:59 srv-dell crmd: [13708]: info: crm_update_quorum:
Updating quorum status to true (call=148)
Jun 3 10:06:59 srv-dell cib: [13704]: info: ais_dispatch: Membership
192: quorum retained
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/144,
version=0.23.3): ok (rc=0)
Jun 3 10:06:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=local/crmd/145, version=0.23.4): ok (rc=0)
Jun 3 10:06:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/146, version=0.23.4): ok (rc=0)
Jun 3 10:06:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:06:59 srv-dell cib: [13704]: info: log_data_element:
cib:diff: - <cib have-quorum="0" admin_epoch="0" epoch="23"
num_updates="5" />
Jun 3 10:06:59 srv-dell cib: [13704]: info: log_data_element:
cib:diff: + <cib have-quorum="1" admin_epoch="0" epoch="24"
num_updates="1" />
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/148, version=0.24.1): ok (rc=0)
Jun 3 10:06:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_sync_one for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.24.1): ok (rc=0)
Jun 3 10:06:59 srv-dell crmd: [13708]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=cluster_ip_address_monitor_0,
magic=0:7;5:7:7:8c22a74b-522f-4be1-ab50-02fcd5915fa2) : Resource op
removal
Jun 3 10:06:59 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/150, version=0.24.1): ok (rc=0)
Jun 3 10:06:59 srv-dell crmd: [13708]: info: abort_transition_graph:
te_update_diff:157 - Triggered transition abort (complete=1,
tag=transient_attributes, id=srv-lenovo, magic=NA) : Transient
attribute: removal
Jun 3 10:06:59 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/transient_attributes":
ok (rc=0)
Jun 3 10:06:59 srv-dell crmd: [13708]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 10:06:59 srv-dell crmd: [13708]: info: need_abort: Aborting on
change to have-quorum
Jun 3 10:06:59 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 10:06:59 srv-dell crmd: [13708]: info: do_state_transition:
Membership changed: 184 -> 192 - join restart
Jun 3 10:06:59 srv-dell crmd: [13708]: info: do_pe_invoke: Query 151:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 10:06:59 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=do_state_transition ]
Jun 3 10:06:59 srv-dell crmd: [13708]: info: update_dc: Unset DC srv-dell
Jun 3 10:06:59 srv-dell crmd: [13708]: info: join_make_offer: Making
join offers based on membership 192
Jun 3 10:06:59 srv-dell crmd: [13708]: info: do_dc_join_offer_all:
join-6: Waiting on 2 outstanding join acks
Jun 3 10:06:59 srv-dell crmd: [13708]: info: ais_dispatch: Membership
192: quorum retained
Jun 3 10:06:59 srv-dell cib: [25838]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-24.raw
Jun 3 10:06:59 srv-dell cib: [25838]: info: write_cib_contents: Wrote
version 0.24.0 of the CIB to disk (digest:
bb0d1debfe4eeb24acefaa1aee868979)
Jun 3 10:06:59 srv-dell cib: [25838]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.7Yykm1 (digest:
/var/lib/heartbeat/crm/cib.FOPOfK)
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/152, version=0.24.1): ok (rc=0)
Jun 3 10:06:59 srv-dell crmd: [13708]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 10:06:59 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/155, version=0.24.1): ok (rc=0)
Jun 3 10:07:00 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 10:07:00 srv-dell crmd: [13708]: info: update_dc: Unset DC srv-dell
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_dc_join_offer_all: A
new node joined the cluster
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_dc_join_offer_all:
join-7: Waiting on 2 outstanding join acks
Jun 3 10:07:00 srv-dell crmd: [13708]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes responded to the join offer.
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_dc_join_finalize:
join-7: Syncing the CIB from srv-dell to the rest of the cluster
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/158, version=0.24.1): ok (rc=0)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_dc_join_ack: join-7:
Updating node state to member for srv-dell
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/159, version=0.24.1): ok (rc=0)
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/160, version=0.24.1): ok (rc=0)
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/161,
version=0.24.2): ok (rc=0)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 10:07:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=srv-lenovo/crmd/7, version=0.24.3): ok (rc=0)
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=srv-lenovo/crmd/8,
version=0.24.3): ok (rc=0)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_dc_join_ack: join-7:
Updating node state to member for srv-lenovo
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/163,
version=0.24.3): ok (rc=0)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 3 10:07:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:00 srv-dell crmd: [13708]: info: crm_update_quorum:
Updating quorum status to true (call=167)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_pe_invoke: Query 168:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 10:07:00 srv-dell attrd: [13706]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 10:07:00 srv-dell attrd: [13706]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/165, version=0.24.4): ok (rc=0)
Jun 3 10:07:00 srv-dell cib: [13704]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/167, version=0.24.4): ok (rc=0)
Jun 3 10:07:00 srv-dell attrd: [13706]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244038020-105, seq=192, quorate=1
Jun 3 10:07:00 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 10:07:00 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 10:07:00 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:07:00 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 10:07:00 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 10:07:00 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:07:00 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 10:07:00 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 10:07:00 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 10:07:00 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 10:07:00 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell ]
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:1 ]
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: RecurringOp: Start
recurring monitor (5s) for primitiveresource_cluster_ip on srv-dell
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 10:07:00 srv-dell pengine: [13707]: notice: LogActions: Start
cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 10:07:00 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 20: 9 actions in 9 synapses
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 20 (ref=pe_calc-dc-1244038020-105) derived from
/var/lib/pengine/pe-input-15.bz2
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 5: monitor primitiveresource_cluster_ip_monitor_0 on
srv-lenovo
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 6: monitor
cloneresource_cluster_fencing_null:1_monitor_0 on srv-lenovo
Jun 3 10:07:00 srv-dell pengine: [13707]: info: process_pe_message:
Transition 20: PEngine Input stored in:
/var/lib/pengine/pe-input-15.bz2
Jun 3 10:07:00 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 10:07:00 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:1_monitor_0 (6) confirmed on
srv-lenovo (rc=0)
Jun 3 10:07:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:00 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_monitor_0 (5) confirmed on
srv-lenovo (rc=0)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 4: probe_complete probe_complete on srv-lenovo - no
waiting
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 7: start primitiveresource_cluster_ip_start_0 on
srv-dell (local)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=7:20:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_start_0 )
Jun 3 10:07:00 srv-dell lrmd: [13705]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 12 fired and confirmed
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 11: start
cloneresource_cluster_fencing_null:1_start_0 on srv-lenovo
Jun 3 10:07:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:00 srv-dell crmd: [13708]: info: match_graph_event:
Action cloneresource_cluster_fencing_null:1_start_0 (11) confirmed on
srv-lenovo (rc=0)
Jun 3 10:07:00 srv-dell crmd: [13708]: info: te_pseudo_action: Pseudo
action 13 fired and confirmed
Jun 3 10:07:01 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 10:07:01 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_10:07:01 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 10:07:01 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 10:07:01 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_10:07:01 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 10:07:01 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 10:07:01 srv-dell lrmd: [13705]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_10:07:01 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 10:07:01 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=18, rc=0,
cib-update=169, confirmed=true) complete ok
Jun 3 10:07:01 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_start_0 (7) confirmed on srv-dell
(rc=0)
Jun 3 10:07:01 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:01 srv-dell crmd: [13708]: info: te_rsc_command:
Initiating action 8: monitor primitiveresource_cluster_ip_monitor_5000
on srv-dell (local)
Jun 3 10:07:01 srv-dell crmd: [13708]: info: do_lrm_rsc_op:
Performing key=8:20:0:8c22a74b-522f-4be1-ab50-02fcd5915fa2
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 10:07:01 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 10:07:02 srv-dell crmd: [13708]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=19, rc=0,
cib-update=170, confirmed=false) complete ok
Jun 3 10:07:02 srv-dell crmd: [13708]: info: match_graph_event:
Action primitiveresource_cluster_ip_monitor_5000 (8) confirmed on
srv-dell (rc=0)
Jun 3 10:07:02 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 10:07:02 srv-dell crmd: [13708]: notice: run_graph: Transition
20 (Complete=9, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-15.bz2): Complete
Jun 3 10:07:02 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 20 is now complete
Jun 3 10:07:02 srv-dell crmd: [13708]: info: notify_crmd: Transition
20 status: done - <null>
Jun 3 10:07:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 10:07:02 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 10:07:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 10:07:02 srv-dell mgmtd: [13709]: info: CIB query: cib
Jun 3 10:08:28 srv-dell kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Jun 3 10:08:28 srv-dell kernel: ISO 9660 Extensions: RRIP_1991A
Jun 3 10:13:32 srv-dell cib: [13704]: info: cib_stats: Processed 49
operations (4489.00us average, 0% utilization) in the last 10min
Jun 3 10:14:16 srv-dell kernel: drbd: initialised. Version: 8.2.7
(api:88/proto:86-88)
Jun 3 10:14:16 srv-dell kernel: drbd: GIT-hash:
a1b440e8b3011a1318d8bff1bb7edc763ef995b0 build by lmb at hermes,
2009-02-20 13:35:59
Jun 3 10:14:16 srv-dell kernel: drbd: registered as block device major 147
Jun 3 10:14:16 srv-dell kernel: drbd: minor_table @ 0xffff8801eaa42ec0
Jun 3 10:19:44 srv-dell kernel: drbd: module cleanup done.
Jun 3 10:19:44 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 10:19:44 srv-dell kernel: drbd: initialised. Version: 8.2.7
(api:88/proto:86-88)
Jun 3 10:19:44 srv-dell kernel: drbd: GIT-hash:
a1b440e8b3011a1318d8bff1bb7edc763ef995b0 build by lmb at hermes,
2009-02-20 13:35:59
Jun 3 10:19:44 srv-dell kernel: drbd: registered as block device major 147
Jun 3 10:19:44 srv-dell kernel: drbd: minor_table @ 0xffff88018ec631c0
Jun 3 10:19:44 srv-dell kernel: drbd0: disk( Diskless -> Attaching )
Jun 3 10:19:44 srv-dell kernel: drbd0: Starting worker thread (from
cqueue [48])
Jun 3 10:19:44 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 10:19:44 srv-dell kernel: drbd0: Error while reading metadata,
magic not found.
Jun 3 10:19:44 srv-dell kernel: drbd0: disk( Attaching -> Diskless )
Jun 3 10:19:44 srv-dell kernel: drbd0: drbd_bm_resize called with capacity == 0
Jun 3 10:19:44 srv-dell kernel: drbd0: worker terminated
Jun 3 10:19:44 srv-dell kernel: drbd0: Terminating worker thread
Jun 3 10:22:02 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 10:22:02 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_pe_invoke: Query 171:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244038922-112, seq=192, quorate=1
Jun 3 10:22:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 10:22:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 10:22:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:22:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 10:22:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 10:22:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:22:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 10:22:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 10:22:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 10:22:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 10:22:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 10:22:02 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 10:22:02 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 10:22:02 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 10:22:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 10:22:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 10:22:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 10:22:02 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 21: 0 actions in 0 synapses
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 21 (ref=pe_calc-dc-1244038922-112) derived from
/var/lib/pengine/pe-input-16.bz2
Jun 3 10:22:02 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 10:22:02 srv-dell crmd: [13708]: notice: run_graph: Transition
21 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-16.bz2): Complete
Jun 3 10:22:02 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 21 is now complete
Jun 3 10:22:02 srv-dell crmd: [13708]: info: notify_crmd: Transition
21 status: done - <null>
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 10:22:02 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 10:22:02 srv-dell pengine: [13707]: info: process_pe_message:
Transition 21: PEngine Input stored in:
/var/lib/pengine/pe-input-16.bz2
Jun 3 10:22:02 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 10:23:32 srv-dell cib: [13704]: info: cib_stats: Processed 1
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 10:26:42 srv-dell kernel: drbd0: disk( Diskless -> Attaching )
Jun 3 10:26:42 srv-dell kernel: drbd0: Starting worker thread (from
cqueue [48])
Jun 3 10:26:42 srv-dell kernel: drbd0: No usable activity log found.
Jun 3 10:26:42 srv-dell kernel: drbd0: Method to ensure write ordering: barrier
Jun 3 10:26:42 srv-dell kernel: drbd0: Backing device's
merge_bvec_fn() = ffffffffa00f2798
Jun 3 10:26:42 srv-dell kernel: drbd0: max_segment_size ( = BIO size ) = 4096
Jun 3 10:26:42 srv-dell kernel: drbd0: drbd_bm_resize called with
capacity == 41925304
Jun 3 10:26:42 srv-dell kernel: drbd0: resync bitmap: bits=5240663 words=81886
Jun 3 10:26:42 srv-dell kernel: drbd0: size = 20 GB (20962652 KB)
Jun 3 10:26:42 srv-dell kernel: drbd0: Writing the whole bitmap, size changed
Jun 3 10:26:42 srv-dell kernel: drbd0: meta data flush failed with
status -95, disabling md-flushes
Jun 3 10:26:42 srv-dell kernel: drbd0: 20 GB (5240663 bits) marked
out-of-sync by on disk bit-map.
Jun 3 10:26:42 srv-dell kernel: drbd0: recounting of set bits took
additional 0 jiffies
Jun 3 10:26:42 srv-dell kernel: drbd0: 20 GB (5240663 bits) marked
out-of-sync by on disk bit-map.
Jun 3 10:26:42 srv-dell kernel: drbd0: disk( Attaching -> Inconsistent )
Jun 3 10:26:42 srv-dell kernel: drbd0: conn( StandAlone -> Unconnected )
Jun 3 10:26:42 srv-dell kernel: drbd0: Starting receiver thread (from
drbd0_worker [31600])
Jun 3 10:26:42 srv-dell kernel: drbd0: receiver (re)started
Jun 3 10:26:42 srv-dell kernel: drbd0: conn( Unconnected -> WFConnection )
Jun 3 10:27:30 srv-dell kernel: drbd0: Handshake successful: Agreed
network protocol version 88
Jun 3 10:27:30 srv-dell kernel: drbd0: conn( WFConnection -> WFReportParams )
Jun 3 10:27:30 srv-dell kernel: drbd0: Starting asender thread (from
drbd0_receiver [31602])
Jun 3 10:27:30 srv-dell kernel: drbd0: data-integrity-alg: <not-used>
Jun 3 10:27:30 srv-dell kernel: drbd0: drbd_sync_handshake:
Jun 3 10:27:30 srv-dell kernel: drbd0: self
0000000000000004:0000000000000000:0000000000000000:0000000000000000
Jun 3 10:27:30 srv-dell kernel: drbd0: peer
0000000000000004:0000000000000000:0000000000000000:0000000000000000
Jun 3 10:27:30 srv-dell kernel: drbd0: uuid_compare()=0 by rule 1
Jun 3 10:27:30 srv-dell kernel: drbd0: No resync, but 5240663 bits in bitmap!
Jun 3 10:27:30 srv-dell kernel: drbd0: peer( Unknown -> Secondary )
conn( WFReportParams -> Connected ) pdsk( DUnknown -> Inconsistent )
Jun 3 10:29:25 srv-dell kernel: drbd0: role( Secondary -> Primary )
disk( Inconsistent -> UpToDate )
Jun 3 10:29:25 srv-dell kernel: drbd0: Forced to consider local data
as UpToDate!
Jun 3 10:29:25 srv-dell kernel: drbd0: Creating new current UUID
Jun 3 10:29:25 srv-dell kernel: drbd0: drbd_sync_handshake:
Jun 3 10:29:25 srv-dell kernel: drbd0: self
2E25EF88A8250665:0000000000000004:0000000000000000:0000000000000000
Jun 3 10:29:25 srv-dell kernel: drbd0: peer
0000000000000004:0000000000000000:0000000000000000:0000000000000000
Jun 3 10:29:25 srv-dell kernel: drbd0: uuid_compare()=2 by rule 3
Jun 3 10:29:25 srv-dell kernel: drbd0: Becoming sync source due to disk states.
Jun 3 10:29:25 srv-dell kernel: drbd0: Writing the whole bitmap, full
sync required after drbd_sync_handshake.
Jun 3 10:29:25 srv-dell kernel: drbd0: 20 GB (5240663 bits) marked
out-of-sync by on disk bit-map.
Jun 3 10:29:25 srv-dell kernel: drbd0: conn( Connected -> WFBitMapS )
Jun 3 10:29:25 srv-dell kernel: drbd0: conn( WFBitMapS -> SyncSource )
Jun 3 10:29:25 srv-dell kernel: drbd0: Began resync as SyncSource
(will sync 20962652 KB [5240663 bits set]).
Jun 3 10:37:02 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 10:37:02 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_pe_invoke: Query 172:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244039822-113, seq=192, quorate=1
Jun 3 10:37:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 10:37:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 10:37:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:37:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 10:37:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 10:37:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:37:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 10:37:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 10:37:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 10:37:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 10:37:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 10:37:02 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 10:37:02 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 10:37:02 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 10:37:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 10:37:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 10:37:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 10:37:02 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 22: 0 actions in 0 synapses
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 22 (ref=pe_calc-dc-1244039822-113) derived from
/var/lib/pengine/pe-input-17.bz2
Jun 3 10:37:02 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 10:37:02 srv-dell crmd: [13708]: notice: run_graph: Transition
22 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-17.bz2): Complete
Jun 3 10:37:02 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 22 is now complete
Jun 3 10:37:02 srv-dell crmd: [13708]: info: notify_crmd: Transition
22 status: done - <null>
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 10:37:02 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 10:37:02 srv-dell pengine: [13707]: info: process_pe_message:
Transition 22: PEngine Input stored in:
/var/lib/pengine/pe-input-17.bz2
Jun 3 10:37:02 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 10:43:32 srv-dell cib: [13704]: info: cib_stats: Processed 1
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 10:44:13 srv-dell kernel: VFS: Can't find ext3 filesystem on dev drbd0.
Jun 3 10:47:49 srv-dell kernel: kjournald starting. Commit interval 5 seconds
Jun 3 10:47:49 srv-dell kernel: EXT3 FS on drbd0, internal journal
Jun 3 10:47:49 srv-dell kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Jun 3 10:48:00 srv-dell kernel: JBD: barrier-based sync failed on
drbd0 - disabling barriers
Jun 3 10:52:02 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 10:52:02 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_pe_invoke: Query 173:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244040722-114, seq=192, quorate=1
Jun 3 10:52:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 10:52:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 10:52:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:52:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 10:52:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 10:52:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 10:52:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 10:52:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 10:52:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 10:52:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 10:52:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 10:52:02 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 10:52:02 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 10:52:02 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 10:52:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 10:52:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 10:52:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 10:52:02 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 23: 0 actions in 0 synapses
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 23 (ref=pe_calc-dc-1244040722-114) derived from
/var/lib/pengine/pe-input-18.bz2
Jun 3 10:52:02 srv-dell pengine: [13707]: info: process_pe_message:
Transition 23: PEngine Input stored in:
/var/lib/pengine/pe-input-18.bz2
Jun 3 10:52:02 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 10:52:02 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 10:52:02 srv-dell crmd: [13708]: notice: run_graph: Transition
23 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-18.bz2): Complete
Jun 3 10:52:02 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 23 is now complete
Jun 3 10:52:02 srv-dell crmd: [13708]: info: notify_crmd: Transition
23 status: done - <null>
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 10:52:02 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 10:52:25 srv-dell kernel: drbd0: State change failed: Device is
held open by someone
Jun 3 10:52:25 srv-dell kernel: drbd0: state = { cs:SyncSource
st:Primary/Secondary ds:UpToDate/Inconsistent r--- }
Jun 3 10:52:25 srv-dell kernel: drbd0: wanted = { cs:SyncSource
st:Secondary/Secondary ds:UpToDate/Inconsistent r--- }
Jun 3 10:52:43 srv-dell kernel: drbd0: State change failed: Device is
held open by someone
Jun 3 10:52:43 srv-dell kernel: drbd0: state = { cs:SyncSource
st:Primary/Secondary ds:UpToDate/Inconsistent r--- }
Jun 3 10:52:43 srv-dell kernel: drbd0: wanted = { cs:SyncSource
st:Secondary/Secondary ds:UpToDate/Inconsistent r--- }
Jun 3 10:53:32 srv-dell cib: [13704]: info: cib_stats: Processed 1
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 10:53:35 srv-dell kernel: drbd0: peer( Secondary -> Unknown )
conn( SyncSource -> TearDown )
Jun 3 10:53:35 srv-dell kernel: drbd0: meta connection shut down by peer.
Jun 3 10:53:35 srv-dell kernel: drbd0: asender terminated
Jun 3 10:53:35 srv-dell kernel: drbd0: Terminating asender thread
Jun 3 10:53:35 srv-dell kernel: drbd0: Connection closed
Jun 3 10:53:35 srv-dell kernel: drbd0: conn( TearDown -> Unconnected )
Jun 3 10:53:35 srv-dell kernel: drbd0: receiver terminated
Jun 3 10:53:35 srv-dell kernel: drbd0: Restarting receiver thread
Jun 3 10:53:35 srv-dell kernel: drbd0: receiver (re)started
Jun 3 10:53:35 srv-dell kernel: drbd0: conn( Unconnected -> WFConnection )
Jun 3 10:54:12 srv-dell kernel: drbd0: State change failed: Device is
held open by someone
Jun 3 10:54:12 srv-dell kernel: drbd0: state = { cs:WFConnection
st:Primary/Unknown ds:UpToDate/Inconsistent r--- }
Jun 3 10:54:12 srv-dell kernel: drbd0: wanted = { cs:WFConnection
st:Secondary/Unknown ds:UpToDate/Inconsistent r--- }
Jun 3 10:54:58 srv-dell kernel: drbd0: State change failed: Refusing
to be Primary without at least one UpToDate disk
Jun 3 10:54:58 srv-dell kernel: drbd0: state = { cs:WFConnection
st:Primary/Unknown ds:UpToDate/Inconsistent r--- }
Jun 3 10:54:58 srv-dell kernel: drbd0: wanted = { cs:WFConnection
st:Primary/Unknown ds:Diskless/Inconsistent r--- }
Jun 3 10:55:57 srv-dell kernel: drbd0: State change failed: Device is
held open by someone
Jun 3 10:55:57 srv-dell kernel: drbd0: state = { cs:WFConnection
st:Primary/Unknown ds:UpToDate/Inconsistent r--- }
Jun 3 10:55:57 srv-dell kernel: drbd0: wanted = { cs:WFConnection
st:Secondary/Unknown ds:UpToDate/Inconsistent r--- }
Jun 3 10:56:21 srv-dell kernel: drbd0: role( Primary -> Secondary )
Jun 3 10:56:21 srv-dell kernel: drbd0: conn( WFConnection -> Disconnecting )
Jun 3 10:56:21 srv-dell kernel: drbd0: Discarding network configuration.
Jun 3 10:56:21 srv-dell kernel: drbd0: Connection closed
Jun 3 10:56:21 srv-dell kernel: drbd0: conn( Disconnecting -> StandAlone )
Jun 3 10:56:21 srv-dell kernel: drbd0: receiver terminated
Jun 3 10:56:21 srv-dell kernel: drbd0: Terminating receiver thread
Jun 3 10:56:21 srv-dell kernel: drbd0: disk( UpToDate -> Diskless )
pdsk( Inconsistent -> DUnknown )
Jun 3 10:56:21 srv-dell kernel: drbd0: drbd_bm_resize called with capacity == 0
Jun 3 10:56:21 srv-dell kernel: drbd0: worker terminated
Jun 3 10:56:21 srv-dell kernel: drbd0: Terminating worker thread
Jun 3 11:05:22 srv-dell syslog-ng[3116]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=3476', processed='center(received)=3208',
processed='destination(newsnotice)=0',
processed='destination(acpid)=0',
processed='destination(firewall)=54', processed='destination(null)=0',
processed='destination(mail)=8', processed='destination(mailinfo)=8',
processed='destination(console)=39',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=3146',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=39',
processed='destination(warn)=180', processed='source(src)=3208'
Jun 3 11:07:02 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 11:07:02 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_pe_invoke: Query 174:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244041622-115, seq=192, quorate=1
Jun 3 11:07:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 11:07:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 11:07:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 11:07:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 11:07:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 11:07:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 11:07:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 11:07:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 11:07:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 11:07:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 11:07:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 11:07:02 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 11:07:02 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 11:07:02 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 11:07:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 11:07:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 11:07:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 11:07:02 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 24: 0 actions in 0 synapses
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 24 (ref=pe_calc-dc-1244041622-115) derived from
/var/lib/pengine/pe-input-19.bz2
Jun 3 11:07:02 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 11:07:02 srv-dell crmd: [13708]: notice: run_graph: Transition
24 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-19.bz2): Complete
Jun 3 11:07:02 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 24 is now complete
Jun 3 11:07:02 srv-dell crmd: [13708]: info: notify_crmd: Transition
24 status: done - <null>
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 11:07:02 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 11:07:02 srv-dell pengine: [13707]: info: process_pe_message:
Transition 24: PEngine Input stored in:
/var/lib/pengine/pe-input-19.bz2
Jun 3 11:07:02 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 11:08:16 srv-dell kernel: drbd: module cleanup done.
Jun 3 11:11:11 srv-dell kernel: kjournald starting. Commit interval 5 seconds
Jun 3 11:11:11 srv-dell kernel: EXT3 FS on sda4, internal journal
Jun 3 11:11:11 srv-dell kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Jun 3 11:13:32 srv-dell cib: [13704]: info: cib_stats: Processed 1
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 11:15:10 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 11:15:10 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 11:15:10 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 11:15:10 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 11:19:55 srv-dell kernel: iscsi_trgt: fileio_make_request(63)
I/O error 512, -5
Jun 3 11:19:55 srv-dell kernel: iscsi_trgt: fileio_make_request(63)
I/O error 512, -5
Jun 3 11:19:55 srv-dell kernel: iscsi_trgt: fileio_make_request(63)
I/O error 512, -5
Jun 3 11:19:55 srv-dell kernel: iscsi_trgt: fileio_make_request(63)
I/O error 512, -5
Jun 3 11:19:55 srv-dell kernel: iscsi_trgt: fileio_make_request(63)
I/O error 512, -5
Jun 3 11:19:55 srv-dell kernel: iscsi_trgt: fileio_make_request(63)
I/O error 512, -5
Jun 3 11:21:22 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 3 11:21:22 srv-dell kernel: iscsi: registered transport (tcp)
Jun 3 11:21:22 srv-dell kernel: iscsi: registered transport (iser)
Jun 3 11:21:22 srv-dell iscsid: iSCSI logger with pid=12788 started!
Jun 3 11:21:23 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 3 11:21:23 srv-dell iscsid: iSCSI daemon with pid=12789 started!
Jun 3 11:22:02 srv-dell crmd: [13708]: info: crm_timer_popped:
PEngine Recheck Timer (I_PE_CALC) just popped!
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 11:22:02 srv-dell crmd: [13708]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_pe_invoke: Query 175:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244042522-116, seq=192, quorate=1
Jun 3 11:22:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-dell is online
Jun 3 11:22:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
cluster_ip_address running on srv-dell
Jun 3 11:22:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="cluster_ip_address" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 11:22:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan cluster_ip_address is
stopped
Jun 3 11:22:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource
resource_clusterip running on srv-dell
Jun 3 11:22:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="resource_clusterip" type="IPaddr" class="ocf" provider="heartbeat"
/>
Jun 3 11:22:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan resource_clusterip is
stopped
Jun 3 11:22:02 srv-dell pengine: [13707]: WARN:
process_orphan_resource: Nothing known about resource st-null:0
running on srv-dell
Jun 3 11:22:02 srv-dell pengine: [13707]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="st-null:0"
type="null" class="stonith" />
Jun 3 11:22:02 srv-dell pengine: [13707]: info:
process_orphan_resource: Making sure orphan st-null:0 is stopped
Jun 3 11:22:02 srv-dell pengine: [13707]: info:
determine_online_status: Node srv-lenovo is online
Jun 3 11:22:02 srv-dell pengine: [13707]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 11:22:02 srv-dell pengine: [13707]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 11:22:02 srv-dell pengine: [13707]: notice: print_list:
Started: [ srv-dell srv-lenovo ]
Jun 3 11:22:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 3 11:22:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 11:22:02 srv-dell pengine: [13707]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-lenovo)
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 11:22:02 srv-dell crmd: [13708]: info: unpack_graph: Unpacked
transition 25: 0 actions in 0 synapses
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_te_invoke: Processing
graph 25 (ref=pe_calc-dc-1244042522-116) derived from
/var/lib/pengine/pe-input-20.bz2
Jun 3 11:22:02 srv-dell crmd: [13708]: info: run_graph:
====================================================
Jun 3 11:22:02 srv-dell crmd: [13708]: notice: run_graph: Transition
25 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-20.bz2): Complete
Jun 3 11:22:02 srv-dell pengine: [13707]: info: process_pe_message:
Transition 25: PEngine Input stored in:
/var/lib/pengine/pe-input-20.bz2
Jun 3 11:22:02 srv-dell crmd: [13708]: info: te_graph_trigger:
Transition 25 is now complete
Jun 3 11:22:02 srv-dell pengine: [13707]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 11:22:02 srv-dell crmd: [13708]: info: notify_crmd: Transition
25 status: done - <null>
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 11:22:02 srv-dell crmd: [13708]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 11:23:32 srv-dell cib: [13704]: info: cib_stats: Processed 1
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 11:23:45 srv-dell kernel: scsi4 : iSCSI Initiator over TCP/IP
Jun 3 11:23:45 srv-dell kernel: scsi 4:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] Write Protect is off
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] Mode Sense: 77 00 00 08
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] Write Protect is off
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] Mode Sense: 77 00 00 08
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 11:23:45 srv-dell kernel: sdc: sdc1
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: [sdc] Attached SCSI disk
Jun 3 11:23:45 srv-dell kernel: sd 4:0:0:0: Attached scsi generic sg4 type 0
Jun 3 11:23:45 srv-dell kernel: kjournald starting. Commit interval 5 seconds
Jun 3 11:23:45 srv-dell kernel: EXT3 FS on sdc1, internal journal
Jun 3 11:23:45 srv-dell kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Jun 3 11:23:45 srv-dell hald: mounted /dev/sdc1 on behalf of uid 0
Jun 3 11:23:45 srv-dell iscsid: connection1:0 is operational now
Jun 3 11:24:05 srv-dell hald: unmounted /dev/sdc1 from
'/media/disk-1' on behalf of uid 0
Jun 3 11:32:37 srv-dell console-kit-daemon[3244]:
GLib-GObject-WARNING: IA__g_object_get_valist: value location for
`gchararray' passed as NULL
Jun 3 11:32:43 srv-dell polkit-grant-helper[15917]: granted
authorization for org.freedesktop.consolekit.system.restart to pid
5480 [uid=0] [auth=root]
Jun 3 11:32:43 srv-dell console-kit-daemon[3244]:
GLib-GObject-WARNING: IA__g_object_get_valist: value location for
`gchararray' passed as NULL
Jun 3 11:32:43 srv-dell shutdown[15943]: shutting down for system reboot
Jun 3 11:32:43 srv-dell init: Switching to runlevel: 6
Jun 3 11:32:45 srv-dell libvirtd: Shutting down on signal 15
Jun 3 11:32:45 srv-dell auditd[4339]: Error sending signal_info
request (Operation not supported)
Jun 3 11:32:45 srv-dell auditd[4339]: The audit daemon is exiting.
Jun 3 11:32:45 srv-dell sshd[5004]: Received signal 15; terminating.
Jun 3 11:32:46 srv-dell rpcbind: rpcbind terminating on signal.
Restart with "rpcbind -w"
Jun 3 11:32:46 srv-dell kernel: Kernel logging (proc) stopped.
Jun 3 11:32:46 srv-dell kernel: Kernel log daemon terminating.
Jun 3 11:32:46 srv-dell syslog-ng[3116]: Termination requested via
signal, terminating;
Jun 3 11:32:46 srv-dell syslog-ng[3116]: syslog-ng shutting down;
version='2.0.9'
Jun 3 11:51:14 srv-dell syslog-ng[3118]: syslog-ng starting up; version='2.0.9'
Jun 3 11:51:14 srv-dell rchal: CPU frequency scaling is not supported
by your processor.
Jun 3 11:51:14 srv-dell rchal: boot with 'CPUFREQ=no' in to avoid this warning.
Jun 3 11:51:14 srv-dell rchal: Cannot load cpufreq governors - No
cpufreq driver available
Jun 3 11:51:14 srv-dell ifup: lo
Jun 3 11:51:14 srv-dell ifup: lo
Jun 3 11:51:14 srv-dell ifup: IP address: 127.0.0.1/8
Jun 3 11:51:14 srv-dell ifup:
Jun 3 11:51:14 srv-dell ifup:
Jun 3 11:51:14 srv-dell ifup: IP address: 127.0.0.2/8
Jun 3 11:51:14 srv-dell ifup:
Jun 3 11:51:15 srv-dell ifup: eth0 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 11:51:15 srv-dell ifup: No configuration found for eth0
Jun 3 11:51:15 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 11:51:15 srv-dell ifup-dhcp: eth1 Starting DHCP4 client
Jun 3 11:51:15 srv-dell dhcpcd[3634]: eth1: dhcpcd 3.2.3 starting
Jun 3 11:51:15 srv-dell dhcpcd[3634]: eth1: hardware address =
00:19:b9:d7:20:d7
Jun 3 11:51:15 srv-dell dhcpcd[3634]: eth1: broadcasting for a lease
Jun 3 11:51:16 srv-dell ifup-dhcp: .
Jun 3 11:51:18 srv-dell ifup-dhcp: .
Jun 3 11:51:19 srv-dell kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.182:2):
operation="profile_load" name="/bin/ping" name2="default" pid=2885
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.212:3):
operation="profile_load" name="/sbin/klogd" name2="default" pid=2900
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.269:4):
operation="profile_load" name="/sbin/syslog-ng" name2="default"
pid=2917
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.326:5):
operation="profile_load" name="/sbin/syslogd" name2="default" pid=2934
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.386:6):
operation="profile_load" name="/usr/sbin/avahi-daemon" name2="default"
pid=2949
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.438:7):
operation="profile_load" name="/usr/sbin/identd" name2="default"
pid=2962
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.491:8):
operation="profile_load" name="/usr/sbin/mdnsd" name2="default"
pid=2963
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.578:9):
operation="profile_load" name="/usr/sbin/nscd" name2="default"
pid=2964
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.694:10):
operation="profile_load" name="/usr/sbin/ntpd" name2="default"
pid=2965
Jun 3 11:51:19 srv-dell kernel: type=1505 audit(1244044272.741:11):
operation="profile_load" name="/usr/sbin/traceroute" name2="default"
pid=2966
Jun 3 11:51:19 srv-dell kernel: IA-32 Microcode Update Driver:
v1.14a-xen <tigran at aivazian.fsnet.co.uk>
Jun 3 11:51:19 srv-dell kernel: firmware: requesting intel-ucode/06-0f-07
Jun 3 11:51:19 srv-dell kernel: bnx2: eth1: using MSI
Jun 3 11:51:19 srv-dell kernel: NET: Registered protocol family 17
Jun 3 11:51:19 srv-dell kernel: bnx2: eth1 NIC Copper Link is Up,
1000 Mbps full duplex, receive & transmit flow control ON
Jun 3 11:51:20 srv-dell ifup-dhcp: .
Jun 3 11:51:21 srv-dell dhcpcd[3634]: eth1: offered 192.168.2.220
from 192.168.2.1
Jun 3 11:51:21 srv-dell dhcpcd[3634]: eth1: checking 192.168.2.220 is
available on attached networks
Jun 3 11:51:22 srv-dell ifup-dhcp: .
Jun 3 11:51:22 srv-dell dhcpcd[3634]: eth1: leased 192.168.2.220 for
86400 seconds
Jun 3 11:51:22 srv-dell dhcpcd[3634]: eth1: no renewal time supplied,
assuming 43200 seconds
Jun 3 11:51:22 srv-dell dhcpcd[3634]: eth1: no rebind time supplied,
assuming 75600 seconds
Jun 3 11:51:22 srv-dell dhcpcd[3634]: eth1: adding IP address 192.168.2.220/24
Jun 3 11:51:22 srv-dell dhcpcd[3634]: eth1: adding default route via
192.168.2.1 metric 0
Jun 3 11:51:22 srv-dell dns-resolver: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 11:51:22 srv-dell dns-resolver: You can find my version in
/etc/resolv.conf.netconfig
Jun 3 11:51:23 srv-dell dhcpcd-hook: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 11:51:23 srv-dell dhcpcd-hook: You can find my version in
/etc/resolv.conf.netconfig ...
Jun 3 11:51:23 srv-dell syslog-ng[3118]: Configuration reload request
received, reloading configuration;
Jun 3 11:51:23 srv-dell syslog-ng[3118]: New configuration initialized;
Jun 3 11:51:23 srv-dell ifdown: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 11:51:23 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 11:51:23 srv-dell ifup-dhcp:
Jun 3 11:51:23 srv-dell ifup-dhcp: eth1 IP address: 192.168.2.220/24
Jun 3 11:51:24 srv-dell SuSEfirewall2: SuSEfirewall2 not active
Jun 3 11:51:24 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 3 11:51:24 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 11:51:24 srv-dell kernel: iscsi: registered transport (tcp)
Jun 3 11:51:24 srv-dell auditd[4189]: Started dispatcher:
/sbin/audispd pid: 4191
Jun 3 11:51:24 srv-dell audispd: priority_boost_parser called with: 4
Jun 3 11:51:24 srv-dell audispd: af_unix plugin initialized
Jun 3 11:51:24 srv-dell audispd: audispd initialized with q_depth=80
and 1 active plugins
Jun 3 11:51:24 srv-dell kernel: NET: Registered protocol family 10
Jun 3 11:51:24 srv-dell kernel: lo: Disabled Privacy Extensions
Jun 3 11:51:24 srv-dell auditd[4189]: Init complete, auditd 1.7.7
listening for events (startup state disable)
Jun 3 11:51:24 srv-dell kernel: iscsi: registered transport (iser)
Jun 3 11:51:24 srv-dell iscsid: iSCSI logger with pid=4319 started!
Jun 3 11:51:24 srv-dell dhcpcd[3634]: eth1: setting hostname to `srv-dell'
Jun 3 11:51:24 srv-dell dhcpcd[3634]: eth1: exiting
Jun 3 11:51:25 srv-dell smartd[4693]: smartd 5.39 2008-10-24 22:33
[x86_64-suse-linux-gnu] (openSUSE RPM) Copyright (C) 2002-8 by Bruce
Allen, http://smartmontools.sourceforge.net
Jun 3 11:51:25 srv-dell smartd[4693]: Opened configuration file
/etc/smartd.conf
Jun 3 11:51:25 srv-dell smartd[4693]: Drive: DEVICESCAN, implied '-a'
Directive on line 26 of file /etc/smartd.conf
Jun 3 11:51:25 srv-dell smartd[4693]: Configuration file
/etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Jun 3 11:51:25 srv-dell smartd[4693]: Device: /dev/sda, opened
Jun 3 11:51:25 srv-dell smartd[4693]: Device: /dev/sda, IE (SMART)
not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on
SMART features
Jun 3 11:51:25 srv-dell smartd[4693]: Unable to monitor any SMART
enabled devices. Try debug (-d) option. Exiting...
Jun 3 11:51:25 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 3 11:51:25 srv-dell iscsid: iSCSI daemon with pid=4320 started!
Jun 3 11:51:25 srv-dell sshd[4950]: Server listening on 0.0.0.0 port 22.
Jun 3 11:51:25 srv-dell sshd[4950]: Server listening on :: port 22.
Jun 3 11:51:25 srv-dell xenstored: Checking store ...
Jun 3 11:51:25 srv-dell xenstored: Checking store complete.
Jun 3 11:51:25 srv-dell kernel: suspend: event channel 55
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:795: blktapctrl: v1.0.0
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [raw image (aio)]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [raw image (sync)]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [vmware image (vmdk)]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [ramdisk image (ram)]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [qcow disk (qcow)]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [qcow2 disk (qcow2)]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [ioemu disk]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl.c:797: Found
driver: [raw image (cdrom)]
Jun 3 11:51:25 srv-dell BLKTAPCTRL[4963]: blktapctrl_linux.c:23:
/dev/xen/blktap0 device already exists
Jun 3 11:51:25 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/Xservers
Jun 3 11:51:25 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/xdm-config
Jun 3 11:51:26 srv-dell kernel: drbd: initialised. Version: 8.2.7
(api:88/proto:86-88)
Jun 3 11:51:26 srv-dell kernel: drbd: GIT-hash:
a1b440e8b3011a1318d8bff1bb7edc763ef995b0 build by lmb at hermes,
2009-02-20 13:35:59
Jun 3 11:51:26 srv-dell kernel: drbd: registered as block device major 147
Jun 3 11:51:26 srv-dell kernel: drbd: minor_table @ 0xffff8801eac6abc0
Jun 3 11:51:26 srv-dell kernel: Bridge firewalling registered
Jun 3 11:51:26 srv-dell kernel: drbd0: conn( StandAlone -> Unconnected )
Jun 3 11:51:26 srv-dell kernel: drbd0: Starting worker thread (from
cqueue [48])
Jun 3 11:51:26 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 11:51:26 srv-dell kernel: drbd0: Starting receiver thread (from
drbd0_worker [5099])
Jun 3 11:51:26 srv-dell kernel: drbd0: receiver (re)started
Jun 3 11:51:26 srv-dell kernel: drbd0: conn( Unconnected -> WFConnection )
Jun 3 11:51:26 srv-dell /usr/sbin/cron[5136]: (CRON) STARTUP (V5.0)
Jun 3 11:51:26 srv-dell kernel: vendor=8086 device=244e
Jun 3 11:51:26 srv-dell kernel: pci 0000:10:0d.0: PCI INT A -> GSI 19
(level, low) -> IRQ 19
Jun 3 11:51:33 srv-dell gdm-simple-greeter[5245]: libglade-WARNING:
Unexpected element <requires-version> inside <glade-interface>.
Jun 3 11:51:35 srv-dell kernel: eth1: no IPv6 routers present
Jun 3 11:52:44 srv-dell python: hp-systray(init)[5420]: error:
hp-systray cannot be run as root. Exiting.
Jun 3 11:53:27 srv-dell kernel: drbd0: conn( WFConnection -> Disconnecting )
Jun 3 11:53:27 srv-dell kernel: drbd0: Discarding network configuration.
Jun 3 11:53:27 srv-dell kernel: drbd0: Connection closed
Jun 3 11:53:27 srv-dell kernel: drbd0: conn( Disconnecting -> StandAlone )
Jun 3 11:53:27 srv-dell kernel: drbd0: receiver terminated
Jun 3 11:53:27 srv-dell kernel: drbd0: Terminating receiver thread
Jun 3 11:53:27 srv-dell kernel: drbd0: drbd_bm_resize called with capacity == 0
Jun 3 11:53:27 srv-dell kernel: drbd0: worker terminated
Jun 3 11:53:27 srv-dell kernel: drbd0: Terminating worker thread
Jun 3 11:53:27 srv-dell kernel: drbd: module cleanup done.
Jun 3 11:53:40 srv-dell kernel: st: Version 20080504, fixed bufsize
32768, s/g segs 256
Jun 3 11:53:58 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 11:53:58 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 11:53:58 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 11:53:58 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 11:53:59 srv-dell kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jun 3 11:54:21 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 11:54:21 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 11:54:21 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 11:54:21 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 11:54:27 srv-dell kernel: No iBFT detected.
Jun 3 11:54:49 srv-dell kernel: scsi3 : iSCSI Initiator over TCP/IP
Jun 3 11:54:50 srv-dell kernel: scsi 3:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 11:54:50 srv-dell kernel: sdb: sdb1
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: [sdb] Attached SCSI disk
Jun 3 11:54:50 srv-dell kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Jun 3 11:54:50 srv-dell iscsid: connection1:0 is operational now
Jun 3 11:54:50 srv-dell kernel: kjournald starting. Commit interval 5 seconds
Jun 3 11:54:50 srv-dell kernel: EXT3 FS on sdb1, internal journal
Jun 3 11:54:50 srv-dell kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Jun 3 11:54:50 srv-dell hald: mounted /dev/sdb1 on behalf of uid 0
Jun 3 11:55:01 srv-dell hald: unmounted /dev/sdb1 from
'/media/disk-1' on behalf of uid 0
Jun 3 11:55:24 srv-dell openais[6510]: [MAIN ] AIS Executive Service
RELEASE 'subrev 1152 version 0.80'
Jun 3 11:55:24 srv-dell openais[6510]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Jun 3 11:55:24 srv-dell openais[6510]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Jun 3 11:55:24 srv-dell openais[6510]: [MAIN ] AIS Executive Service:
started and ready to provide service.
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] Token Timeout (5000
ms) retransmit timeout (490 ms)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] token hold (382 ms)
retransmits before loss (10 retrans)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] join (1000 ms)
send_join (45 ms) consensus (2500 ms) merge (200 ms)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] seqno unchanged const
(30 rotations) Maximum network MTU 1500
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] send threads (0 threads)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] RRP token expired
timeout (490 ms)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] RRP token problem
counter (2000 ms)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] RRP threshold (10 problem count)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] RRP mode set to none.
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] heartbeat_failures_allowed (0)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] max_network_delay (50 ms)
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] HeartBeat is Disabled.
To enable set heartbeat_failures_allowed > 0
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] The network interface
[192.168.2.220] is now up.
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] Created or loaded
sequence id 192.192.168.2.220 for this ring.
Jun 3 11:55:24 srv-dell openais[6510]: [TOTEM] entering GATHER state from 15.
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info:
process_ais_conf: Reading configure
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info:
config_find_next: Processing additional logging options...
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: get_config_opt:
Found 'off' for option: debug
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: get_config_opt:
Found 'yes' for option: to_syslog
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: get_config_opt:
Found 'daemon' for option: syslog_facility
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info:
config_find_next: Processing additional service options...
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_logd
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_mgmtd
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info:
pcmk_plugin_init: CRM: Initialized
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] Logging: Initialized
pcmk_plugin_init
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info:
pcmk_plugin_init: Service: 9
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info:
pcmk_plugin_init: Local node id: 1543678144
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info:
pcmk_plugin_init: Local hostname: srv-dell
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: update_member:
Creating entry for node 1543678144 born on 0
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: update_member:
0x7fca4c0341d0 Node 1543678144 now known as srv-dell (was: (null))
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: update_member:
Node srv-dell now has 1 quorum votes (was 0)
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: update_member:
Node 1543678144/srv-dell is now: member
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: spawn_child:
Forked child 6516 for process stonithd
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: spawn_child:
Forked child 6517 for process cib
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: spawn_child:
Forked child 6518 for process lrmd
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: spawn_child:
Forked child 6519 for process attrd
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: spawn_child:
Forked child 6520 for process pengine
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: spawn_child:
Forked child 6521 for process crmd
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: spawn_child:
Forked child 6522 for process mgmtd
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info: pcmk_startup:
CRM: Initialized
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] Service initialized
'Pacemaker Cluster Manager'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais extended virtual synchrony service'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais cluster membership service B.01.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais availability management framework B.01.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais checkpoint service B.01.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais event service B.01.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais distributed locking service B.01.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais message service B.01.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais configuration service'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais cluster closed process group service v1.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SERV ] Service initialized
'openais cluster config database access v1.01'
Jun 3 11:55:25 srv-dell openais[6510]: [SYNC ] Not using a virtual
synchrony filter.
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] Saving state aru 0
high seq received 0
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] Storing new sequence
id for ring c4
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] entering COMMIT state.
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] entering RECOVERY state.
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] previous ring seq 192
rep 192.168.2.220
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] aru 0 high delivered 0
received flag 1
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] Sending initial ORF token
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] New Configuration:
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] Members Left:
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] Members Joined:
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 196: memb=0,
new=0, lost=0
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] New Configuration:
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] Members Left:
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] Members Joined:
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 196: memb=1, new=1,
lost=0
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info:
pcmk_peer_update: NEW: srv-dell 1543678144
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 11:55:25 srv-dell openais[6510]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000053312
(340754)
Jun 3 11:55:25 srv-dell openais[6510]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 11:55:25 srv-dell openais[6510]: [TOTEM] entering OPERATIONAL state.
Jun 3 11:55:25 srv-dell openais[6510]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 11:55:25 srv-dell lrmd: [6518]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 11:55:25 srv-dell lrmd: [6518]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Jun 3 11:55:25 srv-dell mgmtd: [6522]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 11:55:25 srv-dell mgmtd: [6522]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 3 11:55:25 srv-dell mgmtd: [6522]: debug: Enabling coredumps
Jun 3 11:55:25 srv-dell mgmtd: [6522]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 11:55:25 srv-dell mgmtd: [6522]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 11:55:25 srv-dell attrd: [6519]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 11:55:25 srv-dell mgmtd: [6522]: WARN: lrm_signon: can not
initiate connection
Jun 3 11:55:25 srv-dell mgmtd: [6522]: info: login to lrm: 0, ret:0
Jun 3 11:55:25 srv-dell pengine: [6520]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 11:55:25 srv-dell cib: [6517]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 11:55:25 srv-dell attrd: [6519]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 11:55:25 srv-dell attrd: [6519]: info: main: Starting up....
Jun 3 11:55:25 srv-dell stonithd: [6516]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 11:55:25 srv-dell stonithd: [6516]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 11:55:25 srv-dell cib: [6517]: WARN: Initializing connection to
logging daemon failed. Logging daemon may not be running
Jun 3 11:55:25 srv-dell crmd: [6521]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 11:55:25 srv-dell crmd: [6521]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 11:55:25 srv-dell pengine: [6520]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 11:55:25 srv-dell stonithd: [6516]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 11:55:25 srv-dell cib: [6517]: info: G_main_add_TriggerHandler:
Added signal manual handler
Jun 3 11:55:25 srv-dell attrd: [6519]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 11:55:25 srv-dell crmd: [6521]: info: main: CRM Hg Version:
0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Jun 3 11:55:25 srv-dell cib: [6517]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 11:55:25 srv-dell attrd: [6519]: info: init_ais_connection: AIS
connection established
Jun 3 11:55:25 srv-dell lrmd: [6518]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 11:55:25 srv-dell crmd: [6521]: info: crmd_init: Starting crmd
Jun 3 11:55:25 srv-dell attrd: [6519]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 11:55:25 srv-dell lrmd: [6518]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jun 3 11:55:25 srv-dell crmd: [6521]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 11:55:25 srv-dell pengine: [6520]: info: main: Starting pengine
Jun 3 11:55:25 srv-dell attrd: [6519]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 11:55:25 srv-dell lrmd: [6518]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jun 3 11:55:25 srv-dell attrd: [6519]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 11:55:25 srv-dell cib: [6517]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Jun 3 11:55:25 srv-dell lrmd: [6518]: info: Started.
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info: pcmk_ipc:
Recorded connection 0x7fca4c0344b0 for attrd/6519
Jun 3 11:55:25 srv-dell stonithd: [6516]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 11:55:25 srv-dell stonithd: [6516]: info: init_ais_connection:
AIS connection established
Jun 3 11:55:25 srv-dell stonithd: [6516]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 11:55:25 srv-dell stonithd: [6516]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 11:55:25 srv-dell stonithd: [6516]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info: pcmk_ipc:
Recorded connection 0x7fca4c036ae0 for stonithd/6516
Jun 3 11:55:25 srv-dell stonithd: [6516]: notice:
/usr/lib64/heartbeat/stonithd start up successfully.
Jun 3 11:55:25 srv-dell stonithd: [6516]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 11:55:25 srv-dell cib: [6517]: info: startCib: CIB
Initialization completed successfully
Jun 3 11:55:25 srv-dell cib: [6517]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 11:55:25 srv-dell cib: [6517]: info: init_ais_connection: AIS
connection established
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info: pcmk_ipc:
Recorded connection 0x7fca4c036eb0 for cib/6517
Jun 3 11:55:25 srv-dell openais[6510]: [crm ] info: pcmk_ipc:
Sending membership update 196 to cib
Jun 3 11:55:25 srv-dell cib: [6517]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 11:55:25 srv-dell cib: [6517]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 11:55:25 srv-dell cib: [6517]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 11:55:25 srv-dell cib: [6517]: info: cib_init: Starting cib mainloop
Jun 3 11:55:25 srv-dell cib: [6517]: info: ais_dispatch: Membership
196: quorum still lost
Jun 3 11:55:25 srv-dell cib: [6517]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=196
proc=00000000000000000000000000053312 (new)
Jun 3 11:55:25 srv-dell cib: [6526]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-25.raw
Jun 3 11:55:25 srv-dell cib: [6526]: info: write_cib_contents: Wrote
version 0.24.0 of the CIB to disk (digest:
93c10646988b02fb7bf4b1b3aa8897f9)
Jun 3 11:55:25 srv-dell cib: [6526]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ts8dBT (digest:
/var/lib/heartbeat/crm/cib.VmGLPh)
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] entering GATHER state from 11.
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] Saving state aru a
high seq received a
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] Storing new sequence
id for ring c8
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] entering COMMIT state.
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] entering RECOVERY state.
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] previous ring seq 196
rep 192.168.2.120
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] aru 25 high delivered
25 received flag 1
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] previous ring seq 196
rep 192.168.2.220
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] aru a high delivered a
received flag 1
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] New Configuration:
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] Members Left:
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] Members Joined:
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 200: memb=1,
new=0, lost=0
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] New Configuration:
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] Members Left:
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] Members Joined:
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 11:55:26 srv-dell cib: [6517]: info: ais_dispatch: Membership
200: quorum still lost
Jun 3 11:55:26 srv-dell cib: [6517]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 11:55:26 srv-dell cib: [6517]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=200 proc=00000000000000000000000000000000
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 200: memb=2, new=1,
lost=0
Jun 3 11:55:26 srv-dell openais[6510]: [MAIN ] info: update_member:
Creating entry for node 2013440192 born on 200
Jun 3 11:55:26 srv-dell openais[6510]: [MAIN ] info: update_member:
Node 2013440192/unknown is now: member
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info:
pcmk_peer_update: NEW: .pending. 2013440192
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info:
pcmk_peer_update: MEMB: .pending. 2013440192
Jun 3 11:55:26 srv-dell cib: [6517]: info: ais_dispatch: Membership
200: quorum still lost
Jun 3 11:55:26 srv-dell cib: [6517]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 11:55:26 srv-dell cib: [6517]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=192 seen=200 proc=00000000000000000000000000053312
(new)
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info:
send_member_notification: Sending membership update 200 to 1 children
Jun 3 11:55:26 srv-dell openais[6510]: [MAIN ] info: update_member:
0x7fca4c0341d0 Node 1543678144 ((null)) born on: 200
Jun 3 11:55:26 srv-dell openais[6510]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 11:55:26 srv-dell openais[6510]: [TOTEM] entering OPERATIONAL state.
Jun 3 11:55:26 srv-dell openais[6510]: [MAIN ] info: update_member:
0x7fca4c036c50 Node 2013440192 (srv-lenovo) born on: 192
Jun 3 11:55:26 srv-dell openais[6510]: [MAIN ] info: update_member:
0x7fca4c036c50 Node 2013440192 now known as srv-lenovo (was: (null))
Jun 3 11:55:26 srv-dell openais[6510]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 3 11:55:26 srv-dell openais[6510]: [MAIN ] info: update_member:
Node srv-lenovo now has 1 quorum votes (was 0)
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info:
send_member_notification: Sending membership update 200 to 1 children
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 11:55:26 srv-dell openais[6510]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 11:55:26 srv-dell cib: [6517]: info: cib_process_diff: Diff
0.26.1 -> 0.26.2 not applied to 0.24.0: current "epoch" is less than
required
Jun 3 11:55:26 srv-dell cib: [6517]: info: cib_server_process_diff:
Requesting re-sync from peer
Jun 3 11:55:26 srv-dell cib: [6517]: WARN: cib_diff_notify:
Local-only Change (client:crmd, call: 38): 0.0.0 (Application of an
update diff failed, requesting a full refresh)
Jun 3 11:55:26 srv-dell cib: [6517]: WARN: cib_server_process_diff:
Not applying diff 0.26.2 -> 0.26.3 (sync in progress)
Jun 3 11:55:26 srv-dell cib: [6517]: WARN: cib_server_process_diff:
Not applying diff 0.26.3 -> 0.26.4 (sync in progress)
Jun 3 11:55:26 srv-dell cib: [6517]: WARN: cib_server_process_diff:
Not applying diff 0.26.4 -> 0.27.1 (sync in progress)
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 11:55:26 srv-dell cib: [6517]: info: cib_replace_notify:
Replaced: 0.24.0 -> 0.27.1 from srv-lenovo
Jun 3 11:55:26 srv-dell cib: [6527]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-26.raw
Jun 3 11:55:26 srv-dell cib: [6527]: info: write_cib_contents: Wrote
version 0.27.0 of the CIB to disk (digest:
4aabe4d9a3ace01b621468016f01ae8c)
Jun 3 11:55:26 srv-dell cib: [6527]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Tih5FJ (digest:
/var/lib/heartbeat/crm/cib.ChwtJ9)
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 11:55:26 srv-dell mgmtd: [6522]: info: init_crm
Jun 3 11:55:26 srv-dell crmd: [6521]: info: do_cib_control: CIB
connection established
Jun 3 11:55:26 srv-dell crmd: [6521]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 11:55:26 srv-dell crmd: [6521]: info: init_ais_connection: AIS
connection established
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info: pcmk_ipc:
Recorded connection 0x748850 for crmd/6521
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info: pcmk_ipc:
Sending membership update 200 to crmd
Jun 3 11:55:26 srv-dell crmd: [6521]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 11:55:26 srv-dell crmd: [6521]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 11:55:26 srv-dell crmd: [6521]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 11:55:26 srv-dell crmd: [6521]: info: do_ha_control: Connected
to the cluster
Jun 3 11:55:26 srv-dell crmd: [6521]: info: do_started: Delaying
start, CCM (0000000000100000) not connected
Jun 3 11:55:26 srv-dell crmd: [6521]: info: crmd_init: Starting crmd's mainloop
Jun 3 11:55:26 srv-dell crmd: [6521]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 11:55:26 srv-dell openais[6510]: [crm ] info:
update_expected_votes: Expected quorum votes 1024 -> 2
Jun 3 11:55:26 srv-dell crmd: [6521]: info: ais_dispatch: Membership
200: quorum still lost
Jun 3 11:55:26 srv-dell crmd: [6521]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 11:55:26 srv-dell crmd: [6521]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 11:55:26 srv-dell crmd: [6521]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=200
proc=00000000000000000000000000053312
Jun 3 11:55:26 srv-dell crmd: [6521]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=200 seen=200
proc=00000000000000000000000000053312 (new)
Jun 3 11:55:26 srv-dell crmd: [6521]: info: do_started: The local CRM
is operational
Jun 3 11:55:26 srv-dell crmd: [6521]: info: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Jun 3 11:55:26 srv-dell mgmtd: [6522]: debug: main: run the loop...
Jun 3 11:55:26 srv-dell mgmtd: [6522]: info: Started.
Jun 3 11:55:27 srv-dell crmd: [6521]: notice: ais_dispatch:
Membership 200: quorum aquired
Jun 3 11:55:27 srv-dell crmd: [6521]: info: update_dc: Set DC to
srv-lenovo (3.0.1)
Jun 3 11:55:27 srv-dell crmd: [6521]: info: update_attrd: Connecting
to attrd...
Jun 3 11:55:27 srv-dell crmd: [6521]: info: update_attrd: Updating
terminate=<none> via attrd for srv-dell
Jun 3 11:55:27 srv-dell crmd: [6521]: info: update_attrd: Updating
shutdown=<none> via attrd for srv-dell
Jun 3 11:55:27 srv-dell crmd: [6521]: info: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 3 11:55:27 srv-dell crmd: [6521]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/transient_attributes": ok
(rc=0)
Jun 3 11:55:27 srv-dell crmd: [6521]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 11:55:27 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=4:3:7:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 11:55:27 srv-dell lrmd: [6518]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 11:55:27 srv-dell lrmd: [6518]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 11:55:27 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=5:3:7:7c754d7f-6d3c-4527-b36c-979453669faf
op=cloneresource_cluster_fencing_null:0_monitor_0 )
Jun 3 11:55:27 srv-dell lrmd: [6518]: info:
rsc:cloneresource_cluster_fencing_null:0: monitor
Jun 3 11:55:27 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_monitor_0 (call=3,
rc=7, cib-update=8, confirmed=true) complete not running
Jun 3 11:55:27 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=2, rc=7,
cib-update=9, confirmed=true) complete not running
Jun 3 11:55:27 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=7:3:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_start_0 )
Jun 3 11:55:27 srv-dell lrmd: [6518]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 11:55:27 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=9:3:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 11:55:27 srv-dell lrmd: [6518]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 11:55:27 srv-dell lrmd: [6551]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 11:55:27 srv-dell stonithd: [6516]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 11:55:27 srv-dell lrmd: [6518]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 11:55:27 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=5, rc=0,
cib-update=13, confirmed=true) complete ok
Jun 3 11:55:27 srv-dell lrmd: [6518]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 11:55:27 srv-dell lrmd: [6518]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_11:55:27 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 11:55:27 srv-dell lrmd: [6518]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 11:55:27 srv-dell lrmd: [6518]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_11:55:27 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 11:55:27 srv-dell lrmd: [6518]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 11:55:27 srv-dell lrmd: [6518]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_11:55:27 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 11:55:27 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=4, rc=0,
cib-update=14, confirmed=true) complete ok
Jun 3 11:55:27 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=8:3:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 11:55:28 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=6, rc=0,
cib-update=15, confirmed=false) complete ok
Jun 3 11:55:35 srv-dell attrd: [6519]: info: main: Sending full refresh
Jun 3 11:55:35 srv-dell attrd: [6519]: info: main: Starting mainloop...
Jun 3 11:55:35 srv-dell attrd: [6519]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 11:55:35 srv-dell attrd: [6519]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 11:55:35 srv-dell attrd: [6519]: info: find_hash_entry:
Creating hash entry for terminate
Jun 3 11:55:35 srv-dell attrd: [6519]: info: find_hash_entry:
Creating hash entry for shutdown
Jun 3 11:55:35 srv-dell attrd: [6519]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 11:55:35 srv-dell attrd: [6519]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 11:55:35 srv-dell attrd: [6519]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 11:57:47 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 11:58:06 srv-dell crm_shadow: [7320]: info: Invoked: crm_shadow
Jun 3 11:58:06 srv-dell crm_shadow: [7334]: info: Invoked: crm_shadow
Jun 3 11:58:06 srv-dell crm_shadow: [7336]: info: Invoked: crm_shadow
Jun 3 11:58:09 srv-dell cibadmin: [7337]: info: Invoked: cibadmin -Ql
Jun 3 11:58:54 srv-dell cibadmin: [7502]: info: Invoked: cibadmin -Ql
Jun 3 11:58:54 srv-dell cibadmin: [7503]: info: Invoked: cibadmin -p -U
Jun 3 11:58:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 11:58:54 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=5:4:7:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:1_monitor_0 )
Jun 3 11:58:54 srv-dell lrmd: [6518]: info: rsc:dlm:1: monitor
Jun 3 11:58:54 srv-dell cib: [7504]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-27.raw
Jun 3 11:58:54 srv-dell cib: [7504]: info: write_cib_contents: Wrote
version 0.28.0 of the CIB to disk (digest:
6e533e684f1bcfa20ebc83cf0ae06489)
Jun 3 11:58:54 srv-dell cib: [7504]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.uXzYlb (digest:
/var/lib/heartbeat/crm/cib.288ExU)
Jun 3 11:58:54 srv-dell cibadmin: [7506]: info: Invoked: cibadmin -Ql
Jun 3 11:58:54 srv-dell lrmd: [6518]: info: RA output:
(dlm:1:monitor:stderr) dlm_controld.pcmk: no process killed
Jun 3 11:58:54 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation dlm:1_monitor_0 (call=7, rc=7, cib-update=16,
confirmed=true) complete not running
Jun 3 11:58:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 11:58:54 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 11:58:54 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=20:4:0:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:1_start_0 )
Jun 3 11:58:54 srv-dell lrmd: [6518]: info: rsc:dlm:1: start
Jun 3 11:58:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 11:58:54 srv-dell lrmd: [6518]: info: RA output:
(dlm:1:start:stderr) dlm_controld.pcmk: no process killed
Jun 3 11:58:54 srv-dell kernel: DLM (built Feb 28 2009 07:25:25) installed
Jun 3 11:58:55 srv-dell cluster-dlm[7558]: main: dlm_controld 1239267206
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info:
init_ais_connection: AIS connection established
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 11:58:55 srv-dell openais[6510]: [crm ] info: pcmk_notify:
Enabling node notifications for child 7558 (0x7fca4c01d0a0)
Jun 3 11:58:55 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 3 11:58:55 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 3 11:58:55 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 3 11:58:55 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 3 11:58:55 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 3 11:58:55 srv-dell cluster-dlm: set_protocol: set_protocol
member_count 1 propose daemon 1.1.1 kernel 1.1.1
Jun 3 11:58:55 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 1543678144
Jun 3 11:58:55 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 3 11:58:55 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 3 11:58:55 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: notice: ais_dispatch:
Membership 200: quorum aquired
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=200
proc=00000000000000000000000000053312
Jun 3 11:58:55 srv-dell cluster-dlm: [7558]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=200 seen=200
proc=00000000000000000000000000053312 (new)
Jun 3 11:58:55 srv-dell cluster-dlm: update_cluster: Processing membership 200
Jun 3 11:58:55 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 3 11:58:55 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 3 11:58:55 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=192, last-seen=200, this-event=200,
last-event=0
Jun 3 11:58:55 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 3 11:58:55 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 3 11:58:55 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=200, last-seen=200, this-event=200,
last-event=0
Jun 3 11:58:55 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 11:58:56 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation dlm:1_start_0 (call=8, rc=0, cib-update=19, confirmed=true)
complete ok
Jun 3 11:58:56 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=21:4:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=dlm:1_monitor_120000 )
Jun 3 11:58:56 srv-dell haclient: on_event:evt:cib_changed
Jun 3 11:58:56 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation dlm:1_monitor_120000 (call=9, rc=0, cib-update=20,
confirmed=false) complete ok
Jun 3 11:58:56 srv-dell haclient: on_event:evt:cib_changed
Jun 3 11:58:56 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 11:58:56 srv-dell haclient: on_event:evt:cib_changed
Jun 3 11:58:56 srv-dell haclient: on_event:evt:cib_changed
Jun 3 11:58:56 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 11:59:48 srv-dell crm_shadow: [7792]: info: Invoked: crm_shadow
Jun 3 11:59:48 srv-dell crm_shadow: [7806]: info: Invoked: crm_shadow
Jun 3 11:59:48 srv-dell crm_shadow: [7808]: info: Invoked: crm_shadow
Jun 3 11:59:53 srv-dell cibadmin: [7827]: info: Invoked: cibadmin -Ql
Jun 3 12:01:22 srv-dell cibadmin: [8183]: info: Invoked: cibadmin -Ql
Jun 3 12:01:22 srv-dell cibadmin: [8184]: info: Invoked: cibadmin -p -U
Jun 3 12:01:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:01:22 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=7:5:7:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:1_monitor_0 )
Jun 3 12:01:22 srv-dell lrmd: [6518]: info: rsc:o2cb:1: monitor
Jun 3 12:01:22 srv-dell cibadmin: [8187]: info: Invoked: cibadmin -Ql
Jun 3 12:01:22 srv-dell cib: [8185]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-28.raw
Jun 3 12:01:22 srv-dell lrmd: [6518]: info: RA output:
(o2cb:1:monitor:stderr) logd is not running
Jun 3 12:01:22 srv-dell lrmd: [6518]: info: RA output:
(o2cb:1:monitor:stderr) 2009/06/03_12:01:22 INFO: Stack glue driver
not loaded
Jun 3 12:01:22 srv-dell cib: [8185]: info: write_cib_contents: Wrote
version 0.29.0 of the CIB to disk (digest:
fe3a164e42049da477a6e0cdabae7337)
Jun 3 12:01:22 srv-dell cib: [8185]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.LQrGfN (digest:
/var/lib/heartbeat/crm/cib.2lIDyJ)
Jun 3 12:01:22 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation o2cb:1_monitor_0 (call=10, rc=7, cib-update=21,
confirmed=true) complete not running
Jun 3 12:01:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:01:22 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:01:23 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=30:5:0:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:1_start_0 )
Jun 3 12:01:23 srv-dell lrmd: [6518]: info: rsc:o2cb:1: start
Jun 3 12:01:23 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:01:23 srv-dell lrmd: [6518]: info: RA output:
(o2cb:1:start:stderr) logd is not running
Jun 3 12:01:23 srv-dell lrmd: [6518]: info: RA output:
(o2cb:1:start:stderr) 2009/06/03_12:01:23 INFO: Stack glue driver not
loaded
Jun 3 12:01:23 srv-dell lrmd: [6518]: info: RA output:
(o2cb:1:start:stderr) logd is not running
Jun 3 12:01:23 srv-dell lrmd: [6518]: info: RA output:
(o2cb:1:start:stderr) 2009/06/03_12:01:23 INFO: Starting o2cb:1
Jun 3 12:01:23 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 3 12:01:23 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 3 12:01:23 srv-dell kernel: OCFS2 1.5.0
Jun 3 12:01:23 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info:
init_ais_connection: AIS connection established
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 12:01:23 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 3 12:01:23 srv-dell openais[6510]: [crm ] info: pcmk_notify:
Enabling node notifications for child 8317 (0x7fca4c0379c0)
Jun 3 12:01:23 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 3 12:01:23 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 3 12:01:23 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:01:23 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:01:23 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 3 12:01:23 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:01:23 srv-dell ocfs2_controld: setup done
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: notice: ais_dispatch:
Membership 200: quorum aquired
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=192 seen=200
proc=00000000000000000000000000053312
Jun 3 12:01:23 srv-dell ocfs2_controld: [8317]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=200 seen=200
proc=00000000000000000000000000053312 (new)
Jun 3 12:01:23 srv-dell ocfs2_controld: confchg called
Jun 3 12:01:23 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 3 12:01:23 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 3 12:01:23 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 3 12:01:23 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Creating section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Created section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 3 12:01:23 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Creating section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:01:23 srv-dell ocfs2_controld: Created section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 3 12:01:23 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 3 12:01:23 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 3 12:01:23 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 3 12:01:23 srv-dell ocfs2_controld: Opening control device
Jun 3 12:01:23 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 3 12:01:23 srv-dell ocfs2_controld: new listening connection 4
Jun 3 12:01:23 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 3 12:01:23 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:01:23 srv-dell ocfs2_controld: confchg called
Jun 3 12:01:23 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 3 12:01:25 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation o2cb:1_start_0 (call=11, rc=0, cib-update=24,
confirmed=true) complete ok
Jun 3 12:01:25 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=31:5:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=o2cb:1_monitor_120000 )
Jun 3 12:01:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:01:25 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation o2cb:1_monitor_120000 (call=12, rc=0, cib-update=25,
confirmed=false) complete ok
Jun 3 12:01:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:01:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:01:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:01:25 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:04:03 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:04:03 srv-dell ocfs2_controld: client msg
Jun 3 12:04:03 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 3 12:04:03 srv-dell ocfs2_controld: client msg
Jun 3 12:04:03 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 12:04:03 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 12:05:25 srv-dell cib: [6517]: info: cib_stats: Processed 88
operations (454.00us average, 0% utilization) in the last 10min
Jun 3 12:06:06 srv-dell crm_shadow: [9554]: info: Invoked: crm_shadow
Jun 3 12:06:06 srv-dell crm_shadow: [9568]: info: Invoked: crm_shadow
Jun 3 12:06:06 srv-dell crm_shadow: [9570]: info: Invoked: crm_shadow
Jun 3 12:06:11 srv-dell cibadmin: [9571]: info: Invoked: cibadmin -Ql
Jun 3 12:09:30 srv-dell cibadmin: [10375]: info: Invoked: cibadmin -Ql
Jun 3 12:09:30 srv-dell cibadmin: [10376]: info: Invoked: cibadmin -p -U
Jun 3 12:09:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:09:30 srv-dell cib: [10377]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-29.raw
Jun 3 12:09:30 srv-dell cib: [10377]: info: write_cib_contents: Wrote
version 0.30.0 of the CIB to disk (digest:
f630eea2d323036c8744313335ac2a1f)
Jun 3 12:09:30 srv-dell cib: [10377]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.1teN02 (digest:
/var/lib/heartbeat/crm/cib.654Zc8)
Jun 3 12:09:30 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=9:6:7:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:1_monitor_0 )
Jun 3 12:09:30 srv-dell lrmd: [6518]: info: rsc:fs:1: monitor
Jun 3 12:09:30 srv-dell cibadmin: [10378]: info: Invoked: cibadmin -Ql
Jun 3 12:09:30 srv-dell crmd: [6521]: info: process_lrm_event: LRM
operation fs:1_monitor_0 (call=13, rc=7, cib-update=26,
confirmed=true) complete not running
Jun 3 12:09:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:09:31 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:09:31 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:09:31 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:09:31 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=40:6:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:1_start_0 )
Jun 3 12:09:31 srv-dell lrmd: [6518]: info: rsc:fs:1: start
Jun 3 12:09:31 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:09:31 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:09:31 srv-dell lrmd: [6518]: info: RA output:
(fs:1:start:stderr) logd is not running
Jun 3 12:09:31 srv-dell lrmd: [6518]: info: RA output:
(fs:1:start:stderr) 2009/06/03_12:09:31 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 12:09:32 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:09:32 srv-dell ocfs2_controld: client msg
Jun 3 12:09:32 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 3 12:09:32 srv-dell ocfs2_controld: start_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", device "/dev/sdb1", service
"ocfs2"
Jun 3 12:09:32 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid 69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell ocfs2_controld: Starting join for group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:09:32 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:09:32 srv-dell ocfs2_controld: start_mount returns 0
Jun 3 12:09:32 srv-dell ocfs2_controld: confchg called
Jun 3 12:09:32 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 2, left 0,
joined 1
Jun 3 12:09:32 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell ocfs2_controld: This node joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell ocfs2_controld: Filling node 2013440192 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell ocfs2_controld: Registering mountgroup
69AB6E551666484FB314D5B168FD6B84 with dlm_controld
Jun 3 12:09:32 srv-dell ocfs2_controld: Registering
"69AB6E551666484FB314D5B168FD6B84" with dlm_controld
Jun 3 12:09:32 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 12:09:32 srv-dell ocfs2_controld: Registration of
"69AB6E551666484FB314D5B168FD6B84" complete
Jun 3 12:09:32 srv-dell ocfs2_controld: Mountgroup
69AB6E551666484FB314D5B168FD6B84 successfully registered with
dlm_controld
Jun 3 12:09:32 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 12:09:32 srv-dell ocfs2_controld: Notified client: 1
Jun 3 12:09:32 srv-dell kernel: dlm: Using TCP for communications
Jun 3 12:09:32 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell cluster-dlm: process_uevent: kernel: add@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell cluster-dlm: process_uevent: kernel: online@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:09:32 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 3 12:09:32 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 3 12:09:32 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 2 joined 1 remove 0 failed 0
Jun 3 12:09:32 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 12:09:32 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 12:09:32 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 12:09:32 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 2 1 0 0
Jun 3 12:09:32 srv-dell cluster-dlm: receive_start: receive_start
2013440192:2 len 80
Jun 3 12:09:32 srv-dell kernel: dlm: got connection from 2013440192
Jun 3 12:09:32 srv-dell cluster-dlm: match_change: match_change
2013440192:2 matches cg 1
Jun 3 12:09:32 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 need 1 of 2
Jun 3 12:09:32 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 80
Jun 3 12:09:32 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 3 12:09:32 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 2
Jun 3 12:09:32 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 2
Jun 3 12:09:32 srv-dell cluster-dlm: do_sysfs: write "1438231596" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/id"
Jun 3 12:09:32 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 12:09:32 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/2013440192"
Jun 3 12:09:32 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 12:09:32 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 12:09:32 srv-dell cluster-dlm: set_plock_ckpt_node:
set_plock_ckpt_node from 0 to 2013440192
Jun 3 12:09:32 srv-dell cluster-dlm: receive_plocks_stored:
receive_plocks_stored 2013440192:2 need_plocks 1
Jun 3 12:09:32 srv-dell kernel: dlm: create_rcom to 2013440192 type 1
len 0 ENOBUFS
Jun 3 12:09:32 srv-dell cluster-dlm: match_change: match_change
2013440192:2 matches cg 1
Jun 3 12:09:32 srv-dell cluster-dlm: retrieve_plocks: retrieve_plocks
Jun 3 12:09:32 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:09:51 srv-dell lrmd: [6518]: WARN: fs:1:start process (PID
10414) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 12:09:51 srv-dell lrmd: [6518]: WARN: operation start[14] on
ocf::Filesystem::fs:1 for client 6521, its parameters:
directory=[/shared] fstype=[ocfs2] device=[/dev/sdb1]
CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [10414] timed out
Jun 3 12:09:51 srv-dell crmd: [6521]: ERROR: process_lrm_event: LRM
operation fs:1_start_0 (14) Timed Out (timeout=20000ms)
Jun 3 12:09:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:09:52 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:09:52 srv-dell crmd: [6521]: info: do_lrm_rsc_op: Performing
key=4:7:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:1_stop_0 )
Jun 3 12:09:52 srv-dell lrmd: [6518]: info: rsc:fs:1: stop
Jun 3 12:09:52 srv-dell lrmd: [6518]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:09:52 srv-dell lrmd: [6518]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:09:52 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:10:12 srv-dell lrmd: [6518]: WARN: fs:1:stop process (PID
10564) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 12:10:12 srv-dell lrmd: [6518]: WARN: operation stop[15] on
ocf::Filesystem::fs:1 for client 6521, its parameters:
directory=[/shared] fstype=[ocfs2] device=[/dev/sdb1]
CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [10564] timed out
Jun 3 12:10:12 srv-dell crmd: [6521]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (15) Timed Out (timeout=20000ms)
Jun 3 12:10:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:10:13 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:10:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:10:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:10:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:10:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:10:13 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:10:14 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:10:15 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:13:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:13:17 srv-dell cib: [11395]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-30.raw
Jun 3 12:13:17 srv-dell cib: [11395]: info: write_cib_contents: Wrote
version 0.31.0 of the CIB to disk (digest:
82fbaa8b9407d69879aa698001e068ee)
Jun 3 12:13:17 srv-dell cib: [11395]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.sN2EYg (digest:
/var/lib/heartbeat/crm/cib.3CANSk)
Jun 3 12:13:18 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:13:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:13:22 srv-dell cib: [11414]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-31.raw
Jun 3 12:13:22 srv-dell cib: [11414]: info: write_cib_contents: Wrote
version 0.32.0 of the CIB to disk (digest:
010741609ce187abfb8b639689bd4645)
Jun 3 12:13:22 srv-dell cib: [11414]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.sHNN4s (digest:
/var/lib/heartbeat/crm/cib.Yx934F)
Jun 3 12:13:22 srv-dell mgmtd: [6522]: info: CIB query: cib
Jun 3 12:13:43 srv-dell openais[6510]: [SERV ] Unloading all openais components
Jun 3 12:13:43 srv-dell openais[6510]: [SERV ] Unloading slot 10:
openais cluster config database access v1.01
Jun 3 12:13:43 srv-dell openais[6510]: [SERV ] Unloading openais
component: openais_confdb v0 (16/10)
Jun 3 12:13:43 srv-dell openais[6510]: [SERV ] Unloading slot 9:
Pacemaker Cluster Manager
Jun 3 12:13:43 srv-dell openais[6510]: [SERV ] Unloading openais
component: pacemaker v0 (2/9)
Jun 3 12:13:43 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
Begining shutdown
Jun 3 12:13:43 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to mgmtd: [6522]
Jun 3 12:13:43 srv-dell mgmtd: [6522]: info: mgmtd is shutting down
Jun 3 12:13:43 srv-dell mgmtd: [6522]: debug: [mgmtd] stopped
Jun 3 12:13:43 srv-dell haclient: on_event:None
Jun 3 12:13:44 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
mgmtd (pid=6522) confirmed dead
Jun 3 12:13:44 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to crmd: [6521]
Jun 3 12:13:44 srv-dell crmd: [6521]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 12:13:44 srv-dell crmd: [6521]: info: crm_shutdown: Requesting shutdown
Jun 3 12:13:44 srv-dell crmd: [6521]: info: do_shutdown_req: Sending
shutdown request to DC: srv-lenovo
Jun 3 12:13:44 srv-dell attrd: [6519]: info: attrd_ais_dispatch:
Update relayed from srv-lenovo
Jun 3 12:13:44 srv-dell attrd: [6519]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 12:13:44 srv-dell attrd: [6519]: info: attrd_perform_update:
Sent update 10: shutdown=1244045756
Jun 3 12:14:13 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
Still waiting for crmd (pid=6521) to terminate...
Jun 3 12:14:30 srv-dell openais[6510]: [SERV ] Unloading all openais components
Jun 3 12:14:30 srv-dell openais[6510]: [SERV ] Unloading slot 9:
Pacemaker Cluster Manager
Jun 3 12:14:30 srv-dell openais[6510]: [SERV ] Unloading openais
component: pacemaker v0 (2/9)
Jun 3 12:14:30 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
Begining shutdown
Jun 3 12:14:30 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
mgmtd (pid=0) confirmed dead
Jun 3 12:14:30 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to crmd: [6521]
Jun 3 12:14:30 srv-dell crmd: [6521]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: crm_shutdown: Escalating
the shutdown
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: do_log: FSA: Input
I_ERROR from crm_shutdown() received in state S_NOT_DC
Jun 3 12:14:30 srv-dell crmd: [6521]: info: do_state_transition:
State transition S_NOT_DC -> S_RECOVERY [ input=I_ERROR
cause=C_SHUTDOWN origin=crm_shutdown ]
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: do_recover: Action
A_RECOVER (0000000001000000) not supported
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: do_log: FSA: Input
I_TERMINATE from do_recover() received in state S_RECOVERY
Jun 3 12:14:30 srv-dell crmd: [6521]: info: do_state_transition:
State transition S_RECOVERY -> S_TERMINATE [ input=I_TERMINATE
cause=C_FSA_INTERNAL origin=do_recover ]
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: verify_stopped: Resource
o2cb:1 was active at shutdown. You may ignore this error if it is
unmanaged.
Jun 3 12:14:30 srv-dell crmd: [6521]: notice:
ghash_print_pending_for_rsc: Recurring action o2cb:1:12
(o2cb:1_monitor_120000) incomplete at shutdown
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: verify_stopped: Resource
dlm:1 was active at shutdown. You may ignore this error if it is
unmanaged.
Jun 3 12:14:30 srv-dell crmd: [6521]: notice:
ghash_print_pending_for_rsc: Recurring action dlm:1:9
(dlm:1_monitor_120000) incomplete at shutdown
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: verify_stopped: Resource
fs:1 was active at shutdown. You may ignore this error if it is
unmanaged.
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: verify_stopped: Resource
primitiveresource_cluster_ip was active at shutdown. You may ignore
this error if it is unmanaged.
Jun 3 12:14:30 srv-dell crmd: [6521]: notice:
ghash_print_pending_for_rsc: Recurring action
primitiveresource_cluster_ip:6
(primitiveresource_cluster_ip_monitor_5000) incomplete at shutdown
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: verify_stopped: Resource
cloneresource_cluster_fencing_null:0 was active at shutdown. You may
ignore this error if it is unmanaged.
Jun 3 12:14:30 srv-dell crmd: [6521]: info: do_lrm_control:
Disconnected from the LRM
Jun 3 12:14:30 srv-dell crmd: [6521]: info: do_ha_control:
Disconnected from OpenAIS
Jun 3 12:14:30 srv-dell crmd: [6521]: info: do_cib_control: Disconnecting CIB
Jun 3 12:14:30 srv-dell crmd: [6521]: info:
crmd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 12:14:30 srv-dell crmd: [6521]: info: do_exit: Performing
A_EXIT_0 - gracefully exiting the CRMd
Jun 3 12:14:30 srv-dell crmd: [6521]: ERROR: do_exit: Could not
recover from internal error
Jun 3 12:14:30 srv-dell crmd: [6521]: info: free_mem: Dropping
I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=do_stop ]
Jun 3 12:14:30 srv-dell crmd: [6521]: info: do_exit: [crmd] stopped (2)
Jun 3 12:14:30 srv-dell cib: [6517]: WARN: send_ipc_message: IPC
Channel to 6521 is not connected
Jun 3 12:14:30 srv-dell cib: [6517]: WARN: send_via_callback_channel:
Delivery of reply to client 6521/bae7b4c6-e5b6-47be-8a0d-7b5d84ca6066
failed
Jun 3 12:14:30 srv-dell cib: [6517]: WARN: do_local_notify: A-Sync
reply to crmd failed: reply failed
Jun 3 12:14:30 srv-dell openais[6510]: [crm ] info: pcmk_ipc_exit:
Client crmd (conn=0x748850, async-conn=0x748850) left
Jun 3 12:14:30 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
crmd (pid=6521) confirmed dead
Jun 3 12:14:30 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to pengine: [6520]
Jun 3 12:14:30 srv-dell pengine: [6520]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 12:14:31 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
crmd (pid=6521) confirmed dead
Jun 3 12:14:31 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to pengine: [6520]
Jun 3 12:14:31 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
pengine (pid=6520) confirmed dead
Jun 3 12:14:31 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to attrd: [6519]
Jun 3 12:14:31 srv-dell attrd: [6519]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 12:14:31 srv-dell attrd: [6519]: info: attrd_shutdown: Exiting
Jun 3 12:14:31 srv-dell attrd: [6519]: info: main: Exiting...
Jun 3 12:14:31 srv-dell attrd: [6519]: info:
attrd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 12:14:31 srv-dell openais[6510]: [crm ] info: pcmk_ipc_exit:
Client attrd (conn=0x7fca4c0344b0, async-conn=0x7fca4c0344b0) left
Jun 3 12:14:31 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
pengine (pid=6520) confirmed dead
Jun 3 12:14:31 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
attrd (pid=0) confirmed dead
Jun 3 12:14:31 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to lrmd: [6518]
Jun 3 12:14:31 srv-dell lrmd: [6518]: info: lrmd is shutting down
Jun 3 12:14:32 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
attrd (pid=6519) confirmed dead
Jun 3 12:14:32 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to lrmd: [6518]
Jun 3 12:14:32 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
lrmd (pid=6518) confirmed dead
Jun 3 12:14:32 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to cib: [6517]
Jun 3 12:14:32 srv-dell openais[6510]: [crm ] notice: pcmk_shutdown:
lrmd (pid=6518) confirmed dead
Jun 3 12:14:32 srv-dell openais[6510]: [MAIN ] notice: stop_child:
Sent -15 to cib: [6517]
Jun 3 12:16:28 srv-dell syslog-ng[3159]: syslog-ng starting up; version='2.0.9'
Jun 3 12:16:28 srv-dell rchal: CPU frequency scaling is not supported
by your processor.
Jun 3 12:16:28 srv-dell rchal: boot with 'CPUFREQ=no' in to avoid this warning.
Jun 3 12:16:28 srv-dell rchal: Cannot load cpufreq governors - No
cpufreq driver available
Jun 3 12:16:29 srv-dell ifup: lo
Jun 3 12:16:29 srv-dell ifup: lo
Jun 3 12:16:29 srv-dell ifup: IP address: 127.0.0.1/8
Jun 3 12:16:29 srv-dell ifup:
Jun 3 12:16:29 srv-dell ifup:
Jun 3 12:16:29 srv-dell ifup: IP address: 127.0.0.2/8
Jun 3 12:16:29 srv-dell ifup:
Jun 3 12:16:29 srv-dell ifup: eth0 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 12:16:29 srv-dell ifup: No configuration found for eth0
Jun 3 12:16:29 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 12:16:30 srv-dell ifup-dhcp: eth1 Starting DHCP4 client
Jun 3 12:16:30 srv-dell dhcpcd[3680]: eth1: dhcpcd 3.2.3 starting
Jun 3 12:16:30 srv-dell dhcpcd[3680]: eth1: hardware address =
00:19:b9:d7:20:d7
Jun 3 12:16:30 srv-dell dhcpcd[3680]: eth1: broadcasting for a lease
Jun 3 12:16:31 srv-dell ifup-dhcp: .
Jun 3 12:16:33 srv-dell ifup-dhcp: .
Jun 3 12:16:33 srv-dell kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.010:2):
operation="profile_load" name="/bin/ping" name2="default" pid=2927
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.042:3):
operation="profile_load" name="/sbin/klogd" name2="default" pid=2951
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.098:4):
operation="profile_load" name="/sbin/syslog-ng" name2="default"
pid=2970
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.154:5):
operation="profile_load" name="/sbin/syslogd" name2="default" pid=2983
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.210:6):
operation="profile_load" name="/usr/sbin/avahi-daemon" name2="default"
pid=2996
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.266:7):
operation="profile_load" name="/usr/sbin/identd" name2="default"
pid=3007
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.322:8):
operation="profile_load" name="/usr/sbin/mdnsd" name2="default"
pid=3008
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.410:9):
operation="profile_load" name="/usr/sbin/nscd" name2="default"
pid=3009
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.534:10):
operation="profile_load" name="/usr/sbin/ntpd" name2="default"
pid=3010
Jun 3 12:16:33 srv-dell kernel: type=1505 audit(1244045787.582:11):
operation="profile_load" name="/usr/sbin/traceroute" name2="default"
pid=3013
Jun 3 12:16:33 srv-dell kernel: IA-32 Microcode Update Driver:
v1.14a-xen <tigran at aivazian.fsnet.co.uk>
Jun 3 12:16:33 srv-dell kernel: firmware: requesting intel-ucode/06-0f-07
Jun 3 12:16:33 srv-dell kernel: bnx2: eth1: using MSI
Jun 3 12:16:33 srv-dell kernel: NET: Registered protocol family 17
Jun 3 12:16:33 srv-dell kernel: bnx2: eth1 NIC Copper Link is Up,
1000 Mbps full duplex, receive & transmit flow control ON
Jun 3 12:16:35 srv-dell ifup-dhcp: .
Jun 3 12:16:36 srv-dell dhcpcd[3680]: eth1: offered 192.168.2.220
from 192.168.2.1
Jun 3 12:16:36 srv-dell dhcpcd[3680]: eth1: checking 192.168.2.220 is
available on attached networks
Jun 3 12:16:37 srv-dell dhcpcd[3680]: eth1: leased 192.168.2.220 for
86400 seconds
Jun 3 12:16:37 srv-dell dhcpcd[3680]: eth1: no renewal time supplied,
assuming 43200 seconds
Jun 3 12:16:37 srv-dell dhcpcd[3680]: eth1: no rebind time supplied,
assuming 75600 seconds
Jun 3 12:16:37 srv-dell dhcpcd[3680]: eth1: adding IP address 192.168.2.220/24
Jun 3 12:16:37 srv-dell dhcpcd[3680]: eth1: adding default route via
192.168.2.1 metric 0
Jun 3 12:16:37 srv-dell ifup-dhcp:
Jun 3 12:16:37 srv-dell dns-resolver: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 12:16:37 srv-dell ifup-dhcp: eth1 IP address: 192.168.2.220/24
Jun 3 12:16:37 srv-dell dns-resolver: You can find my version in
/etc/resolv.conf.netconfig
Jun 3 12:16:37 srv-dell auditd[3979]: Started dispatcher:
/sbin/audispd pid: 3981
Jun 3 12:16:37 srv-dell audispd: priority_boost_parser called with: 4
Jun 3 12:16:37 srv-dell auditd[3979]: Init complete, auditd 1.7.7
listening for events (startup state disable)
Jun 3 12:16:37 srv-dell audispd: af_unix plugin initialized
Jun 3 12:16:37 srv-dell audispd: audispd initialized with q_depth=80
and 1 active plugins
Jun 3 12:16:37 srv-dell dhcpcd-hook: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 12:16:37 srv-dell dhcpcd-hook: You can find my version in
/etc/resolv.conf.netconfig ...
Jun 3 12:16:37 srv-dell kernel: NET: Registered protocol family 10
Jun 3 12:16:37 srv-dell kernel: lo: Disabled Privacy Extensions
Jun 3 12:16:37 srv-dell syslog-ng[3159]: Configuration reload request
received, reloading configuration;
Jun 3 12:16:37 srv-dell syslog-ng[3159]: New configuration initialized;
Jun 3 12:16:38 srv-dell ifdown: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 12:16:38 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 12:16:38 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 12:16:38 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 12:16:38 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 12:16:38 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 12:16:38 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 12:16:38 srv-dell smartd[4498]: smartd 5.39 2008-10-24 22:33
[x86_64-suse-linux-gnu] (openSUSE RPM) Copyright (C) 2002-8 by Bruce
Allen, http://smartmontools.sourceforge.net
Jun 3 12:16:38 srv-dell smartd[4498]: Opened configuration file
/etc/smartd.conf
Jun 3 12:16:38 srv-dell smartd[4498]: Drive: DEVICESCAN, implied '-a'
Directive on line 26 of file /etc/smartd.conf
Jun 3 12:16:38 srv-dell smartd[4498]: Configuration file
/etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Jun 3 12:16:38 srv-dell smartd[4498]: Device: /dev/sda, opened
Jun 3 12:16:38 srv-dell smartd[4498]: Device: /dev/sda, IE (SMART)
not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on
SMART features
Jun 3 12:16:38 srv-dell smartd[4498]: Unable to monitor any SMART
enabled devices. Try debug (-d) option. Exiting...
Jun 3 12:16:39 srv-dell SuSEfirewall2: SuSEfirewall2 not active
Jun 3 12:16:39 srv-dell kernel: drbd: initialised. Version: 8.2.7
(api:88/proto:86-88)
Jun 3 12:16:39 srv-dell kernel: drbd: GIT-hash:
a1b440e8b3011a1318d8bff1bb7edc763ef995b0 build by lmb at hermes,
2009-02-20 13:35:59
Jun 3 12:16:39 srv-dell kernel: drbd: registered as block device major 147
Jun 3 12:16:39 srv-dell kernel: drbd: minor_table @ 0xffff8801e69ee080
Jun 3 12:16:39 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 12:16:39 srv-dell kernel: drbd0: conn( StandAlone -> Unconnected )
Jun 3 12:16:39 srv-dell kernel: drbd0: Starting worker thread (from
cqueue [48])
Jun 3 12:16:39 srv-dell kernel: drbd0: Starting receiver thread (from
drbd0_worker [4852])
Jun 3 12:16:39 srv-dell kernel: drbd0: receiver (re)started
Jun 3 12:16:39 srv-dell kernel: drbd0: conn( Unconnected -> WFConnection )
Jun 3 12:16:39 srv-dell sshd[4877]: Server listening on 0.0.0.0 port 22.
Jun 3 12:16:39 srv-dell sshd[4877]: Server listening on :: port 22.
Jun 3 12:16:39 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/Xservers
Jun 3 12:16:39 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/xdm-config
Jun 3 12:16:39 srv-dell dhcpcd[3680]: eth1: setting hostname to `srv-dell'
Jun 3 12:16:39 srv-dell dhcpcd[3680]: eth1: exiting
Jun 3 12:16:39 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 3 12:16:39 srv-dell kernel: iscsi: registered transport (tcp)
Jun 3 12:16:39 srv-dell kernel: iscsi: registered transport (iser)
Jun 3 12:16:39 srv-dell iscsid: iSCSI logger with pid=5113 started!
Jun 3 12:16:39 srv-dell /usr/sbin/cron[5127]: (CRON) STARTUP (V5.0)
Jun 3 12:16:40 srv-dell kernel: vendor=8086 device=244e
Jun 3 12:16:40 srv-dell kernel: pci 0000:10:0d.0: PCI INT A -> GSI 19
(level, low) -> IRQ 19
Jun 3 12:16:40 srv-dell xenstored: Checking store ...
Jun 3 12:16:40 srv-dell xenstored: Checking store complete.
Jun 3 12:16:40 srv-dell kernel: suspend: event channel 55
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:795: blktapctrl: v1.0.0
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [raw image (aio)]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [raw image (sync)]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [vmware image (vmdk)]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [ramdisk image (ram)]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [qcow disk (qcow)]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [qcow2 disk (qcow2)]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [ioemu disk]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [raw image (cdrom)]
Jun 3 12:16:40 srv-dell BLKTAPCTRL[5201]: blktapctrl_linux.c:23:
/dev/xen/blktap0 device already exists
Jun 3 12:16:40 srv-dell kernel: Bridge firewalling registered
Jun 3 12:16:40 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 3 12:16:40 srv-dell iscsid: iSCSI daemon with pid=5115 started!
Jun 3 12:16:46 srv-dell gdm-simple-greeter[5343]: libglade-WARNING:
Unexpected element <requires-version> inside <glade-interface>.
Jun 3 12:16:48 srv-dell kernel: eth1: no IPv6 routers present
Jun 3 12:16:54 srv-dell pulseaudio[5534]: pid.c: Stale PID file, overwriting.
Jun 3 12:16:54 srv-dell python: hp-systray(init)[5508]: error:
hp-systray cannot be run as root. Exiting.
Jun 3 12:17:11 srv-dell kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] AIS Executive Service
RELEASE 'subrev 1152 version 0.80'
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] AIS Executive Service:
started and ready to provide service.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Token Timeout (5000
ms) retransmit timeout (490 ms)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] token hold (382 ms)
retransmits before loss (10 retrans)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] join (1000 ms)
send_join (45 ms) consensus (2500 ms) merge (200 ms)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] seqno unchanged const
(30 rotations) Maximum network MTU 1500
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] send threads (0 threads)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] RRP token expired
timeout (490 ms)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] RRP token problem
counter (2000 ms)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] RRP threshold (10 problem count)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] RRP mode set to none.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] heartbeat_failures_allowed (0)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] max_network_delay (50 ms)
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] HeartBeat is Disabled.
To enable set heartbeat_failures_allowed > 0
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] The network interface
[192.168.2.220] is now up.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Created or loaded
sequence id 200.192.168.2.220 for this ring.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] entering GATHER state from 15.
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info:
process_ais_conf: Reading configure
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info:
config_find_next: Processing additional logging options...
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: get_config_opt:
Found 'off' for option: debug
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: get_config_opt:
Found 'yes' for option: to_syslog
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: get_config_opt:
Found 'daemon' for option: syslog_facility
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info:
config_find_next: Processing additional service options...
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_logd
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_mgmtd
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info:
pcmk_plugin_init: CRM: Initialized
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] Logging: Initialized
pcmk_plugin_init
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info:
pcmk_plugin_init: Service: 9
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info:
pcmk_plugin_init: Local node id: 1543678144
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info:
pcmk_plugin_init: Local hostname: srv-dell
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: update_member:
Creating entry for node 1543678144 born on 0
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: update_member:
0x736340 Node 1543678144 now known as srv-dell (was: (null))
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: update_member:
Node srv-dell now has 1 quorum votes (was 0)
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: update_member:
Node 1543678144/srv-dell is now: member
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: spawn_child:
Forked child 5798 for process stonithd
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: spawn_child:
Forked child 5799 for process cib
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: spawn_child:
Forked child 5800 for process lrmd
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: spawn_child:
Forked child 5801 for process attrd
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: spawn_child:
Forked child 5802 for process pengine
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: spawn_child:
Forked child 5803 for process crmd
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: spawn_child:
Forked child 5804 for process mgmtd
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info: pcmk_startup:
CRM: Initialized
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] Service initialized
'Pacemaker Cluster Manager'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais extended virtual synchrony service'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais cluster membership service B.01.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais availability management framework B.01.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais checkpoint service B.01.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais event service B.01.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais distributed locking service B.01.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais message service B.01.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais configuration service'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais cluster closed process group service v1.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SERV ] Service initialized
'openais cluster config database access v1.01'
Jun 3 12:17:21 srv-dell openais[5792]: [SYNC ] Not using a virtual
synchrony filter.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Saving state aru 0
high seq received 0
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Storing new sequence
id for ring cc
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] entering COMMIT state.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] entering RECOVERY state.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] previous ring seq 200
rep 192.168.2.220
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] aru 0 high delivered 0
received flag 1
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] Sending initial ORF token
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 204: memb=0,
new=0, lost=0
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 204: memb=1, new=1,
lost=0
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: NEW: srv-dell 1543678144
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 12:17:21 srv-dell openais[5792]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000053312
(340754)
Jun 3 12:17:21 srv-dell openais[5792]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 12:17:21 srv-dell openais[5792]: [TOTEM] entering OPERATIONAL state.
Jun 3 12:17:21 srv-dell openais[5792]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 12:17:21 srv-dell lrmd: [5800]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:21 srv-dell lrmd: [5800]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Jun 3 12:17:21 srv-dell mgmtd: [5804]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:21 srv-dell mgmtd: [5804]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 3 12:17:21 srv-dell mgmtd: [5804]: debug: Enabling coredumps
Jun 3 12:17:21 srv-dell mgmtd: [5804]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 12:17:21 srv-dell mgmtd: [5804]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 12:17:21 srv-dell cib: [5799]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 12:17:21 srv-dell attrd: [5801]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 12:17:21 srv-dell pengine: [5802]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 12:17:21 srv-dell mgmtd: [5804]: WARN: lrm_signon: can not
initiate connection
Jun 3 12:17:21 srv-dell mgmtd: [5804]: info: login to lrm: 0, ret:0
Jun 3 12:17:21 srv-dell cib: [5799]: WARN: Initializing connection to
logging daemon failed. Logging daemon may not be running
Jun 3 12:17:21 srv-dell attrd: [5801]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:21 srv-dell pengine: [5802]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:21 srv-dell cib: [5799]: info: G_main_add_TriggerHandler:
Added signal manual handler
Jun 3 12:17:21 srv-dell stonithd: [5798]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:21 srv-dell stonithd: [5798]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 12:17:21 srv-dell stonithd: [5798]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 12:17:21 srv-dell cib: [5799]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 12:17:21 srv-dell crmd: [5803]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 12:17:21 srv-dell attrd: [5801]: info: main: Starting up....
Jun 3 12:17:21 srv-dell lrmd: [5800]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 12:17:21 srv-dell attrd: [5801]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 12:17:21 srv-dell lrmd: [5800]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jun 3 12:17:21 srv-dell crmd: [5803]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:21 srv-dell cib: [5799]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Jun 3 12:17:21 srv-dell attrd: [5801]: info: init_ais_connection: AIS
connection established
Jun 3 12:17:21 srv-dell crmd: [5803]: info: main: CRM Hg Version:
0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Jun 3 12:17:21 srv-dell lrmd: [5800]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jun 3 12:17:21 srv-dell pengine: [5802]: info: main: Starting pengine
Jun 3 12:17:21 srv-dell lrmd: [5800]: info: Started.
Jun 3 12:17:21 srv-dell stonithd: [5798]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 12:17:21 srv-dell attrd: [5801]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 12:17:21 srv-dell attrd: [5801]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 12:17:21 srv-dell attrd: [5801]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 12:17:21 srv-dell crmd: [5803]: info: crmd_init: Starting crmd
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info: pcmk_ipc:
Recorded connection 0x7fc95c01cff0 for attrd/5801
Jun 3 12:17:21 srv-dell crmd: [5803]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 12:17:21 srv-dell stonithd: [5798]: info: init_ais_connection:
AIS connection established
Jun 3 12:17:21 srv-dell stonithd: [5798]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 12:17:21 srv-dell stonithd: [5798]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info: pcmk_ipc:
Recorded connection 0x7fc95c01d220 for stonithd/5798
Jun 3 12:17:21 srv-dell stonithd: [5798]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 12:17:21 srv-dell stonithd: [5798]: notice:
/usr/lib64/heartbeat/stonithd start up successfully.
Jun 3 12:17:21 srv-dell stonithd: [5798]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 12:17:21 srv-dell cib: [5799]: info: startCib: CIB
Initialization completed successfully
Jun 3 12:17:21 srv-dell cib: [5799]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 12:17:21 srv-dell cib: [5799]: info: init_ais_connection: AIS
connection established
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info: pcmk_ipc:
Recorded connection 0x7fc95c01d450 for cib/5799
Jun 3 12:17:21 srv-dell cib: [5799]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 12:17:21 srv-dell cib: [5799]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 12:17:21 srv-dell cib: [5799]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 12:17:21 srv-dell openais[5792]: [crm ] info: pcmk_ipc:
Sending membership update 204 to cib
Jun 3 12:17:21 srv-dell cib: [5799]: info: cib_init: Starting cib mainloop
Jun 3 12:17:21 srv-dell cib: [5799]: info: ais_dispatch: Membership
204: quorum still lost
Jun 3 12:17:21 srv-dell cib: [5799]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=204
proc=00000000000000000000000000053312 (new)
Jun 3 12:17:21 srv-dell cib: [5808]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-32.raw
Jun 3 12:17:21 srv-dell cib: [5808]: info: write_cib_contents: Wrote
version 0.32.0 of the CIB to disk (digest:
df625237f328ba7a5f85b72052ea249b)
Jun 3 12:17:21 srv-dell cib: [5808]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.TmP0pm (digest:
/var/lib/heartbeat/crm/cib.TyK5zU)
Jun 3 12:17:22 srv-dell mgmtd: [5804]: info: init_crm
Jun 3 12:17:22 srv-dell crmd: [5803]: info: do_cib_control: CIB
connection established
Jun 3 12:17:22 srv-dell crmd: [5803]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 12:17:22 srv-dell crmd: [5803]: info: init_ais_connection: AIS
connection established
Jun 3 12:17:22 srv-dell openais[5792]: [crm ] info: pcmk_ipc:
Recorded connection 0x7fc95c01d720 for crmd/5803
Jun 3 12:17:22 srv-dell openais[5792]: [crm ] info: pcmk_ipc:
Sending membership update 204 to crmd
Jun 3 12:17:22 srv-dell crmd: [5803]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 12:17:22 srv-dell crmd: [5803]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 12:17:22 srv-dell crmd: [5803]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 12:17:22 srv-dell crmd: [5803]: info: do_ha_control: Connected
to the cluster
Jun 3 12:17:22 srv-dell crmd: [5803]: info: do_started: Delaying
start, CCM (0000000000100000) not connected
Jun 3 12:17:22 srv-dell crmd: [5803]: info: crmd_init: Starting crmd's mainloop
Jun 3 12:17:22 srv-dell crmd: [5803]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 12:17:22 srv-dell openais[5792]: [crm ] info:
update_expected_votes: Expected quorum votes 1024 -> 2
Jun 3 12:17:22 srv-dell crmd: [5803]: info: ais_dispatch: Membership
204: quorum still lost
Jun 3 12:17:22 srv-dell crmd: [5803]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=204
proc=00000000000000000000000000053312 (new)
Jun 3 12:17:22 srv-dell crmd: [5803]: info: do_started: The local CRM
is operational
Jun 3 12:17:22 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Jun 3 12:17:23 srv-dell mgmtd: [5804]: debug: main: run the loop...
Jun 3 12:17:23 srv-dell mgmtd: [5804]: info: Started.
Jun 3 12:17:23 srv-dell crmd: [5803]: info: ais_dispatch: Membership
204: quorum still lost
Jun 3 12:17:31 srv-dell attrd: [5801]: info: main: Sending full refresh
Jun 3 12:17:31 srv-dell attrd: [5801]: info: main: Starting mainloop...
Jun 3 12:17:33 srv-dell crmd: [5803]: info: crm_timer_popped:
Election Trigger (I_DC_TIMEOUT) just popped!
Jun 3 12:17:33 srv-dell crmd: [5803]: WARN: do_log: FSA: Input
I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_te_control:
Registering TE UUID: bdd9819c-e3a8-4096-8ecd-03c49ec14d09
Jun 3 12:17:33 srv-dell crmd: [5803]: WARN:
cib_client_add_notify_callback: Callback already present
Jun 3 12:17:33 srv-dell crmd: [5803]: info: set_graph_functions:
Setting custom graph functions
Jun 3 12:17:33 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition -1: 0 actions in 0 synapses
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_dc_takeover: Taking
over DC status for this partition
Jun 3 12:17:33 srv-dell cib: [5799]: info: cib_process_readwrite: We
are now in R/W mode
Jun 3 12:17:33 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_master for section 'all'
(origin=local/crmd/6, version=0.32.0): ok (rc=0)
Jun 3 12:17:33 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/7, version=0.32.0): ok (rc=0)
Jun 3 12:17:33 srv-dell crmd: [5803]: info: join_make_offer: Making
join offers based on membership 204
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_dc_join_offer_all:
join-1: Waiting on 1 outstanding join acks
Jun 3 12:17:33 srv-dell crmd: [5803]: info: ais_dispatch: Membership
204: quorum still lost
Jun 3 12:17:33 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/9, version=0.32.0): ok (rc=0)
Jun 3 12:17:33 srv-dell crmd: [5803]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 12:17:33 srv-dell crmd: [5803]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 12:17:33 srv-dell crmd: [5803]: info: ais_dispatch: Membership
204: quorum still lost
Jun 3 12:17:33 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/12, version=0.32.0): ok (rc=0)
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes responded to the join offer.
Jun 3 12:17:33 srv-dell crmd: [5803]: info: do_dc_join_finalize:
join-1: Syncing the CIB from srv-dell to the rest of the cluster
Jun 3 12:17:33 srv-dell crmd: [5803]: info: te_connect_stonith:
Attempting connection to fencing daemon...
Jun 3 12:17:33 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/15, version=0.32.0): ok (rc=0)
Jun 3 12:17:33 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/16, version=0.32.0): ok (rc=0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: te_connect_stonith: Connected
Jun 3 12:17:34 srv-dell crmd: [5803]: info: update_attrd: Connecting
to attrd...
Jun 3 12:17:34 srv-dell crmd: [5803]: info: update_attrd: Updating
terminate=<none> via attrd for srv-dell
Jun 3 12:17:34 srv-dell crmd: [5803]: info: update_attrd: Updating
shutdown=<none> via attrd for srv-dell
Jun 3 12:17:34 srv-dell attrd: [5801]: info: find_hash_entry:
Creating hash entry for terminate
Jun 3 12:17:34 srv-dell attrd: [5801]: info: find_hash_entry:
Creating hash entry for shutdown
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_dc_join_ack: join-1:
Updating node state to member for srv-dell
Jun 3 12:17:34 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/17, version=0.32.0): ok (rc=0)
Jun 3 12:17:34 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/transient_attributes
(origin=local/crmd/18, version=0.32.0): ok (rc=0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/transient_attributes": ok
(rc=0)
Jun 3 12:17:34 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/19,
version=0.32.0): ok (rc=0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 12:17:34 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/20,
version=0.32.0): ok (rc=0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 3 12:17:34 srv-dell crmd: [5803]: info: crm_update_quorum:
Updating quorum status to false (call=24)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 3 12:17:34 srv-dell attrd: [5801]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_pe_invoke: Query 25:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:17:34 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/22, version=0.32.1): ok (rc=0)
Jun 3 12:17:34 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 12:17:34 srv-dell cib: [5799]: info: log_data_element:
cib:diff: - <cib have-quorum="1" admin_epoch="0" epoch="32"
num_updates="1" />
Jun 3 12:17:34 srv-dell cib: [5799]: info: log_data_element:
cib:diff: + <cib have-quorum="0" dc-uuid="srv-dell" admin_epoch="0"
epoch="33" num_updates="1" />
Jun 3 12:17:34 srv-dell crmd: [5803]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 12:17:34 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/24, version=0.33.1): ok (rc=0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: need_abort: Aborting on
change to have-quorum
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_pe_invoke: Query 26:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:17:34 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244045854-7, seq=204, quorate=0
Jun 3 12:17:34 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 12:17:34 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 12:17:34 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 12:17:34 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 12:17:34 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 12:17:34 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 12:17:34 srv-dell cib: [5902]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-33.raw
Jun 3 12:17:34 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 12:17:34 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:0 with dlm:0 on srv-dell
Jun 3 12:17:34 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating fs:0 with o2cb:0 on srv-dell
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Start
cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 12:17:34 srv-dell cib: [5902]: info: write_cib_contents: Wrote
version 0.33.0 of the CIB to disk (digest:
7b5ff6e345d6ad4a339eb822ff621483)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Stopped)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Start
dlm:0 (srv-dell)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Leave
resource dlm:1 (Stopped)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Start
o2cb:0 (srv-dell)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Leave
resource o2cb:1 (Stopped)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Start
fs:0 (srv-dell)
Jun 3 12:17:34 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 12:17:34 srv-dell cib: [5902]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Glx8EZ (digest:
/var/lib/heartbeat/crm/cib.SKREo0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 12:17:34 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 0: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-13.bz2
Jun 3 12:17:34 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 12:17:34 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 0: 18 actions in 18 synapses
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1244045854-7) derived from
/var/lib/pengine/pe-warn-13.bz2
Jun 3 12:17:34 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 4: monitor primitiveresource_cluster_ip_monitor_0 on
srv-dell (local)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=4:0:7:bdd9819c-e3a8-4096-8ecd-03c49ec14d09
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 12:17:34 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 12:17:34 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 5: monitor
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell (local)
Jun 3 12:17:34 srv-dell lrmd: [5800]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=5:0:7:bdd9819c-e3a8-4096-8ecd-03c49ec14d09
op=cloneresource_cluster_fencing_null:0_monitor_0 )
Jun 3 12:17:34 srv-dell lrmd: [5800]: info:
rsc:cloneresource_cluster_fencing_null:0: monitor
Jun 3 12:17:34 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 6: monitor dlm:0_monitor_0 on srv-dell (local)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=6:0:7:bdd9819c-e3a8-4096-8ecd-03c49ec14d09 op=dlm:0_monitor_0 )
Jun 3 12:17:34 srv-dell lrmd: [5800]: info: rsc:dlm:0: monitor
Jun 3 12:17:34 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 7: monitor o2cb:0_monitor_0 on srv-dell (local)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=7:0:7:bdd9819c-e3a8-4096-8ecd-03c49ec14d09 op=o2cb:0_monitor_0 )
Jun 3 12:17:34 srv-dell lrmd: [5800]: info: rsc:o2cb:0: monitor
Jun 3 12:17:34 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 8: monitor fs:0_monitor_0 on srv-dell (local)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=8:0:7:bdd9819c-e3a8-4096-8ecd-03c49ec14d09 op=fs:0_monitor_0 )
Jun 3 12:17:34 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_monitor_0 (call=3,
rc=7, cib-update=27, confirmed=true) complete not running
Jun 3 12:17:34 srv-dell crmd: [5803]: info: match_graph_event: Action
cloneresource_cluster_fencing_null:0_monitor_0 (5) confirmed on
srv-dell (rc=0)
Jun 3 12:17:34 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:monitor:stderr) logd is not running
Jun 3 12:17:34 srv-dell lrmd: [5800]: info: RA output:
(dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Jun 3 12:17:34 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_0 (call=4, rc=7, cib-update=28,
confirmed=true) complete not running
Jun 3 12:17:34 srv-dell crmd: [5803]: info: match_graph_event: Action
dlm:0_monitor_0 (6) confirmed on srv-dell (rc=0)
Jun 3 12:17:34 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:monitor:stderr) 2009/06/03_12:17:34 INFO: configfs not laoded
Jun 3 12:17:34 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=2, rc=7,
cib-update=29, confirmed=true) complete not running
Jun 3 12:17:34 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_0 (call=5, rc=7, cib-update=30,
confirmed=true) complete not running
Jun 3 12:17:34 srv-dell crmd: [5803]: info: match_graph_event: Action
primitiveresource_cluster_ip_monitor_0 (4) confirmed on srv-dell
(rc=0)
Jun 3 12:17:34 srv-dell crmd: [5803]: info: match_graph_event: Action
o2cb:0_monitor_0 (7) confirmed on srv-dell (rc=0)
Jun 3 12:17:35 srv-dell lrmd: [5800]: info: rsc:fs:0: monitor
Jun 3 12:17:35 srv-dell lrmd: [5800]: info: RA output:
(fs:0:monitor:stderr) logd is not running
Jun 3 12:17:35 srv-dell lrmd: [5800]: info: RA output:
(fs:0:monitor:stderr) 2009/06/03_12:17:35 WARNING: Couldn't find
device [/dev/sdb1]. Expected /dev/??? to exist
Jun 3 12:17:35 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=6, rc=7, cib-update=31, confirmed=true)
complete not running
Jun 3 12:17:35 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:0_monitor_0 (8) confirmed on srv-dell (rc=0)
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 3: probe_complete probe_complete on srv-dell (local)
- no waiting
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 20 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 32 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 33 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 12 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 18 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 30 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 11: start
cloneresource_cluster_fencing_null:0_start_0 on srv-dell (local)
Jun 3 12:17:35 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=11:0:0:bdd9819c-e3a8-4096-8ecd-03c49ec14d09
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 12:17:35 srv-dell lrmd: [5800]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 24 fired and confirmed
Jun 3 12:17:35 srv-dell lrmd: [5991]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 12:17:35 srv-dell stonithd: [5798]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 12:17:35 srv-dell lrmd: [5800]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 12:17:35 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=7, rc=0,
cib-update=35, confirmed=true) complete ok
Jun 3 12:17:35 srv-dell crmd: [5803]: info: match_graph_event: Action
cloneresource_cluster_fencing_null:0_start_0 (11) confirmed on
srv-dell (rc=0)
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 13 fired and confirmed
Jun 3 12:17:35 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 12:17:35 srv-dell crmd: [5803]: notice: run_graph: Transition 0
(Complete=18, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-13.bz2): Complete
Jun 3 12:17:35 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 0 is now complete
Jun 3 12:17:35 srv-dell crmd: [5803]: info: notify_crmd: Transition 0
status: done - <null>
Jun 3 12:17:35 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 12:17:35 srv-dell crmd: [5803]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] entering GATHER state from 11.
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] Saving state aru 29
high seq received 29
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] Storing new sequence
id for ring d0
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] entering COMMIT state.
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] entering RECOVERY state.
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] previous ring seq 204
rep 192.168.2.120
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] aru 46 high delivered
42 received flag 1
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] previous ring seq 204
rep 192.168.2.220
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] aru 29 high delivered
29 received flag 1
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 208: memb=1,
new=0, lost=0
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 12:17:38 srv-dell cib: [5799]: notice: ais_dispatch: Membership
208: quorum aquired
Jun 3 12:17:38 srv-dell cib: [5799]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 12:17:38 srv-dell cib: [5799]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=208 proc=00000000000000000000000000000000
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 208: memb=2, new=1,
lost=0
Jun 3 12:17:38 srv-dell openais[5792]: [MAIN ] info: update_member:
Creating entry for node 2013440192 born on 208
Jun 3 12:17:38 srv-dell crmd: [5803]: notice: ais_dispatch:
Membership 208: quorum aquired
Jun 3 12:17:38 srv-dell crmd: [5803]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 12:17:38 srv-dell crmd: [5803]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=208 proc=00000000000000000000000000000000
Jun 3 12:17:38 srv-dell crmd: [5803]: info: crm_update_quorum:
Updating quorum status to true (call=38)
Jun 3 12:17:38 srv-dell openais[5792]: [MAIN ] info: update_member:
Node 2013440192/unknown is now: member
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: NEW: .pending. 2013440192
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: MEMB: .pending. 2013440192
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] info:
send_member_notification: Sending membership update 208 to 2 children
Jun 3 12:17:38 srv-dell openais[5792]: [MAIN ] info: update_member:
0x736340 Node 1543678144 ((null)) born on: 208
Jun 3 12:17:38 srv-dell openais[5792]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 12:17:38 srv-dell openais[5792]: [TOTEM] entering OPERATIONAL state.
Jun 3 12:17:38 srv-dell openais[5792]: [MAIN ] info: update_member:
0x7fc95c01c7e0 Node 2013440192 (srv-lenovo) born on: 192
Jun 3 12:17:38 srv-dell openais[5792]: [MAIN ] info: update_member:
0x7fc95c01c7e0 Node 2013440192 now known as srv-lenovo (was: (null))
Jun 3 12:17:38 srv-dell openais[5792]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 3 12:17:38 srv-dell openais[5792]: [MAIN ] info: update_member:
Node srv-lenovo now has 1 quorum votes (was 0)
Jun 3 12:17:38 srv-dell openais[5792]: [crm ] info:
send_member_notification: Sending membership update 208 to 2 children
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 12:17:38 srv-dell openais[5792]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 12:17:38 srv-dell openais[5792]: [CPG ] got joinlist message
from node 2013440192
Jun 3 12:17:38 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/36, version=0.33.8): ok (rc=0)
Jun 3 12:17:38 srv-dell cib: [5799]: info: log_data_element:
cib:diff: - <cib have-quorum="0" admin_epoch="0" epoch="33"
num_updates="8" />
Jun 3 12:17:38 srv-dell cib: [5799]: info: log_data_element:
cib:diff: + <cib have-quorum="1" admin_epoch="0" epoch="34"
num_updates="1" />
Jun 3 12:17:38 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/38, version=0.34.1): ok (rc=0)
Jun 3 12:17:38 srv-dell cib: [5799]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 12:17:38 srv-dell crmd: [5803]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 12:17:38 srv-dell crmd: [5803]: info: need_abort: Aborting on
change to have-quorum
Jun 3 12:17:38 srv-dell cib: [5799]: WARN: cib_process_diff: Diff
0.33.15 -> 0.33.16 not applied to 0.34.1: current "epoch" is greater
than required
Jun 3 12:17:38 srv-dell crmd: [5803]: info: ais_dispatch: Membership
208: quorum retained
Jun 3 12:17:38 srv-dell crmd: [5803]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 12:17:38 srv-dell crmd: [5803]: info: ais_status_callback:
status: srv-lenovo is now member
Jun 3 12:17:38 srv-dell crmd: [5803]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=192 seen=208 proc=00000000000000000000000000053312
(new)
Jun 3 12:17:38 srv-dell cib: [5799]: info: ais_dispatch: Membership
208: quorum retained
Jun 3 12:17:38 srv-dell cib: [5799]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=192 seen=208 proc=00000000000000000000000000053312
(new)
Jun 3 12:17:38 srv-dell cib: [5799]: WARN: cib_process_diff: Diff
0.33.16 -> 0.33.17 not applied to 0.34.1: current "epoch" is greater
than required
Jun 3 12:17:38 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/40, version=0.34.1): ok (rc=0)
Jun 3 12:17:38 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/41, version=0.34.1): ok (rc=0)
Jun 3 12:17:38 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 3 12:17:38 srv-dell crmd: [5803]: info: do_state_transition:
Membership changed: 204 -> 208 - join restart
Jun 3 12:17:38 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/44, version=0.34.2): ok (rc=0)
Jun 3 12:17:38 srv-dell crmd: [5803]: info: do_pe_invoke: Query 45:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:17:38 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=do_state_transition ]
Jun 3 12:17:38 srv-dell crmd: [5803]: info: update_dc: Unset DC srv-dell
Jun 3 12:17:38 srv-dell crmd: [5803]: info: join_make_offer: Making
join offers based on membership 208
Jun 3 12:17:38 srv-dell cib: [5993]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-34.raw
Jun 3 12:17:38 srv-dell crmd: [5803]: info: do_dc_join_offer_all:
join-2: Waiting on 2 outstanding join acks
Jun 3 12:17:38 srv-dell crmd: [5803]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 12:17:38 srv-dell cib: [5993]: info: write_cib_contents: Wrote
version 0.34.0 of the CIB to disk (digest:
2765bc4c6a2b881fb1c47bf2c6eadc0f)
Jun 3 12:17:38 srv-dell cib: [5993]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.hzSYMm (digest:
/var/lib/heartbeat/crm/cib.gQIXEw)
Jun 3 12:17:39 srv-dell cib: [5799]: WARN: cib_process_diff: Diff
0.33.17 -> 0.33.18 not applied to 0.34.2: current "epoch" is greater
than required
Jun 3 12:17:39 srv-dell crmd: [5803]: ERROR: crmd_ha_msg_filter:
Another DC detected: srv-lenovo (op=noop)
Jun 3 12:17:39 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_INTEGRATION -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter ]
Jun 3 12:17:39 srv-dell crmd: [5803]: info: update_dc: Unset DC srv-dell
Jun 3 12:17:39 srv-dell cib: [5799]: WARN: cib_process_diff: Diff
0.33.18 -> 0.33.19 not applied to 0.34.2: current "epoch" is greater
than required
Jun 3 12:17:39 srv-dell cib: [5799]: WARN: cib_process_diff: Diff
0.33.19 -> 0.33.20 not applied to 0.34.2: current "epoch" is greater
than required
Jun 3 12:17:39 srv-dell cib: [5799]: WARN: cib_process_diff: Diff
0.33.20 -> 0.34.1 not applied to 0.34.2: current "epoch" is greater
than required
Jun 3 12:17:39 srv-dell crmd: [5803]: info: crmd_ha_msg_filter:
Another DC detected: srv-lenovo (op=noop)
Jun 3 12:17:39 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_ELECTION -> S_RELEASE_DC [ input=I_RELEASE_DC
cause=C_FSA_INTERNAL origin=do_election_count_vote ]
Jun 3 12:17:39 srv-dell crmd: [5803]: info: do_dc_release: DC role released
Jun 3 12:17:39 srv-dell crmd: [5803]: info: pe_connection_destroy:
Connection to the Policy Engine released
Jun 3 12:17:39 srv-dell cib: [5799]: info: cib_process_readwrite: We
are now in R/O mode
Jun 3 12:17:39 srv-dell crmd: [5803]: info: do_te_control:
Transitioner is now inactive
Jun 3 12:17:39 srv-dell crmd: [5803]: info: do_te_control:
Disconnecting STONITH...
Jun 3 12:17:39 srv-dell crmd: [5803]: info:
tengine_stonith_connection_destroy: Fencing daemon disconnected
Jun 3 12:17:39 srv-dell crmd: [5803]: notice: Not currently connected.
Jun 3 12:17:39 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_RELEASE_DC -> S_PENDING [ input=I_RELEASE_SUCCESS
cause=C_FSA_INTERNAL origin=do_dc_release ]
Jun 3 12:17:39 srv-dell crmd: [5803]: info: update_dc: Set DC to
srv-lenovo (3.0.1)
Jun 3 12:17:39 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=srv-lenovo/crmd/178, version=0.34.2): ok (rc=0)
Jun 3 12:17:39 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 3 12:17:39 srv-dell attrd: [5801]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 12:17:39 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 12:17:39 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 12:17:39 srv-dell attrd: [5801]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 12:17:39 srv-dell attrd: [5801]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 12:17:39 srv-dell cib: [5994]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-35.raw
Jun 3 12:17:39 srv-dell cib: [5994]: info: write_cib_contents: Wrote
version 0.35.0 of the CIB to disk (digest:
6c8677fd707c9d209cd3b40f2742cf93)
Jun 3 12:17:39 srv-dell cib: [5994]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.pnmrt1 (digest:
/var/lib/heartbeat/crm/cib.R03fPc)
Jun 3 12:17:40 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=7:19:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_start_0 )
Jun 3 12:17:40 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 12:17:40 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=16:19:0:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:0_start_0 )
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: rsc:dlm:0: start
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: RA output:
(dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Jun 3 12:17:40 srv-dell kernel: DLM (built Feb 28 2009 07:25:25) installed
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:17:40 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:17:40 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 12:17:40 srv-dell cluster-dlm[6082]: main: dlm_controld 1239267206
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info:
init_ais_connection: AIS connection established
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 12:17:40 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 3 12:17:40 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 3 12:17:40 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 3 12:17:40 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 3 12:17:40 srv-dell openais[5792]: [crm ] info: pcmk_notify:
Enabling node notifications for child 6082 (0x7fc95c01f2c0)
Jun 3 12:17:40 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 3 12:17:40 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 2013440192
Jun 3 12:17:40 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 3 12:17:40 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 3 12:17:40 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: notice: ais_dispatch:
Membership 208: quorum aquired
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:17:40 srv-dell cluster-dlm: [6082]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=208 seen=208
proc=00000000000000000000000000053312 (new)
Jun 3 12:17:40 srv-dell cluster-dlm: update_cluster: Processing membership 208
Jun 3 12:17:40 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 3 12:17:40 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 3 12:17:40 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=192, last-seen=208, this-event=208,
last-event=0
Jun 3 12:17:40 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 3 12:17:40 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 3 12:17:40 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=208, last-seen=208, this-event=208,
last-event=0
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:17:40 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:17:40 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 12:17:40 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=8, rc=0,
cib-update=51, confirmed=true) complete ok
Jun 3 12:17:41 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_start_0 (call=9, rc=0, cib-update=52, confirmed=true)
complete ok
Jun 3 12:17:41 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=6:20:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 12:17:41 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=15:20:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=dlm:0_monitor_120000 )
Jun 3 12:17:41 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=20:20:0:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:0_start_0 )
Jun 3 12:17:41 srv-dell lrmd: [5800]: info: rsc:o2cb:0: start
Jun 3 12:17:41 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=11, rc=0, cib-update=53,
confirmed=false) complete ok
Jun 3 12:17:41 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) logd is not running
Jun 3 12:17:41 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) 2009/06/03_12:17:41 INFO: Stack glue driver not
loaded
Jun 3 12:17:41 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) logd is not running
Jun 3 12:17:41 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) 2009/06/03_12:17:41 INFO: Starting o2cb:0
Jun 3 12:17:41 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 3 12:17:41 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 3 12:17:41 srv-dell kernel: OCFS2 1.5.0
Jun 3 12:17:41 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info:
init_ais_connection: AIS connection established
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 12:17:41 srv-dell openais[5792]: [crm ] info: pcmk_notify:
Enabling node notifications for child 6261 (0x7fc95c0217f0)
Jun 3 12:17:41 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 3 12:17:41 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 3 12:17:41 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 3 12:17:41 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:17:41 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 3 12:17:41 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:17:41 srv-dell ocfs2_controld: Stored section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:17:41 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:17:41 srv-dell ocfs2_controld: Stored section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:17:41 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 3 12:17:41 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:17:41 srv-dell ocfs2_controld: setup done
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: notice: ais_dispatch:
Membership 208: quorum aquired
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:17:41 srv-dell ocfs2_controld: [6261]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=208 seen=208
proc=00000000000000000000000000053312 (new)
Jun 3 12:17:41 srv-dell ocfs2_controld: confchg called
Jun 3 12:17:41 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 3 12:17:41 srv-dell ocfs2_controld: CPG is live, we are not the
first daemon
Jun 3 12:17:41 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 3 12:17:41 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 3 12:17:41 srv-dell ocfs2_controld: Reading from section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:17:41 srv-dell ocfs2_controld: Read section
"daemon_protocol" from checkpoint "ocfs2:controld"
Jun 3 12:17:41 srv-dell ocfs2_controld: Reading from section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:17:41 srv-dell ocfs2_controld: Read section "ocfs2_protocol"
from checkpoint "ocfs2:controld"
Jun 3 12:17:41 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 3 12:17:41 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 3 12:17:41 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 3 12:17:41 srv-dell ocfs2_controld: Opening control device
Jun 3 12:17:41 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 3 12:17:41 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 3 12:17:41 srv-dell ocfs2_controld: new listening connection 4
Jun 3 12:17:42 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=10, rc=0,
cib-update=54, confirmed=false) complete ok
Jun 3 12:17:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_start_0 (call=12, rc=0, cib-update=55,
confirmed=true) complete ok
Jun 3 12:17:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=21:20:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=o2cb:0_monitor_120000 )
Jun 3 12:17:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=26:20:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:0_start_0 )
Jun 3 12:17:43 srv-dell lrmd: [5800]: info: rsc:fs:0: start
Jun 3 12:17:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=13, rc=0, cib-update=56,
confirmed=false) complete ok
Jun 3 12:17:43 srv-dell lrmd: [5800]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 12:17:43 srv-dell lrmd: [5800]: info: RA output:
(fs:0:start:stderr) 2009/06/03_12:17:43 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 12:17:43 srv-dell lrmd: [5800]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 12:17:43 srv-dell lrmd: [5800]: info: RA output:
(fs:0:start:stderr) 2009/06/03_12:17:43 ERROR: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 12:17:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:0_start_0 (call=14, rc=5, cib-update=57, confirmed=true)
complete not installed
Jun 3 12:17:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=3:21:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:0_stop_0 )
Jun 3 12:17:44 srv-dell lrmd: [5800]: info: rsc:fs:0: stop
Jun 3 12:17:44 srv-dell lrmd: [5800]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 12:17:44 srv-dell lrmd: [5800]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_12:17:44 WARNING: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 12:17:44 srv-dell lrmd: [5800]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 12:17:44 srv-dell lrmd: [5800]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_12:17:44 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:17:44 srv-dell lrmd: [5800]: info: RA output:
(fs:0:stop:stderr) /dev/sdb1: No such file or directory
Jun 3 12:17:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=15, rc=0, cib-update=58, confirmed=true)
complete ok
Jun 3 12:18:36 srv-dell kernel: No iBFT detected.
Jun 3 12:18:52 srv-dell kernel: scsi3 : iSCSI Initiator over TCP/IP
Jun 3 12:18:52 srv-dell kernel: scsi 3:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 12:18:52 srv-dell kernel: sdb: sdb1
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: [sdb] Attached SCSI disk
Jun 3 12:18:52 srv-dell kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Jun 3 12:18:52 srv-dell iscsid: connection1:0 is operational now
Jun 3 12:18:52 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:18:52 srv-dell ocfs2_controld: client msg
Jun 3 12:18:52 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 3 12:18:52 srv-dell ocfs2_controld: start_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", device "/dev/sdb1", service
"ocfs2"
Jun 3 12:18:52 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid 69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell ocfs2_controld: Starting join for group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:18:52 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:18:52 srv-dell ocfs2_controld: start_mount returns 0
Jun 3 12:18:52 srv-dell ocfs2_controld: confchg called
Jun 3 12:18:52 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 1, left 0,
joined 1
Jun 3 12:18:52 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell ocfs2_controld: This node joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell ocfs2_controld: Registering mountgroup
69AB6E551666484FB314D5B168FD6B84 with dlm_controld
Jun 3 12:18:52 srv-dell ocfs2_controld: Registering
"69AB6E551666484FB314D5B168FD6B84" with dlm_controld
Jun 3 12:18:52 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 12:18:52 srv-dell ocfs2_controld: Registration of
"69AB6E551666484FB314D5B168FD6B84" complete
Jun 3 12:18:52 srv-dell ocfs2_controld: Mountgroup
69AB6E551666484FB314D5B168FD6B84 successfully registered with
dlm_controld
Jun 3 12:18:52 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 12:18:52 srv-dell ocfs2_controld: Notified client: 1
Jun 3 12:18:52 srv-dell kernel: dlm: Using TCP for communications
Jun 3 12:18:52 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell cluster-dlm: process_uevent: kernel: add@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell cluster-dlm: process_uevent: kernel: online@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:18:52 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 3 12:18:52 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 3 12:18:52 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 1 joined 1 remove 0 failed 0
Jun 3 12:18:52 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 12:18:52 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 12:18:52 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 12:18:52 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 1 1 0 0
Jun 3 12:18:52 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 76
Jun 3 12:18:52 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 3 12:18:52 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 1
Jun 3 12:18:52 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 1
Jun 3 12:18:52 srv-dell cluster-dlm: do_sysfs: write "1438231596" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/id"
Jun 3 12:18:52 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 12:18:52 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 12:18:52 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 12:18:52 srv-dell kernel: JBD: Ignoring recovery information on journal
Jun 3 12:18:52 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 3 12:18:52 srv-dell kernel: ocfs2: Mounting device (8,17) on
(node 1543678, slot 1) with ordered data mode.
Jun 3 12:18:52 srv-dell ocfs2_controld: client msg
Jun 3 12:18:52 srv-dell ocfs2_controld: client message 1 from 5: MRESULT
Jun 3 12:18:52 srv-dell ocfs2_controld: complete_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", errcode "0", service "ocfs2"
Jun 3 12:18:52 srv-dell ocfs2_controld: client msg
Jun 3 12:18:52 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 12:18:52 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 12:18:52 srv-dell kernel: (7027,4):ocfs2_replay_journal:1311
Recovering node 2013440192 from slot 0 on device (8,17)
Jun 3 12:18:52 srv-dell ocfs2_hb_ctl[7028]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -P -d /dev/sdb1
Jun 3 12:18:52 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:18:52 srv-dell ocfs2_controld: client msg
Jun 3 12:18:52 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 3 12:18:52 srv-dell ocfs2_controld: client msg
Jun 3 12:18:52 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 12:18:52 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 12:18:52 srv-dell hald: mounted /dev/sdb1 on behalf of uid 0
Jun 3 12:18:55 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 3 12:18:55 srv-dell kernel:
(7027,4):ocfs2_begin_quota_recovery:371 Beginning quota recovery in
slot 0
Jun 3 12:18:55 srv-dell kernel:
(6258,4):ocfs2_finish_quota_recovery:555 Finishing quota recovery in
slot 0
Jun 3 12:19:07 srv-dell cluster-dlm: process_uevent: uevent:
offline@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell cluster-dlm: process_uevent: kernel: offline@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell cluster-dlm: confchg_cb: confchg for our leave
Jun 3 12:19:07 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 0
Jun 3 12:19:07 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 12:19:07 srv-dell cluster-dlm: update_dir_members: dir_member 1543678144
Jun 3 12:19:07 srv-dell cluster-dlm: set_configfs_members:
set_members rmdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 12:19:07 srv-dell cluster-dlm: set_configfs_members:
set_members lockspace rmdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:19:07 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 12:19:07 srv-dell cluster-dlm: purge_plocks: purged 0 plocks
for 1543678144
Jun 3 12:19:07 srv-dell cluster-dlm: process_uevent: uevent:
remove@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell cluster-dlm: process_uevent: kernel: remove@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell ocfs2_hb_ctl[7113]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -K -u 69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:19:07 srv-dell ocfs2_controld: client msg
Jun 3 12:19:07 srv-dell ocfs2_controld: client message 2 from 5: UNMOUNT
Jun 3 12:19:07 srv-dell ocfs2_controld: remove_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", service "ocfs2"
Jun 3 12:19:07 srv-dell ocfs2_controld: Unregistering mountgroup
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell ocfs2_controld: Unregistering
"69AB6E551666484FB314D5B168FD6B84" from dlm_controld
Jun 3 12:19:07 srv-dell ocfs2_controld: time to leave group
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell ocfs2_controld: calling LEAVE for group
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell ocfs2_controld: leaving group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:19:07 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 3 12:19:07 srv-dell ocfs2_controld: confchg called
Jun 3 12:19:07 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 0, left 1,
joined 0
Jun 3 12:19:07 srv-dell ocfs2_controld: Node 1543678144 leaves group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:19:07 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 12:19:07 srv-dell ocfs2_controld: Notified client: 1
Jun 3 12:19:07 srv-dell ocfs2_controld: client 6 fd 14 dead
Jun 3 12:19:07 srv-dell ocfs2_controld: client msg
Jun 3 12:19:07 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 12:19:07 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 12:19:07 srv-dell kernel: ocfs2: Unmounting device (8,17) on
(node 1543678)
Jun 3 12:19:07 srv-dell hald: unmounted /dev/sdb1 from
'/media/disk-1' on behalf of uid 0
Jun 3 12:19:24 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:19:30 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:19:46 srv-dell crm_resource: [7301]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:19:46 srv-dell crm_resource: [7301]: info: Invoked:
crm_resource --meta -r fs -g target-role
Jun 3 12:19:46 srv-dell crm_resource: [7301]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:19:47 srv-dell crm_resource: [7303]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:19:47 srv-dell crm_resource: [7303]: info: Invoked:
crm_resource --meta -r fs-clone -p target-role -v started
Jun 3 12:19:47 srv-dell crm_resource: [7303]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:19:47 srv-dell cib: [7304]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-36.raw
Jun 3 12:19:47 srv-dell cib: [7304]: info: write_cib_contents: Wrote
version 0.36.0 of the CIB to disk (digest:
d1826e7afd95dacba5717810d49e87ac)
Jun 3 12:19:47 srv-dell cib: [7304]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.X6aQtO (digest:
/var/lib/heartbeat/crm/cib.blF6av)
Jun 3 12:19:48 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:19:49 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:19:49 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:20:11 srv-dell crm_resource: [7396]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:20:11 srv-dell crm_resource: [7396]: info: Invoked:
crm_resource --meta -r fs -p target-role -v started
Jun 3 12:20:11 srv-dell crm_resource: [7396]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:20:11 srv-dell cib: [7397]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-37.raw
Jun 3 12:20:11 srv-dell cib: [7397]: info: write_cib_contents: Wrote
version 0.37.0 of the CIB to disk (digest:
e3e414d3fd6f9dac5a5779593425443a)
Jun 3 12:20:11 srv-dell cib: [7397]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.uFypt9 (digest:
/var/lib/heartbeat/crm/cib.FGlqrF)
Jun 3 12:20:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:20:12 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:20:12 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:20:54 srv-dell mgmtd: [5804]: info: CIB replace: resources
Jun 3 12:22:08 srv-dell crm_resource: [7854]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:22:08 srv-dell crm_resource: [7854]: info: Invoked:
crm_resource --meta -r fs -g target-role
Jun 3 12:22:08 srv-dell crm_resource: [7854]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:22:09 srv-dell crm_resource: [7856]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:22:09 srv-dell crm_resource: [7856]: info: Invoked:
crm_resource --meta -r fs-clone -p target-role -v stopped
Jun 3 12:22:09 srv-dell crm_resource: [7856]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:22:22 srv-dell crm_resource: [7911]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:22:22 srv-dell crm_resource: [7911]: info: Invoked:
crm_resource --meta -r fs -g target-role
Jun 3 12:22:22 srv-dell crm_resource: [7911]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:22:23 srv-dell crm_resource: [7913]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:22:23 srv-dell crm_resource: [7913]: info: Invoked:
crm_resource --meta -r fs-clone -p target-role -v started
Jun 3 12:22:23 srv-dell crm_resource: [7913]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:22:59 srv-dell mgmtd: [5804]: info: CIB replace: resources
Jun 3 12:22:59 srv-dell cib: [8040]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-38.raw
Jun 3 12:22:59 srv-dell cib: [8040]: info: write_cib_contents: Wrote
version 0.38.0 of the CIB to disk (digest:
a67783264f8b7c197212ae15d85d960c)
Jun 3 12:22:59 srv-dell cib: [8040]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.vhbnmL (digest:
/var/lib/heartbeat/crm/cib.XQyZHa)
Jun 3 12:22:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:22:59 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:23:01 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:23:04 srv-dell crm_resource: [8059]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:23:04 srv-dell crm_resource: [8059]: info: Invoked:
crm_resource --meta -r fs -g target-role
Jun 3 12:23:04 srv-dell crm_resource: [8059]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:23:05 srv-dell crm_resource: [8061]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:23:05 srv-dell crm_resource: [8061]: info: Invoked:
crm_resource --meta -r fs-clone -p target-role -v started
Jun 3 12:23:05 srv-dell crm_resource: [8061]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:23:13 srv-dell crm_resource: [8099]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:23:13 srv-dell crm_resource: [8099]: info: Invoked:
crm_resource --meta -r fs -p target-role -v started
Jun 3 12:23:13 srv-dell crm_resource: [8099]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:23:43 srv-dell mgmtd: [5804]: info: CIB replace: resources
Jun 3 12:23:43 srv-dell cib: [8221]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-39.raw
Jun 3 12:23:43 srv-dell cib: [8221]: info: write_cib_contents: Wrote
version 0.39.0 of the CIB to disk (digest:
f8d7de440ae86cad793c66c77fad0e2b)
Jun 3 12:23:43 srv-dell cib: [8221]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.s9syG3 (digest:
/var/lib/heartbeat/crm/cib.7xVxd2)
Jun 3 12:23:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:23:44 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:23:46 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:23:50 srv-dell crm_resource: [8260]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:23:50 srv-dell crm_resource: [8260]: info: Invoked:
crm_resource --meta -r fs -p target-role -v started
Jun 3 12:23:50 srv-dell crm_resource: [8260]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:23:50 srv-dell cib: [8261]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-40.raw
Jun 3 12:23:50 srv-dell cib: [8261]: info: write_cib_contents: Wrote
version 0.40.0 of the CIB to disk (digest:
1a9312fc4694c58fbb238e797c308e3e)
Jun 3 12:23:50 srv-dell cib: [8261]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.vkpzSq (digest:
/var/lib/heartbeat/crm/cib.IaxsPE)
Jun 3 12:23:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:23:52 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:23:52 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:27:21 srv-dell cib: [5799]: info: cib_stats: Processed 143
operations (2587.00us average, 0% utilization) in the last 10min
Jun 3 12:27:29 srv-dell pengine: [9103]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/root
Jun 3 12:27:29 srv-dell pengine: [9103]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:27:30 srv-dell crmd: [9104]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/root
Jun 3 12:27:30 srv-dell crmd: [9104]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 12:28:42 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:28:42 srv-dell cib: [9411]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-41.raw
Jun 3 12:28:42 srv-dell cib: [9411]: info: write_cib_contents: Wrote
version 0.41.0 of the CIB to disk (digest:
9c722094d444579b29f509dd7c506ad0)
Jun 3 12:28:42 srv-dell cib: [9411]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.9BrEpC (digest:
/var/lib/heartbeat/crm/cib.OMJal4)
Jun 3 12:28:42 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:28:42 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:28:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=7:27:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 12:28:43 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 12:28:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=18:27:0:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:0_stop_0 )
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: rsc:o2cb:0: stop
Jun 3 12:28:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=10, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 12:28:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=13, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) logd is not running
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) 2009/06/03_12:28:43 INFO: Stopping o2cb:0
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_12:28:43 INFO:
ifconfig eth1:0 down
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) logd is not running
Jun 3 12:28:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=16, rc=0,
cib-update=59, confirmed=true) complete ok
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) 2009/06/03_12:28:43 INFO: Stopping
ocfs2_controld.pcmk
Jun 3 12:28:43 srv-dell ocfs2_controld: Caught signal 15
Jun 3 12:28:43 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 3 12:28:43 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 3 12:28:43 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 3 12:28:43 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 3 12:28:43 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 3 12:28:43 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 3 12:28:43 srv-dell ocfs2_controld: closing cpg connection
Jun 3 12:28:43 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:28:43 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 3 12:28:43 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 3 12:28:43 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 3 12:28:43 srv-dell ocfs2_controld: closing pacemaker connection
Jun 3 12:28:43 srv-dell ocfs2_controld: [6261]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 3 12:28:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:28:43 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 3 12:28:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_stop_0 (call=17, rc=0, cib-update=60, confirmed=true)
complete ok
Jun 3 12:28:43 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:28:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=13:27:0:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:0_stop_0 )
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: rsc:dlm:0: stop
Jun 3 12:28:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=11, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:28:43 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:28:43 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/1543678144"
Jun 3 12:28:43 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 3 12:28:43 srv-dell dlm_controld.pcmk: [6082]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 3 12:28:43 srv-dell kernel: dlm: closing connection to node 1543678144
Jun 3 12:28:43 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 3 12:28:43 srv-dell lrmd: [5800]: info: RA output:
(dlm:0:stop:stderr) dlm_controld.pcmk: no process killed
Jun 3 12:28:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:28:43 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:28:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_stop_0 (call=18, rc=0, cib-update=61, confirmed=true)
complete ok
Jun 3 12:28:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=8:27:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=cloneresource_cluster_fencing_null:0_stop_0 )
Jun 3 12:28:44 srv-dell lrmd: [5800]: info:
rsc:cloneresource_cluster_fencing_null:0: stop
Jun 3 12:28:44 srv-dell lrmd: [9596]: info: Try to stop STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 12:28:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_stop_0 (call=19, rc=0,
cib-update=62, confirmed=true) complete ok
Jun 3 12:28:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:28:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:28:44 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:28:44 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:29:20 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:20 srv-dell cib: [9598]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-42.raw
Jun 3 12:29:20 srv-dell cib: [9598]: info: write_cib_contents: Wrote
version 0.42.0 of the CIB to disk (digest:
1ab7b04dfa22fba48c2824bd44a35d1b)
Jun 3 12:29:20 srv-dell cib: [9598]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.AR8YnK (digest:
/var/lib/heartbeat/crm/cib.uTVRry)
Jun 3 12:29:20 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:29:20 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:29:21 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=4:28:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_start_0 )
Jun 3 12:29:21 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:29:21 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:29:21 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=6:28:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 12:29:21 srv-dell lrmd: [5800]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 12:29:21 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=11:28:0:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:0_start_0 )
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: rsc:dlm:0: start
Jun 3 12:29:21 srv-dell lrmd: [9645]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:29:21 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 12:29:21 srv-dell stonithd: [5798]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 12:29:21 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=21, rc=0,
cib-update=63, confirmed=true) complete ok
Jun 3 12:29:21 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: RA output:
(dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Jun 3 12:29:21 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:29:21 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 12:29:21 srv-dell cluster-dlm[9692]: main: dlm_controld 1239267206
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info:
init_ais_connection: AIS connection established
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 12:29:21 srv-dell openais[5792]: [crm ] info: pcmk_notify:
Enabling node notifications for child 9692 (0x7fc95c01dda0)
Jun 3 12:29:21 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 3 12:29:21 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 3 12:29:21 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 3 12:29:21 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 3 12:29:21 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 3 12:29:21 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 2013440192
Jun 3 12:29:21 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 3 12:29:21 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 3 12:29:21 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: notice: ais_dispatch:
Membership 208: quorum aquired
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:29:21 srv-dell cluster-dlm: [9692]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=208 seen=208
proc=00000000000000000000000000053312 (new)
Jun 3 12:29:21 srv-dell cluster-dlm: update_cluster: Processing membership 208
Jun 3 12:29:21 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 3 12:29:21 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 3 12:29:21 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=192, last-seen=208, this-event=208,
last-event=0
Jun 3 12:29:21 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 3 12:29:21 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 3 12:29:21 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=208, last-seen=208, this-event=208,
last-event=0
Jun 3 12:29:21 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=20, rc=0,
cib-update=64, confirmed=true) complete ok
Jun 3 12:29:21 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=5:28:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 12:29:21 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:22 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:29:22 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:29:22 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_start_0 (call=22, rc=0, cib-update=65, confirmed=true)
complete ok
Jun 3 12:29:22 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=12:28:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=dlm:0_monitor_120000 )
Jun 3 12:29:22 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=17:28:0:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:0_start_0 )
Jun 3 12:29:22 srv-dell lrmd: [5800]: info: rsc:o2cb:0: start
Jun 3 12:29:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:22 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=24, rc=0, cib-update=66,
confirmed=false) complete ok
Jun 3 12:29:22 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) logd is not running
Jun 3 12:29:22 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) 2009/06/03_12:29:22 INFO: Stack glue driver not
loaded
Jun 3 12:29:22 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=23, rc=0,
cib-update=67, confirmed=false) complete ok
Jun 3 12:29:22 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) logd is not running
Jun 3 12:29:22 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:start:stderr) 2009/06/03_12:29:22 INFO: Starting o2cb:0
Jun 3 12:29:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:22 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 3 12:29:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:22 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 3 12:29:22 srv-dell kernel: OCFS2 1.5.0
Jun 3 12:29:22 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info:
init_ais_connection: AIS connection established
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 12:29:22 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 3 12:29:22 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 3 12:29:22 srv-dell openais[5792]: [crm ] info: pcmk_notify:
Enabling node notifications for child 9837 (0x7fc95c022a70)
Jun 3 12:29:22 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 3 12:29:22 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 3 12:29:22 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:29:22 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:29:22 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 3 12:29:22 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:29:22 srv-dell ocfs2_controld: setup done
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: notice: ais_dispatch:
Membership 208: quorum aquired
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:29:22 srv-dell ocfs2_controld: [9837]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=208 seen=208
proc=00000000000000000000000000053312 (new)
Jun 3 12:29:22 srv-dell ocfs2_controld: confchg called
Jun 3 12:29:22 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 3 12:29:22 srv-dell ocfs2_controld: CPG is live, we are not the
first daemon
Jun 3 12:29:22 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 3 12:29:22 srv-dell ocfs2_controld: Reading from section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Read section
"daemon_protocol" from checkpoint "ocfs2:controld"
Jun 3 12:29:22 srv-dell ocfs2_controld: Reading from section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:29:22 srv-dell ocfs2_controld: Read section "ocfs2_protocol"
from checkpoint "ocfs2:controld"
Jun 3 12:29:22 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 3 12:29:22 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 3 12:29:22 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 3 12:29:22 srv-dell ocfs2_controld: Opening control device
Jun 3 12:29:22 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 3 12:29:22 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 3 12:29:22 srv-dell ocfs2_controld: new listening connection 4
Jun 3 12:29:23 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:29:23 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:29:24 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_start_0 (call=25, rc=0, cib-update=68,
confirmed=true) complete ok
Jun 3 12:29:24 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=18:28:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=o2cb:0_monitor_120000 )
Jun 3 12:29:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:24 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=26, rc=0, cib-update=69,
confirmed=false) complete ok
Jun 3 12:29:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:29:25 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:29:25 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:30:45 srv-dell lrmd: [5800]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 12:31:08 srv-dell lrmd: [5800]: ERROR: get_resource_meta:
pclose failed: Resource temporarily unavailable
Jun 3 12:31:08 srv-dell lrmd: [5800]: WARN: on_msg_get_metadata:
empty metadata for ocf::heartbeat::controld.
Jun 3 12:31:08 srv-dell lrmd: [5800]: WARN: G_SIG_dispatch: Dispatch
function for SIGCHLD was delayed 200 ms (> 100 ms) before being called
(GSource: 0x615a60)
Jun 3 12:31:08 srv-dell lrmd: [5800]: info: G_SIG_dispatch: started
at 1718154173 should have started at 1718154153
Jun 3 12:31:08 srv-dell mgmtd: [5804]: ERROR:
lrm_get_rsc_type_metadata(575): got a return code HA_FAIL from a reply
message of rmetadata with function get_ret_from_msg.
Jun 3 12:36:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:36:25 srv-dell cib: [11684]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-43.raw
Jun 3 12:36:25 srv-dell cib: [11684]: info: write_cib_contents: Wrote
version 0.43.0 of the CIB to disk (digest:
2dca94fae4a7aeb3b8821a9a3dd2f0f9)
Jun 3 12:36:25 srv-dell cib: [11684]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.NLBUnA (digest:
/var/lib/heartbeat/crm/cib.XKYVPj)
Jun 3 12:36:25 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op primitiveresource_cluster_ip_monitor_5000 from
3:29:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244046985-25
Jun 3 12:36:25 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op dlm:0_monitor_120000 from
2:29:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244046985-26
Jun 3 12:36:25 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op o2cb:0_monitor_120000 from
1:29:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244046985-27
Jun 3 12:36:25 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:36:25 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=23, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 12:36:25 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=24, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:36:25 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=26, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:36:25 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:36:25 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:36:25 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:36:25 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:36:26 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:36:34 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:36:34 srv-dell cib: [11685]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-44.raw
Jun 3 12:36:34 srv-dell cib: [11685]: info: write_cib_contents: Wrote
version 0.44.0 of the CIB to disk (digest:
ffc3113f092c738ffb946009517a7586)
Jun 3 12:36:34 srv-dell cib: [11685]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.d1m9M9 (digest:
/var/lib/heartbeat/crm/cib.6qQ5ed)
Jun 3 12:36:34 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:36:35 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:36:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:36:43 srv-dell cib: [11686]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-45.raw
Jun 3 12:36:43 srv-dell cib: [11686]: info: write_cib_contents: Wrote
version 0.45.0 of the CIB to disk (digest:
e68a6c0406fc3615d3e7d044a0285746)
Jun 3 12:36:43 srv-dell cib: [11686]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.fLHdtJ (digest:
/var/lib/heartbeat/crm/cib.NPTza5)
Jun 3 12:36:43 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:36:43 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:37:03 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:37:03 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:37:03 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:37:21 srv-dell cib: [5799]: info: cib_stats: Processed 62
operations (2419.00us average, 0% utilization) in the last 10min
Jun 3 12:37:32 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:38:00 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:38:29 srv-dell crm_resource: [11688]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:38:29 srv-dell crm_resource: [11688]: info: Invoked:
crm_resource --meta -r primitiveresource_cluster_ip -p target-role -v
started
Jun 3 12:38:29 srv-dell crm_resource: [11688]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:38:29 srv-dell cib: [11689]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-46.raw
Jun 3 12:38:29 srv-dell cib: [11689]: info: write_cib_contents: Wrote
version 0.46.0 of the CIB to disk (digest:
862e2a021fb0dc5b9af67bbfe273da73)
Jun 3 12:38:29 srv-dell cib: [11689]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.8jncdV (digest:
/var/lib/heartbeat/crm/cib.NnHTA0)
Jun 3 12:38:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:38:30 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:38:30 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:38:45 srv-dell crmd: [5803]: info: do_lrm_invoke: Removing
resource primitiveresource_cluster_ip from the LRM
Jun 3 12:38:45 srv-dell mgmtd: [5804]: info: Delete fail-count for
primitiveresource_cluster_ip from srv-dell
Jun 3 12:38:45 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op primitiveresource_cluster_ip_delete_0 from mgmtd-5804:
lrm_invoke-lrmd-1244047125-29
Jun 3 12:38:45 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:38:45 srv-dell cib: [11690]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-47.raw
Jun 3 12:38:45 srv-dell cib: [11690]: info: write_cib_contents: Wrote
version 0.47.0 of the CIB to disk (digest:
28047b6a34b2a9f715f00ce99ce12cfa)
Jun 3 12:38:45 srv-dell cib: [11690]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.pMbkSp (digest:
/var/lib/heartbeat/crm/cib.J1y023)
Jun 3 12:38:46 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=6:34:7:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 12:38:46 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 12:38:46 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=27, rc=0,
cib-update=80, confirmed=true) complete ok
Jun 3 12:38:47 srv-dell mgmtd: [5804]: info: Delete fail-count for
primitiveresource_cluster_ip from srv-lenovo
Jun 3 12:38:47 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11709 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:38:47 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:38:47 srv-dell openais[5792]: [MAIN ] Msg[86]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=907): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:38:47 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:38:47 srv-dell cib: [11710]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-48.raw
Jun 3 12:38:47 srv-dell cib: [11710]: info: write_cib_contents: Wrote
version 0.48.0 of the CIB to disk (digest:
92eae8cf6fff554a63bc4073557e493a)
Jun 3 12:38:47 srv-dell cib: [11710]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.14a9oy (digest:
/var/lib/heartbeat/crm/cib.Lpz4Yg)
Jun 3 12:38:49 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:38:49 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:38:49 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:38:49 srv-dell cib: [11711]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-49.raw
Jun 3 12:38:49 srv-dell cib: [11711]: info: write_cib_contents: Wrote
version 0.49.0 of the CIB to disk (digest:
711973fbef63fb038a6c53119b0fe15f)
Jun 3 12:38:49 srv-dell cib: [11711]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.SwgaEz (digest:
/var/lib/heartbeat/crm/cib.x0Raym)
Jun 3 12:38:49 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:38:49 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:38:49 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:38:49 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:38:49 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:38:49 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:39:01 srv-dell mgmtd: [5804]: info: Delete fail-count for
cloneresource_cluster_fencing_null:0 from srv-dell
Jun 3 12:39:01 srv-dell crmd: [5803]: info: do_lrm_invoke: Removing
resource cloneresource_cluster_fencing_null:0 from the LRM
Jun 3 12:39:01 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op cloneresource_cluster_fencing_null:0_delete_0 from
mgmtd-5804: lrm_invoke-lrmd-1244047141-32
Jun 3 12:39:01 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:02 srv-dell cib: [11712]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-50.raw
Jun 3 12:39:02 srv-dell cib: [11712]: info: write_cib_contents: Wrote
version 0.50.0 of the CIB to disk (digest:
bbb55bab078a4f800f0e2a1eadf035d4)
Jun 3 12:39:02 srv-dell cib: [11712]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.0GOeJp (digest:
/var/lib/heartbeat/crm/cib.4MfMKD)
Jun 3 12:39:03 srv-dell lrmd: [5800]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 12:39:03 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=6:38:7:7c754d7f-6d3c-4527-b36c-979453669faf
op=cloneresource_cluster_fencing_null:0_monitor_0 )
Jun 3 12:39:03 srv-dell lrmd: [5800]: info:
rsc:cloneresource_cluster_fencing_null:0: monitor
Jun 3 12:39:03 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_monitor_0 (call=28,
rc=0, cib-update=88, confirmed=true) complete ok
Jun 3 12:39:03 srv-dell mgmtd: [5804]: info: Delete fail-count for
cloneresource_cluster_fencing_null:0 from srv-lenovo
Jun 3 12:39:03 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:03 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11715 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:03 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:03 srv-dell openais[5792]: [MAIN ] Msg[92]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=923): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:06 srv-dell crmd: [5803]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244047146-13" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5804_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5804" >
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive
id="cloneresource_cluster_fencing_null:1" />
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 12:39:06 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 12:39:06 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op cloneresource_cluster_fencing_null:1_delete_0 from
mgmtd-5804: lrm_invoke-lrmd-1244047146-35
Jun 3 12:39:06 srv-dell mgmtd: [5804]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-dell
Jun 3 12:39:06 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:06 srv-dell cib: [11716]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-51.raw
Jun 3 12:39:06 srv-dell cib: [11716]: info: write_cib_contents: Wrote
version 0.51.0 of the CIB to disk (digest:
fc768182459cecd32be85082fa89fe8b)
Jun 3 12:39:06 srv-dell cib: [11716]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.oxU85y (digest:
/var/lib/heartbeat/crm/cib.Ck4NQV)
Jun 3 12:39:08 srv-dell mgmtd: [5804]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-lenovo
Jun 3 12:39:08 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:08 srv-dell cib: [11717]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-52.raw
Jun 3 12:39:08 srv-dell cib: [11717]: info: write_cib_contents: Wrote
version 0.52.0 of the CIB to disk (digest:
619c1d79a38ded743e5a5e947f072eec)
Jun 3 12:39:08 srv-dell cib: [11717]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.jikTKD (digest:
/var/lib/heartbeat/crm/cib.2ZCqW4)
Jun 3 12:39:08 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11718 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:08 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:08 srv-dell openais[5792]: [MAIN ] Msg[95]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=923): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:08 srv-dell cib: [11719]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-53.raw
Jun 3 12:39:08 srv-dell cib: [11719]: info: write_cib_contents: Wrote
version 0.53.0 of the CIB to disk (digest:
f0837b2e6a5e99b37806439882af4d9c)
Jun 3 12:39:08 srv-dell cib: [11719]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.z7lCC1 (digest:
/var/lib/heartbeat/crm/cib.2xIzrt)
Jun 3 12:39:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:39:10 srv-dell cib: [11720]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-54.raw
Jun 3 12:39:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:39:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:39:10 srv-dell cib: [11720]: info: write_cib_contents: Wrote
version 0.54.0 of the CIB to disk (digest:
89b447cf9c65c43982a14c2e0f116c23)
Jun 3 12:39:10 srv-dell cib: [11720]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.EK59IM (digest:
/var/lib/heartbeat/crm/cib.euguki)
Jun 3 12:39:10 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:10 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:10 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:10 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:10 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:10 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:39:10 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:39:18 srv-dell crmd: [5803]: info: do_lrm_invoke: Removing
resource dlm:0 from the LRM
Jun 3 12:39:18 srv-dell mgmtd: [5804]: info: Delete fail-count for
dlm:0 from srv-dell
Jun 3 12:39:18 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op dlm:0_delete_0 from mgmtd-5804:
lrm_invoke-lrmd-1244047158-38
Jun 3 12:39:18 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:18 srv-dell cib: [11721]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-55.raw
Jun 3 12:39:18 srv-dell cib: [11721]: info: write_cib_contents: Wrote
version 0.55.0 of the CIB to disk (digest:
42063ed3d312303d1a7b2660ec624af0)
Jun 3 12:39:18 srv-dell cib: [11721]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.vy4Tbe (digest:
/var/lib/heartbeat/crm/cib.kXL1z1)
Jun 3 12:39:19 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=6:44:7:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:0_monitor_0 )
Jun 3 12:39:19 srv-dell lrmd: [5800]: info: rsc:dlm:0: monitor
Jun 3 12:39:19 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_0 (call=29, rc=0, cib-update=98,
confirmed=true) complete ok
Jun 3 12:39:20 srv-dell mgmtd: [5804]: info: Delete fail-count for
dlm:0 from srv-lenovo
Jun 3 12:39:20 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11735 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:20 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:20 srv-dell openais[5792]: [MAIN ] Msg[101]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=861): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:20 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:20 srv-dell cib: [11736]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-56.raw
Jun 3 12:39:20 srv-dell cib: [11736]: info: write_cib_contents: Wrote
version 0.56.0 of the CIB to disk (digest:
da60aadac1a7146093fbd8c62a1fa0f5)
Jun 3 12:39:20 srv-dell cib: [11736]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.u9pHqm (digest:
/var/lib/heartbeat/crm/cib.02exde)
Jun 3 12:39:22 srv-dell crmd: [5803]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244047162-25" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5804_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5804" >
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive id="dlm:1" />
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 12:39:22 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 12:39:22 srv-dell mgmtd: [5804]: info: Delete fail-count for
dlm:1 from srv-dell
Jun 3 12:39:22 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op dlm:1_delete_0 from mgmtd-5804:
lrm_invoke-lrmd-1244047162-41
Jun 3 12:39:22 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:22 srv-dell cib: [11737]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-57.raw
Jun 3 12:39:22 srv-dell cib: [11737]: info: write_cib_contents: Wrote
version 0.57.0 of the CIB to disk (digest:
61d68289c820763eec7414fb2d9a88cc)
Jun 3 12:39:22 srv-dell cib: [11737]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.x8xeRn (digest:
/var/lib/heartbeat/crm/cib.KBK1Yj)
Jun 3 12:39:24 srv-dell mgmtd: [5804]: info: Delete fail-count for
dlm:1 from srv-lenovo
Jun 3 12:39:24 srv-dell cib: [11738]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-58.raw
Jun 3 12:39:24 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:24 srv-dell cib: [11738]: info: write_cib_contents: Wrote
version 0.58.0 of the CIB to disk (digest:
35421b321c93f378593e28e36ce95148)
Jun 3 12:39:24 srv-dell cib: [11738]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ESmoUt (digest:
/var/lib/heartbeat/crm/cib.YO7rru)
Jun 3 12:39:24 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11739 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:24 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:24 srv-dell openais[5792]: [MAIN ] Msg[105]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=861): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:26 srv-dell cib: [11740]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-59.raw
Jun 3 12:39:26 srv-dell cib: [11740]: info: write_cib_contents: Wrote
version 0.59.0 of the CIB to disk (digest:
698b4a8be14d1a84436f3b0efe1e0bc8)
Jun 3 12:39:26 srv-dell cib: [11740]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ah7tHA (digest:
/var/lib/heartbeat/crm/cib.8jZTEF)
Jun 3 12:39:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:26 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:39:26 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:39:26 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:39:34 srv-dell mgmtd: [5804]: info: Delete fail-count for
o2cb:0 from srv-dell
Jun 3 12:39:34 srv-dell crmd: [5803]: info: do_lrm_invoke: Removing
resource o2cb:0 from the LRM
Jun 3 12:39:34 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op o2cb:0_delete_0 from mgmtd-5804:
lrm_invoke-lrmd-1244047174-44
Jun 3 12:39:34 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:34 srv-dell cib: [11741]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-60.raw
Jun 3 12:39:34 srv-dell cib: [11741]: info: write_cib_contents: Wrote
version 0.60.0 of the CIB to disk (digest:
eb117f4a4277900a24f9da094b77eedc)
Jun 3 12:39:34 srv-dell cib: [11741]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.fPSFXF (digest:
/var/lib/heartbeat/crm/cib.mVUAZ1)
Jun 3 12:39:34 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=5:50:7:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:0_monitor_0 )
Jun 3 12:39:34 srv-dell lrmd: [5800]: info: rsc:o2cb:0: monitor
Jun 3 12:39:34 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_0 (call=30, rc=0, cib-update=108,
confirmed=true) complete ok
Jun 3 12:39:36 srv-dell mgmtd: [5804]: info: Delete fail-count for
o2cb:0 from srv-lenovo
Jun 3 12:39:36 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11761 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:36 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:36 srv-dell openais[5792]: [MAIN ] Msg[110]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=863): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:36 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:36 srv-dell cib: [11762]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-61.raw
Jun 3 12:39:36 srv-dell cib: [11762]: info: write_cib_contents: Wrote
version 0.61.0 of the CIB to disk (digest:
78cb79406315b71b081ca917070b613f)
Jun 3 12:39:36 srv-dell cib: [11762]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.W0qnvO (digest:
/var/lib/heartbeat/crm/cib.u0KyWe)
Jun 3 12:39:38 srv-dell crmd: [5803]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 12:39:38 srv-dell mgmtd: [5804]: info: Delete fail-count for
o2cb:1 from srv-dell
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244047178-37" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5804_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5804" >
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive id="o2cb:1" />
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 12:39:38 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 12:39:38 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op o2cb:1_delete_0 from mgmtd-5804:
lrm_invoke-lrmd-1244047178-47
Jun 3 12:39:38 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:38 srv-dell cib: [11763]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-62.raw
Jun 3 12:39:38 srv-dell cib: [11763]: info: write_cib_contents: Wrote
version 0.62.0 of the CIB to disk (digest:
62a7303b3af6e53a64e62e392b2fc2e0)
Jun 3 12:39:38 srv-dell cib: [11763]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.LOW3TR (digest:
/var/lib/heartbeat/crm/cib.MgAEFm)
Jun 3 12:39:40 srv-dell mgmtd: [5804]: info: Delete fail-count for
o2cb:1 from srv-lenovo
Jun 3 12:39:40 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:40 srv-dell cib: [11764]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-63.raw
Jun 3 12:39:40 srv-dell cib: [11764]: info: write_cib_contents: Wrote
version 0.63.0 of the CIB to disk (digest:
a84929b9d6953c3019f53b263ef1529b)
Jun 3 12:39:40 srv-dell cib: [11764]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.AaYQBY (digest:
/var/lib/heartbeat/crm/cib.eDdMNx)
Jun 3 12:39:40 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11765 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:40 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:40 srv-dell openais[5792]: [MAIN ] Msg[114]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=863): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:42 srv-dell cib: [11766]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-64.raw
Jun 3 12:39:42 srv-dell cib: [11766]: info: write_cib_contents: Wrote
version 0.64.0 of the CIB to disk (digest:
dd28211a411e45dffc2169df506afeef)
Jun 3 12:39:42 srv-dell cib: [11766]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.sc0Zr2 (digest:
/var/lib/heartbeat/crm/cib.InIa3F)
Jun 3 12:39:42 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:42 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:39:42 srv-dell mgmtd: [5804]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 12:39:46 srv-dell crmd: [5803]: info: do_lrm_invoke: Removing
resource fs:0 from the LRM
Jun 3 12:39:46 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op fs:0_delete_0 from mgmtd-5804:
lrm_invoke-lrmd-1244047186-50
Jun 3 12:39:46 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:46 srv-dell mgmtd: [5804]: info: Delete fail-count for
fs:0 from srv-dell
Jun 3 12:39:46 srv-dell cib: [11767]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-65.raw
Jun 3 12:39:46 srv-dell cib: [11767]: info: write_cib_contents: Wrote
version 0.65.0 of the CIB to disk (digest:
0b0d868ad59539232ae058b0a89f5ad7)
Jun 3 12:39:46 srv-dell cib: [11767]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.FdhimO (digest:
/var/lib/heartbeat/crm/cib.Q2m1KB)
Jun 3 12:39:47 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=4:56:7:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:0_monitor_0 )
Jun 3 12:39:47 srv-dell lrmd: [5800]: info: rsc:fs:0: monitor
Jun 3 12:39:47 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=31, rc=7, cib-update=118,
confirmed=true) complete not running
Jun 3 12:39:48 srv-dell mgmtd: [5804]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 12:39:48 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11803 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:48 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:48 srv-dell openais[5792]: [MAIN ] Msg[119]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:48 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:48 srv-dell cib: [11804]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-66.raw
Jun 3 12:39:48 srv-dell cib: [11804]: info: write_cib_contents: Wrote
version 0.66.0 of the CIB to disk (digest:
5d87ba064a0e2b6b17a7007798b0becc)
Jun 3 12:39:48 srv-dell cib: [11804]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.sDyeiV (digest:
/var/lib/heartbeat/crm/cib.yy8i7M)
Jun 3 12:39:50 srv-dell crmd: [5803]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 12:39:50 srv-dell mgmtd: [5804]: info: Delete fail-count for
fs:1 from srv-dell
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244047190-49" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5804_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5804" >
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive id="fs:1" />
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 12:39:50 srv-dell crmd: [5803]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 12:39:50 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op fs:1_delete_0 from mgmtd-5804:
lrm_invoke-lrmd-1244047190-53
Jun 3 12:39:50 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:50 srv-dell cib: [11805]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-67.raw
Jun 3 12:39:50 srv-dell cib: [11805]: info: write_cib_contents: Wrote
version 0.67.0 of the CIB to disk (digest:
8cbffa1dbb76064bf673ca3fae732c04)
Jun 3 12:39:50 srv-dell cib: [11805]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.FFUeJW (digest:
/var/lib/heartbeat/crm/cib.yfQhTS)
Jun 3 12:39:52 srv-dell mgmtd: [5804]: info: Delete fail-count for
fs:1 from srv-lenovo
Jun 3 12:39:52 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:39:52 srv-dell cib: [11806]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-68.raw
Jun 3 12:39:52 srv-dell cib: [11806]: info: write_cib_contents: Wrote
version 0.68.0 of the CIB to disk (digest:
6ea299e3b6420a110d20b62b42b9388c)
Jun 3 12:39:52 srv-dell cib: [11806]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.4S8Cw3 (digest:
/var/lib/heartbeat/crm/cib.wED263)
Jun 3 12:39:53 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 11807 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:39:53 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:39:53 srv-dell openais[5792]: [MAIN ] Msg[123]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 12:39:54 srv-dell cib: [11808]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-69.raw
Jun 3 12:39:54 srv-dell cib: [11808]: info: write_cib_contents: Wrote
version 0.69.0 of the CIB to disk (digest:
1cc48ef6725ea82468917e758f180696)
Jun 3 12:39:54 srv-dell cib: [11808]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.QeWpS9 (digest:
/var/lib/heartbeat/crm/cib.Q8sDSe)
Jun 3 12:39:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:39:55 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:40:07 srv-dell crm_resource: [11810]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:40:07 srv-dell crm_resource: [11810]: info: Invoked:
crm_resource -R -H srv-dell
Jun 3 12:40:07 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:40:24 srv-dell crm_attribute: [11812]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:40:24 srv-dell crm_attribute: [11812]: info: Invoked:
crm_attribute -t op_defaults -n record-pending -v true
Jun 3 12:40:24 srv-dell cib: [11813]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-70.raw
Jun 3 12:40:24 srv-dell cib: [11813]: info: write_cib_contents: Wrote
version 0.70.0 of the CIB to disk (digest:
fafaf1f7d864a9b665162d3b43d238c0)
Jun 3 12:40:24 srv-dell cib: [11813]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Dzmgp4 (digest:
/var/lib/heartbeat/crm/cib.cM7kpc)
Jun 3 12:40:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:40:25 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:40:56 srv-dell crm_resource: [11815]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:40:56 srv-dell crm_resource: [11815]: info: Invoked:
crm_resource --meta -r primitiveresource_cluster_ip -p target-role -v
started
Jun 3 12:41:22 srv-dell crm_resource: [11817]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:22 srv-dell crm_resource: [11817]: info: Invoked:
crm_resource --meta -r cloneresource_cluster_fencing -p is-managed -v
true
Jun 3 12:41:29 srv-dell crm_resource: [11819]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:29 srv-dell crm_resource: [11819]: info: Invoked:
crm_resource --meta -r cloneresource_cluster_fencing_null -p
is-managed -v true
Jun 3 12:41:30 srv-dell cib: [11820]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-71.raw
Jun 3 12:41:30 srv-dell cib: [11820]: info: write_cib_contents: Wrote
version 0.71.0 of the CIB to disk (digest:
e89274c7c882a27d2ae9a867de9a37de)
Jun 3 12:41:30 srv-dell cib: [11820]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.wZH9BA (digest:
/var/lib/heartbeat/crm/cib.M8GeX0)
Jun 3 12:41:31 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:41:31 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:41:33 srv-dell crm_resource: [11822]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:33 srv-dell crm_resource: [11822]: info: Invoked:
crm_resource --meta -r cloneresource_cluster_fencing_null -p
is-managed -v true
Jun 3 12:41:38 srv-dell crm_resource: [11824]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:38 srv-dell crm_resource: [11824]: info: Invoked:
crm_resource --meta -r dlm-clone -p is-managed -v true
Jun 3 12:41:38 srv-dell cib: [11825]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-72.raw
Jun 3 12:41:38 srv-dell cib: [11825]: info: write_cib_contents: Wrote
version 0.72.0 of the CIB to disk (digest:
81581a164dd4794b120453318c0e9ca9)
Jun 3 12:41:38 srv-dell cib: [11825]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.D6nsvg (digest:
/var/lib/heartbeat/crm/cib.8JSsZY)
Jun 3 12:41:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:41:39 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:41:42 srv-dell crm_resource: [11827]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:42 srv-dell crm_resource: [11827]: info: Invoked:
crm_resource --meta -r dlm -p is-managed -v true
Jun 3 12:41:42 srv-dell cib: [11828]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-73.raw
Jun 3 12:41:42 srv-dell cib: [11828]: info: write_cib_contents: Wrote
version 0.73.0 of the CIB to disk (digest:
fae4f5aa0e77ca3558f00f08071bab25)
Jun 3 12:41:42 srv-dell cib: [11828]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.qOxktU (digest:
/var/lib/heartbeat/crm/cib.GLQ5zM)
Jun 3 12:41:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:41:43 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:41:46 srv-dell crm_resource: [11830]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:46 srv-dell crm_resource: [11830]: info: Invoked:
crm_resource --meta -r dlm -p is-managed -v true
Jun 3 12:41:50 srv-dell crm_resource: [11832]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:50 srv-dell crm_resource: [11832]: info: Invoked:
crm_resource --meta -r o2cb-clone -p is-managed -v true
Jun 3 12:41:50 srv-dell cib: [11833]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-74.raw
Jun 3 12:41:50 srv-dell cib: [11833]: info: write_cib_contents: Wrote
version 0.74.0 of the CIB to disk (digest:
f80819d5a9483b30304ae66658b6fb4c)
Jun 3 12:41:50 srv-dell cib: [11833]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.rCNISb (digest:
/var/lib/heartbeat/crm/cib.kNNzpl)
Jun 3 12:41:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:41:51 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:41:54 srv-dell crm_resource: [11835]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:54 srv-dell crm_resource: [11835]: info: Invoked:
crm_resource --meta -r o2cb -p is-managed -v true
Jun 3 12:41:54 srv-dell cib: [11836]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-75.raw
Jun 3 12:41:54 srv-dell cib: [11836]: info: write_cib_contents: Wrote
version 0.75.0 of the CIB to disk (digest:
0ac7bb1cb997322f5bfe2949e42d873a)
Jun 3 12:41:54 srv-dell cib: [11836]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.2eStF9 (digest:
/var/lib/heartbeat/crm/cib.KnT4vr)
Jun 3 12:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:41:55 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:41:58 srv-dell crm_resource: [11838]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:41:58 srv-dell crm_resource: [11838]: info: Invoked:
crm_resource --meta -r o2cb -p is-managed -v true
Jun 3 12:42:36 srv-dell crm_attribute: [11840]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:42:36 srv-dell crm_attribute: [11840]: info: Invoked:
crm_attribute -t crm_config -D -n maintenance-mode
Jun 3 12:42:36 srv-dell cib: [11841]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-76.raw
Jun 3 12:42:36 srv-dell cib: [11841]: info: write_cib_contents: Wrote
version 0.76.0 of the CIB to disk (digest:
889b69f0a3948d38b2c3805352bde9b0)
Jun 3 12:42:36 srv-dell cib: [11841]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.dy3rRJ (digest:
/var/lib/heartbeat/crm/cib.8D6Trv)
Jun 3 12:42:37 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=7:68:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 12:42:37 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=17:68:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=dlm:0_monitor_120000 )
Jun 3 12:42:37 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=27:68:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=o2cb:0_monitor_120000 )
Jun 3 12:42:37 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=33, rc=0, cib-update=125,
confirmed=false) complete ok
Jun 3 12:42:37 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=34, rc=0, cib-update=126,
confirmed=false) complete ok
Jun 3 12:42:37 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=38:68:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:1_start_0 )
Jun 3 12:42:37 srv-dell lrmd: [5800]: info: rsc:fs:1: start
Jun 3 12:42:37 srv-dell lrmd: [5800]: info: RA output:
(fs:1:start:stderr) logd is not running
Jun 3 12:42:37 srv-dell lrmd: [5800]: info: RA output:
(fs:1:start:stderr) 2009/06/03_12:42:37 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 12:42:37 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:42:37 srv-dell ocfs2_controld: client msg
Jun 3 12:42:37 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 3 12:42:37 srv-dell ocfs2_controld: start_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", device "/dev/sdb1", service
"ocfs2"
Jun 3 12:42:37 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid 69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell ocfs2_controld: Starting join for group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:42:37 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:42:37 srv-dell ocfs2_controld: start_mount returns 0
Jun 3 12:42:37 srv-dell ocfs2_controld: confchg called
Jun 3 12:42:37 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 1, left 0,
joined 1
Jun 3 12:42:37 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell ocfs2_controld: This node joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell ocfs2_controld: Registering mountgroup
69AB6E551666484FB314D5B168FD6B84 with dlm_controld
Jun 3 12:42:37 srv-dell ocfs2_controld: Registering
"69AB6E551666484FB314D5B168FD6B84" with dlm_controld
Jun 3 12:42:37 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 12:42:37 srv-dell ocfs2_controld: Registration of
"69AB6E551666484FB314D5B168FD6B84" complete
Jun 3 12:42:37 srv-dell ocfs2_controld: Mountgroup
69AB6E551666484FB314D5B168FD6B84 successfully registered with
dlm_controld
Jun 3 12:42:37 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 12:42:37 srv-dell ocfs2_controld: Notified client: 1
Jun 3 12:42:37 srv-dell kernel: dlm: Using TCP for communications
Jun 3 12:42:37 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell cluster-dlm: process_uevent: kernel: add@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell cluster-dlm: process_uevent: kernel: online@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:42:37 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 3 12:42:37 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 3 12:42:37 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 1 joined 1 remove 0 failed 0
Jun 3 12:42:37 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 12:42:37 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 12:42:37 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 12:42:37 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 1 1 0 0
Jun 3 12:42:37 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 76
Jun 3 12:42:37 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 3 12:42:37 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 1
Jun 3 12:42:37 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 1
Jun 3 12:42:37 srv-dell cluster-dlm: do_sysfs: write "1438231596" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/id"
Jun 3 12:42:37 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 12:42:37 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 12:42:37 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 12:42:37 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 3 12:42:37 srv-dell kernel: ocfs2: Mounting device (8,17) on
(node 1543678, slot 0) with ordered data mode.
Jun 3 12:42:37 srv-dell ocfs2_controld: client msg
Jun 3 12:42:37 srv-dell ocfs2_controld: client message 1 from 5: MRESULT
Jun 3 12:42:37 srv-dell ocfs2_controld: complete_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", errcode "0", service "ocfs2"
Jun 3 12:42:37 srv-dell ocfs2_controld: client msg
Jun 3 12:42:37 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 12:42:37 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 12:42:37 srv-dell ocfs2_hb_ctl[11944]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -P -d /dev/sdb1
Jun 3 12:42:37 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:42:37 srv-dell ocfs2_controld: client msg
Jun 3 12:42:37 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 3 12:42:37 srv-dell ocfs2_controld: client msg
Jun 3 12:42:37 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 12:42:37 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 12:42:37 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:1_start_0 (call=35, rc=0, cib-update=127, confirmed=true)
complete ok
Jun 3 12:42:37 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=39:68:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=fs:1_monitor_120000 )
Jun 3 12:42:37 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:1_monitor_120000 (call=36, rc=0, cib-update=128,
confirmed=false) complete ok
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:37 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:42:38 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=32, rc=0,
cib-update=129, confirmed=false) complete ok
Jun 3 12:42:38 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:42:38 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:44:30 srv-dell crm_resource: [12401]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:44:30 srv-dell crm_resource: [12401]: info: Invoked:
crm_resource -M -r fs -H srv-lenovo
Jun 3 12:46:04 srv-dell lrmd: [5800]: ERROR: get_resource_meta:
pclose failed: Resource temporarily unavailable
Jun 3 12:46:04 srv-dell lrmd: [5800]: WARN: on_msg_get_metadata:
empty metadata for ocf::heartbeat::controld.
Jun 3 12:46:04 srv-dell lrmd: [5800]: WARN: G_SIG_dispatch: Dispatch
function for SIGCHLD was delayed 200 ms (> 100 ms) before being called
(GSource: 0x615a60)
Jun 3 12:46:04 srv-dell lrmd: [5800]: info: G_SIG_dispatch: started
at 1718243801 should have started at 1718243781
Jun 3 12:46:04 srv-dell mgmtd: [5804]: ERROR:
lrm_get_rsc_type_metadata(575): got a return code HA_FAIL from a reply
message of rmetadata with function get_ret_from_msg.
Jun 3 12:46:36 srv-dell crm_resource: [12944]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:46:36 srv-dell crm_resource: [12944]: info: Invoked:
crm_resource -M -r fs -H srv-lenovo
Jun 3 12:47:21 srv-dell cib: [5799]: info: cib_stats: Processed 316
operations (601.00us average, 0% utilization) in the last 10min
Jun 3 12:48:14 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:48:14 srv-dell cib: [13354]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-77.raw
Jun 3 12:48:14 srv-dell cib: [13354]: info: write_cib_contents: Wrote
version 0.77.0 of the CIB to disk (digest:
41fc9aad5c63bc8e8e6b2bcf7cc2e020)
Jun 3 12:48:14 srv-dell cib: [13354]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.iFVCGM (digest:
/var/lib/heartbeat/crm/cib.UAYzNo)
Jun 3 12:48:14 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:48:14 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:48:14 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:48:34 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:48:34 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:48:35 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:48:35 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:48:35 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:48:36 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:48:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:48:55 srv-dell cib: [13571]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-78.raw
Jun 3 12:48:55 srv-dell cib: [13571]: info: write_cib_contents: Wrote
version 0.78.0 of the CIB to disk (digest:
1e772bda36aeae546cae36cacd5a2f62)
Jun 3 12:48:55 srv-dell cib: [13571]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.w5Y2SD (digest:
/var/lib/heartbeat/crm/cib.zafIYH)
Jun 3 12:48:55 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:49:20 srv-dell crm_resource: [13663]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:49:20 srv-dell crm_resource: [13663]: info: Invoked:
crm_resource -R -H srv-lenovo
Jun 3 12:49:21 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:21 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:49:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:22 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:49:49 srv-dell crm_attribute: [13773]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:49:49 srv-dell crm_attribute: [13773]: info: Invoked:
crm_attribute -t crm_config -n maintenance-mode -v true
Jun 3 12:49:49 srv-dell cib: [13774]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-79.raw
Jun 3 12:49:49 srv-dell cib: [13774]: info: write_cib_contents: Wrote
version 0.79.0 of the CIB to disk (digest:
a07e7fffb6db69288735d3e4edba6e76)
Jun 3 12:49:49 srv-dell cib: [13774]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.TZhJtx (digest:
/var/lib/heartbeat/crm/cib.LfF69u)
Jun 3 12:49:50 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op primitiveresource_cluster_ip_monitor_5000 from
1:75:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047790-57
Jun 3 12:49:50 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op dlm:0_monitor_120000 from
2:75:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047790-58
Jun 3 12:49:50 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op o2cb:0_monitor_120000 from
3:75:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047790-59
Jun 3 12:49:50 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op fs:1_monitor_120000 from
4:75:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047790-60
Jun 3 12:49:50 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=32, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 12:49:50 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=33, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:49:50 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=34, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:49:50 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:1_monitor_120000 (call=36, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:50 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:49:56 srv-dell crm_attribute: [13776]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:49:56 srv-dell crm_attribute: [13776]: info: Invoked:
crm_attribute -t crm_config -D -n maintenance-mode
Jun 3 12:49:56 srv-dell cib: [13777]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-80.raw
Jun 3 12:49:56 srv-dell cib: [13777]: info: write_cib_contents: Wrote
version 0.80.0 of the CIB to disk (digest:
212e8789956dac8abc7c43215d614e82)
Jun 3 12:49:56 srv-dell cib: [13777]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.zsZlNr (digest:
/var/lib/heartbeat/crm/cib.83vfaE)
Jun 3 12:49:56 srv-dell crmd: [5803]: info: cancel_op: No pending op
found for primitiveresource_cluster_ip:-1
Jun 3 12:49:56 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op primitiveresource_cluster_ip_cancel_5000 from
2:76:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047796-61
Jun 3 12:49:56 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=10:76:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 12:49:56 srv-dell crmd: [5803]: info: cancel_op: No pending op
found for dlm:0:-1
Jun 3 12:49:56 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op dlm:0_cancel_120000 from
3:76:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047796-62
Jun 3 12:49:56 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=19:76:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=dlm:0_monitor_120000 )
Jun 3 12:49:56 srv-dell crmd: [5803]: info: cancel_op: No pending op
found for o2cb:0:-1
Jun 3 12:49:56 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op o2cb:0_cancel_120000 from
4:76:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047796-63
Jun 3 12:49:56 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=26:76:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=o2cb:0_monitor_120000 )
Jun 3 12:49:56 srv-dell crmd: [5803]: info: cancel_op: No pending op
found for fs:1:-1
Jun 3 12:49:56 srv-dell crmd: [5803]: info: send_direct_ack: ACK'ing
resource op fs:1_cancel_120000 from
5:76:0:7c754d7f-6d3c-4527-b36c-979453669faf:
lrm_invoke-lrmd-1244047796-64
Jun 3 12:49:56 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=33:76:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=fs:1_monitor_120000 )
Jun 3 12:49:56 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=38, rc=0, cib-update=138,
confirmed=false) complete ok
Jun 3 12:49:56 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=39, rc=0, cib-update=139,
confirmed=false) complete ok
Jun 3 12:49:56 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:1_monitor_120000 (call=40, rc=0, cib-update=140,
confirmed=false) complete ok
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:57 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:49:57 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=37, rc=0,
cib-update=141, confirmed=false) complete ok
Jun 3 12:49:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:49:58 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:50:19 srv-dell crm_resource: [13936]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:50:19 srv-dell crm_resource: [13936]: info: Invoked:
crm_resource -P -H srv-lenovo
Jun 3 12:51:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:51:02 srv-dell cib: [14081]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-81.raw
Jun 3 12:51:02 srv-dell cib: [14081]: info: write_cib_contents: Wrote
version 0.81.0 of the CIB to disk (digest:
d377a25a76d66180a4a4523d75b707c6)
Jun 3 12:51:02 srv-dell cib: [14081]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.rBPn6x (digest:
/var/lib/heartbeat/crm/cib.AV1gD5)
Jun 3 12:51:02 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:51:11 srv-dell cib: [14118]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-82.raw
Jun 3 12:51:11 srv-dell cib: [14118]: info: write_cib_contents: Wrote
version 0.82.0 of the CIB to disk (digest:
57bd30bbe8a3f5d987101a5fbb374924)
Jun 3 12:51:11 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:51:11 srv-dell cib: [14118]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ztw4Dv (digest:
/var/lib/heartbeat/crm/cib.9EqG7l)
Jun 3 12:51:11 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:02 srv-dell mgmtd: [5804]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-lenovo
Jun 3 12:52:02 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 14366 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:52:02 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:52:02 srv-dell openais[5792]: [MAIN ] Msg[143]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=923): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:52:02 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:52:02 srv-dell cib: [14367]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-83.raw
Jun 3 12:52:02 srv-dell cib: [14367]: info: write_cib_contents: Wrote
version 0.83.0 of the CIB to disk (digest:
8c74543cc06636917d916aa23befa62e)
Jun 3 12:52:02 srv-dell cib: [14367]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.3mNq99 (digest:
/var/lib/heartbeat/crm/cib.WQQBmN)
Jun 3 12:52:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:04 srv-dell cib: [14369]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-84.raw
Jun 3 12:52:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:04 srv-dell cib: [14369]: info: write_cib_contents: Wrote
version 0.84.0 of the CIB to disk (digest:
cc564595deecc2f7d50fd8b18170fe7c)
Jun 3 12:52:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:04 srv-dell cib: [14369]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.lgUURb (digest:
/var/lib/heartbeat/crm/cib.ob2hmT)
Jun 3 12:52:04 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:13 srv-dell mgmtd: [5804]: info: Delete fail-count for
dlm:1 from srv-lenovo
Jun 3 12:52:13 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 14405 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:52:13 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:52:13 srv-dell openais[5792]: [MAIN ] Msg[146]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=861): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:52:13 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:52:15 srv-dell cib: [14424]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-85.raw
Jun 3 12:52:15 srv-dell cib: [14424]: info: write_cib_contents: Wrote
version 0.85.0 of the CIB to disk (digest:
55e3bee3898ace09f2f8b41c734b041e)
Jun 3 12:52:15 srv-dell cib: [14424]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.zsIHeu (digest:
/var/lib/heartbeat/crm/cib.X4tPAz)
Jun 3 12:52:15 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:15 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:15 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:15 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:15 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:15 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:25 srv-dell mgmtd: [5804]: info: Delete fail-count for
o2cb:1 from srv-lenovo
Jun 3 12:52:25 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 14461 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:52:25 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:52:25 srv-dell openais[5792]: [MAIN ] Msg[148]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=863): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:52:25 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:52:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:27 srv-dell cib: [14462]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-86.raw
Jun 3 12:52:27 srv-dell cib: [14462]: info: write_cib_contents: Wrote
version 0.86.0 of the CIB to disk (digest:
67440a934e6d9b85afec1d7338a7f9bf)
Jun 3 12:52:27 srv-dell cib: [14462]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.1DnKHo (digest:
/var/lib/heartbeat/crm/cib.tVXK0U)
Jun 3 12:52:27 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:35 srv-dell mgmtd: [5804]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 12:52:35 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Child 14499 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 12:52:35 srv-dell openais[5792]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 12:52:35 srv-dell openais[5792]: [MAIN ] Msg[150]
(dest=local:unknown, from=srv-lenovo:crmd.8207, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 12:52:35 srv-dell crmd: [5803]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 12:52:35 srv-dell cib: [14500]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-87.raw
Jun 3 12:52:35 srv-dell cib: [14500]: info: write_cib_contents: Wrote
version 0.87.0 of the CIB to disk (digest:
080a30baadb80d24106e65cb11f195f9)
Jun 3 12:52:35 srv-dell cib: [14500]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.JLT9Iv (digest:
/var/lib/heartbeat/crm/cib.ds3m9i)
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:37 srv-dell cib: [14501]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-88.raw
Jun 3 12:52:37 srv-dell cib: [14501]: info: write_cib_contents: Wrote
version 0.88.0 of the CIB to disk (digest:
c2f58361b75923871795502025afe706)
Jun 3 12:52:37 srv-dell cib: [14501]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.qgVe6x (digest:
/var/lib/heartbeat/crm/cib.xF9HOp)
Jun 3 12:52:37 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:50 srv-dell cib: [14556]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-89.raw
Jun 3 12:52:50 srv-dell cib: [14556]: info: write_cib_contents: Wrote
version 0.89.0 of the CIB to disk (digest:
b34e9499e46f74ed04c498d7f832a925)
Jun 3 12:52:50 srv-dell cib: [14556]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.2AxDjQ (digest:
/var/lib/heartbeat/crm/cib.ek4U29)
Jun 3 12:52:50 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:51 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=12:91:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 12:52:51 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 12:52:51 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=38:91:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:1_stop_0 )
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 12:52:51 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=37, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 12:52:51 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:1_monitor_120000 (call=40, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_12:52:51 INFO:
ifconfig eth1:0 down
Jun 3 12:52:51 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=41, rc=0,
cib-update=146, confirmed=true) complete ok
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:52:51 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:52:51 INFO: Trying to unmount /shared
Jun 3 12:52:51 srv-dell cluster-dlm: process_uevent: uevent:
offline@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell cluster-dlm: process_uevent: kernel: offline@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell cluster-dlm: confchg_cb: confchg for our leave
Jun 3 12:52:51 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 0
Jun 3 12:52:51 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 12:52:51 srv-dell cluster-dlm: update_dir_members: dir_member 1543678144
Jun 3 12:52:51 srv-dell cluster-dlm: set_configfs_members:
set_members rmdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 12:52:51 srv-dell cluster-dlm: set_configfs_members:
set_members lockspace rmdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:52:51 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 12:52:51 srv-dell cluster-dlm: purge_plocks: purged 0 plocks
for 1543678144
Jun 3 12:52:51 srv-dell cluster-dlm: process_uevent: uevent:
remove@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell cluster-dlm: process_uevent: kernel: remove@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell ocfs2_hb_ctl[14638]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -K -u 69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:52:51 srv-dell ocfs2_controld: client msg
Jun 3 12:52:51 srv-dell ocfs2_controld: client message 2 from 5: UNMOUNT
Jun 3 12:52:51 srv-dell ocfs2_controld: remove_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", service "ocfs2"
Jun 3 12:52:51 srv-dell ocfs2_controld: Unregistering mountgroup
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell ocfs2_controld: Unregistering
"69AB6E551666484FB314D5B168FD6B84" from dlm_controld
Jun 3 12:52:51 srv-dell ocfs2_controld: time to leave group
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell ocfs2_controld: calling LEAVE for group
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell ocfs2_controld: leaving group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:52:51 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 3 12:52:51 srv-dell ocfs2_controld: confchg called
Jun 3 12:52:51 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 0, left 1,
joined 0
Jun 3 12:52:51 srv-dell ocfs2_controld: Node 1543678144 leaves group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:52:51 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 12:52:51 srv-dell ocfs2_controld: Notified client: 1
Jun 3 12:52:51 srv-dell ocfs2_controld: client 6 fd 14 dead
Jun 3 12:52:51 srv-dell ocfs2_controld: client msg
Jun 3 12:52:51 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 12:52:51 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 12:52:51 srv-dell kernel: ocfs2: Unmounting device (8,17) on
(node 1543678)
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:52:51 INFO: unmounted /shared
successfully
Jun 3 12:52:51 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:1_stop_0 (call=42, rc=0, cib-update=147, confirmed=true)
complete ok
Jun 3 12:52:51 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=29:91:0:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:0_stop_0 )
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: rsc:o2cb:0: stop
Jun 3 12:52:51 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=39, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) logd is not running
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) 2009/06/03_12:52:51 INFO: Stopping o2cb:0
Jun 3 12:52:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) logd is not running
Jun 3 12:52:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:0:stop:stderr) 2009/06/03_12:52:51 INFO: Stopping
ocfs2_controld.pcmk
Jun 3 12:52:51 srv-dell ocfs2_controld: Caught signal 15
Jun 3 12:52:51 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 3 12:52:51 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 3 12:52:51 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 3 12:52:51 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 3 12:52:51 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 3 12:52:51 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 3 12:52:51 srv-dell ocfs2_controld: closing cpg connection
Jun 3 12:52:51 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:52:51 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 3 12:52:51 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 3 12:52:51 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 3 12:52:51 srv-dell ocfs2_controld: closing pacemaker connection
Jun 3 12:52:51 srv-dell ocfs2_controld: [9837]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 3 12:52:51 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:52 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 3 12:52:52 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:0_stop_0 (call=43, rc=0, cib-update=148,
confirmed=true) complete ok
Jun 3 12:52:52 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:91:0:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:0_stop_0 )
Jun 3 12:52:52 srv-dell lrmd: [5800]: info: rsc:dlm:0: stop
Jun 3 12:52:52 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=38, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 12:52:52 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/1543678144"
Jun 3 12:52:52 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 3 12:52:52 srv-dell kernel: dlm: closing connection to node 1543678144
Jun 3 12:52:52 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 3 12:52:52 srv-dell dlm_controld.pcmk: [9692]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 3 12:52:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:52 srv-dell lrmd: [5800]: info: RA output:
(dlm:0:stop:stderr) dlm_controld.pcmk: no process killed
Jun 3 12:52:52 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:53 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:52:53 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:0_stop_0 (call=44, rc=0, cib-update=149, confirmed=true)
complete ok
Jun 3 12:52:53 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=15:91:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=cloneresource_cluster_fencing_null:0_stop_0 )
Jun 3 12:52:53 srv-dell lrmd: [5800]: info:
rsc:cloneresource_cluster_fencing_null:0: stop
Jun 3 12:52:53 srv-dell lrmd: [14811]: info: Try to stop STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 12:52:53 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_stop_0 (call=45, rc=0,
cib-update=150, confirmed=true) complete ok
Jun 3 12:52:53 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:53 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:53 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:52:53 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:54:18 srv-dell kernel: st: Version 20080504, fixed bufsize
32768, s/g segs 256
Jun 3 12:57:21 srv-dell cib: [5799]: info: cib_stats: Processed 180
operations (1444.00us average, 0% utilization) in the last 10min
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell cib: [14975]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-90.raw
Jun 3 12:57:43 srv-dell cib: [14975]: info: write_cib_contents: Wrote
version 0.90.0 of the CIB to disk (digest:
02c1bc511b5d7475dedd3af2991ebf37)
Jun 3 12:57:43 srv-dell cib: [14975]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.UgoWlR (digest:
/var/lib/heartbeat/crm/cib.tW54Gs)
Jun 3 12:57:43 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:57:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=12:92:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 12:57:43 srv-dell lrmd: [5800]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 12:57:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=21:92:0:7c754d7f-6d3c-4527-b36c-979453669faf op=dlm:1_start_0 )
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: rsc:dlm:1: start
Jun 3 12:57:43 srv-dell lrmd: [14976]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 12:57:43 srv-dell stonithd: [5798]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 12:57:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=46, rc=0,
cib-update=151, confirmed=true) complete ok
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: RA output:
(dlm:1:start:stderr) dlm_controld.pcmk: no process killed
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell cluster-dlm[14994]: main: dlm_controld 1239267206
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info:
init_ais_connection: AIS connection established
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 12:57:43 srv-dell openais[5792]: [crm ] info: pcmk_notify:
Enabling node notifications for child 14994 (0x7fc95c01dda0)
Jun 3 12:57:43 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 3 12:57:43 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 3 12:57:43 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 3 12:57:43 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 3 12:57:43 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 3 12:57:43 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 2013440192
Jun 3 12:57:43 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 3 12:57:43 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 3 12:57:43 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: notice: ais_dispatch:
Membership 208: quorum aquired
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:57:43 srv-dell cluster-dlm: [14994]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=208 seen=208
proc=00000000000000000000000000053312 (new)
Jun 3 12:57:43 srv-dell cluster-dlm: update_cluster: Processing membership 208
Jun 3 12:57:43 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 3 12:57:43 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 3 12:57:43 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=192, last-seen=208, this-event=208,
last-event=0
Jun 3 12:57:43 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 3 12:57:43 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 3 12:57:43 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=208, last-seen=208, this-event=208,
last-event=0
Jun 3 12:57:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=10:92:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_start_0 )
Jun 3 12:57:43 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:57:43 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:57:43 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 12:57:43 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_12:57:43 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 12:57:43 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=48, rc=0,
cib-update=152, confirmed=true) complete ok
Jun 3 12:57:43 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=11:92:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:43 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:57:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:1_start_0 (call=47, rc=0, cib-update=153,
confirmed=true) complete ok
Jun 3 12:57:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:92:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=dlm:1_monitor_120000 )
Jun 3 12:57:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=29:92:0:7c754d7f-6d3c-4527-b36c-979453669faf op=o2cb:1_start_0 )
Jun 3 12:57:44 srv-dell lrmd: [5800]: info: rsc:o2cb:1: start
Jun 3 12:57:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation dlm:1_monitor_120000 (call=50, rc=0, cib-update=154,
confirmed=false) complete ok
Jun 3 12:57:44 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:start:stderr) logd is not running
Jun 3 12:57:44 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:start:stderr) 2009/06/03_12:57:44 INFO: Stack glue driver not
loaded
Jun 3 12:57:44 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:start:stderr) logd is not running
Jun 3 12:57:44 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:start:stderr) 2009/06/03_12:57:44 INFO: Starting o2cb:1
Jun 3 12:57:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:44 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 3 12:57:44 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 3 12:57:44 srv-dell kernel: OCFS2 1.5.0
Jun 3 12:57:44 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info:
init_ais_connection: AIS connection established
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info:
get_ais_nodeid: Server details: id=1543678144 uname=srv-dell
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 12:57:44 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 3 12:57:44 srv-dell openais[5792]: [crm ] info: pcmk_notify:
Enabling node notifications for child 15196 (0x7fc95c022a70)
Jun 3 12:57:44 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 3 12:57:44 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 3 12:57:44 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:57:44 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 12:57:44 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 3 12:57:44 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:57:44 srv-dell ocfs2_controld: setup done
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: notice:
ais_dispatch: Membership 208: quorum aquired
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:57:44 srv-dell ocfs2_controld: [15196]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=208 seen=208
proc=00000000000000000000000000053312 (new)
Jun 3 12:57:44 srv-dell ocfs2_controld: confchg called
Jun 3 12:57:44 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 3 12:57:44 srv-dell ocfs2_controld: CPG is live, we are not the
first daemon
Jun 3 12:57:44 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 3 12:57:44 srv-dell ocfs2_controld: Reading from section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Read section
"daemon_protocol" from checkpoint "ocfs2:controld"
Jun 3 12:57:44 srv-dell ocfs2_controld: Reading from section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 12:57:44 srv-dell ocfs2_controld: Read section "ocfs2_protocol"
from checkpoint "ocfs2:controld"
Jun 3 12:57:44 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 3 12:57:44 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 3 12:57:44 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 3 12:57:44 srv-dell ocfs2_controld: Opening control device
Jun 3 12:57:44 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 3 12:57:44 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 3 12:57:44 srv-dell ocfs2_controld: new listening connection 4
Jun 3 12:57:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=49, rc=0,
cib-update=155, confirmed=false) complete ok
Jun 3 12:57:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:44 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:57:46 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:1_start_0 (call=51, rc=0, cib-update=156,
confirmed=true) complete ok
Jun 3 12:57:46 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=30:92:0:7c754d7f-6d3c-4527-b36c-979453669faf
op=o2cb:1_monitor_120000 )
Jun 3 12:57:46 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=37:92:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:1_start_0 )
Jun 3 12:57:46 srv-dell lrmd: [5800]: info: rsc:fs:1: start
Jun 3 12:57:46 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:46 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:46 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:46 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:1_monitor_120000 (call=52, rc=0, cib-update=157,
confirmed=false) complete ok
Jun 3 12:57:46 srv-dell lrmd: [5800]: info: RA output:
(fs:1:start:stderr) logd is not running
Jun 3 12:57:46 srv-dell lrmd: [5800]: info: RA output:
(fs:1:start:stderr) 2009/06/03_12:57:46 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 12:57:46 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:57:46 srv-dell ocfs2_controld: new client connection 5
Jun 3 12:57:46 srv-dell ocfs2_controld: client msg
Jun 3 12:57:46 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 3 12:57:46 srv-dell ocfs2_controld: start_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", device "/dev/sdb1", service
"ocfs2"
Jun 3 12:57:46 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid 69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell ocfs2_controld: Starting join for group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:57:46 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 12:57:46 srv-dell ocfs2_controld: start_mount returns 0
Jun 3 12:57:46 srv-dell ocfs2_controld: confchg called
Jun 3 12:57:46 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 2, left 0,
joined 1
Jun 3 12:57:46 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell ocfs2_controld: This node joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell ocfs2_controld: Filling node 2013440192 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell ocfs2_controld: Registering mountgroup
69AB6E551666484FB314D5B168FD6B84 with dlm_controld
Jun 3 12:57:46 srv-dell ocfs2_controld: Registering
"69AB6E551666484FB314D5B168FD6B84" with dlm_controld
Jun 3 12:57:46 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 12:57:46 srv-dell ocfs2_controld: Registration of
"69AB6E551666484FB314D5B168FD6B84" complete
Jun 3 12:57:46 srv-dell ocfs2_controld: Mountgroup
69AB6E551666484FB314D5B168FD6B84 successfully registered with
dlm_controld
Jun 3 12:57:46 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 12:57:46 srv-dell ocfs2_controld: Notified client: 1
Jun 3 12:57:46 srv-dell kernel: dlm: Using TCP for communications
Jun 3 12:57:46 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell cluster-dlm: process_uevent: kernel: add@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell cluster-dlm: process_uevent: kernel: online@
69AB6E551666484FB314D5B168FD6B84
Jun 3 12:57:46 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 3 12:57:46 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 3 12:57:46 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 2 joined 1 remove 0 failed 0
Jun 3 12:57:46 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 12:57:46 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 12:57:46 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 12:57:46 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 2 1 0 0
Jun 3 12:57:46 srv-dell cluster-dlm: receive_start: receive_start
2013440192:2 len 80
Jun 3 12:57:46 srv-dell cluster-dlm: match_change: match_change
2013440192:2 matches cg 1
Jun 3 12:57:46 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 need 1 of 2
Jun 3 12:57:46 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 80
Jun 3 12:57:46 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 3 12:57:46 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 2
Jun 3 12:57:46 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 2
Jun 3 12:57:46 srv-dell cluster-dlm: do_sysfs: write "1438231596" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/id"
Jun 3 12:57:46 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 12:57:46 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/2013440192"
Jun 3 12:57:46 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 12:57:46 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 12:57:46 srv-dell cluster-dlm: set_plock_ckpt_node:
set_plock_ckpt_node from 0 to 2013440192
Jun 3 12:57:46 srv-dell cluster-dlm: receive_plocks_stored:
receive_plocks_stored 2013440192:2 need_plocks 1
Jun 3 12:57:46 srv-dell cluster-dlm: match_change: match_change
2013440192:2 matches cg 1
Jun 3 12:57:46 srv-dell cluster-dlm: retrieve_plocks: retrieve_plocks
Jun 3 12:57:46 srv-dell kernel: dlm: got connection from 2013440192
Jun 3 12:57:46 srv-dell kernel: dlm: create_rcom to 2013440192 type 1
len 0 ENOBUFS
Jun 3 12:57:46 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:58:05 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:05 srv-dell cib: [15381]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-91.raw
Jun 3 12:58:05 srv-dell cib: [15381]: info: write_cib_contents: Wrote
version 0.91.0 of the CIB to disk (digest:
87da42cdf9fa7d2f4b1e11e67c995d20)
Jun 3 12:58:05 srv-dell cib: [15381]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.EszyOd (digest:
/var/lib/heartbeat/crm/cib.L1Mb5B)
Jun 3 12:58:05 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:58:06 srv-dell lrmd: [5800]: WARN: fs:1:start process (PID
15223) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 12:58:06 srv-dell lrmd: [5800]: WARN: operation start[53] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15223] timed out
Jun 3 12:58:06 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_start_0 (53) Timed Out (timeout=20000ms)
Jun 3 12:58:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:06 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:58:07 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=2:93:0:7c754d7f-6d3c-4527-b36c-979453669faf op=fs:1_stop_0 )
Jun 3 12:58:07 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 12:58:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:07 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:58:07 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:58:07 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:58:07 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:58:27 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
15382) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 12:58:27 srv-dell lrmd: [5800]: WARN: operation stop[54] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15382] timed out
Jun 3 12:58:27 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (54) Timed Out (timeout=20000ms)
Jun 3 12:58:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:27 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:58:29 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:29 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:29 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:29 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:29 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:29 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:58:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 12:58:30 srv-dell mgmtd: [5804]: info: CIB query: cib
Jun 3 12:58:40 srv-dell openais[5792]: [TOTEM] The token was lost in
the OPERATIONAL state.
Jun 3 12:58:40 srv-dell openais[5792]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 12:58:40 srv-dell openais[5792]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 12:58:40 srv-dell openais[5792]: [TOTEM] entering GATHER state from 2.
Jun 3 12:58:42 srv-dell crm_attribute: [15554]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 12:58:42 srv-dell crm_attribute: [15554]: info: Invoked:
crm_attribute -t crm_config -n maintenance-mode -v true
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] entering GATHER state from 0.
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] Saving state aru 4bf
high seq received 4bf
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] Storing new sequence
id for ring d4
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] entering COMMIT state.
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] entering RECOVERY state.
Jun 3 12:58:42 srv-dell cib: [5799]: notice: ais_dispatch: Membership
212: quorum lost
Jun 3 12:58:42 srv-dell cib: [5799]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=192 seen=208 proc=00000000000000000000000000053312
Jun 3 12:58:42 srv-dell crmd: [5803]: notice: ais_dispatch:
Membership 212: quorum lost
Jun 3 12:58:42 srv-dell crmd: [5803]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=192 seen=208 proc=00000000000000000000000000053312
Jun 3 12:58:42 srv-dell crmd: [5803]: info: erase_node_from_join:
Removed node srv-lenovo from join calculations: welcomed=1 itegrated=0
finalized=0 confirmed=0
Jun 3 12:58:42 srv-dell crmd: [5803]: WARN: check_dead_member: Our DC
node (srv-lenovo) left the cluster
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] previous ring seq 208
rep 192.168.2.120
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] aru 4bf high delivered
4bf received flag 1
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=check_dead_member ]
Jun 3 12:58:42 srv-dell crmd: [5803]: info: update_dc: Unset DC srv-lenovo
Jun 3 12:58:42 srv-dell cluster-dlm: [14994]: notice: ais_dispatch:
Membership 212: quorum lost
Jun 3 12:58:42 srv-dell ocfs2_controld: [15196]: notice:
ais_dispatch: Membership 212: quorum lost
Jun 3 12:58:42 srv-dell cluster-dlm: [14994]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=lost (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Jun 3 12:58:42 srv-dell cluster-dlm: update_cluster: Processing membership 212
Jun 3 12:58:42 srv-dell cluster-dlm: del_configfs_node:
del_configfs_node rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_te_control:
Registering TE UUID: 9f20fbee-34bd-4374-9bc1-2ea89443936d
Jun 3 12:58:42 srv-dell ocfs2_controld: [15196]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=lost (new)
addr=r(0) ip(192.168.2.120) votes=1 born=192 seen=208
proc=00000000000000000000000000053312
Jun 3 12:58:42 srv-dell ocfs2_controld: confchg called
Jun 3 12:58:42 srv-dell cluster-dlm: dlm_process_node: Removed
inactive node 2013440192: born-on=192, last-seen=208, this-event=212,
last-event=208
Jun 3 12:58:42 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 3 12:58:42 srv-dell crmd: [5803]: WARN:
cib_client_add_notify_callback: Callback already present
Jun 3 12:58:42 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 1, joined 0
Jun 3 12:58:42 srv-dell cluster-dlm: dlm_process_node: Skipped active
node 1543678144: born-on=208, last-seen=212, this-event=212,
last-event=208
Jun 3 12:58:42 srv-dell crmd: [5803]: info: set_graph_functions:
Setting custom graph functions
Jun 3 12:58:42 srv-dell cluster-dlm: add_change: add_change cg 2
remove nodeid 2013440192 reason 3
Jun 3 12:58:42 srv-dell ocfs2_controld: node daemon left 2013440192
Jun 3 12:58:42 srv-dell cluster-dlm: add_change: add_change cg 2
counts member 1 joined 0 remove 1 failed 1
Jun 3 12:58:42 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition -1: 0 actions in 0 synapses
Jun 3 12:58:42 srv-dell ocfs2_controld: node down 2013440192
Jun 3 12:58:42 srv-dell ocfs2_controld: Node 2013440192 has left
mountgroup 69AB6E551666484FB314D5B168FD6B84
Jun 3 12:58:42 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 2
Jun 3 12:58:42 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 12:58:42 srv-dell ocfs2_controld: Sending notification of node
2013440192 for "69AB6E551666484FB314D5B168FD6B84"
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_dc_takeover: Taking
over DC status for this partition
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] Sending initial ORF token
Jun 3 12:58:42 srv-dell cluster-dlm: purge_plocks: purged 0 plocks
for 2013440192
Jun 3 12:58:42 srv-dell ocfs2_controld: confchg called
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 12:58:42 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 1, left 1,
joined 0
Jun 3 12:58:42 srv-dell cib: [5799]: info: cib_process_readwrite: We
are now in R/W mode
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 12:58:42 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 212: memb=1,
new=0, lost=1
Jun 3 12:58:42 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 12:58:42 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: lost: srv-lenovo 2013440192
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 12:58:42 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 212: memb=1, new=0,
lost=0
Jun 3 12:58:42 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 12:58:42 srv-dell openais[5792]: [crm ] info:
ais_mark_unseen_peer_dead: Node srv-lenovo was not seen in the
previous transition
Jun 3 12:58:42 srv-dell openais[5792]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: lost
Jun 3 12:58:42 srv-dell openais[5792]: [crm ] info:
send_member_notification: Sending membership update 212 to 4 children
Jun 3 12:58:42 srv-dell openais[5792]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 12:58:42 srv-dell openais[5792]: [TOTEM] entering OPERATIONAL state.
Jun 3 12:58:42 srv-dell openais[5792]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 12:58:42 srv-dell openais[5792]: [CPG ] got joinlist message
from node 1543678144
Jun 3 12:58:42 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_master for section 'all'
(origin=local/crmd/161, version=0.91.13): ok (rc=0)
Jun 3 12:58:42 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/162, version=0.91.13): ok (rc=0)
Jun 3 12:58:42 srv-dell crmd: [5803]: info: join_make_offer: Making
join offers based on membership 212
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_dc_join_offer_all:
join-3: Waiting on 1 outstanding join acks
Jun 3 12:58:42 srv-dell crmd: [5803]: info: ais_dispatch: Membership
212: quorum still lost
Jun 3 12:58:42 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/164, version=0.91.13): ok (rc=0)
Jun 3 12:58:42 srv-dell crmd: [5803]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 12:58:42 srv-dell crmd: [5803]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 12:58:42 srv-dell crmd: [5803]: info: ais_dispatch: Membership
212: quorum still lost
Jun 3 12:58:42 srv-dell cluster-dlm: fence_node_time: Node
2013440192/srv-lenovo has not been shot yet
Jun 3 12:58:42 srv-dell cluster-dlm: check_fencing_done:
check_fencing 2013440192 not fenced add 1244048266 fence 0
Jun 3 12:58:42 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/167, version=0.91.13): ok (rc=0)
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 12:58:42 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/170, version=0.91.13): ok (rc=0)
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes responded to the join offer.
Jun 3 12:58:42 srv-dell crmd: [5803]: info: do_dc_join_finalize:
join-3: Syncing the CIB from srv-dell to the rest of the cluster
Jun 3 12:58:42 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 12:58:42 srv-dell ocfs2_controld: Notified for
"69AB6E551666484FB314D5B168FD6B84", node 2013440192, status 0
Jun 3 12:58:42 srv-dell crmd: [5803]: info: te_connect_stonith:
Attempting connection to fencing daemon...
Jun 3 12:58:42 srv-dell ocfs2_controld: Completing notification on
"69AB6E551666484FB314D5B168FD6B84" for node 2013440192
Jun 3 12:58:42 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/171, version=0.91.13): ok (rc=0)
Jun 3 12:58:43 srv-dell crmd: [5803]: info: te_connect_stonith: Connected
Jun 3 12:58:43 srv-dell crmd: [5803]: info: do_dc_join_ack: join-3:
Updating node state to member for srv-dell
Jun 3 12:58:43 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/172, version=0.91.13): ok (rc=0)
Jun 3 12:58:43 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/173,
version=0.91.13): ok (rc=0)
Jun 3 12:58:43 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 12:58:43 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 12:58:43 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 12:58:43 srv-dell crmd: [5803]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 3 12:58:43 srv-dell crmd: [5803]: info: crm_update_quorum:
Updating quorum status to false (call=177)
Jun 3 12:58:43 srv-dell crmd: [5803]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 3 12:58:43 srv-dell attrd: [5801]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 12:58:43 srv-dell crmd: [5803]: info: do_pe_invoke: Query 178:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:58:43 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 12:58:44 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/175, version=0.91.14): ok (rc=0)
Jun 3 12:58:44 srv-dell crmd: [5803]: WARN: match_down_event: No
match for shutdown action on srv-lenovo
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_update_diff:
Stonith/shutdown of srv-lenovo not matched
Jun 3 12:58:44 srv-dell cib: [5799]: info: log_data_element:
cib:diff: - <cib have-quorum="1" dc-uuid="srv-lenovo" admin_epoch="0"
epoch="91" num_updates="15" />
Jun 3 12:58:44 srv-dell crmd: [5803]: info: abort_transition_graph:
te_update_diff:191 - Triggered transition abort (complete=1,
tag=node_state, id=srv-lenovo, magic=NA) : Node failure
Jun 3 12:58:44 srv-dell cib: [5799]: info: log_data_element:
cib:diff: + <cib have-quorum="0" dc-uuid="srv-dell" admin_epoch="0"
epoch="92" num_updates="1" />
Jun 3 12:58:44 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/177, version=0.92.1): ok (rc=0)
Jun 3 12:58:44 srv-dell crmd: [5803]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 12:58:44 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 12:58:44 srv-dell crmd: [5803]: info: need_abort: Aborting on
change to have-quorum
Jun 3 12:58:44 srv-dell cib: [15555]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-92.raw
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048324-76, seq=212, quorate=0
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_pe_invoke: Query 179:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_pe_invoke: Query 180:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:58:44 srv-dell cib: [15555]: info: write_cib_contents: Wrote
version 0.92.0 of the CIB to disk (digest:
94769702e7b27cbc5898bfe9af4a68da)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 12:58:44 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 12:58:44 srv-dell cib: [15555]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.HoktFX (digest:
/var/lib/heartbeat/crm/cib.KpMVLH)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 12:58:44 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048324-77, seq=212, quorate=0
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource primitiveresource_cluster_ip (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 12:58:44 srv-dell crmd: [5803]: info: handle_response: pe_calc
calculation pe_calc-dc-1244048324-76 is obsolete
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 1: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-14.bz2
Jun 3 12:58:44 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 12:58:44 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 12:58:44 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 12:58:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 12:58:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource primitiveresource_cluster_ip (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 12:58:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 12:58:44 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 2: 13 actions in 13 synapses
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 2 (ref=pe_calc-dc-1244048324-77) derived from
/var/lib/pengine/pe-warn-15.bz2
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 5: stop primitiveresource_cluster_ip_stop_0 on
srv-dell (local)
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=5:2:0:9f20fbee-34bd-4374-9bc1-2ea89443936d
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 12:58:44 srv-dell lrmd: [5800]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 12:58:44 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 2: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-15.bz2
Jun 3 12:58:44 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op primitiveresource_cluster_ip_stop_0 in the CIB
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 10 fired and confirmed
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 16 fired and confirmed
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 27 fired and confirmed
Jun 3 12:58:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=49, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 24: stop fs:1_stop_0 on srv-dell (local)
Jun 3 12:58:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=24:2:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 12:58:44 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 12:58:44 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 12:58:44 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 12:58:44 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 12:58:44 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 12:58:44 srv-dell lrmd: [5800]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_12:58:44 INFO:
ifconfig eth1:0 down
Jun 3 12:58:44 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=55, rc=0,
cib-update=183, confirmed=true) complete ok
Jun 3 12:58:44 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:58:44 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:58:44 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:58:44 srv-dell crmd: [5803]: info: match_graph_event: Action
primitiveresource_cluster_ip_stop_0 (5) confirmed on srv-dell (rc=0)
Jun 3 12:59:04 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
15557) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 12:59:04 srv-dell lrmd: [5800]: WARN: operation stop[56] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15557] timed out
Jun 3 12:59:04 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (56) Timed Out (timeout=20000ms)
Jun 3 12:59:04 srv-dell crmd: [5803]: WARN: status_from_rc: Action 24
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 12:59:04 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048344)
Jun 3 12:59:04 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;24:2:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 12:59:04 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 12:59:04 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 12:59:04 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (24) confirmed on srv-dell (rc=4)
Jun 3 12:59:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 28 fired and confirmed
Jun 3 12:59:04 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 12:59:04 srv-dell crmd: [5803]: notice: run_graph: Transition 2
(Complete=6, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-15.bz2): Stopped
Jun 3 12:59:04 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 2 is now complete
Jun 3 12:59:04 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 12:59:04 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 12:59:04 srv-dell crmd: [5803]: info: do_pe_invoke: Query 191:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:59:04 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048344-80, seq=212, quorate=0
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 12:59:04 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 12:59:04 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 12:59:04 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 12:59:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 12:59:04 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 12:59:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 12:59:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 12:59:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 12:59:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 12:59:04 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 12:59:04 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 3: 12 actions in 12 synapses
Jun 3 12:59:04 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 3 (ref=pe_calc-dc-1244048344-80) derived from
/var/lib/pengine/pe-warn-16.bz2
Jun 3 12:59:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 12:59:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 12:59:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 12:59:04 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 3: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-16.bz2
Jun 3 12:59:04 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 12:59:04 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 12:59:04 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:3:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 12:59:04 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 12:59:04 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 12:59:04 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:59:04 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:59:04 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:59:24 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
15627) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 12:59:24 srv-dell lrmd: [5800]: WARN: operation stop[57] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15627] timed out
Jun 3 12:59:24 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (57) Timed Out (timeout=20000ms)
Jun 3 12:59:24 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 12:59:24 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048364)
Jun 3 12:59:24 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:3:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 12:59:24 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 12:59:24 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 12:59:24 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 12:59:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 12:59:24 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 12:59:24 srv-dell crmd: [5803]: notice: run_graph: Transition 3
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-16.bz2): Stopped
Jun 3 12:59:24 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 3 is now complete
Jun 3 12:59:24 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 12:59:24 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 12:59:24 srv-dell crmd: [5803]: info: do_pe_invoke: Query 198:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:59:24 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048364-82, seq=212, quorate=0
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 12:59:24 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 12:59:24 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 12:59:24 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 12:59:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 12:59:24 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 12:59:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 12:59:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 12:59:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 12:59:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 12:59:24 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 12:59:24 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 4: 12 actions in 12 synapses
Jun 3 12:59:24 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 4 (ref=pe_calc-dc-1244048364-82) derived from
/var/lib/pengine/pe-warn-17.bz2
Jun 3 12:59:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 12:59:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 12:59:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 12:59:24 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 4: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-17.bz2
Jun 3 12:59:24 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 12:59:24 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 12:59:24 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:4:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 12:59:24 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 12:59:24 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 12:59:24 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:59:24 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:59:24 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:59:44 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
15672) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 12:59:44 srv-dell lrmd: [5800]: WARN: operation stop[58] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15672] timed out
Jun 3 12:59:44 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (58) Timed Out (timeout=20000ms)
Jun 3 12:59:44 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 12:59:44 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048384)
Jun 3 12:59:44 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:4:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 12:59:44 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 12:59:44 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 12:59:44 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 12:59:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 12:59:44 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 12:59:44 srv-dell crmd: [5803]: notice: run_graph: Transition 4
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-17.bz2): Stopped
Jun 3 12:59:44 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 4 is now complete
Jun 3 12:59:44 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 12:59:44 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 12:59:44 srv-dell crmd: [5803]: info: do_pe_invoke: Query 205:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 12:59:44 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048384-84, seq=212, quorate=0
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 12:59:44 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 12:59:44 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 12:59:44 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 12:59:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 12:59:44 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 12:59:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 12:59:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 12:59:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 12:59:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 12:59:44 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 12:59:44 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 5: 12 actions in 12 synapses
Jun 3 12:59:44 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 5 (ref=pe_calc-dc-1244048384-84) derived from
/var/lib/pengine/pe-warn-18.bz2
Jun 3 12:59:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 12:59:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 12:59:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 12:59:44 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 5: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-18.bz2
Jun 3 12:59:44 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 12:59:44 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 12:59:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:5:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 12:59:44 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 12:59:44 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 12:59:44 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 12:59:44 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_12:59:44 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 12:59:45 srv-dell openais[5792]: [SERV ] Unloading all openais components
Jun 3 12:59:45 srv-dell openais[5792]: [SERV ] Unloading slot 10:
openais cluster config database access v1.01
Jun 3 12:59:45 srv-dell openais[5792]: [SERV ] Unloading openais
component: openais_confdb v0 (16/10)
Jun 3 12:59:45 srv-dell openais[5792]: [SERV ] Unloading slot 9:
Pacemaker Cluster Manager
Jun 3 12:59:45 srv-dell openais[5792]: [SERV ] Unloading openais
component: pacemaker v0 (2/9)
Jun 3 12:59:45 srv-dell openais[5792]: [crm ] notice: pcmk_shutdown:
Begining shutdown
Jun 3 12:59:45 srv-dell openais[5792]: [MAIN ] notice: stop_child:
Sent -15 to mgmtd: [5804]
Jun 3 13:00:04 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
15717) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 13:00:04 srv-dell lrmd: [5800]: WARN: operation stop[59] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15717] timed out
Jun 3 13:00:04 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (59) Timed Out (timeout=20000ms)
Jun 3 13:00:04 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 13:00:04 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048404)
Jun 3 13:00:04 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:5:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 13:00:04 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 13:00:04 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 13:00:04 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 13:00:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 13:00:04 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 13:00:04 srv-dell crmd: [5803]: notice: run_graph: Transition 5
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-18.bz2): Stopped
Jun 3 13:00:04 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 5 is now complete
Jun 3 13:00:04 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 13:00:04 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 13:00:04 srv-dell crmd: [5803]: info: do_pe_invoke: Query 212:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:00:04 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048404-86, seq=212, quorate=0
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 13:00:04 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 13:00:04 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 13:00:04 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 13:00:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 13:00:04 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 13:00:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 13:00:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 13:00:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 13:00:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 13:00:04 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 13:00:04 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 6: 12 actions in 12 synapses
Jun 3 13:00:04 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 6 (ref=pe_calc-dc-1244048404-86) derived from
/var/lib/pengine/pe-warn-19.bz2
Jun 3 13:00:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 13:00:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 13:00:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 13:00:04 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 13:00:04 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:6:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 13:00:04 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 13:00:04 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 13:00:04 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 6: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-19.bz2
Jun 3 13:00:04 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 13:00:04 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 13:00:04 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_13:00:04 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:00:14 srv-dell openais[5792]: [crm ] notice: pcmk_shutdown:
Still waiting for mgmtd (pid=5804) to terminate...
Jun 3 13:00:24 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
15871) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 13:00:24 srv-dell lrmd: [5800]: WARN: operation stop[60] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15871] timed out
Jun 3 13:00:24 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (60) Timed Out (timeout=20000ms)
Jun 3 13:00:24 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 13:00:24 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048424)
Jun 3 13:00:24 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:6:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 13:00:24 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 13:00:24 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 13:00:24 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 13:00:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 13:00:24 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 13:00:24 srv-dell crmd: [5803]: notice: run_graph: Transition 6
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-19.bz2): Stopped
Jun 3 13:00:24 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 6 is now complete
Jun 3 13:00:24 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 13:00:24 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 13:00:24 srv-dell crmd: [5803]: info: do_pe_invoke: Query 219:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:00:24 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048424-88, seq=212, quorate=0
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 13:00:24 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is online
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 13:00:24 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 13:00:24 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 13:00:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 13:00:24 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 13:00:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 13:00:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 13:00:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 13:00:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 13:00:24 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 13:00:24 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 7: 12 actions in 12 synapses
Jun 3 13:00:24 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 7 (ref=pe_calc-dc-1244048424-88) derived from
/var/lib/pengine/pe-warn-20.bz2
Jun 3 13:00:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 13:00:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 13:00:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 13:00:24 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 7: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-20.bz2
Jun 3 13:00:24 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 13:00:24 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 13:00:24 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:7:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 13:00:24 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 13:00:24 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 13:00:24 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 13:00:24 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_13:00:24 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:00:43 srv-dell mgmtd: [5804]: info: mgmtd is shutting down
Jun 3 13:00:43 srv-dell mgmtd: [5804]: debug: [mgmtd] stopped
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:00:43 srv-dell haclient: on_event:None
Jun 3 13:00:43 srv-dell openais[5792]: [crm ] notice: pcmk_shutdown:
mgmtd (pid=5804) confirmed dead
Jun 3 13:00:43 srv-dell openais[5792]: [MAIN ] notice: stop_child:
Sent -15 to crmd: [5803]
Jun 3 13:00:43 srv-dell crmd: [5803]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 13:00:43 srv-dell crmd: [5803]: info: crm_shutdown: Requesting shutdown
Jun 3 13:00:43 srv-dell crmd: [5803]: WARN: do_log: FSA: Input
I_SHUTDOWN from crm_shutdown() received in state S_TRANSITION_ENGINE
Jun 3 13:00:43 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]
Jun 3 13:00:43 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 13:00:43 srv-dell crmd: [5803]: info: do_shutdown_req: Sending
shutdown request to DC: srv-dell
Jun 3 13:00:43 srv-dell crmd: [5803]: info: handle_shutdown_request:
Creating shutdown request for srv-dell (state=S_POLICY_ENGINE)
Jun 3 13:00:43 srv-dell crmd: [5803]: info: update_attrd: Updating
shutdown=1244048443 via attrd for srv-dell
Jun 3 13:00:43 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 13:00:43 srv-dell attrd: [5801]: info: attrd_perform_update:
Sent update 20: shutdown=1244048443
Jun 3 13:00:43 srv-dell crmd: [5803]: info: abort_transition_graph:
te_update_diff:146 - Triggered transition abort (complete=0,
tag=transient_attributes, id=srv-dell, magic=NA) : Transient
attribute: update
Jun 3 13:00:43 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1000000
Jun 3 13:00:43 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 13:00:44 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
15954) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 13:00:44 srv-dell lrmd: [5800]: WARN: operation stop[61] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [15954] timed out
Jun 3 13:00:44 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (61) Timed Out (timeout=20000ms)
Jun 3 13:00:44 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 13:00:44 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048444)
Jun 3 13:00:44 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:7:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 13:00:44 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 13:00:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 13:00:44 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 13:00:44 srv-dell crmd: [5803]: notice: run_graph: Transition 7
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-20.bz2): Stopped
Jun 3 13:00:44 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 7 is now complete
Jun 3 13:00:44 srv-dell crmd: [5803]: info: do_pe_invoke: Query 227:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:00:44 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048444-91, seq=212, quorate=0
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 13:00:44 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is shutting down
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 13:00:44 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 13:00:44 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 13:00:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 13:00:44 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:00:44 srv-dell pengine: [5802]: info: stage6: Scheduling
Node srv-dell for shutdown
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 13:00:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 13:00:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 13:00:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 13:00:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 13:00:44 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 13:00:44 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 8: 13 actions in 13 synapses
Jun 3 13:00:44 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 8 (ref=pe_calc-dc-1244048444-91) derived from
/var/lib/pengine/pe-warn-21.bz2
Jun 3 13:00:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 13:00:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 13:00:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 13:00:44 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 8: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-21.bz2
Jun 3 13:00:44 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 13:00:44 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 13:00:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:8:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 13:00:44 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 13:00:44 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 13:00:44 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 13:00:44 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_13:00:44 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:01:04 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
16033) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 13:01:04 srv-dell lrmd: [5800]: WARN: operation stop[62] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [16033] timed out
Jun 3 13:01:04 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (62) Timed Out (timeout=20000ms)
Jun 3 13:01:04 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 13:01:04 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048464)
Jun 3 13:01:04 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:8:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 13:01:04 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 13:01:04 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 13:01:04 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 13:01:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 13:01:04 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 13:01:04 srv-dell crmd: [5803]: notice: run_graph: Transition 8
(Complete=5, Pending=0, Fired=0, Skipped=5, Incomplete=3,
Source=/var/lib/pengine/pe-warn-21.bz2): Stopped
Jun 3 13:01:04 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 8 is now complete
Jun 3 13:01:04 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 13:01:04 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 13:01:04 srv-dell crmd: [5803]: info: do_pe_invoke: Query 234:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:01:04 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048464-93, seq=212, quorate=0
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 13:01:04 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is shutting down
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 13:01:04 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 13:01:04 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 13:01:04 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 13:01:04 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:04 srv-dell pengine: [5802]: info: stage6: Scheduling
Node srv-dell for shutdown
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 13:01:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 13:01:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 13:01:04 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 13:01:04 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 13:01:04 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 13:01:04 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 9: 13 actions in 13 synapses
Jun 3 13:01:04 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 9 (ref=pe_calc-dc-1244048464-93) derived from
/var/lib/pengine/pe-warn-22.bz2
Jun 3 13:01:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 13:01:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 13:01:04 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 13:01:04 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 9: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-22.bz2
Jun 3 13:01:04 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 13:01:04 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 13:01:04 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:9:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 13:01:04 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 13:01:04 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 13:01:04 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 13:01:04 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_13:01:04 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:01:13 srv-dell openais[5792]: [crm ] notice: pcmk_shutdown:
Still waiting for crmd (pid=5803) to terminate...
Jun 3 13:01:24 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
16078) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 13:01:24 srv-dell lrmd: [5800]: WARN: operation stop[63] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [16078] timed out
Jun 3 13:01:24 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (63) Timed Out (timeout=20000ms)
Jun 3 13:01:24 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 13:01:24 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048484)
Jun 3 13:01:24 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:9:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event failed
Jun 3 13:01:24 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 13:01:24 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 13:01:24 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 13:01:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 13:01:24 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 13:01:24 srv-dell crmd: [5803]: notice: run_graph: Transition 9
(Complete=5, Pending=0, Fired=0, Skipped=5, Incomplete=3,
Source=/var/lib/pengine/pe-warn-22.bz2): Stopped
Jun 3 13:01:24 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 9 is now complete
Jun 3 13:01:24 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 13:01:24 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 13:01:24 srv-dell crmd: [5803]: info: do_pe_invoke: Query 241:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:01:24 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048484-95, seq=212, quorate=0
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 13:01:24 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is shutting down
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 13:01:24 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 13:01:24 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 13:01:24 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 13:01:24 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:24 srv-dell pengine: [5802]: info: stage6: Scheduling
Node srv-dell for shutdown
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 13:01:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 13:01:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 13:01:24 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 13:01:24 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 13:01:24 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 13:01:24 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 10: 13 actions in 13 synapses
Jun 3 13:01:24 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 10 (ref=pe_calc-dc-1244048484-95) derived from
/var/lib/pengine/pe-warn-23.bz2
Jun 3 13:01:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 13:01:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 13:01:24 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 10: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-23.bz2
Jun 3 13:01:24 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 13:01:24 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 13:01:24 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 13:01:24 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:10:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 13:01:24 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 13:01:24 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 13:01:24 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 13:01:24 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_13:01:24 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:01:43 srv-dell openais[5792]: [crm ] notice: pcmk_shutdown:
Still waiting for crmd (pid=5803) to terminate...
Jun 3 13:01:44 srv-dell lrmd: [5800]: WARN: fs:1:stop process (PID
16125) timed out (try 1). Killing with signal SIGTERM (15).
Jun 3 13:01:44 srv-dell lrmd: [5800]: WARN: operation stop[64] on
ocf::Filesystem::fs:1 for client 5803, its parameters:
directory=[/shared] fstype=[ocfs2] CRM_meta_record_pending=[true]
device=[/dev/sdb1] CRM_meta_timeout=[20000] CRM_meta_clone_max=[2]
crm_feature_set=[3.0.1] CRM_meta_globally_unique=[false]
CRM_meta_clone=[1] : pid [16125] timed out
Jun 3 13:01:44 srv-dell crmd: [5803]: ERROR: process_lrm_event: LRM
operation fs:1_stop_0 (64) Timed Out (timeout=20000ms)
Jun 3 13:01:44 srv-dell crmd: [5803]: WARN: status_from_rc: Action 22
(fs:1_stop_0) on srv-dell failed (target: 0 vs. rc: -2): Error
Jun 3 13:01:44 srv-dell crmd: [5803]: WARN: update_failcount:
Updating failcount for fs:1 on srv-dell after failed stop: rc=-2
(update=INFINITY, time=1244048504)
Jun 3 13:01:44 srv-dell crmd: [5803]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:1_stop_0,
magic=2:-2;22:10:0:9f20fbee-34bd-4374-9bc1-2ea89443936d) : Event
failed
Jun 3 13:01:44 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 13:01:44 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 13:01:44 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 13:01:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 13:01:44 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 13:01:44 srv-dell crmd: [5803]: notice: run_graph: Transition
10 (Complete=5, Pending=0, Fired=0, Skipped=5, Incomplete=3,
Source=/var/lib/pengine/pe-warn-23.bz2): Stopped
Jun 3 13:01:44 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 10 is now complete
Jun 3 13:01:44 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 13:01:44 srv-dell crmd: [5803]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 13:01:44 srv-dell crmd: [5803]: info: do_pe_invoke: Query 248:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:01:44 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048504-97, seq=212, quorate=0
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 13:01:44 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is shutting down
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_stop_0 on srv-dell: unknown exec error
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 13:01:44 srv-dell pengine: [5802]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-lenovo
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-lenovo
Jun 3 13:01:44 srv-dell pengine: [5802]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 13:01:44 srv-dell pengine: [5802]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-lenovo returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-lenovo
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 13:01:44 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: custom_action: Action
dlm:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: custom_action: Action
o2cb:0_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 13:01:44 srv-dell pengine: [5802]: info: stage6: Scheduling
Node srv-dell for shutdown
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 13:01:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 13:01:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-lenovo
Jun 3 13:01:44 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating o2cb:1 with fs:1 on srv-dell
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:0 (srv-lenovo)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:0 (srv-lenovo)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 13:01:44 srv-dell pengine: [5802]: notice: LogActions: Stop
resource fs:1 (srv-dell)
Jun 3 13:01:44 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 13:01:44 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 11: 13 actions in 13 synapses
Jun 3 13:01:44 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 11 (ref=pe_calc-dc-1244048504-97) derived from
/var/lib/pengine/pe-warn-24.bz2
Jun 3 13:01:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 13:01:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 13:01:44 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 13:01:44 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 11: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-24.bz2
Jun 3 13:01:44 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 13:01:44 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop fs:1_stop_0 on srv-dell (local)
Jun 3 13:01:44 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:11:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=fs:1_stop_0 )
Jun 3 13:01:44 srv-dell lrmd: [5800]: info: rsc:fs:1: stop
Jun 3 13:01:44 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:1_stop_0 in the CIB
Jun 3 13:01:45 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) logd is not running
Jun 3 13:01:45 srv-dell lrmd: [5800]: info: RA output:
(fs:1:stop:stderr) 2009/06/03_13:01:45 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:01:47 srv-dell openais[5792]: [TOTEM] entering GATHER state from 11.
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] Saving state aru 87
high seq received 87
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] Storing new sequence
id for ring d8
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] entering COMMIT state.
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] entering RECOVERY state.
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] previous ring seq 212
rep 192.168.2.120
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] aru a high delivered a
received flag 1
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] previous ring seq 212
rep 192.168.2.220
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] aru 87 high delivered
87 received flag 1
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 216: memb=1,
new=0, lost=0
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] New Configuration:
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 13:01:49 srv-dell cib: [5799]: notice: ais_dispatch: Membership
216: quorum aquired
Jun 3 13:01:49 srv-dell ocfs2_controld: [15196]: notice:
ais_dispatch: Membership 216: quorum aquired
Jun 3 13:01:49 srv-dell cluster-dlm: [14994]: notice: ais_dispatch:
Membership 216: quorum aquired
Jun 3 13:01:49 srv-dell ocfs2_controld: [15196]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=192 seen=216
proc=00000000000000000000000000053312
Jun 3 13:01:49 srv-dell cib: [5799]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=216
proc=00000000000000000000000000053312
Jun 3 13:01:49 srv-dell crmd: [5803]: notice: ais_dispatch:
Membership 216: quorum aquired
Jun 3 13:01:49 srv-dell cluster-dlm: [14994]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=216
proc=00000000000000000000000000053312
Jun 3 13:01:49 srv-dell cluster-dlm: update_cluster: Processing membership 216
Jun 3 13:01:49 srv-dell crmd: [5803]: info: ais_status_callback:
status: srv-lenovo is now member (was lost)
Jun 3 13:01:49 srv-dell ocfs2_controld: [15196]: info: ais_dispatch:
Membership 216: quorum retained
Jun 3 13:01:49 srv-dell cib: [5799]: info: ais_dispatch: Membership
216: quorum retained
Jun 3 13:01:49 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 3 13:01:49 srv-dell crmd: [5803]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=192 seen=216
proc=00000000000000000000000000053312
Jun 3 13:01:49 srv-dell ocfs2_controld: [15196]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member addr=r(0)
ip(192.168.2.220) votes=1 born=208 seen=216
proc=00000000000000000000000000013312 (new)
Jun 3 13:01:49 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 3 13:01:49 srv-dell cib: [5799]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member addr=r(0) ip(192.168.2.220)
votes=1 born=208 seen=216 proc=00000000000000000000000000013312 (new)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: crm_update_quorum:
Updating quorum status to true (call=254)
Jun 3 13:01:49 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=192, last-seen=216, this-event=216,
last-event=212
Jun 3 13:01:49 srv-dell cluster-dlm: dlm_process_node: Skipped active
node 1543678144: born-on=208, last-seen=216, this-event=216,
last-event=212
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] Members Left:
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] Members Joined:
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 216: memb=2, new=1,
lost=0
Jun 3 13:01:49 srv-dell openais[5792]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: member
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: NEW: srv-lenovo 2013440192
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: MEMB: srv-lenovo 2013440192
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] info:
send_member_notification: Sending membership update 216 to 4 children
Jun 3 13:01:49 srv-dell openais[5792]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000013312
(78610)
Jun 3 13:01:49 srv-dell openais[5792]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 13:01:49 srv-dell openais[5792]: [TOTEM] entering OPERATIONAL state.
Jun 3 13:01:49 srv-dell openais[5792]: [MAIN ] info: update_member:
0x7fc95c01c7e0 Node 2013440192 (srv-lenovo) born on: 216
Jun 3 13:01:49 srv-dell openais[5792]: [crm ] info:
send_member_notification: Sending membership update 216 to 4 children
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 13:01:49 srv-dell openais[5792]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 13:01:49 srv-dell openais[5792]: [CPG ] got joinlist message
from node 1543678144
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/250,
version=0.92.35): ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=local/crmd/251, version=0.92.36): ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/252, version=0.92.36): ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: log_data_element:
cib:diff: - <cib have-quorum="0" admin_epoch="0" epoch="92"
num_updates="37" />
Jun 3 13:01:49 srv-dell cib: [5799]: info: log_data_element:
cib:diff: + <cib have-quorum="1" admin_epoch="0" epoch="93"
num_updates="1" />
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/254, version=0.93.1): ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_sync_one for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.93.1): ok (rc=0)
Jun 3 13:01:49 srv-dell cluster-dlm: fence_node_time: Node
2013440192/srv-lenovo was last shot 'now'
Jun 3 13:01:49 srv-dell cluster-dlm: check_fencing_done:
check_fencing 2013440192 1244048266 fenced at 1244048509
Jun 3 13:01:49 srv-dell crmd: [5803]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=o2cb:0_monitor_0,
magic=0:0;8:57:7:7c754d7f-6d3c-4527-b36c-979453669faf) : Resource op
removal
Jun 3 13:01:49 srv-dell crmd: [5803]: info: update_abort_priority:
Abort priority upgraded from 0 to 1000000
Jun 3 13:01:49 srv-dell crmd: [5803]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 13:01:49 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: abort_transition_graph:
te_update_diff:157 - Triggered transition abort (complete=0,
tag=transient_attributes, id=srv-lenovo, magic=NA) : Transient
attribute: removal
Jun 3 13:01:49 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/transient_attributes":
ok (rc=0)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=0) : Non-status
change
Jun 3 13:01:49 srv-dell crmd: [5803]: info: need_abort: Aborting on
change to have-quorum
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/256, version=0.93.1): ok (rc=0)
Jun 3 13:01:49 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 13:01:49 srv-dell crmd: [5803]: info: ais_dispatch: Membership
216: quorum retained
Jun 3 13:01:49 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 13:01:49 srv-dell crmd: [5803]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member addr=r(0) ip(192.168.2.220)
votes=1 born=208 seen=216 proc=00000000000000000000000000013312 (new)
Jun 3 13:01:49 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 13:01:49 srv-dell cluster-dlm: send_info: send_start cg 2 flags
2 counts 1 1 0 1 1
Jun 3 13:01:49 srv-dell cluster-dlm: [14994]: info: ais_dispatch:
Membership 216: quorum retained
Jun 3 13:01:49 srv-dell cib: [16247]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-93.raw
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/257, version=0.93.1): ok (rc=0)
Jun 3 13:01:49 srv-dell cluster-dlm: [14994]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member addr=r(0) ip(192.168.2.220)
votes=1 born=208 seen=216 proc=00000000000000000000000000013312 (new)
Jun 3 13:01:49 srv-dell cib: [16247]: info: write_cib_contents: Wrote
version 0.93.0 of the CIB to disk (digest:
77d155fb655de5f5509b17a066379e28)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/260, version=0.93.1): ok (rc=0)
Jun 3 13:01:49 srv-dell cluster-dlm: receive_start: receive_start
1543678144:2 len 76
Jun 3 13:01:49 srv-dell cluster-dlm: match_change: match_change
1543678144:2 matches cg 2
Jun 3 13:01:49 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 2 got all 1
Jun 3 13:01:49 srv-dell cluster-dlm: start_kernel: start_kernel cg 2
member_count 1
Jun 3 13:01:49 srv-dell cib: [16247]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Gg0ig3 (digest:
/var/lib/heartbeat/crm/cib.5aqAfj)
Jun 3 13:01:49 srv-dell cluster-dlm: update_dir_members: dir_member 2013440192
Jun 3 13:01:49 srv-dell cluster-dlm: update_dir_members: dir_member 1543678144
Jun 3 13:01:49 srv-dell cluster-dlm: set_configfs_members:
set_members rmdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/2013440192"
Jun 3 13:01:49 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 13:01:49 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 3 13:01:49 srv-dell ocfs2_controld: client msg
Jun 3 13:01:49 srv-dell ocfs2_controld: client message 1 from 5: MRESULT
Jun 3 13:01:49 srv-dell ocfs2_controld: complete_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", errcode "0", service "ocfs2"
Jun 3 13:01:49 srv-dell ocfs2_controld: client msg
Jun 3 13:01:49 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 13:01:49 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 13:01:49 srv-dell kernel: ocfs2: Mounting device (8,17) on
(node 1543678, slot 1) with ordered data mode.
Jun 3 13:01:49 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation fs:1_stop_0 (call=65, rc=0, cib-update=261, confirmed=true)
complete ok
Jun 3 13:01:49 srv-dell ocfs2_hb_ctl[16251]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -P -d /dev/sdb1
Jun 3 13:01:49 srv-dell kernel: (16250,0):ocfs2_replay_journal:1311
Recovering node 2013440192 from slot 0 on device (8,17)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:1_stop_0 (22) confirmed on srv-dell (rc=0)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 13:01:49 srv-dell crmd: [5803]: info: run_graph:
====================================================
Jun 3 13:01:49 srv-dell crmd: [5803]: notice: run_graph: Transition
11 (Complete=5, Pending=0, Fired=0, Skipped=5, Incomplete=3,
Source=/var/lib/pengine/pe-warn-24.bz2): Stopped
Jun 3 13:01:49 srv-dell crmd: [5803]: info: te_graph_trigger:
Transition 11 is now complete
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_state_transition:
Membership changed: 212 -> 216 - join restart
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_pe_invoke: Query 262:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=do_state_transition ]
Jun 3 13:01:49 srv-dell crmd: [5803]: info: update_dc: Unset DC srv-dell
Jun 3 13:01:49 srv-dell crmd: [5803]: info: join_make_offer: Making
join offers based on membership 216
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_dc_join_offer_all:
join-4: Waiting on 2 outstanding join acks
Jun 3 13:01:49 srv-dell crmd: [5803]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 13:01:49 srv-dell ocfs2_controld: new client connection 5
Jun 3 13:01:49 srv-dell ocfs2_controld: client msg
Jun 3 13:01:49 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 3 13:01:49 srv-dell ocfs2_controld: client msg
Jun 3 13:01:49 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 13:01:49 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 13:01:49 srv-dell lrmd: [5800]: info: RA output:
(fs:1:start:stderr) Terminated
Jun 3 13:01:49 srv-dell crmd: [5803]: info: update_dc: Unset DC srv-dell
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_dc_join_offer_all: A
new node joined the cluster
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_dc_join_offer_all:
join-5: Waiting on 2 outstanding join acks
Jun 3 13:01:49 srv-dell crmd: [5803]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_state_transition: All
2 cluster nodes responded to the join offer.
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_dc_join_finalize:
join-5: Syncing the CIB from srv-dell to the rest of the cluster
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/265, version=0.93.2): ok (rc=0)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: do_dc_join_ack: join-5:
Updating node state to member for srv-dell
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/266, version=0.93.2): ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/267, version=0.93.2): ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/268,
version=0.93.3): ok (rc=0)
Jun 3 13:01:49 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=srv-lenovo/crmd/7, version=0.93.4): ok (rc=0)
Jun 3 13:01:49 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=srv-lenovo/crmd/8,
version=0.93.4): ok (rc=0)
Jun 3 13:01:50 srv-dell crmd: [5803]: info: do_dc_join_ack: join-5:
Updating node state to member for srv-lenovo
Jun 3 13:01:50 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/270,
version=0.93.4): ok (rc=0)
Jun 3 13:01:50 srv-dell crmd: [5803]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 3 13:01:50 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 13:01:50 srv-dell crmd: [5803]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 3 13:01:50 srv-dell crmd: [5803]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 3 13:01:50 srv-dell crmd: [5803]: info: crm_update_quorum:
Updating quorum status to true (call=274)
Jun 3 13:01:50 srv-dell crmd: [5803]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 3 13:01:50 srv-dell crmd: [5803]: info: do_pe_invoke: Query 275:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 13:01:50 srv-dell attrd: [5801]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 13:01:50 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 13:01:50 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/272, version=0.93.5): ok (rc=0)
Jun 3 13:01:50 srv-dell cib: [5799]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/274, version=0.93.5): ok (rc=0)
Jun 3 13:01:50 srv-dell attrd: [5801]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 13:01:50 srv-dell crmd: [5803]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244048510-110, seq=216, quorate=1
Jun 3 13:01:50 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-dell is shutting down
Jun 3 13:01:50 srv-dell pengine: [5802]: info: unpack_rsc_op:
o2cb:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation o2cb:0_monitor_0 found resource o2cb:0 active on srv-dell
Jun 3 13:01:50 srv-dell pengine: [5802]: info: unpack_rsc_op:
dlm:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation dlm:0_monitor_0 found resource dlm:0 active on srv-dell
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Processing failed op fs:1_start_0 on srv-dell: unknown exec error
Jun 3 13:01:50 srv-dell pengine: [5802]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-dell returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-dell
Jun 3 13:01:50 srv-dell pengine: [5802]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-dell returned 0
(ok) instead of the expected value: 7 (not running)
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-dell
Jun 3 13:01:50 srv-dell pengine: [5802]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 13:01:50 srv-dell pengine: [5802]: info:
determine_online_status: Node srv-lenovo is standby
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: print_list:
Started: [ srv-dell ]
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:1 ]
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: print_list:
Started: [ srv-dell ]
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ dlm:0 ]
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: print_list:
Started: [ srv-dell ]
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ o2cb:0 ]
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: clone_print: Clone
Set: fs-clone
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 13:01:50 srv-dell pengine: [5802]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 13:01:50 srv-dell pengine: [5802]: info: stage6: Scheduling
Node srv-dell for shutdown
Jun 3 13:01:50 srv-dell pengine: [5802]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-dell
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Stopped)
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Leave
resource dlm:0 (Stopped)
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Stop
resource dlm:1 (srv-dell)
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Leave
resource o2cb:0 (Stopped)
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Stop
resource o2cb:1 (srv-dell)
Jun 3 13:01:50 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:0 (Stopped)
Jun 3 13:01:51 srv-dell pengine: [5802]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 13:01:51 srv-dell crmd: [5803]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 13:01:51 srv-dell crmd: [5803]: info: unpack_graph: Unpacked
transition 12: 20 actions in 20 synapses
Jun 3 13:01:51 srv-dell crmd: [5803]: info: do_te_invoke: Processing
graph 12 (ref=pe_calc-dc-1244048510-110) derived from
/var/lib/pengine/pe-warn-25.bz2
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 7: monitor primitiveresource_cluster_ip_monitor_0 on
srv-lenovo
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op primitiveresource_cluster_ip_monitor_0 in the CIB
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 8: monitor
cloneresource_cluster_fencing_null:0_monitor_0 on srv-lenovo
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op cloneresource_cluster_fencing_null:0_monitor_0 in the CIB
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 9: monitor dlm:0_monitor_0 on srv-lenovo
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op dlm:0_monitor_0 in the CIB
Jun 3 13:01:51 srv-dell pengine: [5802]: WARN: process_pe_message:
Transition 12: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-25.bz2
Jun 3 13:01:51 srv-dell pengine: [5802]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 10: monitor o2cb:0_monitor_0 on srv-lenovo
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op o2cb:0_monitor_0 in the CIB
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 11: monitor fs:0_monitor_0 on srv-lenovo
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op fs:0_monitor_0 in the CIB
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 30 fired and confirmed
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 5: probe_complete probe_complete on srv-dell (local)
- no waiting
Jun 3 13:01:51 srv-dell crmd: [5803]: info: match_graph_event: Action
cloneresource_cluster_fencing_null:0_monitor_0 (8) confirmed on
srv-lenovo (rc=0)
Jun 3 13:01:51 srv-dell crmd: [5803]: info: match_graph_event: Action
primitiveresource_cluster_ip_monitor_0 (7) confirmed on srv-lenovo
(rc=0)
Jun 3 13:01:51 srv-dell crmd: [5803]: info: match_graph_event: Action
dlm:0_monitor_0 (9) confirmed on srv-lenovo (rc=0)
Jun 3 13:01:51 srv-dell crmd: [5803]: info: match_graph_event: Action
o2cb:0_monitor_0 (10) confirmed on srv-lenovo (rc=0)
Jun 3 13:01:51 srv-dell crmd: [5803]: info: match_graph_event: Action
fs:0_monitor_0 (11) confirmed on srv-lenovo (rc=0)
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 6: probe_complete probe_complete on srv-lenovo - no
waiting
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 4 fired and confirmed
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 15 fired and confirmed
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 20 fired and confirmed
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command:
Initiating action 22: stop o2cb:1_stop_0 on srv-dell (local)
Jun 3 13:01:51 srv-dell crmd: [5803]: info: do_lrm_rsc_op: Performing
key=22:12:0:9f20fbee-34bd-4374-9bc1-2ea89443936d op=o2cb:1_stop_0 )
Jun 3 13:01:51 srv-dell lrmd: [5800]: info: rsc:o2cb:1: stop
Jun 3 13:01:51 srv-dell crmd: [5803]: info: te_rsc_command: Recording
pending op o2cb:1_stop_0 in the CIB
Jun 3 13:01:51 srv-dell crmd: [5803]: info: process_lrm_event: LRM
operation o2cb:1_monitor_120000 (call=52, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 13:01:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:stop:stderr) logd is not running
Jun 3 13:01:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:stop:stderr) 2009/06/03_13:01:51 INFO: Stopping o2cb:1
Jun 3 13:01:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:stop:stderr) logd is not running
Jun 3 13:01:51 srv-dell lrmd: [5800]: info: RA output:
(o2cb:1:stop:stderr) 2009/06/03_13:01:51 INFO: Stopping
ocfs2_controld.pcmk
Jun 3 13:01:51 srv-dell ocfs2_controld: Caught signal 15
Jun 3 13:01:51 srv-dell ocfs2_controld: Caught signal 15, but mounts
exist. Ignoring.
Jun 3 13:01:51 srv-dell ocfs2_controld: Caught signal 15, but mounts
exist. Ignoring.
Jun 3 13:01:52 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 3 13:01:52 srv-dell kernel:
(16250,5):ocfs2_begin_quota_recovery:371 Beginning quota recovery in
slot 0
Jun 3 13:01:52 srv-dell kernel:
(15193,7):ocfs2_finish_quota_recovery:555 Finishing quota recovery in
slot 0
Jun 3 13:03:54 srv-dell syslog-ng[3195]: syslog-ng starting up; version='2.0.9'
Jun 3 13:03:55 srv-dell ifup: lo
Jun 3 13:03:55 srv-dell ifup: lo
Jun 3 13:03:55 srv-dell ifup: IP address: 127.0.0.1/8
Jun 3 13:03:55 srv-dell ifup:
Jun 3 13:03:55 srv-dell ifup:
Jun 3 13:03:55 srv-dell ifup: IP address: 127.0.0.2/8
Jun 3 13:03:55 srv-dell ifup:
Jun 3 13:03:55 srv-dell rchal: CPU frequency scaling is not supported
by your processor.
Jun 3 13:03:55 srv-dell rchal: boot with 'CPUFREQ=no' in to avoid this warning.
Jun 3 13:03:55 srv-dell rchal: Cannot load cpufreq governors - No
cpufreq driver available
Jun 3 13:03:56 srv-dell ifup: eth0 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 13:03:56 srv-dell ifup: No configuration found for eth0
Jun 3 13:03:56 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 13:03:56 srv-dell ifup-dhcp: eth1 Starting DHCP4 client
Jun 3 13:03:56 srv-dell dhcpcd[3683]: eth1: dhcpcd 3.2.3 starting
Jun 3 13:03:56 srv-dell dhcpcd[3683]: eth1: hardware address =
00:19:b9:d7:20:d7
Jun 3 13:03:56 srv-dell dhcpcd[3683]: eth1: broadcasting for a lease
Jun 3 13:03:57 srv-dell ifup-dhcp: .
Jun 3 13:03:59 srv-dell ifup-dhcp: .
Jun 3 13:03:59 srv-dell kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.178:2):
operation="profile_load" name="/bin/ping" name2="default" pid=2963
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.221:3):
operation="profile_load" name="/sbin/klogd" name2="default" pid=3003
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.278:4):
operation="profile_load" name="/sbin/syslog-ng" name2="default"
pid=3015
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.332:5):
operation="profile_load" name="/sbin/syslogd" name2="default" pid=3028
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.393:6):
operation="profile_load" name="/usr/sbin/avahi-daemon" name2="default"
pid=3039
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.442:7):
operation="profile_load" name="/usr/sbin/identd" name2="default"
pid=3040
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.501:8):
operation="profile_load" name="/usr/sbin/mdnsd" name2="default"
pid=3041
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.574:9):
operation="profile_load" name="/usr/sbin/nscd" name2="default"
pid=3042
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.688:10):
operation="profile_load" name="/usr/sbin/ntpd" name2="default"
pid=3043
Jun 3 13:03:59 srv-dell kernel: type=1505 audit(1244048633.738:11):
operation="profile_load" name="/usr/sbin/traceroute" name2="default"
pid=3046
Jun 3 13:03:59 srv-dell kernel: IA-32 Microcode Update Driver:
v1.14a-xen <tigran at aivazian.fsnet.co.uk>
Jun 3 13:03:59 srv-dell kernel: firmware: requesting intel-ucode/06-0f-07
Jun 3 13:03:59 srv-dell kernel: bnx2: eth1: using MSI
Jun 3 13:03:59 srv-dell kernel: NET: Registered protocol family 17
Jun 3 13:03:59 srv-dell kernel: bnx2: eth1 NIC Copper Link is Up,
1000 Mbps full duplex, receive & transmit flow control ON
Jun 3 13:04:01 srv-dell ifup-dhcp: .
Jun 3 13:04:02 srv-dell dhcpcd[3683]: eth1: offered 192.168.2.220
from 192.168.2.1
Jun 3 13:04:02 srv-dell dhcpcd[3683]: eth1: checking 192.168.2.220 is
available on attached networks
Jun 3 13:04:03 srv-dell ifup-dhcp: .
Jun 3 13:04:03 srv-dell dhcpcd[3683]: eth1: leased 192.168.2.220 for
86400 seconds
Jun 3 13:04:03 srv-dell dhcpcd[3683]: eth1: no renewal time supplied,
assuming 43200 seconds
Jun 3 13:04:03 srv-dell dhcpcd[3683]: eth1: no rebind time supplied,
assuming 75600 seconds
Jun 3 13:04:03 srv-dell dhcpcd[3683]: eth1: adding IP address 192.168.2.220/24
Jun 3 13:04:03 srv-dell dhcpcd[3683]: eth1: adding default route via
192.168.2.1 metric 0
Jun 3 13:04:03 srv-dell dns-resolver: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 13:04:03 srv-dell dns-resolver: You can find my version in
/etc/resolv.conf.netconfig
Jun 3 13:04:03 srv-dell dhcpcd-hook: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 13:04:03 srv-dell dhcpcd-hook: You can find my version in
/etc/resolv.conf.netconfig ...
Jun 3 13:04:03 srv-dell syslog-ng[3195]: Configuration reload request
received, reloading configuration;
Jun 3 13:04:03 srv-dell syslog-ng[3195]: New configuration initialized;
Jun 3 13:04:04 srv-dell ifdown: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 13:04:04 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 13:04:04 srv-dell ifup-dhcp:
Jun 3 13:04:04 srv-dell ifup-dhcp: eth1 IP address: 192.168.2.220/24
Jun 3 13:04:04 srv-dell SuSEfirewall2: SuSEfirewall2 not active
Jun 3 13:04:04 srv-dell auditd[4224]: Started dispatcher:
/sbin/audispd pid: 4226
Jun 3 13:04:05 srv-dell audispd: priority_boost_parser called with: 4
Jun 3 13:04:05 srv-dell audispd: af_unix plugin initialized
Jun 3 13:04:05 srv-dell audispd: audispd initialized with q_depth=80
and 1 active plugins
Jun 3 13:04:05 srv-dell auditd[4224]: Init complete, auditd 1.7.7
listening for events (startup state disable)
Jun 3 13:04:05 srv-dell kernel: NET: Registered protocol family 10
Jun 3 13:04:05 srv-dell kernel: lo: Disabled Privacy Extensions
Jun 3 13:04:05 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 13:04:05 srv-dell dhcpcd[3683]: eth1: setting hostname to `srv-dell'
Jun 3 13:04:05 srv-dell dhcpcd[3683]: eth1: exiting
Jun 3 13:04:06 srv-dell smartd[4613]: smartd 5.39 2008-10-24 22:33
[x86_64-suse-linux-gnu] (openSUSE RPM) Copyright (C) 2002-8 by Bruce
Allen, http://smartmontools.sourceforge.net
Jun 3 13:04:06 srv-dell smartd[4613]: Opened configuration file
/etc/smartd.conf
Jun 3 13:04:06 srv-dell smartd[4613]: Drive: DEVICESCAN, implied '-a'
Directive on line 26 of file /etc/smartd.conf
Jun 3 13:04:06 srv-dell smartd[4613]: Configuration file
/etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Jun 3 13:04:06 srv-dell smartd[4613]: Device: /dev/sda, opened
Jun 3 13:04:06 srv-dell smartd[4613]: Device: /dev/sda, IE (SMART)
not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on
SMART features
Jun 3 13:04:06 srv-dell smartd[4613]: Unable to monitor any SMART
enabled devices. Try debug (-d) option. Exiting...
Jun 3 13:04:06 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 13:04:06 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 13:04:06 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 13:04:06 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 13:04:06 srv-dell sshd[4876]: Server listening on 0.0.0.0 port 22.
Jun 3 13:04:06 srv-dell sshd[4876]: Server listening on :: port 22.
Jun 3 13:04:06 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/Xservers
Jun 3 13:04:06 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/xdm-config
Jun 3 13:04:06 srv-dell kernel: drbd: initialised. Version: 8.2.7
(api:88/proto:86-88)
Jun 3 13:04:06 srv-dell kernel: drbd: GIT-hash:
a1b440e8b3011a1318d8bff1bb7edc763ef995b0 build by lmb at hermes,
2009-02-20 13:35:59
Jun 3 13:04:06 srv-dell kernel: drbd: registered as block device major 147
Jun 3 13:04:06 srv-dell kernel: drbd: minor_table @ 0xffff8801e918c980
Jun 3 13:04:06 srv-dell kernel: drbd0: conn( StandAlone -> Unconnected )
Jun 3 13:04:06 srv-dell kernel: drbd0: Starting worker thread (from
cqueue [48])
Jun 3 13:04:06 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 13:04:06 srv-dell kernel: drbd0: Starting receiver thread (from
drbd0_worker [4943])
Jun 3 13:04:06 srv-dell kernel: drbd0: receiver (re)started
Jun 3 13:04:06 srv-dell kernel: drbd0: conn( Unconnected -> WFConnection )
Jun 3 13:04:07 srv-dell kernel: vendor=8086 device=244e
Jun 3 13:04:07 srv-dell kernel: pci 0000:10:0d.0: PCI INT A -> GSI 19
(level, low) -> IRQ 19
Jun 3 13:04:07 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 3 13:04:07 srv-dell /usr/sbin/cron[5038]: (CRON) STARTUP (V5.0)
Jun 3 13:04:07 srv-dell kernel: iscsi: registered transport (tcp)
Jun 3 13:04:07 srv-dell kernel: iscsi: registered transport (iser)
Jun 3 13:04:07 srv-dell iscsid: iSCSI logger with pid=5148 started!
Jun 3 13:04:07 srv-dell xenstored: Checking store ...
Jun 3 13:04:07 srv-dell xenstored: Checking store complete.
Jun 3 13:04:07 srv-dell kernel: suspend: event channel 55
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:795: blktapctrl: v1.0.0
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [raw image (aio)]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [raw image (sync)]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [vmware image (vmdk)]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [ramdisk image (ram)]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [qcow disk (qcow)]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [qcow2 disk (qcow2)]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [ioemu disk]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl.c:797: Found
driver: [raw image (cdrom)]
Jun 3 13:04:07 srv-dell BLKTAPCTRL[5226]: blktapctrl_linux.c:23:
/dev/xen/blktap0 device already exists
Jun 3 13:04:07 srv-dell kernel: Bridge firewalling registered
Jun 3 13:04:08 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 3 13:04:08 srv-dell iscsid: iSCSI daemon with pid=5149 started!
Jun 3 13:04:13 srv-dell gdm-simple-greeter[5383]: libglade-WARNING:
Unexpected element <requires-version> inside <glade-interface>.
Jun 3 13:04:13 srv-dell gdm-session-worker[5385]: PAM pam_putenv:
NULL pam handle passed
Jun 3 13:04:16 srv-dell kernel: eth1: no IPv6 routers present
Jun 3 13:04:20 srv-dell kernel: metacity[5393]: segfault at 0 ip
00000000004644be sp 00007fff9e1485d0 error 4 in metacity[400000+88000]
Jun 3 13:04:24 srv-dell pulseaudio[5592]: pid.c: Stale PID file, overwriting.
Jun 3 13:04:24 srv-dell python: hp-systray(init)[5570]: error:
hp-systray cannot be run as root. Exiting.
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] AIS Executive Service
RELEASE 'subrev 1152 version 0.80'
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] AIS Executive Service:
started and ready to provide service.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Token Timeout (5000
ms) retransmit timeout (490 ms)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] token hold (382 ms)
retransmits before loss (10 retrans)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] join (1000 ms)
send_join (45 ms) consensus (2500 ms) merge (200 ms)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] seqno unchanged const
(30 rotations) Maximum network MTU 1500
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] send threads (0 threads)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] RRP token expired
timeout (490 ms)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] RRP token problem
counter (2000 ms)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] RRP threshold (10 problem count)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] RRP mode set to none.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] heartbeat_failures_allowed (0)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] max_network_delay (50 ms)
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] HeartBeat is Disabled.
To enable set heartbeat_failures_allowed > 0
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] The network interface
[192.168.2.220] is now up.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Created or loaded
sequence id 216.192.168.2.220 for this ring.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] entering GATHER state from 15.
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info:
process_ais_conf: Reading configure
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info:
config_find_next: Processing additional logging options...
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: get_config_opt:
Found 'off' for option: debug
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: get_config_opt:
Found 'yes' for option: to_syslog
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: get_config_opt:
Found 'daemon' for option: syslog_facility
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info:
config_find_next: Processing additional service options...
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_logd
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_mgmtd
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info:
pcmk_plugin_init: CRM: Initialized
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] Logging: Initialized
pcmk_plugin_init
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info:
pcmk_plugin_init: Service: 9
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info:
pcmk_plugin_init: Local node id: 1543678144
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info:
pcmk_plugin_init: Local hostname: srv-dell
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: update_member:
Creating entry for node 1543678144 born on 0
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: update_member:
0x73e580 Node 1543678144 now known as srv-dell (was: (null))
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: update_member:
Node srv-dell now has 1 quorum votes (was 0)
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: update_member:
Node 1543678144/srv-dell is now: member
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: spawn_child:
Forked child 5667 for process stonithd
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: spawn_child:
Forked child 5668 for process cib
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: spawn_child:
Forked child 5669 for process lrmd
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: spawn_child:
Forked child 5670 for process attrd
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: spawn_child:
Forked child 5671 for process pengine
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: spawn_child:
Forked child 5672 for process crmd
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: spawn_child:
Forked child 5673 for process mgmtd
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info: pcmk_startup:
CRM: Initialized
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] Service initialized
'Pacemaker Cluster Manager'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais extended virtual synchrony service'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais cluster membership service B.01.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais availability management framework B.01.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais checkpoint service B.01.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais event service B.01.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais distributed locking service B.01.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais message service B.01.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais configuration service'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais cluster closed process group service v1.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SERV ] Service initialized
'openais cluster config database access v1.01'
Jun 3 13:04:42 srv-dell openais[5661]: [SYNC ] Not using a virtual
synchrony filter.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Saving state aru 0
high seq received 0
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Storing new sequence
id for ring dc
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] entering COMMIT state.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] entering RECOVERY state.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] previous ring seq 216
rep 192.168.2.220
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] aru 0 high delivered 0
received flag 1
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] Sending initial ORF token
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] New Configuration:
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] Members Left:
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] Members Joined:
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 220: memb=0,
new=0, lost=0
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] New Configuration:
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] Members Left:
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] Members Joined:
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 220: memb=1, new=1,
lost=0
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: NEW: srv-dell 1543678144
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 13:04:42 srv-dell openais[5661]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000053312
(340754)
Jun 3 13:04:42 srv-dell openais[5661]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 13:04:42 srv-dell openais[5661]: [TOTEM] entering OPERATIONAL state.
Jun 3 13:04:42 srv-dell openais[5661]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 13:04:42 srv-dell lrmd: [5669]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:42 srv-dell lrmd: [5669]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Jun 3 13:04:42 srv-dell mgmtd: [5673]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:42 srv-dell mgmtd: [5673]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 3 13:04:42 srv-dell mgmtd: [5673]: debug: Enabling coredumps
Jun 3 13:04:42 srv-dell mgmtd: [5673]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 13:04:42 srv-dell mgmtd: [5673]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 13:04:42 srv-dell attrd: [5670]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 13:04:42 srv-dell pengine: [5671]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 13:04:42 srv-dell cib: [5668]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 13:04:42 srv-dell mgmtd: [5673]: WARN: lrm_signon: can not
initiate connection
Jun 3 13:04:42 srv-dell mgmtd: [5673]: info: login to lrm: 0, ret:0
Jun 3 13:04:42 srv-dell attrd: [5670]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:42 srv-dell pengine: [5671]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:42 srv-dell cib: [5668]: WARN: Initializing connection to
logging daemon failed. Logging daemon may not be running
Jun 3 13:04:42 srv-dell stonithd: [5667]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:42 srv-dell stonithd: [5667]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 13:04:42 srv-dell crmd: [5672]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 13:04:42 srv-dell attrd: [5670]: info: main: Starting up....
Jun 3 13:04:42 srv-dell stonithd: [5667]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 13:04:42 srv-dell attrd: [5670]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 13:04:42 srv-dell cib: [5668]: info: G_main_add_TriggerHandler:
Added signal manual handler
Jun 3 13:04:42 srv-dell cib: [5668]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 13:04:42 srv-dell crmd: [5672]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:42 srv-dell attrd: [5670]: info: init_ais_connection: AIS
connection established
Jun 3 13:04:42 srv-dell pengine: [5671]: info: main: Starting pengine
Jun 3 13:04:42 srv-dell crmd: [5672]: info: main: CRM Hg Version:
0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Jun 3 13:04:42 srv-dell stonithd: [5667]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 13:04:42 srv-dell crmd: [5672]: info: crmd_init: Starting crmd
Jun 3 13:04:42 srv-dell crmd: [5672]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 13:04:42 srv-dell lrmd: [5669]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 13:04:42 srv-dell lrmd: [5669]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jun 3 13:04:42 srv-dell lrmd: [5669]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jun 3 13:04:42 srv-dell lrmd: [5669]: info: Started.
Jun 3 13:04:42 srv-dell attrd: [5670]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 13:04:42 srv-dell cib: [5668]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Jun 3 13:04:42 srv-dell stonithd: [5667]: info: init_ais_connection:
AIS connection established
Jun 3 13:04:42 srv-dell attrd: [5670]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 13:04:42 srv-dell attrd: [5670]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 13:04:42 srv-dell stonithd: [5667]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 13:04:42 srv-dell stonithd: [5667]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f44f001dc50 for attrd/5670
Jun 3 13:04:42 srv-dell stonithd: [5667]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 13:04:42 srv-dell stonithd: [5667]: notice:
/usr/lib64/heartbeat/stonithd start up successfully.
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f44f001d6a0 for stonithd/5667
Jun 3 13:04:42 srv-dell stonithd: [5667]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 13:04:42 srv-dell cib: [5668]: info: startCib: CIB
Initialization completed successfully
Jun 3 13:04:42 srv-dell cib: [5668]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 13:04:42 srv-dell cib: [5668]: info: init_ais_connection: AIS
connection established
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f44f001dd50 for cib/5668
Jun 3 13:04:42 srv-dell openais[5661]: [crm ] info: pcmk_ipc:
Sending membership update 220 to cib
Jun 3 13:04:42 srv-dell cib: [5668]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 13:04:42 srv-dell cib: [5668]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 13:04:42 srv-dell cib: [5668]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 13:04:42 srv-dell cib: [5668]: info: cib_init: Starting cib mainloop
Jun 3 13:04:42 srv-dell cib: [5668]: info: ais_dispatch: Membership
220: quorum still lost
Jun 3 13:04:42 srv-dell cib: [5668]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=220
proc=00000000000000000000000000053312 (new)
Jun 3 13:04:42 srv-dell cib: [5677]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-94.raw
Jun 3 13:04:42 srv-dell cib: [5677]: info: write_cib_contents: Wrote
version 0.93.0 of the CIB to disk (digest:
1716ba49c9d92880b1d18373c1f95271)
Jun 3 13:04:42 srv-dell cib: [5677]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.BzIALo (digest:
/var/lib/heartbeat/crm/cib.qoKGjK)
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] entering GATHER state from 11.
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] Saving state aru a
high seq received a
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] Storing new sequence
id for ring e0
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] entering COMMIT state.
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] entering RECOVERY state.
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] previous ring seq 220
rep 192.168.2.120
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] aru 51 high delivered
51 received flag 1
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] previous ring seq 220
rep 192.168.2.220
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] aru a high delivered a
received flag 1
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] New Configuration:
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] Members Left:
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] Members Joined:
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 224: memb=1,
new=0, lost=0
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] New Configuration:
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] Members Left:
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] Members Joined:
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 224: memb=2, new=1,
lost=0
Jun 3 13:04:43 srv-dell cib: [5668]: info: ais_dispatch: Membership
224: quorum still lost
Jun 3 13:04:43 srv-dell cib: [5668]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 13:04:43 srv-dell cib: [5668]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=224 proc=00000000000000000000000000000000
Jun 3 13:04:43 srv-dell openais[5661]: [MAIN ] info: update_member:
Creating entry for node 2013440192 born on 224
Jun 3 13:04:43 srv-dell openais[5661]: [MAIN ] info: update_member:
Node 2013440192/unknown is now: member
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: NEW: .pending. 2013440192
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: MEMB: .pending. 2013440192
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 13:04:43 srv-dell cib: [5668]: info: ais_dispatch: Membership
224: quorum still lost
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info:
send_member_notification: Sending membership update 224 to 1 children
Jun 3 13:04:43 srv-dell openais[5661]: [MAIN ] info: update_member:
0x73e580 Node 1543678144 ((null)) born on: 224
Jun 3 13:04:43 srv-dell openais[5661]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 13:04:43 srv-dell openais[5661]: [TOTEM] entering OPERATIONAL state.
Jun 3 13:04:43 srv-dell openais[5661]: [MAIN ] info: update_member:
0x7f44f001d260 Node 2013440192 (srv-lenovo) born on: 216
Jun 3 13:04:43 srv-dell openais[5661]: [MAIN ] info: update_member:
0x7f44f001d260 Node 2013440192 now known as srv-lenovo (was: (null))
Jun 3 13:04:43 srv-dell cib: [5668]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 13:04:43 srv-dell openais[5661]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 3 13:04:43 srv-dell openais[5661]: [MAIN ] info: update_member:
Node srv-lenovo now has 1 quorum votes (was 0)
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info:
send_member_notification: Sending membership update 224 to 1 children
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 13:04:43 srv-dell cib: [5668]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=216 seen=224 proc=00000000000000000000000000053312
(new)
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 13:04:43 srv-dell openais[5661]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 13:04:43 srv-dell cib: [5668]: info: cib_process_diff: Diff
0.97.1 -> 0.97.2 not applied to 0.93.0: current "epoch" is less than
required
Jun 3 13:04:43 srv-dell cib: [5668]: info: cib_server_process_diff:
Requesting re-sync from peer
Jun 3 13:04:43 srv-dell cib: [5668]: WARN: cib_diff_notify:
Local-only Change (client:crmd, call: 53): 0.0.0 (Application of an
update diff failed, requesting a full refresh)
Jun 3 13:04:43 srv-dell cib: [5668]: WARN: cib_server_process_diff:
Not applying diff 0.97.2 -> 0.97.3 (sync in progress)
Jun 3 13:04:43 srv-dell cib: [5668]: WARN: cib_server_process_diff:
Not applying diff 0.97.3 -> 0.97.4 (sync in progress)
Jun 3 13:04:43 srv-dell cib: [5668]: WARN: cib_server_process_diff:
Not applying diff 0.97.4 -> 0.98.1 (sync in progress)
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 13:04:43 srv-dell cib: [5668]: info: cib_replace_notify:
Replaced: 0.93.0 -> 0.98.1 from srv-lenovo
Jun 3 13:04:43 srv-dell cib: [5678]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-95.raw
Jun 3 13:04:43 srv-dell cib: [5678]: info: write_cib_contents: Wrote
version 0.98.0 of the CIB to disk (digest:
01f98d2f74131084034d1217c1f65b68)
Jun 3 13:04:43 srv-dell cib: [5678]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.poWPyj (digest:
/var/lib/heartbeat/crm/cib.nLwX4G)
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 13:04:43 srv-dell mgmtd: [5673]: info: init_crm
Jun 3 13:04:43 srv-dell crmd: [5672]: info: do_cib_control: CIB
connection established
Jun 3 13:04:43 srv-dell crmd: [5672]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 13:04:43 srv-dell crmd: [5672]: info: init_ais_connection: AIS
connection established
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f44f0022bb0 for crmd/5672
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info: pcmk_ipc:
Sending membership update 224 to crmd
Jun 3 13:04:43 srv-dell crmd: [5672]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 13:04:43 srv-dell crmd: [5672]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 13:04:43 srv-dell crmd: [5672]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 13:04:43 srv-dell crmd: [5672]: info: do_ha_control: Connected
to the cluster
Jun 3 13:04:43 srv-dell crmd: [5672]: info: do_started: Delaying
start, CCM (0000000000100000) not connected
Jun 3 13:04:43 srv-dell crmd: [5672]: info: crmd_init: Starting crmd's mainloop
Jun 3 13:04:43 srv-dell crmd: [5672]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 13:04:43 srv-dell openais[5661]: [crm ] info:
update_expected_votes: Expected quorum votes 1024 -> 2
Jun 3 13:04:43 srv-dell crmd: [5672]: info: ais_dispatch: Membership
224: quorum still lost
Jun 3 13:04:43 srv-dell crmd: [5672]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 13:04:43 srv-dell crmd: [5672]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 13:04:43 srv-dell crmd: [5672]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=216 seen=224
proc=00000000000000000000000000053312
Jun 3 13:04:43 srv-dell crmd: [5672]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=224 seen=224
proc=00000000000000000000000000053312 (new)
Jun 3 13:04:43 srv-dell crmd: [5672]: info: do_started: The local CRM
is operational
Jun 3 13:04:43 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Jun 3 13:04:44 srv-dell crmd: [5672]: notice: ais_dispatch:
Membership 224: quorum aquired
Jun 3 13:04:44 srv-dell crmd: [5672]: info: update_dc: Set DC to
srv-lenovo (3.0.1)
Jun 3 13:04:44 srv-dell crmd: [5672]: info: update_attrd: Connecting
to attrd...
Jun 3 13:04:44 srv-dell crmd: [5672]: info: update_attrd: Updating
terminate=<none> via attrd for srv-dell
Jun 3 13:04:44 srv-dell crmd: [5672]: info: update_attrd: Updating
shutdown=<none> via attrd for srv-dell
Jun 3 13:04:44 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 3 13:04:44 srv-dell crmd: [5672]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/transient_attributes": ok
(rc=0)
Jun 3 13:04:44 srv-dell crmd: [5672]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 13:04:44 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=4:4:7:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 13:04:44 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 13:04:44 srv-dell lrmd: [5669]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 13:04:44 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=5:4:7:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=cloneresource_cluster_fencing_null:0_monitor_0 )
Jun 3 13:04:44 srv-dell lrmd: [5669]: info:
rsc:cloneresource_cluster_fencing_null:0: monitor
Jun 3 13:04:44 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=6:4:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=dlm:0_monitor_0 )
Jun 3 13:04:44 srv-dell lrmd: [5669]: info: rsc:dlm:0: monitor
Jun 3 13:04:44 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=7:4:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=o2cb:0_monitor_0 )
Jun 3 13:04:44 srv-dell lrmd: [5669]: info: rsc:o2cb:0: monitor
Jun 3 13:04:44 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=8:4:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_monitor_0 )
Jun 3 13:04:44 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_monitor_0 (call=3,
rc=7, cib-update=8, confirmed=true) complete not running
Jun 3 13:04:44 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:monitor:stderr) logd is not running
Jun 3 13:04:44 srv-dell lrmd: [5669]: info: RA output:
(dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Jun 3 13:04:44 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation dlm:0_monitor_0 (call=4, rc=7, cib-update=9, confirmed=true)
complete not running
Jun 3 13:04:44 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:monitor:stderr) 2009/06/03_13:04:44 INFO: configfs not laoded
Jun 3 13:04:44 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=2, rc=7,
cib-update=10, confirmed=true) complete not running
Jun 3 13:04:44 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation o2cb:0_monitor_0 (call=5, rc=7, cib-update=11,
confirmed=true) complete not running
Jun 3 13:04:45 srv-dell mgmtd: [5673]: debug: main: run the loop...
Jun 3 13:04:45 srv-dell mgmtd: [5673]: info: Started.
Jun 3 13:04:45 srv-dell lrmd: [5669]: info: rsc:fs:0: monitor
Jun 3 13:04:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) logd is not running
Jun 3 13:04:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) 2009/06/03_13:04:45 WARNING: Couldn't find
device [/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:04:46 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=6, rc=7, cib-update=12, confirmed=true)
complete not running
Jun 3 13:04:46 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=12:4:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 13:04:46 srv-dell lrmd: [5669]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 13:04:46 srv-dell lrmd: [5769]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 13:04:46 srv-dell stonithd: [5667]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 13:04:46 srv-dell lrmd: [5669]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 13:04:46 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=7, rc=0,
cib-update=16, confirmed=true) complete ok
Jun 3 13:04:46 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=18:4:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=dlm:0_start_0 )
Jun 3 13:04:46 srv-dell lrmd: [5669]: info: rsc:dlm:0: start
Jun 3 13:04:46 srv-dell lrmd: [5669]: info: RA output:
(dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Jun 3 13:04:46 srv-dell kernel: DLM (built Feb 28 2009 07:25:25) installed
Jun 3 13:04:46 srv-dell cluster-dlm[5810]: main: dlm_controld 1239267206
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info:
init_ais_connection: AIS connection established
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 13:04:46 srv-dell openais[5661]: [crm ] info: pcmk_notify:
Enabling node notifications for child 5810 (0x7f44f001f760)
Jun 3 13:04:46 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 3 13:04:46 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 3 13:04:46 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 3 13:04:46 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 3 13:04:46 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 3 13:04:46 srv-dell cluster-dlm: set_protocol: set_protocol
member_count 1 propose daemon 1.1.1 kernel 1.1.1
Jun 3 13:04:46 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 1543678144
Jun 3 13:04:46 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 3 13:04:46 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 3 13:04:46 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: notice: ais_dispatch:
Membership 224: quorum aquired
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=216 seen=224
proc=00000000000000000000000000053312
Jun 3 13:04:46 srv-dell cluster-dlm: [5810]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=224 seen=224
proc=00000000000000000000000000053312 (new)
Jun 3 13:04:46 srv-dell cluster-dlm: update_cluster: Processing membership 224
Jun 3 13:04:46 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 3 13:04:46 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 3 13:04:46 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=216, last-seen=224, this-event=224,
last-event=0
Jun 3 13:04:46 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 3 13:04:46 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 3 13:04:46 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=224, last-seen=224, this-event=224,
last-event=0
Jun 3 13:04:47 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation dlm:0_start_0 (call=8, rc=0, cib-update=17, confirmed=true)
complete ok
Jun 3 13:04:47 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=19:4:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=dlm:0_monitor_120000 )
Jun 3 13:04:47 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=26:4:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=o2cb:0_start_0 )
Jun 3 13:04:47 srv-dell lrmd: [5669]: info: rsc:o2cb:0: start
Jun 3 13:04:47 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=9, rc=0, cib-update=18,
confirmed=false) complete ok
Jun 3 13:04:47 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:start:stderr) logd is not running
Jun 3 13:04:47 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:start:stderr) 2009/06/03_13:04:47 INFO: Stack glue driver not
loaded
Jun 3 13:04:47 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:start:stderr) logd is not running
Jun 3 13:04:47 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:start:stderr) 2009/06/03_13:04:47 INFO: Starting o2cb:0
Jun 3 13:04:47 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 3 13:04:47 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 3 13:04:47 srv-dell kernel: OCFS2 1.5.0
Jun 3 13:04:47 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info:
init_ais_connection: AIS connection established
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 13:04:47 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 3 13:04:47 srv-dell openais[5661]: [crm ] info: pcmk_notify:
Enabling node notifications for child 5932 (0x7f44f0021cb0)
Jun 3 13:04:47 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 3 13:04:47 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 3 13:04:47 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 13:04:47 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 3 13:04:47 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 13:04:47 srv-dell ocfs2_controld: Stored section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 13:04:47 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 13:04:47 srv-dell ocfs2_controld: Stored section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 13:04:47 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 3 13:04:47 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 13:04:47 srv-dell ocfs2_controld: setup done
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: notice: ais_dispatch:
Membership 224: quorum aquired
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=216 seen=224
proc=00000000000000000000000000053312
Jun 3 13:04:47 srv-dell ocfs2_controld: [5932]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=224 seen=224
proc=00000000000000000000000000053312 (new)
Jun 3 13:04:47 srv-dell ocfs2_controld: confchg called
Jun 3 13:04:47 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 3 13:04:47 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 3 13:04:47 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 3 13:04:47 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 3 13:04:47 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 13:04:47 srv-dell ocfs2_controld: Stored section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 3 13:04:47 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 13:04:47 srv-dell ocfs2_controld: Stored section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 3 13:04:47 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 3 13:04:47 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 3 13:04:47 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 3 13:04:47 srv-dell ocfs2_controld: Opening control device
Jun 3 13:04:47 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 3 13:04:47 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 3 13:04:47 srv-dell ocfs2_controld: new listening connection 4
Jun 3 13:04:47 srv-dell ocfs2_controld: confchg called
Jun 3 13:04:47 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 3 13:04:49 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation o2cb:0_start_0 (call=10, rc=0, cib-update=19,
confirmed=true) complete ok
Jun 3 13:04:49 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=27:4:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=o2cb:0_monitor_120000 )
Jun 3 13:04:49 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=34:4:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_start_0 )
Jun 3 13:04:49 srv-dell lrmd: [5669]: info: rsc:fs:0: start
Jun 3 13:04:49 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=11, rc=0, cib-update=20,
confirmed=false) complete ok
Jun 3 13:04:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:04:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:04:49 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 13:04:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:04:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:04:49 ERROR: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:04:49 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_start_0 (call=12, rc=5, cib-update=21, confirmed=true)
complete not installed
Jun 3 13:04:50 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=3:5:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_stop_0 )
Jun 3 13:04:50 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 13:04:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:04:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:04:50 WARNING: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:04:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:04:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:04:50 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:04:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /dev/sdb1: No such file or directory
Jun 3 13:04:50 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=13, rc=0, cib-update=22, confirmed=true)
complete ok
Jun 3 13:04:52 srv-dell attrd: [5670]: info: main: Sending full refresh
Jun 3 13:04:52 srv-dell attrd: [5670]: info: main: Starting mainloop...
Jun 3 13:04:52 srv-dell attrd: [5670]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 13:04:52 srv-dell attrd: [5670]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 13:04:52 srv-dell attrd: [5670]: info: find_hash_entry:
Creating hash entry for terminate
Jun 3 13:04:52 srv-dell attrd: [5670]: info: find_hash_entry:
Creating hash entry for shutdown
Jun 3 13:04:52 srv-dell attrd: [5670]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 13:04:52 srv-dell attrd: [5670]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 13:04:52 srv-dell attrd: [5670]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 13:05:08 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:05:15 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:05:15 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:05:26 srv-dell kernel: No iBFT detected.
Jun 3 13:05:57 srv-dell crm_resource: [6228]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:05:57 srv-dell crm_resource: [6228]: info: Invoked:
crm_resource -M -r fs -H srv-dell
Jun 3 13:05:57 srv-dell crm_resource: [6228]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:05:57 srv-dell crm_resource: [6228]: ERROR: unpack_rsc_op:
Hard error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:06:04 srv-dell crm_resource: [6230]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:06:04 srv-dell crm_resource: [6230]: info: Invoked:
crm_resource --meta -r fs -p target-role -v started
Jun 3 13:06:04 srv-dell crm_resource: [6230]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:06:04 srv-dell crm_resource: [6230]: ERROR: unpack_rsc_op:
Hard error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:06:13 srv-dell crmd: [5672]: info: do_lrm_invoke: Removing
resource fs:0 from the LRM
Jun 3 13:06:13 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:0_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244048773-5
Jun 3 13:06:13 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:06:13 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-dell
Jun 3 13:06:13 srv-dell cib: [6231]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-96.raw
Jun 3 13:06:13 srv-dell cib: [6231]: info: write_cib_contents: Wrote
version 0.99.0 of the CIB to disk (digest:
6c663f9ab228d2534f70b3d8d5fe3b41)
Jun 3 13:06:13 srv-dell cib: [6231]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.sg57gv (digest:
/var/lib/heartbeat/crm/cib.lq4IP2)
Jun 3 13:06:14 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:7:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_monitor_0 )
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: rsc:fs:0: monitor
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) logd is not running
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) 2009/06/03_13:06:14 WARNING: Couldn't find
device [/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:06:14 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=14, rc=7, cib-update=27,
confirmed=true) complete not running
Jun 3 13:06:14 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=37:7:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_start_0 )
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: rsc:fs:0: start
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:06:14 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:06:14 ERROR: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:06:14 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_start_0 (call=15, rc=5, cib-update=30, confirmed=true)
complete not installed
Jun 3 13:06:14 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=3:8:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_stop_0 )
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:06:14 WARNING: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:06:14 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:06:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /dev/sdb1: No such file or directory
Jun 3 13:06:14 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=16, rc=0, cib-update=31, confirmed=true)
complete ok
Jun 3 13:06:15 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 13:06:15 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 6371 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:06:15 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:06:15 srv-dell openais[5661]: [MAIN ] Msg[47]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=858): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:06:15 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:06:15 srv-dell cib: [6372]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-97.raw
Jun 3 13:06:15 srv-dell cib: [6372]: info: write_cib_contents: Wrote
version 0.100.0 of the CIB to disk (digest:
567a5a31e0743a904837517770b4a41c)
Jun 3 13:06:15 srv-dell cib: [6372]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.f0bqRC (digest:
/var/lib/heartbeat/crm/cib.B7LqRe)
Jun 3 13:06:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:17 srv-dell cib: [6373]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-98.raw
Jun 3 13:06:17 srv-dell cib: [6373]: info: write_cib_contents: Wrote
version 0.101.0 of the CIB to disk (digest:
31503b4e2233c8ffab913e1adc57fd87)
Jun 3 13:06:17 srv-dell cib: [6373]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.0x9rsG (digest:
/var/lib/heartbeat/crm/cib.VN9CNm)
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:17 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:06:18 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:06:18 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:06:29 srv-dell crmd: [5672]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244048789-7" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5673_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5673" >
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive id="fs:1" />
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 13:06:29 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 13:06:29 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-dell
Jun 3 13:06:29 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:1_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244048789-8
Jun 3 13:06:29 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:06:31 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:06:31 srv-dell cib: [6374]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-99.raw
Jun 3 13:06:31 srv-dell cib: [6374]: info: write_cib_contents: Wrote
version 0.102.0 of the CIB to disk (digest:
319c6cd46ca0143951e4089f7d10a262)
Jun 3 13:06:31 srv-dell cib: [6374]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.tdKmrR (digest:
/var/lib/heartbeat/crm/cib.pozkG1)
Jun 3 13:06:31 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 6375 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:06:31 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:06:31 srv-dell openais[5661]: [MAIN ] Msg[51]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=858): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:06:31 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-lenovo
Jun 3 13:06:32 srv-dell cib: [6376]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-100.raw
Jun 3 13:06:32 srv-dell cib: [6376]: info: write_cib_contents: Wrote
version 0.103.0 of the CIB to disk (digest:
9ca96da72e9bc19bd0fd48de838d4c02)
Jun 3 13:06:32 srv-dell cib: [6376]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.el0oEo (digest:
/var/lib/heartbeat/crm/cib.QSkcaA)
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:06:33 srv-dell cib: [6377]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-1.raw
Jun 3 13:06:33 srv-dell cib: [6377]: info: write_cib_contents: Wrote
version 0.104.0 of the CIB to disk (digest:
5c0543b413c644e30dd853cd4e2f45f2)
Jun 3 13:06:33 srv-dell cib: [6377]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.IecNq0 (digest:
/var/lib/heartbeat/crm/cib.VdjXaf)
Jun 3 13:06:33 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:06:33 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:06:33 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:06:46 srv-dell crm_resource: [6379]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:06:46 srv-dell crm_resource: [6379]: info: Invoked:
crm_resource --meta -r fs -p target-role -v started
Jun 3 13:06:46 srv-dell crm_resource: [6379]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:06:46 srv-dell crm_resource: [6379]: ERROR: unpack_rsc_op:
Hard error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:07:09 srv-dell crm_resource: [6413]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:07:09 srv-dell crm_resource: [6413]: info: Invoked:
crm_resource -R -H srv-dell
Jun 3 13:07:09 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:07:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:10 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:07:11 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:07:11 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:07:39 srv-dell crmd: [5672]: info: do_lrm_invoke: Removing
resource fs:0 from the LRM
Jun 3 13:07:39 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:0_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244048859-12
Jun 3 13:07:39 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:07:39 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-dell
Jun 3 13:07:39 srv-dell cib: [6414]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-2.raw
Jun 3 13:07:39 srv-dell cib: [6414]: info: write_cib_contents: Wrote
version 0.105.0 of the CIB to disk (digest:
36b83417036195ef0fbf12f42b0fe73c)
Jun 3 13:07:39 srv-dell cib: [6414]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xFMukR (digest:
/var/lib/heartbeat/crm/cib.DkD3Vo)
Jun 3 13:07:39 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:16:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_monitor_0 )
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: rsc:fs:0: monitor
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) logd is not running
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) 2009/06/03_13:07:39 WARNING: Couldn't find
device [/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:07:39 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=17, rc=7, cib-update=40,
confirmed=true) complete not running
Jun 3 13:07:39 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=37:16:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_start_0 )
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: rsc:fs:0: start
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:07:39 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:07:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:07:39 ERROR: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:07:39 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_start_0 (call=18, rc=5, cib-update=43, confirmed=true)
complete not installed
Jun 3 13:07:40 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=3:17:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_stop_0 )
Jun 3 13:07:40 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 13:07:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:07:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:07:40 WARNING: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:07:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:07:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:07:40 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:07:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /dev/sdb1: No such file or directory
Jun 3 13:07:40 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=19, rc=0, cib-update=44, confirmed=true)
complete ok
Jun 3 13:07:41 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 6554 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:07:41 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:07:41 srv-dell openais[5661]: [MAIN ] Msg[59]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=858): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:07:41 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:07:41 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 13:07:41 srv-dell cib: [6555]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-3.raw
Jun 3 13:07:41 srv-dell cib: [6555]: info: write_cib_contents: Wrote
version 0.106.0 of the CIB to disk (digest:
f20a3cca2dc3266cac9c3545bb6e40aa)
Jun 3 13:07:41 srv-dell cib: [6555]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.IuFJ83 (digest:
/var/lib/heartbeat/crm/cib.5vrEhG)
Jun 3 13:07:43 srv-dell cib: [6556]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-4.raw
Jun 3 13:07:43 srv-dell cib: [6556]: info: write_cib_contents: Wrote
version 0.107.0 of the CIB to disk (digest:
d002f8ff684466c389cbbf01c9f435c7)
Jun 3 13:07:43 srv-dell cib: [6556]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.LNWXr6 (digest:
/var/lib/heartbeat/crm/cib.LiVGTM)
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:07:43 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:07:43 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:07:43 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:08:18 srv-dell crmd: [5672]: info: do_lrm_invoke: Removing
resource fs:0 from the LRM
Jun 3 13:08:18 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:0_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244048898-15
Jun 3 13:08:18 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:08:18 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-dell
Jun 3 13:08:18 srv-dell cib: [6558]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-5.raw
Jun 3 13:08:18 srv-dell cib: [6558]: info: write_cib_contents: Wrote
version 0.108.0 of the CIB to disk (digest:
fceef4bac456100d8184a1661b46f98b)
Jun 3 13:08:18 srv-dell cib: [6558]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.yY7hvm (digest:
/var/lib/heartbeat/crm/cib.osLHdh)
Jun 3 13:08:18 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:21:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_monitor_0 )
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: rsc:fs:0: monitor
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) logd is not running
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:monitor:stderr) 2009/06/03_13:08:18 WARNING: Couldn't find
device [/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:08:18 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=20, rc=7, cib-update=50,
confirmed=true) complete not running
Jun 3 13:08:18 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=37:21:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_start_0 )
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: rsc:fs:0: start
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:08:18 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:08:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:08:18 ERROR: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:08:18 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_start_0 (call=21, rc=5, cib-update=53, confirmed=true)
complete not installed
Jun 3 13:08:19 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=3:22:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_stop_0 )
Jun 3 13:08:19 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 13:08:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:08:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:08:19 WARNING: Couldn't find device
[/dev/sdb1]. Expected /dev/??? to exist
Jun 3 13:08:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:08:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:08:19 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:08:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /dev/sdb1: No such file or directory
Jun 3 13:08:19 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=22, rc=0, cib-update=54, confirmed=true)
complete ok
Jun 3 13:08:20 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:08:20 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 6698 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:08:20 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:08:20 srv-dell openais[5661]: [MAIN ] Msg[67]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=858): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:08:20 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 13:08:20 srv-dell cib: [6699]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-6.raw
Jun 3 13:08:20 srv-dell cib: [6699]: info: write_cib_contents: Wrote
version 0.109.0 of the CIB to disk (digest:
cee7380527c208e6957b54f2e8069ff6)
Jun 3 13:08:20 srv-dell cib: [6699]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.jPhcRv (digest:
/var/lib/heartbeat/crm/cib.4dUf4u)
Jun 3 13:08:22 srv-dell cib: [6700]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-7.raw
Jun 3 13:08:22 srv-dell cib: [6700]: info: write_cib_contents: Wrote
version 0.110.0 of the CIB to disk (digest:
864cc76a454ab777e82e0622c645ce93)
Jun 3 13:08:22 srv-dell cib: [6700]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.0tkjxy (digest:
/var/lib/heartbeat/crm/cib.Y2wR3B)
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:08:22 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:08:22 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-dell
Jun 3 13:08:22 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:0_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:09:20 srv-dell kernel: st: Version 20080504, fixed bufsize
32768, s/g segs 256
Jun 3 13:09:41 srv-dell kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jun 3 13:10:15 srv-dell kernel: scsi3 : iSCSI Initiator over TCP/IP
Jun 3 13:10:15 srv-dell kernel: scsi 3:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 13:10:15 srv-dell kernel: sdb: sdb1
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: [sdb] Attached SCSI disk
Jun 3 13:10:15 srv-dell kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Jun 3 13:10:15 srv-dell ocfs2_controld: new client connection 5
Jun 3 13:10:15 srv-dell ocfs2_controld: client msg
Jun 3 13:10:15 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 3 13:10:15 srv-dell ocfs2_controld: start_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", device "/dev/sdb1", service
"ocfs2"
Jun 3 13:10:15 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid 69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell ocfs2_controld: Starting join for group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 13:10:15 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 13:10:15 srv-dell ocfs2_controld: start_mount returns 0
Jun 3 13:10:15 srv-dell ocfs2_controld: confchg called
Jun 3 13:10:15 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 1, left 0,
joined 1
Jun 3 13:10:15 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell ocfs2_controld: This node joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell ocfs2_controld: Registering mountgroup
69AB6E551666484FB314D5B168FD6B84 with dlm_controld
Jun 3 13:10:15 srv-dell ocfs2_controld: Registering
"69AB6E551666484FB314D5B168FD6B84" with dlm_controld
Jun 3 13:10:15 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 13:10:15 srv-dell ocfs2_controld: Registration of
"69AB6E551666484FB314D5B168FD6B84" complete
Jun 3 13:10:15 srv-dell ocfs2_controld: Mountgroup
69AB6E551666484FB314D5B168FD6B84 successfully registered with
dlm_controld
Jun 3 13:10:15 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 13:10:15 srv-dell ocfs2_controld: Notified client: 1
Jun 3 13:10:15 srv-dell kernel: dlm: Using TCP for communications
Jun 3 13:10:15 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell cluster-dlm: process_uevent: kernel: add@
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell cluster-dlm: process_uevent: kernel: online@
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:15 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 3 13:10:15 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 3 13:10:15 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 1 joined 1 remove 0 failed 0
Jun 3 13:10:15 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 13:10:15 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 13:10:15 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 13:10:15 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 1 1 0 0
Jun 3 13:10:15 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 76
Jun 3 13:10:15 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 3 13:10:15 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 1
Jun 3 13:10:15 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 1
Jun 3 13:10:15 srv-dell cluster-dlm: do_sysfs: write "1438231596" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/id"
Jun 3 13:10:15 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 13:10:15 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 13:10:15 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 13:10:15 srv-dell kernel: (7308,7):ocfs2_find_slot:502 slot 1
is already allocated to this node!
Jun 3 13:10:15 srv-dell kernel: (7308,7):ocfs2_check_volume:2068 File
system was not unmounted cleanly, recovering volume.
Jun 3 13:10:15 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 3 13:10:15 srv-dell kernel: ocfs2: Mounting device (8,17) on
(node 1543678, slot 1) with ordered data mode.
Jun 3 13:10:15 srv-dell ocfs2_controld: client msg
Jun 3 13:10:15 srv-dell ocfs2_controld: client message 1 from 5: MRESULT
Jun 3 13:10:15 srv-dell ocfs2_controld: complete_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", errcode "0", service "ocfs2"
Jun 3 13:10:15 srv-dell ocfs2_controld: client msg
Jun 3 13:10:15 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 13:10:15 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 13:10:15 srv-dell ocfs2_hb_ctl[7328]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -P -d /dev/sdb1
Jun 3 13:10:15 srv-dell ocfs2_controld: new client connection 5
Jun 3 13:10:15 srv-dell ocfs2_controld: client msg
Jun 3 13:10:15 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 3 13:10:15 srv-dell ocfs2_controld: client msg
Jun 3 13:10:15 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 13:10:15 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 13:10:15 srv-dell hald: mounted /dev/sdb1 on behalf of uid 0
Jun 3 13:10:15 srv-dell iscsid: connection1:0 is operational now
Jun 3 13:10:26 srv-dell cluster-dlm: process_uevent: uevent:
offline@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell cluster-dlm: process_uevent: kernel: offline@
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell cluster-dlm: confchg_cb: confchg for our leave
Jun 3 13:10:26 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 0
Jun 3 13:10:26 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 13:10:26 srv-dell cluster-dlm: update_dir_members: dir_member 1543678144
Jun 3 13:10:26 srv-dell cluster-dlm: set_configfs_members:
set_members rmdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 13:10:26 srv-dell cluster-dlm: set_configfs_members:
set_members lockspace rmdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84"
Jun 3 13:10:26 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 13:10:26 srv-dell cluster-dlm: purge_plocks: purged 0 plocks
for 1543678144
Jun 3 13:10:26 srv-dell cluster-dlm: process_uevent: uevent:
remove@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell cluster-dlm: process_uevent: kernel: remove@
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell ocfs2_hb_ctl[7354]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -K -u 69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell ocfs2_controld: new client connection 5
Jun 3 13:10:26 srv-dell ocfs2_controld: client msg
Jun 3 13:10:26 srv-dell ocfs2_controld: client message 2 from 5: UNMOUNT
Jun 3 13:10:26 srv-dell ocfs2_controld: remove_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", service "ocfs2"
Jun 3 13:10:26 srv-dell ocfs2_controld: Unregistering mountgroup
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell ocfs2_controld: Unregistering
"69AB6E551666484FB314D5B168FD6B84" from dlm_controld
Jun 3 13:10:26 srv-dell ocfs2_controld: time to leave group
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell ocfs2_controld: calling LEAVE for group
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell ocfs2_controld: leaving group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 13:10:26 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 3 13:10:26 srv-dell ocfs2_controld: confchg called
Jun 3 13:10:26 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 0, left 1,
joined 0
Jun 3 13:10:26 srv-dell ocfs2_controld: Node 1543678144 leaves group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:26 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 13:10:26 srv-dell ocfs2_controld: Notified client: 1
Jun 3 13:10:26 srv-dell ocfs2_controld: client 6 fd 14 dead
Jun 3 13:10:26 srv-dell ocfs2_controld: client msg
Jun 3 13:10:26 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 13:10:26 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 13:10:26 srv-dell kernel: ocfs2: Unmounting device (8,17) on
(node 1543678)
Jun 3 13:10:26 srv-dell hald: unmounted /dev/sdb1 from
'/media/disk-1' on behalf of uid 0
Jun 3 13:10:39 srv-dell crmd: [5672]: info: do_lrm_invoke: Removing
resource fs:0 from the LRM
Jun 3 13:10:39 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:0_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244049039-18
Jun 3 13:10:39 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:10:39 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-dell
Jun 3 13:10:39 srv-dell cib: [7356]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-8.raw
Jun 3 13:10:39 srv-dell cib: [7356]: info: write_cib_contents: Wrote
version 0.111.0 of the CIB to disk (digest:
f40e82abef17029cd5c91447dd41fbfb)
Jun 3 13:10:39 srv-dell cib: [7356]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.F7hYC5 (digest:
/var/lib/heartbeat/crm/cib.BUMASY)
Jun 3 13:10:40 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:26:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_monitor_0 )
Jun 3 13:10:40 srv-dell lrmd: [5669]: info: rsc:fs:0: monitor
Jun 3 13:10:40 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=23, rc=7, cib-update=60,
confirmed=true) complete not running
Jun 3 13:10:40 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=37:26:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_start_0 )
Jun 3 13:10:40 srv-dell lrmd: [5669]: info: rsc:fs:0: start
Jun 3 13:10:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) logd is not running
Jun 3 13:10:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:start:stderr) 2009/06/03_13:10:40 INFO: Running start for
/dev/sdb1 on /shared
Jun 3 13:10:40 srv-dell ocfs2_controld: new client connection 5
Jun 3 13:10:40 srv-dell ocfs2_controld: client msg
Jun 3 13:10:40 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 3 13:10:40 srv-dell ocfs2_controld: start_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", device "/dev/sdb1", service
"ocfs2"
Jun 3 13:10:40 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid 69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell ocfs2_controld: Starting join for group
"ocfs2:69AB6E551666484FB314D5B168FD6B84"
Jun 3 13:10:40 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 13:10:40 srv-dell ocfs2_controld: start_mount returns 0
Jun 3 13:10:40 srv-dell ocfs2_controld: confchg called
Jun 3 13:10:40 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 1, left 0,
joined 1
Jun 3 13:10:40 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell ocfs2_controld: This node joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell ocfs2_controld: Registering mountgroup
69AB6E551666484FB314D5B168FD6B84 with dlm_controld
Jun 3 13:10:40 srv-dell ocfs2_controld: Registering
"69AB6E551666484FB314D5B168FD6B84" with dlm_controld
Jun 3 13:10:40 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 13:10:40 srv-dell ocfs2_controld: Registration of
"69AB6E551666484FB314D5B168FD6B84" complete
Jun 3 13:10:40 srv-dell ocfs2_controld: Mountgroup
69AB6E551666484FB314D5B168FD6B84 successfully registered with
dlm_controld
Jun 3 13:10:40 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 3 13:10:40 srv-dell ocfs2_controld: Notified client: 1
Jun 3 13:10:40 srv-dell kernel: dlm: Using TCP for communications
Jun 3 13:10:40 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell cluster-dlm: process_uevent: kernel: add@
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell cluster-dlm: process_uevent: kernel: online@
69AB6E551666484FB314D5B168FD6B84
Jun 3 13:10:40 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 3 13:10:40 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 3 13:10:40 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 1 joined 1 remove 0 failed 0
Jun 3 13:10:40 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 13:10:40 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 13:10:40 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 13:10:40 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 1 1 0 0
Jun 3 13:10:40 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 76
Jun 3 13:10:40 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 3 13:10:40 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 1
Jun 3 13:10:40 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 1
Jun 3 13:10:40 srv-dell cluster-dlm: do_sysfs: write "1438231596" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/id"
Jun 3 13:10:40 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/1543678144"
Jun 3 13:10:40 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 13:10:40 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/event_done"
Jun 3 13:10:40 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 3 13:10:40 srv-dell kernel: ocfs2: Mounting device (8,17) on
(node 1543678, slot 0) with ordered data mode.
Jun 3 13:10:40 srv-dell ocfs2_controld: client msg
Jun 3 13:10:40 srv-dell ocfs2_controld: client message 1 from 5: MRESULT
Jun 3 13:10:40 srv-dell ocfs2_controld: complete_mount: uuid
"69AB6E551666484FB314D5B168FD6B84", errcode "0", service "ocfs2"
Jun 3 13:10:40 srv-dell ocfs2_controld: client msg
Jun 3 13:10:40 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 13:10:40 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 13:10:40 srv-dell ocfs2_hb_ctl[7462]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -P -d /dev/sdb1
Jun 3 13:10:40 srv-dell ocfs2_controld: new client connection 5
Jun 3 13:10:40 srv-dell ocfs2_controld: client msg
Jun 3 13:10:40 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 3 13:10:40 srv-dell ocfs2_controld: client msg
Jun 3 13:10:40 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 3 13:10:40 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 3 13:10:40 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_start_0 (call=24, rc=0, cib-update=63, confirmed=true)
complete ok
Jun 3 13:10:40 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=38:26:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=fs:0_monitor_120000 )
Jun 3 13:10:40 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_120000 (call=25, rc=0, cib-update=64,
confirmed=false) complete ok
Jun 3 13:10:41 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 7503 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:10:41 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:10:41 srv-dell openais[5661]: [MAIN ] Msg[75]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:10:41 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:10:41 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 13:10:41 srv-dell cib: [7504]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-9.raw
Jun 3 13:10:41 srv-dell cib: [7504]: info: write_cib_contents: Wrote
version 0.112.0 of the CIB to disk (digest:
2af77e1d8c3b65312e7cff53a4e24ea0)
Jun 3 13:10:41 srv-dell cib: [7504]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.39X9ff (digest:
/var/lib/heartbeat/crm/cib.RppEWc)
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:10:43 srv-dell cib: [7505]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-10.raw
Jun 3 13:10:43 srv-dell cib: [7505]: info: write_cib_contents: Wrote
version 0.113.0 of the CIB to disk (digest:
85e2513b3238a51165aca5867e9f23e0)
Jun 3 13:10:43 srv-dell cib: [7505]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.U1es2f (digest:
/var/lib/heartbeat/crm/cib.jPJQ2h)
Jun 3 13:10:44 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:10:44 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:11:32 srv-dell ocfs2_controld: confchg called
Jun 3 13:11:32 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 2, left 0,
joined 1
Jun 3 13:11:32 srv-dell ocfs2_controld: Node 2013440192 joins group
ocfs2:69AB6E551666484FB314D5B168FD6B84
Jun 3 13:11:32 srv-dell cluster-dlm: add_change: add_change cg 2
joined nodeid 2013440192
Jun 3 13:11:32 srv-dell cluster-dlm: add_change: add_change cg 2
counts member 2 joined 1 remove 0 failed 0
Jun 3 13:11:32 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 2
Jun 3 13:11:32 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 13:11:32 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 3 13:11:32 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 3 13:11:32 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 3 13:11:32 srv-dell cluster-dlm: send_info: send_start cg 2 flags
2 counts 1 2 1 0 0
Jun 3 13:11:32 srv-dell cluster-dlm: receive_start: receive_start
2013440192:1 len 80
Jun 3 13:11:32 srv-dell cluster-dlm: match_change: match_change
2013440192:1 matches cg 2
Jun 3 13:11:32 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 2 need 1 of 2
Jun 3 13:11:32 srv-dell cluster-dlm: receive_start: receive_start
1543678144:2 len 80
Jun 3 13:11:32 srv-dell cluster-dlm: match_change: match_change
1543678144:2 matches cg 2
Jun 3 13:11:32 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 2 got all 2
Jun 3 13:11:32 srv-dell cluster-dlm: start_kernel: start_kernel cg 2
member_count 2
Jun 3 13:11:32 srv-dell cluster-dlm: update_dir_members: dir_member 1543678144
Jun 3 13:11:32 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/69AB6E551666484FB314D5B168FD6B84/nodes/2013440192"
Jun 3 13:11:32 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 13:11:32 srv-dell cluster-dlm: set_plock_ckpt_node:
set_plock_ckpt_node from 1543678144 to 1543678144
Jun 3 13:11:32 srv-dell cluster-dlm: _unlink_checkpoint: unlink ckpt 0
Jun 3 13:11:32 srv-dell kernel: dlm: connecting to 2013440192
Jun 3 13:11:32 srv-dell cluster-dlm: store_plocks: store_plocks:
r_count 0, lock_count 0, pp 40 bytes
Jun 3 13:11:32 srv-dell cluster-dlm: store_plocks: store_plocks:
total 0 bytes, max_section 0 bytes
Jun 3 13:11:32 srv-dell cluster-dlm: store_plocks: store_plocks: open
ckpt handle 1220085400000000
Jun 3 13:11:32 srv-dell cluster-dlm: send_info: send_plocks_stored cg
2 flags 2 counts 1 2 1 0 0
Jun 3 13:11:32 srv-dell cluster-dlm: receive_plocks_stored:
receive_plocks_stored 1543678144:2 need_plocks 0
Jun 3 13:11:47 srv-dell crm_resource: [7539]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:11:47 srv-dell crm_resource: [7539]: info: Invoked:
crm_resource -M -r fs -H srv-lenovo
Jun 3 13:11:47 srv-dell crm_resource: [7539]: ERROR: unpack_rsc_op:
Hard error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:12:51 srv-dell crm_resource: [7610]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:12:51 srv-dell crm_resource: [7610]: info: Invoked:
crm_resource -M -r fs -H srv-dell
Jun 3 13:12:51 srv-dell crm_resource: [7610]: ERROR: unpack_rsc_op:
Hard error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:13:36 srv-dell crm_resource: [7612]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:13:36 srv-dell crm_resource: [7612]: info: Invoked:
crm_resource --meta -r fs -p target-role -v started
Jun 3 13:13:36 srv-dell crm_resource: [7612]: ERROR: unpack_rsc_op:
Hard error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:14:42 srv-dell cib: [5668]: info: cib_stats: Processed 332
operations (512.00us average, 0% utilization) in the last 10min
Jun 3 13:17:21 srv-dell crm_resource: [7802]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:17:21 srv-dell crm_resource: [7802]: info: Invoked:
crm_resource --meta -r fs -p target-role -v stopped
Jun 3 13:17:21 srv-dell crm_resource: [7802]: ERROR: unpack_rsc_op:
Hard error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:17:22 srv-dell cib: [7803]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-11.raw
Jun 3 13:17:22 srv-dell cib: [7803]: info: write_cib_contents: Wrote
version 0.114.0 of the CIB to disk (digest:
9ef4fbfd062d8e80ca46a467710a140a)
Jun 3 13:17:22 srv-dell cib: [7803]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.dSm1Lb (digest:
/var/lib/heartbeat/crm/cib.K8C5Fc)
Jun 3 13:17:22 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=37:30:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_stop_0 )
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 13:17:22 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_120000 (call=25, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:22 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:22 INFO: Trying to unmount /shared
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:22 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:22 INFO: Some processes on /shared
were signalled
Jun 3 13:17:23 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:17:23 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:17:23 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:17:23 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:23 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:23 INFO: Some processes on /shared
were signalled
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:24 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:24 INFO: Some processes on /shared
were signalled
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:25 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:25 INFO: Some processes on /shared
were signalled
Jun 3 13:17:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:17:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:26 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 13:17:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:17:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:17:27 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:17:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:27 INFO: Some processes on /shared
were signalled
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:28 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:28 INFO: Some processes on /shared
were signalled
Jun 3 13:17:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:17:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:17:29 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 13:17:29 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=26, rc=1, cib-update=66, confirmed=true)
complete unknown error
Jun 3 13:17:29 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:17:29 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:17:29 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:17:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:17:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:17:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:17:30 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:17:30 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:17:41 srv-dell crm_resource: [7962]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:17:41 srv-dell crm_resource: [7962]: info: Invoked:
crm_resource --meta -r fs -p target-role -v started
Jun 3 13:17:41 srv-dell crm_resource: [7962]: ERROR: unpack_rsc_op:
Hard error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:17:41 srv-dell cib: [7963]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-12.raw
Jun 3 13:17:41 srv-dell cib: [7963]: info: write_cib_contents: Wrote
version 0.115.0 of the CIB to disk (digest:
162803b1153c493326e05b09c4b02ea7)
Jun 3 13:17:41 srv-dell cib: [7963]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Mb5PqX (digest:
/var/lib/heartbeat/crm/cib.B2lwEF)
Jun 3 13:17:42 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:17:43 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:17:43 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:19:23 srv-dell crmd: [5672]: info: do_lrm_invoke: Removing
resource fs:0 from the LRM
Jun 3 13:19:23 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-dell
Jun 3 13:19:23 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:0_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244049563-21
Jun 3 13:19:23 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:19:23 srv-dell cib: [7998]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-13.raw
Jun 3 13:19:23 srv-dell cib: [7998]: info: write_cib_contents: Wrote
version 0.116.0 of the CIB to disk (digest:
1474961b72a1cf3cd72724f8a882948b)
Jun 3 13:19:23 srv-dell cib: [7998]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.VQzT2w (digest:
/var/lib/heartbeat/crm/cib.zDZqxN)
Jun 3 13:19:23 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:34:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_monitor_0 )
Jun 3 13:19:23 srv-dell lrmd: [5669]: info: rsc:fs:0: monitor
Jun 3 13:19:23 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=27, rc=0, cib-update=71,
confirmed=true) complete ok
Jun 3 13:19:23 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=38:35:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=fs:0_monitor_120000 )
Jun 3 13:19:23 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_120000 (call=28, rc=0, cib-update=77,
confirmed=false) complete ok
Jun 3 13:19:25 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 13:19:25 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:19:25 srv-dell cib: [8069]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-14.raw
Jun 3 13:19:25 srv-dell cib: [8069]: info: write_cib_contents: Wrote
version 0.117.0 of the CIB to disk (digest:
665f26c890225449f9fa4f33a31fce9e)
Jun 3 13:19:25 srv-dell cib: [8069]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.8590Ky (digest:
/var/lib/heartbeat/crm/cib.nlJhxT)
Jun 3 13:19:25 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 8070 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:19:25 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:19:25 srv-dell openais[5661]: [MAIN ] Msg[85]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:27 srv-dell cib: [8071]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-15.raw
Jun 3 13:19:27 srv-dell cib: [8071]: info: write_cib_contents: Wrote
version 0.118.0 of the CIB to disk (digest:
e980e39e54890bbf81650f4b1df602c8)
Jun 3 13:19:27 srv-dell cib: [8071]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.6OmO2F (digest:
/var/lib/heartbeat/crm/cib.7TEVf5)
Jun 3 13:19:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:27 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:19:27 srv-dell mgmtd: [5673]: ERROR: unpack_rsc_op: Hard
error - fs:1_start_0 failed with rc=5: Preventing fs-clone from
re-starting on srv-lenovo
Jun 3 13:19:35 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-dell
Jun 3 13:19:35 srv-dell crmd: [5672]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244049575-37" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5673_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5673" >
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive id="fs:1" />
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 13:19:35 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 13:19:35 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:1_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244049575-24
Jun 3 13:19:35 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:19:37 srv-dell cib: [8072]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-16.raw
Jun 3 13:19:37 srv-dell cib: [8072]: info: write_cib_contents: Wrote
version 0.119.0 of the CIB to disk (digest:
7a686472cd0d4c36bd8f098154690319)
Jun 3 13:19:37 srv-dell cib: [8072]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.nPZlXP (digest:
/var/lib/heartbeat/crm/cib.xHmrqC)
Jun 3 13:19:37 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:19:37 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 8073 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:19:37 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:19:37 srv-dell openais[5661]: [MAIN ] Msg[88]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:19:37 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-lenovo
Jun 3 13:19:38 srv-dell cib: [8074]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-17.raw
Jun 3 13:19:38 srv-dell cib: [8074]: info: write_cib_contents: Wrote
version 0.120.0 of the CIB to disk (digest:
3ecb6240c2c9601665a318b8e8f6a6f0)
Jun 3 13:19:38 srv-dell cib: [8074]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.6aA0i4 (digest:
/var/lib/heartbeat/crm/cib.kHKPtR)
Jun 3 13:19:39 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:39 srv-dell cib: [8075]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-18.raw
Jun 3 13:19:39 srv-dell cib: [8075]: info: write_cib_contents: Wrote
version 0.121.0 of the CIB to disk (digest:
dae4310f91d05c651ce8f282f9531e3d)
Jun 3 13:19:39 srv-dell cib: [8075]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.GDl69X (digest:
/var/lib/heartbeat/crm/cib.xD15aP)
Jun 3 13:19:39 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:40 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:19:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:58 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:19:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:19:59 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:20:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:19 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:20:19 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=11:41:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_start_0 )
Jun 3 13:20:19 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 13:20:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:19 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:20:19 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:20:19 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 13:20:19 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:20:19 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:20:19 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 13:20:19 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:20:19 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:20:19 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 13:20:19 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=29, rc=0,
cib-update=81, confirmed=true) complete ok
Jun 3 13:20:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:20 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:20:20 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:42:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 13:20:20 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:20 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:20:21 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=30, rc=0,
cib-update=82, confirmed=false) complete ok
Jun 3 13:20:21 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:20:21 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:16 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell cib: [8912]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-19.raw
Jun 3 13:23:16 srv-dell cib: [8912]: info: write_cib_contents: Wrote
version 0.122.0 of the CIB to disk (digest:
b12890830d21e8fbe16bdd34d5f64f34)
Jun 3 13:23:16 srv-dell cib: [8912]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.mMchO7 (digest:
/var/lib/heartbeat/crm/cib.gkoEvA)
Jun 3 13:23:16 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:16 srv-dell mgmtd: [5673]: ERROR: native_add_running:
Resource ocf::IPaddr:primitiveresource_cluster_ip appears to be active
on 2 nodes.
Jun 3 13:23:16 srv-dell mgmtd: [5673]: WARN: See
http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more
information.
Jun 3 13:23:16 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=13:43:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 13:23:16 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 13:23:16 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=30, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 13:23:16 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 13:23:16 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 13:23:16 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_13:23:16 INFO:
ifconfig eth1:0 down
Jun 3 13:23:16 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=31, rc=0,
cib-update=84, confirmed=true) complete ok
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:17 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:18 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:36 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:37 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:37 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:57 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:58 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=11:45:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_start_0 )
Jun 3 13:23:58 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 13:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:58 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:58 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:23:58 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 13:23:58 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:23:58 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:23:58 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 13:23:58 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:23:58 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:23:58 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 13:23:58 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=32, rc=0,
cib-update=85, confirmed=true) complete ok
Jun 3 13:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:58 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:46:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 13:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:58 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:23:59 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=33, rc=0,
cib-update=86, confirmed=false) complete ok
Jun 3 13:23:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:23:59 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:24:42 srv-dell cib: [5668]: info: cib_stats: Processed 156
operations (1025.00us average, 0% utilization) in the last 10min
Jun 3 13:26:08 srv-dell mgmtd: [5673]: info: CIB replace: constraints
Jun 3 13:26:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:26:08 srv-dell cib: [9599]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-20.raw
Jun 3 13:26:08 srv-dell cib: [9599]: info: write_cib_contents: Wrote
version 0.123.0 of the CIB to disk (digest:
b1fb753f86f76d8f3995dceda3aa93c3)
Jun 3 13:26:08 srv-dell cib: [9599]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.BkJEhM (digest:
/var/lib/heartbeat/crm/cib.eSeRAi)
Jun 3 13:26:09 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:27:00 srv-dell mgmtd: [5673]: info: CIB replace: constraints
Jun 3 13:27:00 srv-dell cib: [9812]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-21.raw
Jun 3 13:27:00 srv-dell cib: [9812]: info: write_cib_contents: Wrote
version 0.124.0 of the CIB to disk (digest:
e865f0d4f4c25b76f3c80fc65b6bcbc2)
Jun 3 13:27:00 srv-dell cib: [9812]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.AxCJzz (digest:
/var/lib/heartbeat/crm/cib.aunX7U)
Jun 3 13:27:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:27:01 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:27:49 srv-dell mgmtd: [5673]: info: CIB replace: constraints
Jun 3 13:28:53 srv-dell mgmtd: [5673]: info: CIB replace: constraints
Jun 3 13:28:53 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:28:53 srv-dell cib: [10294]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-22.raw
Jun 3 13:28:53 srv-dell cib: [10294]: info: write_cib_contents: Wrote
version 0.125.0 of the CIB to disk (digest:
afd53683c31e7d3eb6dbff0ad289a6bd)
Jun 3 13:28:53 srv-dell cib: [10294]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.d99bid (digest:
/var/lib/heartbeat/crm/cib.TXMjfw)
Jun 3 13:28:53 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:29:02 srv-dell mgmtd: [5673]: info: CIB replace: constraints
Jun 3 13:29:02 srv-dell cib: [10331]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-23.raw
Jun 3 13:29:02 srv-dell cib: [10331]: info: write_cib_contents: Wrote
version 0.126.0 of the CIB to disk (digest:
ee908f06b7d53a6255c688b82c528d77)
Jun 3 13:29:02 srv-dell cib: [10331]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.PPYOKP (digest:
/var/lib/heartbeat/crm/cib.WpM9Rs)
Jun 3 13:29:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:29:03 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:29:41 srv-dell mgmtd: [5673]: info: CIB replace: constraints
Jun 3 13:29:41 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:29:41 srv-dell cib: [10493]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-24.raw
Jun 3 13:29:41 srv-dell cib: [10493]: info: write_cib_contents: Wrote
version 0.127.0 of the CIB to disk (digest:
90d2cef215b7bb9b267b16374e4195e1)
Jun 3 13:29:41 srv-dell cib: [10493]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.IQ6cTb (digest:
/var/lib/heartbeat/crm/cib.Ja1KYb)
Jun 3 13:29:42 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:31:58 srv-dell mgmtd: [5673]: info: CIB delete: constraints
Jun 3 13:31:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:31:58 srv-dell cib: [11071]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-25.raw
Jun 3 13:31:58 srv-dell cib: [11071]: info: write_cib_contents: Wrote
version 0.128.0 of the CIB to disk (digest:
0a9266631b311c4a3c67829afb94910d)
Jun 3 13:31:58 srv-dell cib: [11071]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.RxnHs3 (digest:
/var/lib/heartbeat/crm/cib.8kUGUR)
Jun 3 13:31:59 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:32:01 srv-dell mgmtd: [5673]: info: CIB delete: constraints
Jun 3 13:32:01 srv-dell cib: [11090]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-26.raw
Jun 3 13:32:01 srv-dell cib: [11090]: info: write_cib_contents: Wrote
version 0.129.0 of the CIB to disk (digest:
69d63e4964a4db00a7f075dab59e1f77)
Jun 3 13:32:01 srv-dell cib: [11090]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.bdhhDR (digest:
/var/lib/heartbeat/crm/cib.P7gO4L)
Jun 3 13:32:01 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:01 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:32:04 srv-dell mgmtd: [5673]: info: CIB delete: constraints
Jun 3 13:32:04 srv-dell cib: [11091]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-27.raw
Jun 3 13:32:04 srv-dell cib: [11091]: info: write_cib_contents: Wrote
version 0.130.0 of the CIB to disk (digest:
933175ad6538b18fa4aaa1a262880cce)
Jun 3 13:32:04 srv-dell cib: [11091]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.BS04nz (digest:
/var/lib/heartbeat/crm/cib.SWBgDz)
Jun 3 13:32:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:04 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:32:16 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-lenovo
Jun 3 13:32:16 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 11146 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:32:16 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:32:16 srv-dell openais[5661]: [MAIN ] Msg[98]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:32:16 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:32:16 srv-dell cib: [11147]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-28.raw
Jun 3 13:32:16 srv-dell cib: [11147]: info: write_cib_contents: Wrote
version 0.131.0 of the CIB to disk (digest:
04cf65026bcdb06f31a6575b6ba30412)
Jun 3 13:32:16 srv-dell cib: [11147]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.PUt2yB (digest:
/var/lib/heartbeat/crm/cib.QQAm82)
Jun 3 13:32:17 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=13:55:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 13:32:17 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 13:32:17 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=33, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 13:32:17 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 13:32:17 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 13:32:17 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 13:32:17 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_13:32:17 INFO:
ifconfig eth1:0 down
Jun 3 13:32:17 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=34, rc=0,
cib-update=88, confirmed=true) complete ok
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell cib: [11174]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-29.raw
Jun 3 13:32:18 srv-dell cib: [11174]: info: write_cib_contents: Wrote
version 0.132.0 of the CIB to disk (digest:
323045caf33244297787b2035b407e30)
Jun 3 13:32:18 srv-dell cib: [11174]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.UBPWSG (digest:
/var/lib/heartbeat/crm/cib.mOdDLc)
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:19 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:32:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:37 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:32:38 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:39 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:32:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:59 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:32:59 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=11:57:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_start_0 )
Jun 3 13:32:59 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 13:32:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:59 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:32:59 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:32:59 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 13:32:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:59 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:32:59 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:32:59 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 13:32:59 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:32:59 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:32:59 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 13:32:59 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=35, rc=0,
cib-update=89, confirmed=true) complete ok
Jun 3 13:32:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:32:59 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:58:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 13:32:59 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:33:00 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:33:00 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:33:00 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=36, rc=0,
cib-update=90, confirmed=false) complete ok
Jun 3 13:33:00 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:33:01 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:33:27 srv-dell crm_resource: [11441]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 13:33:27 srv-dell crm_resource: [11441]: info: Invoked:
crm_resource -M -r fs -H srv-dell
Jun 3 13:34:04 srv-dell crmd: [5672]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 13:34:04 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-dell
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244050444-46" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5673_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5673" >
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive id="fs:1" />
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 13:34:04 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 13:34:04 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:1_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244050444-28
Jun 3 13:34:04 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:34:05 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=11:60:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 13:34:05 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 13:34:05 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=36, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_13:34:05 INFO:
ifconfig eth1:0 down
Jun 3 13:34:05 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=37, rc=0,
cib-update=92, confirmed=true) complete ok
Jun 3 13:34:05 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=7:61:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_start_0 )
Jun 3 13:34:05 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:34:05 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:34:05 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:34:05 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 13:34:06 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:34:06 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:34:06 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 13:34:06 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=38, rc=0,
cib-update=93, confirmed=true) complete ok
Jun 3 13:34:06 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=8:61:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 13:34:06 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-lenovo
Jun 3 13:34:06 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 11685 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 13:34:06 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 13:34:06 srv-dell openais[5661]: [MAIN ] Msg[108]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 13:34:06 srv-dell crmd: [5672]: WARN: msg_to_op(1224): failed
to get the value of field lrm_opstatus from a ha_msg
Jun 3 13:34:06 srv-dell crmd: [5672]: info: msg_to_op: Message follows:
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG: Dumping message with 16 fields
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[0] : [lrm_t=op]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[1] :
[lrm_rid=primitiveresource_cluster_ip]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[2] : [lrm_op=monitor]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[3] : [lrm_timeout=21000]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[4] : [lrm_interval=5000]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[5] : [lrm_delay=1000]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[6] : [lrm_copyparams=0]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[7] : [lrm_t_run=0]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[8] : [lrm_t_rcchange=0]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[9] : [lrm_exec_time=0]
Jun 3 13:34:06 srv-dell cib: [11684]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-30.raw
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[10] : [lrm_queue_time=0]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[11] : [lrm_targetrc=-2]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[12] : [lrm_app=crmd]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[13] :
[lrm_userdata=8:61:0:8b0cb443-b118-4b49-86d8-676f1a747dbb]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[14] :
[(2)lrm_param=0x648320(184 224)]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG: Dumping message with 7 fields
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[0] : [CRM_meta_interval=5000]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[1] : [ip=192.168.2.10]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[2] :
[CRM_meta_start_delay=1000]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[3] :
[CRM_meta_record_pending=true]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[4] : [CRM_meta_timeout=21000]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[5] : [crm_feature_set=3.0.1]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[6] : [CRM_meta_name=monitor]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: MSG[15] : [lrm_callid=39]
Jun 3 13:34:06 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:34:06 srv-dell cib: [11684]: info: write_cib_contents: Wrote
version 0.133.0 of the CIB to disk (digest:
6aaa29e27c98cbe82dd864d9985757c2)
Jun 3 13:34:06 srv-dell cib: [11684]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.uxzCGM (digest:
/var/lib/heartbeat/crm/cib.oE1Rs4)
Jun 3 13:34:06 srv-dell cib: [11686]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-31.raw
Jun 3 13:34:06 srv-dell cib: [11686]: info: write_cib_contents: Wrote
version 0.134.0 of the CIB to disk (digest:
8b7c579c7c0049e5007f2a3d2dbcb0a1)
Jun 3 13:34:06 srv-dell cib: [11686]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.g0oumP (digest:
/var/lib/heartbeat/crm/cib.mLOud7)
Jun 3 13:34:07 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=39, rc=0,
cib-update=95, confirmed=false) complete ok
Jun 3 13:34:07 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=13:62:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 13:34:07 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 13:34:07 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=39, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 13:34:07 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 13:34:07 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 13:34:07 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 13:34:07 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_13:34:07 INFO:
ifconfig eth1:0 down
Jun 3 13:34:07 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=40, rc=0,
cib-update=96, confirmed=true) complete ok
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:08 srv-dell cib: [11735]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-32.raw
Jun 3 13:34:08 srv-dell cib: [11735]: info: write_cib_contents: Wrote
version 0.135.0 of the CIB to disk (digest:
a957aa4d2e54e63af43f14dcd8d524f2)
Jun 3 13:34:08 srv-dell cib: [11735]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.DyTpAT (digest:
/var/lib/heartbeat/crm/cib.ygTKNf)
Jun 3 13:34:08 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:34:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:09 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:34:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:27 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:34:28 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:28 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:34:42 srv-dell cib: [5668]: info: cib_stats: Processed 137
operations (1021.00us average, 0% utilization) in the last 10min
Jun 3 13:34:48 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:48 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:34:48 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=11:64:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_start_0 )
Jun 3 13:34:48 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 13:34:48 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:48 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:34:48 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:48 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:34:48 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 13:34:48 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:48 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:34:48 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:34:48 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 13:34:48 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:48 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 13:34:48 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_13:34:48 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 13:34:49 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=41, rc=0,
cib-update=97, confirmed=true) complete ok
Jun 3 13:34:49 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:49 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:34:49 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:65:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 13:34:49 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:49 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:34:50 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=42, rc=0,
cib-update=98, confirmed=false) complete ok
Jun 3 13:34:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:34:50 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:39:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:50 srv-dell cib: [13119]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-33.raw
Jun 3 13:39:50 srv-dell cib: [13119]: info: write_cib_contents: Wrote
version 0.136.0 of the CIB to disk (digest:
4946d272f87979603e251d681c47553e)
Jun 3 13:39:50 srv-dell cib: [13119]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.bpm3hO (digest:
/var/lib/heartbeat/crm/cib.QQxz8a)
Jun 3 13:39:50 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:39:50 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=8:66:0:8b0cb443-b118-4b49-86d8-676f1a747dbb
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 13:39:50 srv-dell lrmd: [5669]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 13:39:50 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=24:66:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_stop_0 )
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 13:39:50 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=42, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 13:39:50 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_120000 (call=28, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 13:39:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_13:39:50 INFO:
ifconfig eth1:0 down
Jun 3 13:39:50 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=43, rc=0,
cib-update=99, confirmed=true) complete ok
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:50 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 13:39:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:50 INFO: Trying to unmount /shared
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:50 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 13:39:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:39:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:39:51 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:39:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:51 INFO: Some processes on /shared
were signalled
Jun 3 13:39:51 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:52 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:52 INFO: Some processes on /shared
were signalled
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:53 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:53 INFO: Some processes on /shared
were signalled
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:54 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:54 INFO: Some processes on /shared
were signalled
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:55 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:55 INFO: Some processes on /shared
were signalled
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:56 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:56 INFO: Some processes on /shared
were signalled
Jun 3 13:39:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 13:39:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_13:39:57 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 13:39:57 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=44, rc=1, cib-update=100, confirmed=true)
complete unknown error
Jun 3 13:39:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:57 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:39:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:39:58 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:40:09 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 13:40:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:09 srv-dell cib: [13303]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-34.raw
Jun 3 13:40:09 srv-dell cib: [13303]: info: write_cib_contents: Wrote
version 0.137.0 of the CIB to disk (digest:
bdc955f96a28f5bd7778535ef6797fc0)
Jun 3 13:40:09 srv-dell cib: [13303]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.EbTZMc (digest:
/var/lib/heartbeat/crm/cib.jeTWlg)
Jun 3 13:40:10 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:40:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:10 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:40:10 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:11 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:40:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:31 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:40:31 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:31 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:40:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:51 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:40:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 13:40:52 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 13:44:42 srv-dell cib: [5668]: info: cib_stats: Processed 72
operations (2222.00us average, 0% utilization) in the last 10min
Jun 3 14:02:13 srv-dell cibadmin: [13844]: info: Invoked: cibadmin -Q
/root/cluster-config.xml
Jun 3 14:02:55 srv-dell cibadmin: [13877]: info: Invoked: cibadmin -Q
Jun 3 14:03:08 srv-dell cibadmin: [13878]: info: Invoked: cibadmin -Q
Jun 3 14:04:04 srv-dell syslog-ng[3195]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=1827', processed='center(received)=1555',
processed='destination(newsnotice)=0',
processed='destination(acpid)=1', processed='destination(firewall)=0',
processed='destination(null)=1', processed='destination(mail)=2',
processed='destination(mailinfo)=2',
processed='destination(console)=70',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=1551',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=70',
processed='destination(warn)=128', processed='source(src)=1555'
Jun 3 14:04:42 srv-dell cib: [5668]: info: cib_stats: Processed 4
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 14:04:56 srv-dell haclient: on_event:evt:cib_changed
Jun 3 14:04:56 srv-dell cib: [13917]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-35.raw
Jun 3 14:04:56 srv-dell cib: [13917]: info: write_cib_contents: Wrote
version 0.138.0 of the CIB to disk (digest:
a6b20efdc6806757df71d0cc4c23a7ef)
Jun 3 14:04:56 srv-dell cib: [13917]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.NxhdHR (digest:
/var/lib/heartbeat/crm/cib.mvbEl8)
Jun 3 14:04:56 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 14:14:42 srv-dell cib: [5668]: info: cib_stats: Processed 5
operations (2000.00us average, 0% utilization) in the last 10min
Jun 3 14:34:42 srv-dell -- MARK --
Jun 3 14:54:42 srv-dell -- MARK --
Jun 3 15:04:04 srv-dell syslog-ng[3195]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=1837', processed='center(received)=1565',
processed='destination(newsnotice)=0',
processed='destination(acpid)=1', processed='destination(firewall)=0',
processed='destination(null)=1', processed='destination(mail)=2',
processed='destination(mailinfo)=2',
processed='destination(console)=70',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=1561',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=70',
processed='destination(warn)=128', processed='source(src)=1565'
Jun 3 15:24:04 srv-dell -- MARK --
Jun 3 15:24:56 srv-dell crm_attribute: [15407]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 15:24:56 srv-dell crm_attribute: [15407]: info: Invoked:
crm_attribute -t op_defaults -D -n record-pending
Jun 3 15:24:56 srv-dell cib: [15408]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-36.raw
Jun 3 15:24:56 srv-dell cib: [15408]: info: write_cib_contents: Wrote
version 0.139.0 of the CIB to disk (digest:
ecf48bc5ebe9742379140b94f0898a12)
Jun 3 15:24:56 srv-dell cib: [15408]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.oNaAwo (digest:
/var/lib/heartbeat/crm/cib.QbWwDf)
Jun 3 15:34:42 srv-dell cib: [5668]: info: cib_stats: Processed 4
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 15:54:42 srv-dell -- MARK --
Jun 3 16:04:04 srv-dell syslog-ng[3195]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=1847', processed='center(received)=1574',
processed='destination(newsnotice)=0',
processed='destination(acpid)=1', processed='destination(firewall)=0',
processed='destination(null)=1', processed='destination(mail)=2',
processed='destination(mailinfo)=2',
processed='destination(console)=70',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=1570',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=70',
processed='destination(warn)=129', processed='source(src)=1574'
Jun 3 16:24:04 srv-dell -- MARK --
Jun 3 16:44:04 srv-dell -- MARK --
Jun 3 17:04:04 srv-dell -- MARK --
Jun 3 17:04:04 srv-dell syslog-ng[3195]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=1851', processed='center(received)=1578',
processed='destination(newsnotice)=0',
processed='destination(acpid)=1', processed='destination(firewall)=0',
processed='destination(null)=1', processed='destination(mail)=2',
processed='destination(mailinfo)=2',
processed='destination(console)=70',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=1574',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=70',
processed='destination(warn)=129', processed='source(src)=1578'
Jun 3 17:24:04 srv-dell -- MARK --
Jun 3 17:40:20 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 17:40:47 srv-dell lrmd: [5669]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 17:44:42 srv-dell cib: [5668]: info: cib_stats: Processed 2
operations (0.00us average, 0% utilization) in the last 10min
Jun 3 18:04:04 srv-dell syslog-ng[3195]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=1856', processed='center(received)=1583',
processed='destination(newsnotice)=0',
processed='destination(acpid)=1', processed='destination(firewall)=0',
processed='destination(null)=1', processed='destination(mail)=2',
processed='destination(mailinfo)=2',
processed='destination(console)=70',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=1579',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=70',
processed='destination(warn)=129', processed='source(src)=1583'
Jun 3 18:16:12 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:16:55 srv-dell mgmtd: [5673]: info: CIB delete: resources
Jun 3 18:18:18 srv-dell mgmtd: [5673]: info: CIB delete: resources
Jun 3 18:18:24 srv-dell mgmtd: [5673]: info: CIB delete: resources
Jun 3 18:19:02 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-dell
Jun 3 18:19:02 srv-dell crmd: [5672]: info: do_lrm_invoke: Removing
resource fs:0 from the LRM
Jun 3 18:19:02 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:0_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244067542-31
Jun 3 18:19:02 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:19:02 srv-dell cib: [19501]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-37.raw
Jun 3 18:19:02 srv-dell cib: [19501]: info: write_cib_contents: Wrote
version 0.140.0 of the CIB to disk (digest:
2db84d080ffe9740747137672148e3f5)
Jun 3 18:19:02 srv-dell cib: [19501]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.VVquHU (digest:
/var/lib/heartbeat/crm/cib.YkTwuG)
Jun 3 18:19:03 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=9:93:7:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_monitor_0 )
Jun 3 18:19:03 srv-dell lrmd: [5669]: info: rsc:fs:0: monitor
Jun 3 18:19:03 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_monitor_0 (call=45, rc=0, cib-update=106,
confirmed=true) complete ok
Jun 3 18:19:04 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=21:94:0:8b0cb443-b118-4b49-86d8-676f1a747dbb op=fs:0_stop_0 )
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:04 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:04 INFO: Trying to unmount /shared
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:04 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551 13341
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:04 INFO: Some processes on /shared
were signalled
Jun 3 18:19:04 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:0 from srv-lenovo
Jun 3 18:19:04 srv-dell crmd: [5672]: WARN: msg_to_op(1224): failed
to get the value of field lrm_opstatus from a ha_msg
Jun 3 18:19:04 srv-dell crmd: [5672]: info: msg_to_op: Message follows:
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG: Dumping message with 16 fields
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[0] : [lrm_t=op]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[1] : [lrm_rid=fs:0]
Jun 3 18:19:04 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 19604 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 18:19:04 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[2] : [lrm_op=stop]
Jun 3 18:19:04 srv-dell openais[5661]: [MAIN ] Msg[123]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[3] : [lrm_timeout=20000]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[4] : [lrm_interval=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[5] : [lrm_delay=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[6] : [lrm_copyparams=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[7] : [lrm_t_run=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[8] : [lrm_t_rcchange=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[9] : [lrm_exec_time=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[10] : [lrm_queue_time=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[11] : [lrm_targetrc=-1]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[12] : [lrm_app=crmd]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[13] :
[lrm_userdata=21:94:0:8b0cb443-b118-4b49-86d8-676f1a747dbb]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[14] :
[(2)lrm_param=0x673560(209 261)]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG: Dumping message with 9 fields
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[0] : [directory=/shared]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[1] : [fstype=ocfs2]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[2] : [CRM_meta_op_target_rc=7]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[3] : [device=/dev/sdb1]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[4] : [CRM_meta_timeout=20000]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[5] : [CRM_meta_clone_max=2]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[6] : [crm_feature_set=3.0.1]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[7] :
[CRM_meta_globally_unique=false]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[8] : [CRM_meta_clone=0]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: MSG[15] : [lrm_callid=46]
Jun 3 18:19:04 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:19:04 srv-dell cib: [19605]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-38.raw
Jun 3 18:19:04 srv-dell cib: [19605]: info: write_cib_contents: Wrote
version 0.141.0 of the CIB to disk (digest:
2343a3f159fac70abea62cc234e093e5)
Jun 3 18:19:04 srv-dell cib: [19605]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.VUn7B0 (digest:
/var/lib/heartbeat/crm/cib.yY9TNQ)
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:05 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551 13341
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:05 INFO: Some processes on /shared
were signalled
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:06 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551 13341
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:06 INFO: Some processes on /shared
were signalled
Jun 3 18:19:06 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-dell
Jun 3 18:19:06 srv-dell crmd: [5672]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244067546-58" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5673_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5673" >
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive id="fs:1" />
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 18:19:06 srv-dell crmd: [5672]: info: send_direct_ack: ACK'ing
resource op fs:1_delete_0 from mgmtd-5673:
lrm_invoke-lrmd-1244067546-34
Jun 3 18:19:06 srv-dell crmd: [5672]: WARN: msg_to_op(1224): failed
to get the value of field lrm_opstatus from a ha_msg
Jun 3 18:19:06 srv-dell crmd: [5672]: info: msg_to_op: Message follows:
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG: Dumping message with 16 fields
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[0] : [lrm_t=op]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[1] : [lrm_rid=fs:0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[2] : [lrm_op=stop]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[3] : [lrm_timeout=20000]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[4] : [lrm_interval=0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[5] : [lrm_delay=0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[6] : [lrm_copyparams=0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[7] : [lrm_t_run=0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[8] : [lrm_t_rcchange=0]
Jun 3 18:19:06 srv-dell cib: [19638]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-39.raw
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[9] : [lrm_exec_time=0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[10] : [lrm_queue_time=0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[11] : [lrm_targetrc=-1]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[12] : [lrm_app=crmd]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[13] :
[lrm_userdata=21:94:0:8b0cb443-b118-4b49-86d8-676f1a747dbb]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[14] :
[(2)lrm_param=0x66e1f0(209 261)]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG: Dumping message with 9 fields
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[0] : [directory=/shared]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[1] : [fstype=ocfs2]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[2] : [CRM_meta_op_target_rc=7]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[3] : [device=/dev/sdb1]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[4] : [CRM_meta_timeout=20000]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[5] : [CRM_meta_clone_max=2]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[6] : [crm_feature_set=3.0.1]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[7] :
[CRM_meta_globally_unique=false]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[8] : [CRM_meta_clone=0]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: MSG[15] : [lrm_callid=46]
Jun 3 18:19:06 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:19:06 srv-dell cib: [19638]: info: write_cib_contents: Wrote
version 0.142.0 of the CIB to disk (digest:
2bbf0d6383e911e39e09b3e2b0dc6a23)
Jun 3 18:19:06 srv-dell cib: [19638]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.6hQjt5 (digest:
/var/lib/heartbeat/crm/cib.W1O21Z)
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:07 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551 13341
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:07 INFO: Some processes on /shared
were signalled
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:08 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:08 INFO: Some processes on /shared
were signalled
Jun 3 18:19:08 srv-dell mgmtd: [5673]: info: Delete fail-count for
fs:1 from srv-lenovo
Jun 3 18:19:08 srv-dell crmd: [5672]: WARN: msg_to_op(1224): failed
to get the value of field lrm_opstatus from a ha_msg
Jun 3 18:19:08 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Child 19672 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 18:19:08 srv-dell openais[5661]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 18:19:08 srv-dell openais[5661]: [MAIN ] Msg[127]
(dest=local:unknown, from=srv-lenovo:crmd.4828, remote=true,
size=859): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 18:19:08 srv-dell crmd: [5672]: info: msg_to_op: Message follows:
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG: Dumping message with 16 fields
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[0] : [lrm_t=op]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[1] : [lrm_rid=fs:0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[2] : [lrm_op=stop]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[3] : [lrm_timeout=20000]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[4] : [lrm_interval=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[5] : [lrm_delay=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[6] : [lrm_copyparams=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[7] : [lrm_t_run=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[8] : [lrm_t_rcchange=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[9] : [lrm_exec_time=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[10] : [lrm_queue_time=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[11] : [lrm_targetrc=-1]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[12] : [lrm_app=crmd]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[13] :
[lrm_userdata=21:94:0:8b0cb443-b118-4b49-86d8-676f1a747dbb]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[14] :
[(2)lrm_param=0x640d30(209 261)]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG: Dumping message with 9 fields
Jun 3 18:19:08 srv-dell cib: [19671]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-40.raw
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[0] : [directory=/shared]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[1] : [fstype=ocfs2]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[2] : [CRM_meta_op_target_rc=7]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[3] : [device=/dev/sdb1]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[4] : [CRM_meta_timeout=20000]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[5] : [CRM_meta_clone_max=2]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[6] : [crm_feature_set=3.0.1]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[7] :
[CRM_meta_globally_unique=false]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[8] : [CRM_meta_clone=0]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: MSG[15] : [lrm_callid=46]
Jun 3 18:19:08 srv-dell crmd: [5672]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:19:08 srv-dell cib: [19671]: info: write_cib_contents: Wrote
version 0.143.0 of the CIB to disk (digest:
b68c65c3b8ab21764477c128c236a96b)
Jun 3 18:19:08 srv-dell cib: [19671]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.fqrcjc (digest:
/var/lib/heartbeat/crm/cib.oDHDib)
Jun 3 18:19:08 srv-dell cib: [19673]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-41.raw
Jun 3 18:19:08 srv-dell cib: [19673]: info: write_cib_contents: Wrote
version 0.144.0 of the CIB to disk (digest:
02dbc77e228fd366d7b2b93ef9ec5c0d)
Jun 3 18:19:08 srv-dell cib: [19673]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.nv7b9e (digest:
/var/lib/heartbeat/crm/cib.OIShde)
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:09 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:09 INFO: Some processes on /shared
were signalled
Jun 3 18:19:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:10 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:19:10 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=46, rc=1, cib-update=115, confirmed=true)
complete unknown error
Jun 3 18:19:11 srv-dell cib: [19700]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-42.raw
Jun 3 18:19:11 srv-dell cib: [19700]: info: write_cib_contents: Wrote
version 0.145.0 of the CIB to disk (digest:
9d373f56ba781d887a2bcb6e8ab12798)
Jun 3 18:19:11 srv-dell cib: [19700]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.JBEkLk (digest:
/var/lib/heartbeat/crm/cib.XwVCvo)
Jun 3 18:19:11 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:19:11 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:19:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:19:12 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:19:22 srv-dell openais[5661]: [TOTEM] The token was lost in
the OPERATIONAL state.
Jun 3 18:19:22 srv-dell openais[5661]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 18:19:22 srv-dell openais[5661]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 18:19:23 srv-dell openais[5661]: [TOTEM] entering GATHER state from 2.
Jun 3 18:19:24 srv-dell mgmtd: [5673]: info: CIB delete: resources
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] entering GATHER state from 0.
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] Saving state aru 5d6
high seq received 5d6
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] Storing new sequence
id for ring e4
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] entering COMMIT state.
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] entering RECOVERY state.
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] previous ring seq 224
rep 192.168.2.120
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] aru 5d6 high delivered
5d6 received flag 1
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] Sending initial ORF token
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] New Configuration:
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] Members Left:
Jun 3 18:19:25 srv-dell cib: [5668]: info: ais_dispatch: Membership
228: quorum still lost
Jun 3 18:19:25 srv-dell crmd: [5672]: notice: ais_dispatch:
Membership 228: quorum lost
Jun 3 18:19:25 srv-dell cib: [5668]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=216 seen=224 proc=00000000000000000000000000053312
Jun 3 18:19:25 srv-dell crmd: [5672]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=216 seen=224 proc=00000000000000000000000000053312
Jun 3 18:19:25 srv-dell cluster-dlm: [5810]: notice: ais_dispatch:
Membership 228: quorum lost
Jun 3 18:19:25 srv-dell crmd: [5672]: WARN: check_dead_member: Our DC
node (srv-lenovo) left the cluster
Jun 3 18:19:25 srv-dell ocfs2_controld: [5932]: notice: ais_dispatch:
Membership 228: quorum lost
Jun 3 18:19:25 srv-dell cluster-dlm: [5810]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=lost (new) addr=r(0)
ip(192.168.2.120) votes=1 born=216 seen=224
proc=00000000000000000000000000053312
Jun 3 18:19:25 srv-dell cluster-dlm: update_cluster: Processing membership 228
Jun 3 18:19:25 srv-dell ocfs2_controld: [5932]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=lost (new)
addr=r(0) ip(192.168.2.120) votes=1 born=216 seen=224
proc=00000000000000000000000000053312
Jun 3 18:19:25 srv-dell ocfs2_controld: confchg called
Jun 3 18:19:25 srv-dell cluster-dlm: del_configfs_node:
del_configfs_node rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=check_dead_member ]
Jun 3 18:19:25 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 1, joined 0
Jun 3 18:19:25 srv-dell cluster-dlm: dlm_process_node: Removed
inactive node 2013440192: born-on=216, last-seen=224, this-event=228,
last-event=224
Jun 3 18:19:25 srv-dell crmd: [5672]: info: update_dc: Unset DC srv-lenovo
Jun 3 18:19:25 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 3 18:19:25 srv-dell ocfs2_controld: node daemon left 2013440192
Jun 3 18:19:25 srv-dell ocfs2_controld: node down 2013440192
Jun 3 18:19:25 srv-dell cluster-dlm: dlm_process_node: Skipped active
node 1543678144: born-on=224, last-seen=228, this-event=228,
last-event=224
Jun 3 18:19:25 srv-dell ocfs2_controld: Node 2013440192 has left
mountgroup 69AB6E551666484FB314D5B168FD6B84
Jun 3 18:19:25 srv-dell cluster-dlm: add_change: add_change cg 3
remove nodeid 2013440192 reason 3
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Jun 3 18:19:25 srv-dell cluster-dlm: add_change: add_change cg 3
counts member 1 joined 0 remove 1 failed 1
Jun 3 18:19:25 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 3
Jun 3 18:19:25 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/69AB6E551666484FB314D5B168FD6B84/control"
Jun 3 18:19:25 srv-dell cluster-dlm: purge_plocks: purged 0 plocks
for 2013440192
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_te_control:
Registering TE UUID: b3f3f044-df77-4154-bab3-7011030af129
Jun 3 18:19:25 srv-dell crmd: [5672]: WARN:
cib_client_add_notify_callback: Callback already present
Jun 3 18:19:25 srv-dell crmd: [5672]: info: set_graph_functions:
Setting custom graph functions
Jun 3 18:19:25 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition -1: 0 actions in 0 synapses
Jun 3 18:19:25 srv-dell ocfs2_controld: Sending notification of node
2013440192 for "69AB6E551666484FB314D5B168FD6B84"
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_dc_takeover: Taking
over DC status for this partition
Jun 3 18:19:25 srv-dell cluster-dlm: fence_node_time: Node
2013440192/srv-lenovo has not been shot yet
Jun 3 18:19:25 srv-dell ocfs2_controld: confchg called
Jun 3 18:19:25 srv-dell cluster-dlm: check_fencing_done:
check_fencing 2013440192 not fenced add 1244049092 fence 0
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 18:19:25 srv-dell ocfs2_controld: group
"ocfs2:69AB6E551666484FB314D5B168FD6B84" confchg: members 1, left 1,
joined 0
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] Members Joined:
Jun 3 18:19:25 srv-dell openais[5661]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 228: memb=1,
new=0, lost=1
Jun 3 18:19:25 srv-dell cib: [5668]: info: cib_process_readwrite: We
are now in R/W mode
Jun 3 18:19:25 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 18:19:25 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: lost: srv-lenovo 2013440192
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] New Configuration:
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] Members Left:
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] Members Joined:
Jun 3 18:19:25 srv-dell openais[5661]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 228: memb=1, new=0,
lost=0
Jun 3 18:19:25 srv-dell openais[5661]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 18:19:25 srv-dell openais[5661]: [crm ] info:
ais_mark_unseen_peer_dead: Node srv-lenovo was not seen in the
previous transition
Jun 3 18:19:25 srv-dell openais[5661]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: lost
Jun 3 18:19:25 srv-dell openais[5661]: [crm ] info:
send_member_notification: Sending membership update 228 to 4 children
Jun 3 18:19:25 srv-dell openais[5661]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 18:19:25 srv-dell openais[5661]: [TOTEM] entering OPERATIONAL state.
Jun 3 18:19:25 srv-dell openais[5661]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 18:19:25 srv-dell openais[5661]: [CPG ] got joinlist message
from node 1543678144
Jun 3 18:19:25 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_master for section 'all'
(origin=local/crmd/117, version=0.145.7): ok (rc=0)
Jun 3 18:19:25 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/118, version=0.145.7): ok (rc=0)
Jun 3 18:19:25 srv-dell crmd: [5672]: info: join_make_offer: Making
join offers based on membership 228
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_dc_join_offer_all:
join-1: Waiting on 1 outstanding join acks
Jun 3 18:19:25 srv-dell crmd: [5672]: info: ais_dispatch: Membership
228: quorum still lost
Jun 3 18:19:25 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/120, version=0.145.7): ok (rc=0)
Jun 3 18:19:25 srv-dell crmd: [5672]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 18:19:25 srv-dell crmd: [5672]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 18:19:25 srv-dell crmd: [5672]: info: ais_dispatch: Membership
228: quorum still lost
Jun 3 18:19:25 srv-dell ocfs2_controld: message from dlmcontrol
Jun 3 18:19:25 srv-dell ocfs2_controld: Notified for
"69AB6E551666484FB314D5B168FD6B84", node 2013440192, status 0
Jun 3 18:19:25 srv-dell ocfs2_controld: Completing notification on
"69AB6E551666484FB314D5B168FD6B84" for node 2013440192
Jun 3 18:19:25 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/123, version=0.145.7): ok (rc=0)
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes responded to the join offer.
Jun 3 18:19:25 srv-dell crmd: [5672]: info: do_dc_join_finalize:
join-1: Syncing the CIB from srv-dell to the rest of the cluster
Jun 3 18:19:25 srv-dell crmd: [5672]: info: te_connect_stonith:
Attempting connection to fencing daemon...
Jun 3 18:19:25 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/126, version=0.145.7): ok (rc=0)
Jun 3 18:19:25 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/127, version=0.145.7): ok (rc=0)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: te_connect_stonith: Connected
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_dc_join_ack: join-1:
Updating node state to member for srv-dell
Jun 3 18:19:26 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/128, version=0.145.7): ok (rc=0)
Jun 3 18:19:26 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/129,
version=0.145.7): ok (rc=0)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 3 18:19:26 srv-dell crmd: [5672]: info: crm_update_quorum:
Updating quorum status to false (call=133)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 3 18:19:26 srv-dell attrd: [5670]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_pe_invoke: Query 134:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:19:26 srv-dell attrd: [5670]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 18:19:26 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/131, version=0.145.8): ok (rc=0)
Jun 3 18:19:26 srv-dell crmd: [5672]: WARN: match_down_event: No
match for shutdown action on srv-lenovo
Jun 3 18:19:26 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <cib have-quorum="1" dc-uuid="srv-lenovo" admin_epoch="0"
epoch="145" num_updates="9" />
Jun 3 18:19:26 srv-dell crmd: [5672]: info: te_update_diff:
Stonith/shutdown of srv-lenovo not matched
Jun 3 18:19:26 srv-dell cib: [5668]: info: log_data_element:
cib:diff: + <cib have-quorum="0" dc-uuid="srv-dell" admin_epoch="0"
epoch="146" num_updates="1" />
Jun 3 18:19:26 srv-dell crmd: [5672]: info: abort_transition_graph:
te_update_diff:191 - Triggered transition abort (complete=1,
tag=node_state, id=srv-lenovo, magic=NA) : Node failure
Jun 3 18:19:26 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/133, version=0.146.1): ok (rc=0)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 18:19:26 srv-dell crmd: [5672]: info: need_abort: Aborting on
change to have-quorum
Jun 3 18:19:26 srv-dell attrd: [5670]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067566-42, seq=228, quorate=0
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_pe_invoke: Query 135:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_pe_invoke: Query 136:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:19:26 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:19:26 srv-dell cib: [19702]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-43.raw
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:19:26 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:19:26 srv-dell cib: [19702]: info: write_cib_contents: Wrote
version 0.146.0 of the CIB to disk (digest:
9654ff7d27e55c8d594bb801a905fd17)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067566-43, seq=228, quorate=0
Jun 3 18:19:26 srv-dell cib: [19702]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.0xUCUz (digest:
/var/lib/heartbeat/crm/cib.KrzuHb)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: handle_response: pe_calc
calculation pe_calc-dc-1244067566-42 is obsolete
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 0: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-26.bz2
Jun 3 18:19:26 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:19:26 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:19:26 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:19:26 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:19:26 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:19:26 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:19:26 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 1: 12 actions in 12 synapses
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 1 (ref=pe_calc-dc-1244067566-43) derived from
/var/lib/pengine/pe-warn-27.bz2
Jun 3 18:19:26 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:19:26 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 1: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-27.bz2
Jun 3 18:19:26 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:19:26 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:19:26 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:19:26 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:19:26 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:1:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:26 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:26 INFO: Trying to unmount /shared
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:26 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:27 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:27 INFO: Some processes on /shared
were signalled
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:28 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:28 INFO: Some processes on /shared
were signalled
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:29 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:29 INFO: Some processes on /shared
were signalled
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:30 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:30 INFO: Some processes on /shared
were signalled
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:31 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:31 INFO: Some processes on /shared
were signalled
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:32 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:32 INFO: Some processes on /shared
were signalled
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:33 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:19:33 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=47, rc=1, cib-update=137, confirmed=true)
complete unknown error
Jun 3 18:19:33 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:19:33 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067573)
Jun 3 18:19:33 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:1:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:19:33 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:19:33 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:19:33 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:19:33 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:19:33 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:19:33 srv-dell crmd: [5672]: notice: run_graph: Transition 1
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-27.bz2): Stopped
Jun 3 18:19:33 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 1 is now complete
Jun 3 18:19:33 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:19:33 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:19:33 srv-dell crmd: [5672]: info: do_pe_invoke: Query 144:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:19:33 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067573-45, seq=228, quorate=0
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:19:33 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:19:33 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:19:33 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:19:33 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:19:33 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:19:33 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:19:33 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:19:33 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:19:33 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:19:33 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:19:33 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:19:33 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:19:33 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:19:33 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:19:33 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:19:33 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:19:33 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 2: 12 actions in 12 synapses
Jun 3 18:19:33 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 2 (ref=pe_calc-dc-1244067573-45) derived from
/var/lib/pengine/pe-warn-28.bz2
Jun 3 18:19:33 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:19:33 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 2: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-28.bz2
Jun 3 18:19:33 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:19:33 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:19:33 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:19:33 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:19:33 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:2:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:33 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:33 INFO: Trying to unmount /shared
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:33 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:33 INFO: Some processes on /shared
were signalled
Jun 3 18:19:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:34 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:34 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:35 INFO: Some processes on /shared
were signalled
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:36 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:36 INFO: Some processes on /shared
were signalled
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:37 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:37 INFO: Some processes on /shared
were signalled
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:38 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:38 INFO: Some processes on /shared
were signalled
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:39 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:39 INFO: Some processes on /shared
were signalled
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:40 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:19:40 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=48, rc=1, cib-update=145, confirmed=true)
complete unknown error
Jun 3 18:19:40 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:19:40 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067580)
Jun 3 18:19:40 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:2:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:19:40 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:19:40 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:19:40 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:19:40 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:19:40 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:19:40 srv-dell crmd: [5672]: notice: run_graph: Transition 2
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-28.bz2): Stopped
Jun 3 18:19:40 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 2 is now complete
Jun 3 18:19:40 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:19:40 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:19:40 srv-dell crmd: [5672]: info: do_pe_invoke: Query 150:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:19:40 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067580-47, seq=228, quorate=0
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:19:40 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:19:40 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:19:40 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:19:40 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:19:40 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:19:40 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:19:40 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:19:40 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:19:40 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:19:40 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:19:40 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:19:40 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:19:40 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:19:40 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:19:40 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:19:40 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:19:40 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 3: 12 actions in 12 synapses
Jun 3 18:19:40 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 3 (ref=pe_calc-dc-1244067580-47) derived from
/var/lib/pengine/pe-warn-29.bz2
Jun 3 18:19:40 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:19:40 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:19:40 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 3: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-29.bz2
Jun 3 18:19:40 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:19:40 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:19:40 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:19:40 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:3:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:40 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:40 INFO: Trying to unmount /shared
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:40 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:40 INFO: Some processes on /shared
were signalled
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:41 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:41 INFO: Some processes on /shared
were signalled
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:42 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:42 INFO: Some processes on /shared
were signalled
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:44 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:44 INFO: Some processes on /shared
were signalled
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:45 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:45 INFO: Some processes on /shared
were signalled
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:46 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:46 INFO: Some processes on /shared
were signalled
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:47 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:19:47 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=49, rc=1, cib-update=151, confirmed=true)
complete unknown error
Jun 3 18:19:47 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:19:47 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067587)
Jun 3 18:19:47 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:3:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:19:47 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:19:47 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:19:47 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:19:47 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:19:47 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:19:47 srv-dell crmd: [5672]: notice: run_graph: Transition 3
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-29.bz2): Stopped
Jun 3 18:19:47 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 3 is now complete
Jun 3 18:19:47 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:19:47 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:19:47 srv-dell crmd: [5672]: info: do_pe_invoke: Query 156:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:19:47 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067587-49, seq=228, quorate=0
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:19:47 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:19:47 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:19:47 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:19:47 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:19:47 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:19:47 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:19:47 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:19:47 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:19:47 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:19:47 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:19:47 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:19:47 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:19:47 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:19:47 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:19:47 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:19:47 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:19:47 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 4: 12 actions in 12 synapses
Jun 3 18:19:47 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 4 (ref=pe_calc-dc-1244067587-49) derived from
/var/lib/pengine/pe-warn-30.bz2
Jun 3 18:19:47 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:19:47 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 4: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-30.bz2
Jun 3 18:19:47 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:19:47 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:19:47 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:19:47 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:19:47 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:4:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:47 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:47 INFO: Trying to unmount /shared
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:47 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:47 INFO: Some processes on /shared
were signalled
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:48 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:48 INFO: Some processes on /shared
were signalled
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:49 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:49 INFO: Some processes on /shared
were signalled
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:50 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:50 INFO: Some processes on /shared
were signalled
Jun 3 18:19:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:52 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:52 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:52 INFO: Some processes on /shared
were signalled
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:53 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:53 INFO: Some processes on /shared
were signalled
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:54 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:19:54 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=50, rc=1, cib-update=157, confirmed=true)
complete unknown error
Jun 3 18:19:54 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:19:54 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067594)
Jun 3 18:19:54 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:4:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:19:54 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:19:54 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:19:54 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:19:54 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:19:54 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:19:54 srv-dell crmd: [5672]: notice: run_graph: Transition 4
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-30.bz2): Stopped
Jun 3 18:19:54 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 4 is now complete
Jun 3 18:19:54 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:19:54 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:19:54 srv-dell crmd: [5672]: info: do_pe_invoke: Query 162:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:19:54 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067594-51, seq=228, quorate=0
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:19:54 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:19:54 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:19:54 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:19:54 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:19:54 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:19:54 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:19:54 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:19:54 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:19:54 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:19:54 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:19:54 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:19:54 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:19:54 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:19:54 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:19:54 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:19:54 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:19:54 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 5: 12 actions in 12 synapses
Jun 3 18:19:54 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 5 (ref=pe_calc-dc-1244067594-51) derived from
/var/lib/pengine/pe-warn-31.bz2
Jun 3 18:19:54 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:19:54 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:19:54 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:19:54 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:19:54 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:5:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:19:54 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 5: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-31.bz2
Jun 3 18:19:54 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:54 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:54 INFO: Trying to unmount /shared
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:54 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:54 INFO: Some processes on /shared
were signalled
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:55 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:55 INFO: Some processes on /shared
were signalled
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:56 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:56 INFO: Some processes on /shared
were signalled
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:57 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:57 INFO: Some processes on /shared
were signalled
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:58 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:58 INFO: Some processes on /shared
were signalled
Jun 3 18:19:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:19:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:19:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:19:59 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:00 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:00 INFO: Some processes on /shared
were signalled
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:01 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:01 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=51, rc=1, cib-update=163, confirmed=true)
complete unknown error
Jun 3 18:20:01 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:01 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067601)
Jun 3 18:20:01 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:5:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:01 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:01 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:01 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:01 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:01 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:01 srv-dell crmd: [5672]: notice: run_graph: Transition 5
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-31.bz2): Stopped
Jun 3 18:20:01 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 5 is now complete
Jun 3 18:20:01 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:01 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:01 srv-dell crmd: [5672]: info: do_pe_invoke: Query 168:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:01 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067601-53, seq=228, quorate=0
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:01 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:01 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:01 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:01 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:01 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:01 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:01 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:01 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:01 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:01 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:01 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:01 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:01 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:01 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:01 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:01 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:01 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 6: 12 actions in 12 synapses
Jun 3 18:20:01 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 6 (ref=pe_calc-dc-1244067601-53) derived from
/var/lib/pengine/pe-warn-32.bz2
Jun 3 18:20:01 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:01 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 6: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-32.bz2
Jun 3 18:20:01 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:01 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:01 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:01 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:01 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:6:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:01 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:01 INFO: Trying to unmount /shared
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:01 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:01 INFO: Some processes on /shared
were signalled
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:02 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:02 INFO: Some processes on /shared
were signalled
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:03 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:03 INFO: Some processes on /shared
were signalled
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:04 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:04 INFO: Some processes on /shared
were signalled
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:05 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:05 INFO: Some processes on /shared
were signalled
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:06 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:06 INFO: Some processes on /shared
were signalled
Jun 3 18:20:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:07 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:07 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=52, rc=1, cib-update=169, confirmed=true)
complete unknown error
Jun 3 18:20:07 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:07 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067607)
Jun 3 18:20:08 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:6:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:08 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:08 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:08 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:08 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:08 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:08 srv-dell crmd: [5672]: notice: run_graph: Transition 6
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-32.bz2): Stopped
Jun 3 18:20:08 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 6 is now complete
Jun 3 18:20:08 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:08 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:08 srv-dell crmd: [5672]: info: do_pe_invoke: Query 174:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:08 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067608-55, seq=228, quorate=0
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:08 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:08 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:08 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:08 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:08 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:08 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:08 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:08 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:08 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:08 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:08 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:08 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:08 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:08 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:08 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:08 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:08 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 7: 12 actions in 12 synapses
Jun 3 18:20:08 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 7 (ref=pe_calc-dc-1244067608-55) derived from
/var/lib/pengine/pe-warn-33.bz2
Jun 3 18:20:08 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:08 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:08 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 7: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-33.bz2
Jun 3 18:20:08 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:08 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:08 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:08 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:7:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:08 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:08 INFO: Trying to unmount /shared
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:08 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:08 INFO: Some processes on /shared
were signalled
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:09 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:09 INFO: Some processes on /shared
were signalled
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:10 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:10 INFO: Some processes on /shared
were signalled
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:11 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:11 INFO: Some processes on /shared
were signalled
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:12 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:12 INFO: Some processes on /shared
were signalled
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:13 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:13 INFO: Some processes on /shared
were signalled
Jun 3 18:20:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:14 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:14 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=53, rc=1, cib-update=175, confirmed=true)
complete unknown error
Jun 3 18:20:14 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:14 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067614)
Jun 3 18:20:14 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:7:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:14 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:14 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:14 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:14 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:14 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:14 srv-dell crmd: [5672]: notice: run_graph: Transition 7
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-33.bz2): Stopped
Jun 3 18:20:14 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 7 is now complete
Jun 3 18:20:14 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:14 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:14 srv-dell crmd: [5672]: info: do_pe_invoke: Query 180:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:14 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067614-57, seq=228, quorate=0
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:14 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:14 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:14 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:14 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:14 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:14 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:14 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:14 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:14 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:14 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:14 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:14 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:14 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:14 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:14 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:14 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:14 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 8: 12 actions in 12 synapses
Jun 3 18:20:14 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 8 (ref=pe_calc-dc-1244067614-57) derived from
/var/lib/pengine/pe-warn-34.bz2
Jun 3 18:20:14 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:14 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:14 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 8: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-34.bz2
Jun 3 18:20:14 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:14 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:14 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:14 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:8:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:14 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:15 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:15 INFO: Trying to unmount /shared
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:15 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:15 INFO: Some processes on /shared
were signalled
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:16 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:16 INFO: Some processes on /shared
were signalled
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:17 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:17 INFO: Some processes on /shared
were signalled
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:18 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:18 INFO: Some processes on /shared
were signalled
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:19 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:19 INFO: Some processes on /shared
were signalled
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:20 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:20 INFO: Some processes on /shared
were signalled
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:21 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:21 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=54, rc=1, cib-update=181, confirmed=true)
complete unknown error
Jun 3 18:20:21 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:21 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067621)
Jun 3 18:20:21 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:8:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:21 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:21 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:21 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:21 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:21 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:21 srv-dell crmd: [5672]: notice: run_graph: Transition 8
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-34.bz2): Stopped
Jun 3 18:20:21 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 8 is now complete
Jun 3 18:20:21 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:21 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:21 srv-dell crmd: [5672]: info: do_pe_invoke: Query 186:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:21 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067621-59, seq=228, quorate=0
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:21 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:21 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:21 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:21 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:21 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:21 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:21 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:21 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:21 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:21 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:21 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:21 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:21 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:21 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:21 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:21 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:21 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 9: 12 actions in 12 synapses
Jun 3 18:20:21 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 9 (ref=pe_calc-dc-1244067621-59) derived from
/var/lib/pengine/pe-warn-35.bz2
Jun 3 18:20:21 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:21 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:21 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 9: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-35.bz2
Jun 3 18:20:21 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:21 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:21 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:21 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:9:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:21 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:21 INFO: Trying to unmount /shared
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:21 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:22 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:22 INFO: Some processes on /shared
were signalled
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:23 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:23 INFO: Some processes on /shared
were signalled
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:24 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:24 INFO: Some processes on /shared
were signalled
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:25 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:25 INFO: Some processes on /shared
were signalled
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:26 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:26 INFO: Some processes on /shared
were signalled
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:27 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:27 INFO: Some processes on /shared
were signalled
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:28 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:28 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=55, rc=1, cib-update=187, confirmed=true)
complete unknown error
Jun 3 18:20:28 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:28 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067628)
Jun 3 18:20:28 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:9:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:28 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:28 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:28 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:28 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:28 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:28 srv-dell crmd: [5672]: notice: run_graph: Transition 9
(Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-35.bz2): Stopped
Jun 3 18:20:28 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 9 is now complete
Jun 3 18:20:28 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:28 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:28 srv-dell crmd: [5672]: info: do_pe_invoke: Query 192:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:28 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067628-61, seq=228, quorate=0
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:28 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:28 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:28 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:28 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:28 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:28 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:28 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:28 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:28 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:28 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:28 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:28 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:28 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:28 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:28 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:28 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:28 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 10: 12 actions in 12 synapses
Jun 3 18:20:28 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 10 (ref=pe_calc-dc-1244067628-61) derived from
/var/lib/pengine/pe-warn-36.bz2
Jun 3 18:20:28 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:28 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 10: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-36.bz2
Jun 3 18:20:28 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:28 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:28 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:28 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:28 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:10:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:28 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:28 INFO: Trying to unmount /shared
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:28 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:28 INFO: Some processes on /shared
were signalled
Jun 3 18:20:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:29 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:30 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:30 INFO: Some processes on /shared
were signalled
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:31 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:31 INFO: Some processes on /shared
were signalled
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:32 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:32 INFO: Some processes on /shared
were signalled
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:33 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:33 INFO: Some processes on /shared
were signalled
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:34 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:34 INFO: Some processes on /shared
were signalled
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:35 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:35 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=56, rc=1, cib-update=193, confirmed=true)
complete unknown error
Jun 3 18:20:35 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:35 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067635)
Jun 3 18:20:35 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:10:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:35 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:35 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:35 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:35 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:35 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:35 srv-dell crmd: [5672]: notice: run_graph: Transition
10 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-36.bz2): Stopped
Jun 3 18:20:35 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 10 is now complete
Jun 3 18:20:35 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:35 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:35 srv-dell crmd: [5672]: info: do_pe_invoke: Query 198:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:35 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067635-63, seq=228, quorate=0
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:35 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:35 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:35 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:35 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:35 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:35 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:35 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:35 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:35 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:35 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:35 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:35 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:35 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:35 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:35 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:35 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:35 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 11: 12 actions in 12 synapses
Jun 3 18:20:35 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 11 (ref=pe_calc-dc-1244067635-63) derived from
/var/lib/pengine/pe-warn-37.bz2
Jun 3 18:20:35 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:35 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:35 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 11: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-37.bz2
Jun 3 18:20:35 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:35 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:35 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:35 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:11:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:35 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:35 INFO: Trying to unmount /shared
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:35 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:35 INFO: Some processes on /shared
were signalled
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:36 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:36 INFO: Some processes on /shared
were signalled
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:37 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:37 INFO: Some processes on /shared
were signalled
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:39 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:39 INFO: Some processes on /shared
were signalled
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:40 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:40 INFO: Some processes on /shared
were signalled
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:41 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:41 INFO: Some processes on /shared
were signalled
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:42 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:42 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=57, rc=1, cib-update=199, confirmed=true)
complete unknown error
Jun 3 18:20:42 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:42 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067642)
Jun 3 18:20:42 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:11:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:42 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:42 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:42 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:42 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:42 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:42 srv-dell crmd: [5672]: notice: run_graph: Transition
11 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-37.bz2): Stopped
Jun 3 18:20:42 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 11 is now complete
Jun 3 18:20:42 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:42 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:42 srv-dell crmd: [5672]: info: do_pe_invoke: Query 204:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:42 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067642-65, seq=228, quorate=0
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:42 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:42 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:42 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:42 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:42 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:42 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:42 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:42 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:42 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:42 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:42 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:42 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:42 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:42 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:42 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:42 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:42 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 12: 12 actions in 12 synapses
Jun 3 18:20:42 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 12 (ref=pe_calc-dc-1244067642-65) derived from
/var/lib/pengine/pe-warn-38.bz2
Jun 3 18:20:42 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:42 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:42 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:42 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 12: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-38.bz2
Jun 3 18:20:42 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:42 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:42 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:12:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:42 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:42 INFO: Trying to unmount /shared
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:42 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:42 INFO: Some processes on /shared
were signalled
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:43 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:43 INFO: Some processes on /shared
were signalled
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:44 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:44 INFO: Some processes on /shared
were signalled
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:45 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:45 INFO: Some processes on /shared
were signalled
Jun 3 18:20:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:46 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:47 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:47 INFO: Some processes on /shared
were signalled
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:48 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:48 INFO: Some processes on /shared
were signalled
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:49 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:49 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=58, rc=1, cib-update=205, confirmed=true)
complete unknown error
Jun 3 18:20:49 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:49 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067649)
Jun 3 18:20:49 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:12:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:49 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:49 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:49 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:49 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:49 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:49 srv-dell crmd: [5672]: notice: run_graph: Transition
12 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-38.bz2): Stopped
Jun 3 18:20:49 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 12 is now complete
Jun 3 18:20:49 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:49 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:49 srv-dell crmd: [5672]: info: do_pe_invoke: Query 210:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:49 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067649-67, seq=228, quorate=0
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:49 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:49 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:49 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:49 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:49 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:49 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:49 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:49 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:49 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:49 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:49 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:49 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:49 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:49 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:49 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:49 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:49 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 13: 12 actions in 12 synapses
Jun 3 18:20:49 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 13 (ref=pe_calc-dc-1244067649-67) derived from
/var/lib/pengine/pe-warn-39.bz2
Jun 3 18:20:49 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:49 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:49 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 13: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-39.bz2
Jun 3 18:20:49 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:49 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:49 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:49 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:13:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:49 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:49 INFO: Trying to unmount /shared
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:49 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr)
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:49 INFO: Some processes on /shared
were signalled
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:50 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:50 INFO: Some processes on /shared
were signalled
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:51 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:51 INFO: Some processes on /shared
were signalled
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:52 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:52 INFO: Some processes on /shared
were signalled
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:53 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:53 INFO: Some processes on /shared
were signalled
Jun 3 18:20:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:54 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:55 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:55 INFO: Some processes on /shared
were signalled
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:56 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:20:56 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=59, rc=1, cib-update=211, confirmed=true)
complete unknown error
Jun 3 18:20:56 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:20:56 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067656)
Jun 3 18:20:56 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:13:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:20:56 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:20:56 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:20:56 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:20:56 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:20:56 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:20:56 srv-dell crmd: [5672]: notice: run_graph: Transition
13 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-39.bz2): Stopped
Jun 3 18:20:56 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 13 is now complete
Jun 3 18:20:56 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:20:56 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:20:56 srv-dell crmd: [5672]: info: do_pe_invoke: Query 216:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:20:56 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067656-69, seq=228, quorate=0
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:20:56 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:20:56 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:20:56 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:20:56 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:20:56 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:20:56 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:20:56 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:20:56 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:20:56 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:20:56 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:20:56 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:20:56 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:20:56 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:20:56 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:20:56 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:20:56 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:20:56 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 14: 12 actions in 12 synapses
Jun 3 18:20:56 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 14 (ref=pe_calc-dc-1244067656-69) derived from
/var/lib/pengine/pe-warn-40.bz2
Jun 3 18:20:56 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:20:56 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:20:56 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 14: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-40.bz2
Jun 3 18:20:56 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:20:56 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:20:56 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:20:56 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:14:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:56 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:56 INFO: Trying to unmount /shared
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:56 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:56 INFO: Some processes on /shared
were signalled
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:57 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:57 INFO: Some processes on /shared
were signalled
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:58 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:58 INFO: Some processes on /shared
were signalled
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:59 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:20:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:20:59 INFO: Some processes on /shared
were signalled
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:00 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:00 INFO: Some processes on /shared
were signalled
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:01 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:01 INFO: Some processes on /shared
were signalled
Jun 3 18:21:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:02 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:02 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=60, rc=1, cib-update=217, confirmed=true)
complete unknown error
Jun 3 18:21:02 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:02 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067662)
Jun 3 18:21:02 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:14:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:02 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:02 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:02 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:02 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:02 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:02 srv-dell crmd: [5672]: notice: run_graph: Transition
14 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-40.bz2): Stopped
Jun 3 18:21:02 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 14 is now complete
Jun 3 18:21:02 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:02 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:02 srv-dell crmd: [5672]: info: do_pe_invoke: Query 222:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:03 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067663-71, seq=228, quorate=0
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:03 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:03 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:03 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:03 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:03 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:03 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:03 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:03 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:03 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:03 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:03 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:03 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:03 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:03 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:03 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:03 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:03 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 15: 12 actions in 12 synapses
Jun 3 18:21:03 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 15 (ref=pe_calc-dc-1244067663-71) derived from
/var/lib/pengine/pe-warn-41.bz2
Jun 3 18:21:03 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:03 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 15: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-41.bz2
Jun 3 18:21:03 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:03 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:03 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:03 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:03 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:15:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:03 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:03 INFO: Trying to unmount /shared
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:03 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:03 INFO: Some processes on /shared
were signalled
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:04 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:04 INFO: Some processes on /shared
were signalled
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:05 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:05 INFO: Some processes on /shared
were signalled
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:06 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:06 INFO: Some processes on /shared
were signalled
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:07 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:07 INFO: Some processes on /shared
were signalled
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:08 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:08 INFO: Some processes on /shared
were signalled
Jun 3 18:21:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:09 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:09 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=61, rc=1, cib-update=223, confirmed=true)
complete unknown error
Jun 3 18:21:09 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:09 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067669)
Jun 3 18:21:09 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:15:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:09 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:09 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:09 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:09 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:09 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:09 srv-dell crmd: [5672]: notice: run_graph: Transition
15 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-41.bz2): Stopped
Jun 3 18:21:09 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 15 is now complete
Jun 3 18:21:09 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:09 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:09 srv-dell crmd: [5672]: info: do_pe_invoke: Query 228:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:09 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067669-73, seq=228, quorate=0
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:09 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:09 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:09 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:09 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:09 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:09 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:09 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:09 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:09 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:09 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:09 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:09 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:09 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:09 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:09 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:09 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:09 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 16: 12 actions in 12 synapses
Jun 3 18:21:09 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 16 (ref=pe_calc-dc-1244067669-73) derived from
/var/lib/pengine/pe-warn-42.bz2
Jun 3 18:21:09 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:09 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 16: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-42.bz2
Jun 3 18:21:09 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:09 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:09 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:09 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:09 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:16:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:09 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:10 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:10 INFO: Trying to unmount /shared
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:10 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:10 INFO: Some processes on /shared
were signalled
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:11 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:11 INFO: Some processes on /shared
were signalled
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:12 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:12 INFO: Some processes on /shared
were signalled
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:13 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:13 INFO: Some processes on /shared
were signalled
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:14 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:14 INFO: Some processes on /shared
were signalled
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:15 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:15 INFO: Some processes on /shared
were signalled
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:16 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:16 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=62, rc=1, cib-update=229, confirmed=true)
complete unknown error
Jun 3 18:21:16 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:16 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067676)
Jun 3 18:21:16 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:16:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:16 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:16 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:16 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:16 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:16 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:16 srv-dell crmd: [5672]: notice: run_graph: Transition
16 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-42.bz2): Stopped
Jun 3 18:21:16 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 16 is now complete
Jun 3 18:21:16 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:16 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:16 srv-dell crmd: [5672]: info: do_pe_invoke: Query 234:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:16 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067676-75, seq=228, quorate=0
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:16 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:16 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:16 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:16 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:16 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:16 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:16 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:16 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:16 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:16 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:16 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:16 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:16 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:16 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:16 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:16 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:16 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 17: 12 actions in 12 synapses
Jun 3 18:21:16 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 17 (ref=pe_calc-dc-1244067676-75) derived from
/var/lib/pengine/pe-warn-43.bz2
Jun 3 18:21:16 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:16 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 17: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-43.bz2
Jun 3 18:21:16 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:16 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:16 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:16 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:16 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:17:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:16 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:16 INFO: Trying to unmount /shared
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:16 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:16 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:17 INFO: Some processes on /shared
were signalled
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:18 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:18 INFO: Some processes on /shared
were signalled
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:19 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:19 INFO: Some processes on /shared
were signalled
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:20 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:20 INFO: Some processes on /shared
were signalled
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:21 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:21 INFO: Some processes on /shared
were signalled
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:22 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:22 INFO: Some processes on /shared
were signalled
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:23 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:23 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=63, rc=1, cib-update=235, confirmed=true)
complete unknown error
Jun 3 18:21:23 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:23 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067683)
Jun 3 18:21:23 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:17:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:23 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:23 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:23 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:23 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:23 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:23 srv-dell crmd: [5672]: notice: run_graph: Transition
17 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-43.bz2): Stopped
Jun 3 18:21:23 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 17 is now complete
Jun 3 18:21:23 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:23 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:23 srv-dell crmd: [5672]: info: do_pe_invoke: Query 240:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:23 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067683-77, seq=228, quorate=0
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:23 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:23 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:23 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:23 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:23 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:23 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:23 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:23 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:23 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:23 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:23 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:23 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:23 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:23 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:23 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:23 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:23 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 18: 12 actions in 12 synapses
Jun 3 18:21:23 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 18 (ref=pe_calc-dc-1244067683-77) derived from
/var/lib/pengine/pe-warn-44.bz2
Jun 3 18:21:23 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:23 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 18: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-44.bz2
Jun 3 18:21:23 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:23 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:23 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:23 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:23 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:18:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:23 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:23 INFO: Trying to unmount /shared
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:23 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:23 INFO: Some processes on /shared
were signalled
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:24 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:24 INFO: Some processes on /shared
were signalled
Jun 3 18:21:25 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:26 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:26 INFO: Some processes on /shared
were signalled
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:27 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:27 INFO: Some processes on /shared
were signalled
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:28 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:28 INFO: Some processes on /shared
were signalled
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:29 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:29 INFO: Some processes on /shared
were signalled
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:30 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:30 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=64, rc=1, cib-update=241, confirmed=true)
complete unknown error
Jun 3 18:21:30 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:30 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067690)
Jun 3 18:21:30 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:30 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:18:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:30 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:30 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:30 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:30 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:30 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:30 srv-dell crmd: [5672]: notice: run_graph: Transition
18 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-44.bz2): Stopped
Jun 3 18:21:30 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 18 is now complete
Jun 3 18:21:30 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:30 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:30 srv-dell crmd: [5672]: info: do_pe_invoke: Query 246:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:30 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067690-79, seq=228, quorate=0
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:30 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:30 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:30 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:30 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:30 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:30 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:30 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:30 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:30 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:30 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:30 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:30 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:30 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:30 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:30 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:30 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:30 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 19: 12 actions in 12 synapses
Jun 3 18:21:30 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 19 (ref=pe_calc-dc-1244067690-79) derived from
/var/lib/pengine/pe-warn-45.bz2
Jun 3 18:21:30 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:30 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:30 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 19: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-45.bz2
Jun 3 18:21:30 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:30 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:30 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:30 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:19:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:30 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:30 INFO: Trying to unmount /shared
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:30 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:30 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:30 INFO: Some processes on /shared
were signalled
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:31 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:31 INFO: Some processes on /shared
were signalled
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:32 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:32 INFO: Some processes on /shared
were signalled
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:34 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:34 INFO: Some processes on /shared
were signalled
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:35 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:35 INFO: Some processes on /shared
were signalled
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:36 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:36 INFO: Some processes on /shared
were signalled
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:37 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:37 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=65, rc=1, cib-update=247, confirmed=true)
complete unknown error
Jun 3 18:21:37 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:37 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067697)
Jun 3 18:21:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:37 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:19:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:37 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:37 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:37 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:37 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:37 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:37 srv-dell crmd: [5672]: notice: run_graph: Transition
19 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-45.bz2): Stopped
Jun 3 18:21:37 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 19 is now complete
Jun 3 18:21:37 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:37 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:37 srv-dell crmd: [5672]: info: do_pe_invoke: Query 252:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:37 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067697-81, seq=228, quorate=0
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:37 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:37 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:37 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:37 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:37 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:37 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:37 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:37 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:37 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:37 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:37 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:37 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:37 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:37 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:37 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:37 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:37 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 20: 12 actions in 12 synapses
Jun 3 18:21:37 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 20 (ref=pe_calc-dc-1244067697-81) derived from
/var/lib/pengine/pe-warn-46.bz2
Jun 3 18:21:37 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:37 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 20: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-46.bz2
Jun 3 18:21:37 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:37 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:37 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:37 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:37 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:20:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:37 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:37 INFO: Trying to unmount /shared
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:37 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:37 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:37 INFO: Some processes on /shared
were signalled
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:38 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:38 INFO: Some processes on /shared
were signalled
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:39 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:39 INFO: Some processes on /shared
were signalled
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:40 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:40 INFO: Some processes on /shared
were signalled
Jun 3 18:21:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:42 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:42 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:42 INFO: Some processes on /shared
were signalled
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:43 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:43 INFO: Some processes on /shared
were signalled
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:44 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:44 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=66, rc=1, cib-update=253, confirmed=true)
complete unknown error
Jun 3 18:21:44 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:44 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067704)
Jun 3 18:21:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:44 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:20:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:44 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:44 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:44 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:44 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:44 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:44 srv-dell crmd: [5672]: notice: run_graph: Transition
20 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-46.bz2): Stopped
Jun 3 18:21:44 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 20 is now complete
Jun 3 18:21:44 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:44 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:44 srv-dell crmd: [5672]: info: do_pe_invoke: Query 258:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:44 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067704-83, seq=228, quorate=0
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:44 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:44 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:44 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:44 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:44 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:44 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:44 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:44 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:44 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:44 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:44 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:44 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:44 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:44 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:44 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:44 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:44 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 21: 12 actions in 12 synapses
Jun 3 18:21:44 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 21 (ref=pe_calc-dc-1244067704-83) derived from
/var/lib/pengine/pe-warn-47.bz2
Jun 3 18:21:44 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:44 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 21: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-47.bz2
Jun 3 18:21:44 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:44 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:44 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:44 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:44 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:21:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:44 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:44 INFO: Trying to unmount /shared
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:44 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:44 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:44 INFO: Some processes on /shared
were signalled
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:45 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:45 INFO: Some processes on /shared
were signalled
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:46 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:46 INFO: Some processes on /shared
were signalled
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:47 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:47 INFO: Some processes on /shared
were signalled
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:48 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:48 INFO: Some processes on /shared
were signalled
Jun 3 18:21:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:50 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:50 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:50 INFO: Some processes on /shared
were signalled
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:51 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:51 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=67, rc=1, cib-update=259, confirmed=true)
complete unknown error
Jun 3 18:21:51 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:51 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067711)
Jun 3 18:21:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:51 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:21:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:51 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:51 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:51 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:51 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:51 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:51 srv-dell crmd: [5672]: notice: run_graph: Transition
21 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-47.bz2): Stopped
Jun 3 18:21:51 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 21 is now complete
Jun 3 18:21:51 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:51 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:51 srv-dell crmd: [5672]: info: do_pe_invoke: Query 264:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:51 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067711-85, seq=228, quorate=0
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:51 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:51 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:51 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:51 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:51 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:51 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:51 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:51 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:51 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:51 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:51 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:51 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:51 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:51 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:51 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:51 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:51 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 22: 12 actions in 12 synapses
Jun 3 18:21:51 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 22 (ref=pe_calc-dc-1244067711-85) derived from
/var/lib/pengine/pe-warn-48.bz2
Jun 3 18:21:51 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:51 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 22: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-48.bz2
Jun 3 18:21:51 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:51 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:51 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:51 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:51 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:22:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:51 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:51 INFO: Trying to unmount /shared
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:51 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:51 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:51 INFO: Some processes on /shared
were signalled
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:52 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:52 INFO: Some processes on /shared
were signalled
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:53 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:53 INFO: Some processes on /shared
were signalled
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:54 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:54 INFO: Some processes on /shared
were signalled
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:55 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:55 INFO: Some processes on /shared
were signalled
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:56 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:56 INFO: Some processes on /shared
were signalled
Jun 3 18:21:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:57 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:21:57 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=68, rc=1, cib-update=265, confirmed=true)
complete unknown error
Jun 3 18:21:58 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:21:58 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067718)
Jun 3 18:21:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:21:58 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:22:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:21:58 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:21:58 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:21:58 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:21:58 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:21:58 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:21:58 srv-dell crmd: [5672]: notice: run_graph: Transition
22 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-48.bz2): Stopped
Jun 3 18:21:58 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 22 is now complete
Jun 3 18:21:58 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:21:58 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:21:58 srv-dell crmd: [5672]: info: do_pe_invoke: Query 270:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:21:58 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067718-87, seq=228, quorate=0
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:21:58 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:21:58 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:21:58 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:21:58 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:21:58 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:21:58 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:21:58 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:21:58 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:21:58 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:21:58 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:21:58 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:21:58 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:21:58 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:21:58 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:21:58 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:21:58 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:21:58 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 23: 12 actions in 12 synapses
Jun 3 18:21:58 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 23 (ref=pe_calc-dc-1244067718-87) derived from
/var/lib/pengine/pe-warn-49.bz2
Jun 3 18:21:58 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:21:58 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 23: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-49.bz2
Jun 3 18:21:58 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:21:58 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:21:58 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:21:58 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:21:58 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:23:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:58 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:58 INFO: Trying to unmount /shared
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:58 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:58 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:58 INFO: Some processes on /shared
were signalled
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:59 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:21:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:21:59 INFO: Some processes on /shared
were signalled
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:00 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:00 INFO: Some processes on /shared
were signalled
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:01 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:01 INFO: Some processes on /shared
were signalled
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:02 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:02 INFO: Some processes on /shared
were signalled
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:03 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:03 INFO: Some processes on /shared
were signalled
Jun 3 18:22:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:04 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:04 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=69, rc=1, cib-update=271, confirmed=true)
complete unknown error
Jun 3 18:22:04 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:04 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067724)
Jun 3 18:22:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:04 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:23:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:04 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:04 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:04 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:04 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:04 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:04 srv-dell crmd: [5672]: notice: run_graph: Transition
23 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-49.bz2): Stopped
Jun 3 18:22:04 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 23 is now complete
Jun 3 18:22:04 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:04 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:04 srv-dell crmd: [5672]: info: do_pe_invoke: Query 276:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:04 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067724-89, seq=228, quorate=0
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:04 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:04 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:04 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:04 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:04 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:04 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:04 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:04 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:04 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:04 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:04 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:04 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:04 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:04 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:04 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:04 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:04 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 24: 12 actions in 12 synapses
Jun 3 18:22:04 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 24 (ref=pe_calc-dc-1244067724-89) derived from
/var/lib/pengine/pe-warn-50.bz2
Jun 3 18:22:04 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:04 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:04 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:04 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:04 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:24:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:04 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:04 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 24: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-50.bz2
Jun 3 18:22:04 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:05 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:05 INFO: Trying to unmount /shared
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:05 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:05 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:05 INFO: Some processes on /shared
were signalled
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:06 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:06 INFO: Some processes on /shared
were signalled
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:07 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:07 INFO: Some processes on /shared
were signalled
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:08 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:08 INFO: Some processes on /shared
were signalled
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:09 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:09 INFO: Some processes on /shared
were signalled
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:10 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:10 INFO: Some processes on /shared
were signalled
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:11 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:11 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=70, rc=1, cib-update=277, confirmed=true)
complete unknown error
Jun 3 18:22:11 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:11 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067731)
Jun 3 18:22:11 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:11 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:24:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:11 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:11 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:11 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:11 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:11 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:11 srv-dell crmd: [5672]: notice: run_graph: Transition
24 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-50.bz2): Stopped
Jun 3 18:22:11 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 24 is now complete
Jun 3 18:22:11 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:11 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:11 srv-dell crmd: [5672]: info: do_pe_invoke: Query 282:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:11 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067731-91, seq=228, quorate=0
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:11 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:11 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:11 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:11 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:11 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:11 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:11 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:11 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:11 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:11 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:11 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:11 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:11 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:11 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:11 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:11 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:11 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 25: 12 actions in 12 synapses
Jun 3 18:22:11 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 25 (ref=pe_calc-dc-1244067731-91) derived from
/var/lib/pengine/pe-warn-51.bz2
Jun 3 18:22:11 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:11 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:11 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:11 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:11 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:25:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:11 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 25: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-51.bz2
Jun 3 18:22:11 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:11 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:11 INFO: Trying to unmount /shared
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:11 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:11 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:12 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:12 INFO: Some processes on /shared
were signalled
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:13 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:13 INFO: Some processes on /shared
were signalled
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:14 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:14 INFO: Some processes on /shared
were signalled
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:15 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:15 INFO: Some processes on /shared
were signalled
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:16 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:16 INFO: Some processes on /shared
were signalled
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:17 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:17 INFO: Some processes on /shared
were signalled
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:18 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:18 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=71, rc=1, cib-update=283, confirmed=true)
complete unknown error
Jun 3 18:22:18 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:18 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067738)
Jun 3 18:22:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:18 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:25:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:18 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:18 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:18 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:18 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:18 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:18 srv-dell crmd: [5672]: notice: run_graph: Transition
25 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-51.bz2): Stopped
Jun 3 18:22:18 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 25 is now complete
Jun 3 18:22:18 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:18 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:18 srv-dell crmd: [5672]: info: do_pe_invoke: Query 288:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:18 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067738-93, seq=228, quorate=0
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:18 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:18 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:18 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:18 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:18 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:18 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:18 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:18 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:18 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:18 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:18 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:18 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:18 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:18 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:18 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:18 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:18 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 26: 12 actions in 12 synapses
Jun 3 18:22:18 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 26 (ref=pe_calc-dc-1244067738-93) derived from
/var/lib/pengine/pe-warn-52.bz2
Jun 3 18:22:18 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:18 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:18 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 26: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-52.bz2
Jun 3 18:22:18 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:18 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:18 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:18 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:26:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:18 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:18 INFO: Trying to unmount /shared
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:18 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:18 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:18 INFO: Some processes on /shared
were signalled
Jun 3 18:22:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:19 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:19 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:20 INFO: Some processes on /shared
were signalled
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:21 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:21 INFO: Some processes on /shared
were signalled
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:22 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:22 INFO: Some processes on /shared
were signalled
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:23 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:23 INFO: Some processes on /shared
were signalled
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:24 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:24 INFO: Some processes on /shared
were signalled
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:25 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:25 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=72, rc=1, cib-update=289, confirmed=true)
complete unknown error
Jun 3 18:22:25 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:25 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067745)
Jun 3 18:22:25 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:25 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:26:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:25 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:25 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:25 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:25 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:25 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:25 srv-dell crmd: [5672]: notice: run_graph: Transition
26 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-52.bz2): Stopped
Jun 3 18:22:25 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 26 is now complete
Jun 3 18:22:25 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:25 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:25 srv-dell crmd: [5672]: info: do_pe_invoke: Query 294:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:25 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067745-95, seq=228, quorate=0
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:25 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:25 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:25 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:25 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:25 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:25 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:25 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:25 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:25 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:25 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:25 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:25 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:25 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:25 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:25 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:25 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:25 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 27: 12 actions in 12 synapses
Jun 3 18:22:25 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 27 (ref=pe_calc-dc-1244067745-95) derived from
/var/lib/pengine/pe-warn-53.bz2
Jun 3 18:22:25 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:25 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:25 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:25 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:25 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:27:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:25 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 27: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-53.bz2
Jun 3 18:22:25 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:25 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:25 INFO: Trying to unmount /shared
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:25 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:25 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:25 INFO: Some processes on /shared
were signalled
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:26 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:26 INFO: Some processes on /shared
were signalled
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:27 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:27 INFO: Some processes on /shared
were signalled
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:29 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:29 INFO: Some processes on /shared
were signalled
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:30 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:30 INFO: Some processes on /shared
were signalled
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:31 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:31 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:31 INFO: Some processes on /shared
were signalled
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:32 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:32 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=73, rc=1, cib-update=295, confirmed=true)
complete unknown error
Jun 3 18:22:32 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:32 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067752)
Jun 3 18:22:32 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:32 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:27:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:32 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:32 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:32 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:32 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:32 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:32 srv-dell crmd: [5672]: notice: run_graph: Transition
27 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-53.bz2): Stopped
Jun 3 18:22:32 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 27 is now complete
Jun 3 18:22:32 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:32 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:32 srv-dell crmd: [5672]: info: do_pe_invoke: Query 300:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:32 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067752-97, seq=228, quorate=0
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:32 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:32 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:32 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:32 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:32 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:32 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:32 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:32 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:32 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:32 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:32 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:32 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:32 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:32 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:32 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:32 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:32 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 28: 12 actions in 12 synapses
Jun 3 18:22:32 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 28 (ref=pe_calc-dc-1244067752-97) derived from
/var/lib/pengine/pe-warn-54.bz2
Jun 3 18:22:32 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:32 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 28: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-54.bz2
Jun 3 18:22:32 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:32 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:32 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:32 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:32 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:28:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:32 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:32 INFO: Trying to unmount /shared
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:32 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:32 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:32 INFO: Some processes on /shared
were signalled
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:33 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:33 INFO: Some processes on /shared
were signalled
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:34 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:34 INFO: Some processes on /shared
were signalled
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:35 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:35 INFO: Some processes on /shared
were signalled
Jun 3 18:22:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:37 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:37 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:37 INFO: Some processes on /shared
were signalled
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:38 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:38 INFO: Some processes on /shared
were signalled
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:39 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:39 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=74, rc=1, cib-update=301, confirmed=true)
complete unknown error
Jun 3 18:22:39 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:39 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067759)
Jun 3 18:22:39 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:39 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:28:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:39 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:39 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:39 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:39 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:39 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:39 srv-dell crmd: [5672]: notice: run_graph: Transition
28 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-54.bz2): Stopped
Jun 3 18:22:39 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 28 is now complete
Jun 3 18:22:39 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:39 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:39 srv-dell crmd: [5672]: info: do_pe_invoke: Query 306:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:39 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067759-99, seq=228, quorate=0
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:39 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:39 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:39 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:39 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:39 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:39 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:39 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:39 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:39 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:39 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:39 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:39 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:39 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:39 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:39 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:39 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:39 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 29: 12 actions in 12 synapses
Jun 3 18:22:39 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 29 (ref=pe_calc-dc-1244067759-99) derived from
/var/lib/pengine/pe-warn-55.bz2
Jun 3 18:22:39 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:39 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 29: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-55.bz2
Jun 3 18:22:39 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:39 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:39 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:39 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:39 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:29:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:39 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:39 INFO: Trying to unmount /shared
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:39 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:39 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:39 INFO: Some processes on /shared
were signalled
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:40 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:40 INFO: Some processes on /shared
were signalled
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:41 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:41 INFO: Some processes on /shared
were signalled
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:42 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:42 INFO: Some processes on /shared
were signalled
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:43 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:43 INFO: Some processes on /shared
were signalled
Jun 3 18:22:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:44 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:45 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:45 INFO: Some processes on /shared
were signalled
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:46 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:46 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=75, rc=1, cib-update=307, confirmed=true)
complete unknown error
Jun 3 18:22:46 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:46 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067766)
Jun 3 18:22:46 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:46 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:29:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:46 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:46 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:46 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:46 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:46 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:46 srv-dell crmd: [5672]: notice: run_graph: Transition
29 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-55.bz2): Stopped
Jun 3 18:22:46 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 29 is now complete
Jun 3 18:22:46 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:46 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:46 srv-dell crmd: [5672]: info: do_pe_invoke: Query 312:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:46 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067766-101, seq=228, quorate=0
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:46 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:46 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:46 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:46 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:46 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:46 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:46 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:46 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:46 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:46 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:46 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:46 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:46 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:46 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:46 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:46 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:46 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 30: 12 actions in 12 synapses
Jun 3 18:22:46 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 30 (ref=pe_calc-dc-1244067766-101) derived from
/var/lib/pengine/pe-warn-56.bz2
Jun 3 18:22:46 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 30: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-56.bz2
Jun 3 18:22:46 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:46 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:46 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:46 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:46 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:46 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:30:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:46 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:46 INFO: Trying to unmount /shared
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:46 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:46 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:46 INFO: Some processes on /shared
were signalled
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:47 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:47 INFO: Some processes on /shared
were signalled
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:48 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:48 INFO: Some processes on /shared
were signalled
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:49 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:49 INFO: Some processes on /shared
were signalled
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:50 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:50 INFO: Some processes on /shared
were signalled
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:51 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:51 INFO: Some processes on /shared
were signalled
Jun 3 18:22:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:52 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:52 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:52 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=76, rc=1, cib-update=313, confirmed=true)
complete unknown error
Jun 3 18:22:53 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:53 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067773)
Jun 3 18:22:53 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:53 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:30:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:53 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:53 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:53 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:53 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:53 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:53 srv-dell crmd: [5672]: notice: run_graph: Transition
30 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-56.bz2): Stopped
Jun 3 18:22:53 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 30 is now complete
Jun 3 18:22:53 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:53 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:53 srv-dell crmd: [5672]: info: do_pe_invoke: Query 318:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:53 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067773-103, seq=228, quorate=0
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:53 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:53 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:53 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:53 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:53 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:53 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:53 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:53 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:53 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:53 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:53 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:53 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:53 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:53 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:53 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:53 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:53 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 31: 12 actions in 12 synapses
Jun 3 18:22:53 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 31 (ref=pe_calc-dc-1244067773-103) derived from
/var/lib/pengine/pe-warn-57.bz2
Jun 3 18:22:53 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:53 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 31: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-57.bz2
Jun 3 18:22:53 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:53 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:53 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:53 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:53 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:31:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:53 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:53 INFO: Trying to unmount /shared
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:53 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:53 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:53 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:53 INFO: Some processes on /shared
were signalled
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:54 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:54 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:54 INFO: Some processes on /shared
were signalled
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:55 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:55 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:55 INFO: Some processes on /shared
were signalled
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:56 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:56 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:56 INFO: Some processes on /shared
were signalled
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:57 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:57 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:57 INFO: Some processes on /shared
were signalled
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:58 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:58 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:58 INFO: Some processes on /shared
were signalled
Jun 3 18:22:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:22:59 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:22:59 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:22:59 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=77, rc=1, cib-update=319, confirmed=true)
complete unknown error
Jun 3 18:22:59 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:22:59 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067779)
Jun 3 18:22:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:22:59 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:31:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:22:59 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:22:59 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:22:59 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:22:59 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:22:59 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:22:59 srv-dell crmd: [5672]: notice: run_graph: Transition
31 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-57.bz2): Stopped
Jun 3 18:22:59 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 31 is now complete
Jun 3 18:22:59 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:22:59 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:22:59 srv-dell crmd: [5672]: info: do_pe_invoke: Query 324:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:22:59 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067779-105, seq=228, quorate=0
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:22:59 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:22:59 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:22:59 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:22:59 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:22:59 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:22:59 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:22:59 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:22:59 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:22:59 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:22:59 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:22:59 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:22:59 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:22:59 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:22:59 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:22:59 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:22:59 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:22:59 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 32: 12 actions in 12 synapses
Jun 3 18:22:59 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 32 (ref=pe_calc-dc-1244067779-105) derived from
/var/lib/pengine/pe-warn-58.bz2
Jun 3 18:22:59 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:22:59 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 32: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-58.bz2
Jun 3 18:22:59 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:22:59 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:22:59 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:22:59 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:22:59 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:32:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:22:59 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:00 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:00 INFO: Trying to unmount /shared
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:00 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:00 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:00 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:00 INFO: Some processes on /shared
were signalled
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:01 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:01 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:01 INFO: Some processes on /shared
were signalled
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:02 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:02 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:02 INFO: Some processes on /shared
were signalled
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:03 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:03 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:03 INFO: Some processes on /shared
were signalled
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:04 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:04 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:04 INFO: Some processes on /shared
were signalled
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:05 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:05 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:05 INFO: Some processes on /shared
were signalled
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:06 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:23:06 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=78, rc=1, cib-update=325, confirmed=true)
complete unknown error
Jun 3 18:23:06 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:23:06 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067786)
Jun 3 18:23:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:06 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:32:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:23:06 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:23:06 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:06 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:23:06 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:23:06 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:23:06 srv-dell crmd: [5672]: notice: run_graph: Transition
32 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-58.bz2): Stopped
Jun 3 18:23:06 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 32 is now complete
Jun 3 18:23:06 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:23:06 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:23:06 srv-dell crmd: [5672]: info: do_pe_invoke: Query 330:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:23:06 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067786-107, seq=228, quorate=0
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:23:06 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:23:06 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:23:06 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:23:06 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:23:06 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:23:06 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:23:06 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:23:06 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:23:06 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:23:06 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:23:06 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:23:06 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:23:06 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:23:06 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:23:06 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:23:06 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:23:06 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 33: 12 actions in 12 synapses
Jun 3 18:23:06 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 33 (ref=pe_calc-dc-1244067786-107) derived from
/var/lib/pengine/pe-warn-59.bz2
Jun 3 18:23:06 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:23:06 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:23:06 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 33: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-59.bz2
Jun 3 18:23:06 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:23:06 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:23:06 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:23:06 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:33:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:06 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:06 INFO: Trying to unmount /shared
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:06 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:06 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:06 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:07 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:07 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:07 INFO: Some processes on /shared
were signalled
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:08 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:08 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:08 INFO: Some processes on /shared
were signalled
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:09 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:09 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:09 INFO: Some processes on /shared
were signalled
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:10 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:10 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:10 INFO: Some processes on /shared
were signalled
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:11 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:11 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:11 INFO: Some processes on /shared
were signalled
Jun 3 18:23:12 srv-dell mgmtd: [5673]: info: CIB replace: resources
Jun 3 18:23:12 srv-dell cib: [5668]: ERROR: Expecting an element
meta_attributes, got nothing
Jun 3 18:23:12 srv-dell cib: [5668]: ERROR: Invalid sequence in interleave
Jun 3 18:23:12 srv-dell cib: [5668]: ERROR: Element clone failed to
validate content
Jun 3 18:23:12 srv-dell cib: [5668]: ERROR: Element resources has
extra content: primitive
Jun 3 18:23:12 srv-dell cib: [5668]: ERROR: Invalid sequence in interleave
Jun 3 18:23:12 srv-dell cib: [5668]: ERROR: Element cib failed to
validate content
Jun 3 18:23:12 srv-dell cib: [5668]: WARN: cib_perform_op: Updated
CIB does not validate against pacemaker-1.0 schema/dtd
Jun 3 18:23:12 srv-dell cib: [5668]: WARN: cib_diff_notify: Update
(client: mgmtd, call:253): 0.146.66 -> 0.147.1 (Update does not
conform to the configured schema/DTD)
Jun 3 18:23:12 srv-dell cib: [5668]: WARN: cib_process_request:
Operation complete: op cib_replace for section resources
(origin=local/mgmtd/253, version=0.146.66): Update does not conform to
the configured schema/DTD (rc=-47)
Jun 3 18:23:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:12 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:12 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:12 INFO: Some processes on /shared
were signalled
Jun 3 18:23:12 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:13 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:23:13 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=79, rc=1, cib-update=331, confirmed=true)
complete unknown error
Jun 3 18:23:13 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:23:13 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067793)
Jun 3 18:23:13 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:13 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:33:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:23:13 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:23:13 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:13 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:23:13 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:23:13 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:23:13 srv-dell crmd: [5672]: notice: run_graph: Transition
33 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-59.bz2): Stopped
Jun 3 18:23:13 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 33 is now complete
Jun 3 18:23:13 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:23:13 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:23:13 srv-dell crmd: [5672]: info: do_pe_invoke: Query 336:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:23:13 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067793-109, seq=228, quorate=0
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:23:13 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:23:13 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:23:13 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:23:13 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:23:13 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:23:13 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:23:13 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:23:13 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:23:13 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:23:13 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:23:13 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:23:13 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:23:13 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:23:13 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:23:13 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:23:13 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:23:13 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 34: 12 actions in 12 synapses
Jun 3 18:23:13 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 34 (ref=pe_calc-dc-1244067793-109) derived from
/var/lib/pengine/pe-warn-60.bz2
Jun 3 18:23:13 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:23:13 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:23:13 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 34: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-60.bz2
Jun 3 18:23:13 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:23:13 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:23:13 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:23:13 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:34:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:13 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:13 INFO: Trying to unmount /shared
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:13 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:13 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:13 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:13 INFO: Some processes on /shared
were signalled
Jun 3 18:23:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:14 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:14 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:15 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:15 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:15 INFO: Some processes on /shared
were signalled
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:16 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:16 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:16 INFO: Some processes on /shared
were signalled
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:17 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:17 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:17 INFO: Some processes on /shared
were signalled
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:18 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:18 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:18 INFO: Some processes on /shared
were signalled
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:19 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:19 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:19 INFO: Some processes on /shared
were signalled
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:20 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:23:20 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=80, rc=1, cib-update=337, confirmed=true)
complete unknown error
Jun 3 18:23:20 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:23:20 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067800)
Jun 3 18:23:20 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:20 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:34:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:23:20 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:23:20 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:20 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:23:20 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:23:20 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:23:20 srv-dell crmd: [5672]: notice: run_graph: Transition
34 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-60.bz2): Stopped
Jun 3 18:23:20 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 34 is now complete
Jun 3 18:23:20 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:23:20 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:23:20 srv-dell crmd: [5672]: info: do_pe_invoke: Query 342:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:23:20 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067800-111, seq=228, quorate=0
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:23:20 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:23:20 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:23:20 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:23:20 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:23:20 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:23:20 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:23:20 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:23:20 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:23:20 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:23:20 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:23:20 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:23:20 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:23:20 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:23:20 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:23:20 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:23:20 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:23:20 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 35: 12 actions in 12 synapses
Jun 3 18:23:20 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 35 (ref=pe_calc-dc-1244067800-111) derived from
/var/lib/pengine/pe-warn-61.bz2
Jun 3 18:23:20 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:23:20 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:23:20 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:23:20 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:23:20 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:35:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:20 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 35: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-61.bz2
Jun 3 18:23:20 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:20 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:20 INFO: Trying to unmount /shared
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:20 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:20 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:20 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:20 INFO: Some processes on /shared
were signalled
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:21 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:21 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:21 INFO: Some processes on /shared
were signalled
Jun 3 18:23:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:22 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:22 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:22 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:23 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:23 INFO: Some processes on /shared
were signalled
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:24 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:24 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:24 INFO: Some processes on /shared
were signalled
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:25 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:25 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:25 INFO: Some processes on /shared
were signalled
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:26 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:26 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:26 INFO: Some processes on /shared
were signalled
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:27 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:23:27 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=81, rc=1, cib-update=343, confirmed=true)
complete unknown error
Jun 3 18:23:27 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:23:27 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067807)
Jun 3 18:23:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:27 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:35:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:23:27 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:23:27 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:27 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:23:27 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:23:27 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:23:27 srv-dell crmd: [5672]: notice: run_graph: Transition
35 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-61.bz2): Stopped
Jun 3 18:23:27 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 35 is now complete
Jun 3 18:23:27 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:23:27 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:23:27 srv-dell crmd: [5672]: info: do_pe_invoke: Query 348:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:23:27 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067807-113, seq=228, quorate=0
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:23:27 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:23:27 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:23:27 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:23:27 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:23:27 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:23:27 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:23:27 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:23:27 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:23:27 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:23:27 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:23:27 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:23:27 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:23:27 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:23:27 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:23:27 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:23:27 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:23:27 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 36: 12 actions in 12 synapses
Jun 3 18:23:27 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 36 (ref=pe_calc-dc-1244067807-113) derived from
/var/lib/pengine/pe-warn-62.bz2
Jun 3 18:23:27 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:23:27 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 36: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-62.bz2
Jun 3 18:23:27 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:23:27 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:23:27 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:23:27 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:23:27 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:36:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:27 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:27 INFO: Trying to unmount /shared
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:27 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:27 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:27 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:27 INFO: Some processes on /shared
were signalled
Jun 3 18:23:28 srv-dell mgmtd: [5673]: info: CIB delete: resources
Jun 3 18:23:28 srv-dell cib: [5668]: ERROR: IDREF attribute with-rsc
references an unknown ID "dlm-clone"
Jun 3 18:23:28 srv-dell cib: [5668]: ERROR: IDREF attribute with-rsc
references an unknown ID "dlm-clone"
Jun 3 18:23:28 srv-dell cib: [5668]: ERROR: IDREF attribute first
references an unknown ID "dlm-clone"
Jun 3 18:23:28 srv-dell cib: [5668]: WARN: cib_perform_op: Updated
CIB does not validate against pacemaker-1.0 schema/dtd
Jun 3 18:23:28 srv-dell cib: [5668]: WARN: cib_diff_notify: Update
(client: mgmtd, call:258): 0.146.72 -> 0.147.1 (Update does not
conform to the configured schema/DTD)
Jun 3 18:23:28 srv-dell cib: [5668]: WARN: cib_process_request:
Operation complete: op cib_delete for section resources
(origin=local/mgmtd/258, version=0.146.72): Update does not conform to
the configured schema/DTD (rc=-47)
Jun 3 18:23:28 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:28 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:28 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:28 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:28 INFO: Some processes on /shared
were signalled
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:29 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:29 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:29 INFO: Some processes on /shared
were signalled
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:30 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:30 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:30 INFO: Some processes on /shared
were signalled
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:32 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:32 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:32 INFO: Some processes on /shared
were signalled
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:33 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:33 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:33 INFO: Some processes on /shared
were signalled
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:34 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:23:34 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=82, rc=1, cib-update=349, confirmed=true)
complete unknown error
Jun 3 18:23:34 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:23:34 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067814)
Jun 3 18:23:34 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:34 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:36:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:23:34 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1
Jun 3 18:23:34 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:34 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:23:34 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:23:34 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:23:34 srv-dell crmd: [5672]: notice: run_graph: Transition
36 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-62.bz2): Stopped
Jun 3 18:23:34 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 36 is now complete
Jun 3 18:23:34 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:23:34 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:23:34 srv-dell crmd: [5672]: info: do_pe_invoke: Query 354:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:23:34 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067814-115, seq=228, quorate=0
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:23:34 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:23:34 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:23:34 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:23:34 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:23:34 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:23:34 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:23:34 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:23:34 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:23:34 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:23:34 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:23:34 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: info: native_merge_weights:
o2cb-clone: Rolling back scores from fs-clone
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:23:34 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:23:34 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:23:34 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:23:34 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:23:34 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:23:34 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 37: 12 actions in 12 synapses
Jun 3 18:23:34 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 37 (ref=pe_calc-dc-1244067814-115) derived from
/var/lib/pengine/pe-warn-63.bz2
Jun 3 18:23:34 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:23:34 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:23:34 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:23:34 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:23:34 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:37:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:34 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 37: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-63.bz2
Jun 3 18:23:34 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:34 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:34 INFO: Trying to unmount /shared
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:34 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:34 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:34 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:34 INFO: Some processes on /shared
were signalled
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:35 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:35 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:35 INFO: Some processes on /shared
were signalled
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:36 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:36 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:36 INFO: Some processes on /shared
were signalled
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:37 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:37 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:37 INFO: Some processes on /shared
were signalled
Jun 3 18:23:38 srv-dell mgmtd: [5673]: info: CIB delete: constraints
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="146" num_updates="74" >
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <constraints >
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <rsc_colocation id="fs-with-o2cb" rsc="fs-clone"
score="INFINITY" with-rsc="o2cb-clone"
__crm_diff_marker__="removed:top" />
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </constraints>
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 18:23:38 srv-dell crmd: [5672]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=0) : Non-status
change
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element: cib:diff: - </cib>
Jun 3 18:23:38 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1000000
Jun 3 18:23:38 srv-dell cib: [5668]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="147" num_updates="1" />
Jun 3 18:23:38 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:38 srv-dell crmd: [5672]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 18:23:38 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_delete for section constraints
(origin=local/mgmtd/261, version=0.147.1): ok (rc=0)
Jun 3 18:23:38 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:38 srv-dell cib: [25535]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-44.raw
Jun 3 18:23:38 srv-dell cib: [25535]: info: write_cib_contents: Wrote
version 0.147.0 of the CIB to disk (digest:
a97a0af07a338dc3f98f6eab834c5115)
Jun 3 18:23:38 srv-dell cib: [25535]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.PuF8a1 (digest:
/var/lib/heartbeat/crm/cib.ChFRUs)
Jun 3 18:23:38 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:38 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:38 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:38 INFO: Some processes on /shared
were signalled
Jun 3 18:23:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:39 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:39 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:40 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:40 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:40 INFO: Some processes on /shared
were signalled
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:41 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:23:41 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=83, rc=1, cib-update=355, confirmed=true)
complete unknown error
Jun 3 18:23:41 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:23:41 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067821)
Jun 3 18:23:41 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:41 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:37:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:23:41 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:23:41 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:23:41 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:23:41 srv-dell crmd: [5672]: notice: run_graph: Transition
37 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-63.bz2): Stopped
Jun 3 18:23:41 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 37 is now complete
Jun 3 18:23:41 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:23:41 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:23:41 srv-dell crmd: [5672]: info: do_pe_invoke: Query 360:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:23:41 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067821-117, seq=228, quorate=0
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:23:41 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:23:41 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:23:41 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:23:41 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:23:41 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:23:41 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:23:41 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:23:41 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:23:41 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:23:41 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:23:41 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: info: native_merge_weights:
dlm-clone: Rolling back scores from o2cb-clone
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:23:41 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:23:41 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:23:41 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating o2cb:0 with fs:0 on srv-dell
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:23:41 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:23:41 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:23:41 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 38: 12 actions in 12 synapses
Jun 3 18:23:41 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 38 (ref=pe_calc-dc-1244067821-117) derived from
/var/lib/pengine/pe-warn-64.bz2
Jun 3 18:23:41 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:23:41 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:23:41 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:23:41 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:23:41 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:38:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:41 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 38: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-64.bz2
Jun 3 18:23:41 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:41 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:41 INFO: Trying to unmount /shared
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:41 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:41 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:41 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:41 INFO: Some processes on /shared
were signalled
Jun 3 18:23:41 srv-dell mgmtd: [5673]: info: CIB delete: constraints
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="147" num_updates="3" >
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <constraints >
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <rsc_colocation id="o2cb-with-dlm" rsc="o2cb-clone"
score="INFINITY" with-rsc="dlm-clone"
__crm_diff_marker__="removed:top" />
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </constraints>
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element: cib:diff: - </cib>
Jun 3 18:23:41 srv-dell crmd: [5672]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=0) : Non-status
change
Jun 3 18:23:41 srv-dell cib: [5668]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="148" num_updates="1" />
Jun 3 18:23:41 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1000000
Jun 3 18:23:41 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_delete for section constraints
(origin=local/mgmtd/264, version=0.148.1): ok (rc=0)
Jun 3 18:23:41 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:41 srv-dell crmd: [5672]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 18:23:41 srv-dell cib: [25645]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-45.raw
Jun 3 18:23:41 srv-dell cib: [25645]: info: write_cib_contents: Wrote
version 0.148.0 of the CIB to disk (digest:
d2b0a13bda22ae7e744f8d0e9b566bcf)
Jun 3 18:23:41 srv-dell cib: [25645]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.8algBo (digest:
/var/lib/heartbeat/crm/cib.xUZOmX)
Jun 3 18:23:41 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:41 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:42 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:42 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:42 INFO: Some processes on /shared
were signalled
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:43 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:43 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:43 INFO: Some processes on /shared
were signalled
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:44 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:44 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:44 INFO: Some processes on /shared
were signalled
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:45 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:45 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:45 INFO: Some processes on /shared
were signalled
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:46 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:46 srv-dell mgmtd: [5673]: info: CIB delete: constraints
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="148" num_updates="1" >
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <constraints >
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <rsc_order first="o2cb-clone"
id="start-fs-after-o2cb" score="INFINITY" then="fs-clone"
__crm_diff_marker__="removed:top" />
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </constraints>
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 18:23:46 srv-dell crmd: [5672]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=0) : Non-status
change
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element: cib:diff: - </cib>
Jun 3 18:23:46 srv-dell crmd: [5672]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 18:23:46 srv-dell cib: [5668]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="149" num_updates="1" />
Jun 3 18:23:46 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_delete for section constraints
(origin=local/mgmtd/266, version=0.149.1): ok (rc=0)
Jun 3 18:23:46 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:46 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:46 INFO: Some processes on /shared
were signalled
Jun 3 18:23:46 srv-dell cib: [25720]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-46.raw
Jun 3 18:23:46 srv-dell cib: [25720]: info: write_cib_contents: Wrote
version 0.149.0 of the CIB to disk (digest:
d07fbf62ffa2188e89e2122b8a055744)
Jun 3 18:23:46 srv-dell cib: [25720]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.YnQBpZ (digest:
/var/lib/heartbeat/crm/cib.KceCTJ)
Jun 3 18:23:47 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:47 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:47 ERROR: Couldn't unmount
/shared, giving up!
Jun 3 18:23:47 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation fs:0_stop_0 (call=84, rc=1, cib-update=361, confirmed=true)
complete unknown error
Jun 3 18:23:47 srv-dell crmd: [5672]: WARN: status_from_rc: Action 22
(fs:0_stop_0) on srv-dell failed (target: 0 vs. rc: 1): Error
Jun 3 18:23:47 srv-dell crmd: [5672]: WARN: update_failcount:
Updating failcount for fs:0 on srv-dell after failed stop: rc=1
(update=INFINITY, time=1244067827)
Jun 3 18:23:47 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:48 srv-dell crmd: [5672]: info: abort_transition_graph:
match_graph_event:282 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=fs:0_stop_0,
magic=0:1;22:38:0:b3f3f044-df77-4154-bab3-7011030af129) : Event failed
Jun 3 18:23:48 srv-dell crmd: [5672]: info: match_graph_event: Action
fs:0_stop_0 (22) confirmed on srv-dell (rc=4)
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 3 18:23:48 srv-dell crmd: [5672]: info: run_graph:
====================================================
Jun 3 18:23:48 srv-dell crmd: [5672]: notice: run_graph: Transition
38 (Complete=5, Pending=0, Fired=0, Skipped=4, Incomplete=3,
Source=/var/lib/pengine/pe-warn-64.bz2): Stopped
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_graph_trigger:
Transition 38 is now complete
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [
input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_pe_invoke: Query 366:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244067828-119, seq=228, quorate=0
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 18:23:48 srv-dell pengine: [5671]: info: unpack_status: Node
srv-dell is in standby-mode
Jun 3 18:23:48 srv-dell pengine: [5671]: info:
determine_online_status: Node srv-dell is standby
Jun 3 18:23:48 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_monitor_0 on srv-dell returned 0 (ok) instead of the expected
value: 7 (not running)
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Operation fs:0_monitor_0 found resource fs:0 active on srv-dell
Jun 3 18:23:48 srv-dell pengine: [5671]: info: unpack_rsc_op:
fs:0_stop_0 on srv-dell returned 1 (unknown error) instead of the
expected value: 0 (ok)
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: unpack_rsc_op:
Processing failed op fs:0_stop_0 on srv-dell: unknown error
Jun 3 18:23:48 srv-dell pengine: [5671]: info: unpack_status: Node
srv-lenovo is in standby-mode
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN:
determine_online_status_fencing: Node srv-lenovo (srv-lenovo) is
un-expectedly down
Jun 3 18:23:48 srv-dell pengine: [5671]: info:
determine_online_status_fencing: ha_state=active, ccm_state=false,
crm_state=online, join_state=pending, expected=member
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN:
determine_online_status: Node srv-lenovo is unclean
Jun 3 18:23:48 srv-dell pengine: [5671]: info: find_clone: Internally
renamed o2cb:0 on srv-lenovo to o2cb:1
Jun 3 18:23:48 srv-dell pengine: [5671]: info: find_clone: Internally
renamed dlm:0 on srv-lenovo to dlm:1
Jun 3 18:23:48 srv-dell pengine: [5671]: info: find_clone: Internally
renamed cloneresource_cluster_fencing_null:0 on srv-lenovo to
cloneresource_cluster_fencing_null:1
Jun 3 18:23:48 srv-dell pengine: [5671]: info: find_clone: Internally
renamed fs:0 on srv-lenovo to fs:1
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0
cloneresource_cluster_fencing_null:1 ]
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: dlm-clone
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: o2cb-clone
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ o2cb:0 o2cb:1 ]
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: clone_print: Clone
Set: fs-clone
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: print_list:
Stopped: [ fs:0 fs:1 ]
Jun 3 18:23:48 srv-dell pengine: [5671]: info: get_failcount:
fs-clone has failed 1000000 times on srv-dell
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN:
common_apply_stickiness: Forcing fs-clone away from srv-dell after
1000000 failures (max=1000000)
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:0 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:0 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
o2cb:1 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:1 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: native_color: Resource
fs:0 cannot run anywhere
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:1_stop_0 on srv-lenovo is
unrunnable (offline)
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: custom_action: Action
dlm:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: custom_action: Action
o2cb:1_stop_0 on srv-lenovo is unrunnable (offline)
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: custom_action: Marking
node srv-lenovo unclean
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: stage6: Node srv-dell
is unclean!
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: stage6: Node
srv-lenovo is unclean!
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: stage6: Cannot fence
unclean nodes until quorum is attained (or no-quorum-policy is set to
ignore)
Jun 3 18:23:48 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:0 with o2cb:0 on srv-dell
Jun 3 18:23:48 srv-dell pengine: [5671]: info: find_compatible_child:
Colocating dlm:1 with o2cb:1 on srv-lenovo
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:1 (srv-lenovo)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Stop
resource dlm:1 (srv-lenovo)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:0 (srv-dell)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Stop
resource o2cb:1 (srv-lenovo)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Leave
resource fs:1 (Stopped)
Jun 3 18:23:48 srv-dell pengine: [5671]: notice: LogActions: Stop
resource fs:0 (srv-dell)
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 18:23:48 srv-dell crmd: [5672]: info: unpack_graph: Unpacked
transition 39: 12 actions in 12 synapses
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_te_invoke: Processing
graph 39 (ref=pe_calc-dc-1244067828-119) derived from
/var/lib/pengine/pe-warn-65.bz2
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 20 fired and confirmed
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 16: stop o2cb:0_stop_0 on srv-dell (local)
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=16:39:0:b3f3f044-df77-4154-bab3-7011030af129 op=o2cb:0_stop_0 )
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: rsc:o2cb:0: stop
Jun 3 18:23:48 srv-dell crmd: [5672]: info: te_rsc_command:
Initiating action 22: stop fs:0_stop_0 on srv-dell (local)
Jun 3 18:23:48 srv-dell crmd: [5672]: info: do_lrm_rsc_op: Performing
key=22:39:0:b3f3f044-df77-4154-bab3-7011030af129 op=fs:0_stop_0 )
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: rsc:fs:0: stop
Jun 3 18:23:48 srv-dell crmd: [5672]: info: process_lrm_event: LRM
operation o2cb:0_monitor_120000 (call=11, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 18:23:48 srv-dell pengine: [5671]: WARN: process_pe_message:
Transition 39: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-65.bz2
Jun 3 18:23:48 srv-dell pengine: [5671]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:stop:stderr) logd is not running
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:stop:stderr) 2009/06/03_18:23:48 INFO: Stopping o2cb:0
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:48 INFO: Running stop for
/dev/sdb1 on /shared
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:stop:stderr) logd is not running
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(o2cb:0:stop:stderr) 2009/06/03_18:23:48 INFO: Stopping
ocfs2_controld.pcmk
Jun 3 18:23:48 srv-dell ocfs2_controld: Caught signal 15
Jun 3 18:23:48 srv-dell ocfs2_controld: Caught signal 15, but mounts
exist. Ignoring.
Jun 3 18:23:48 srv-dell ocfs2_controld: Caught signal 15, but mounts
exist. Ignoring.
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:48 INFO: Trying to unmount /shared
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:48 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:48 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:48 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:48 INFO: Some processes on /shared
were signalled
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:49 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:49 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:49 INFO: Some processes on /shared
were signalled
Jun 3 18:23:49 srv-dell mgmtd: [5673]: info: CIB delete: constraints
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="149" num_updates="3" >
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <configuration >
Jun 3 18:23:49 srv-dell crmd: [5672]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=0) : Non-status
change
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <constraints >
Jun 3 18:23:49 srv-dell crmd: [5672]: info: update_abort_priority:
Abort priority upgraded from 0 to 1000000
Jun 3 18:23:49 srv-dell crmd: [5672]: info: update_abort_priority:
Abort action done superceeded by restart
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - <rsc_order first="dlm-clone"
id="start-o2cb-after-dlm" score="INFINITY" then="o2cb-clone"
__crm_diff_marker__="removed:top" />
Jun 3 18:23:49 srv-dell crmd: [5672]: info: need_abort: Aborting on
change to admin_epoch
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </constraints>
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element:
cib:diff: - </configuration>
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element: cib:diff: - </cib>
Jun 3 18:23:49 srv-dell cib: [5668]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="150" num_updates="1" />
Jun 3 18:23:49 srv-dell cib: [5668]: info: cib_process_request:
Operation complete: op cib_delete for section constraints
(origin=local/mgmtd/269, version=0.150.1): ok (rc=0)
Jun 3 18:23:49 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:23:49 srv-dell cib: [25852]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-47.raw
Jun 3 18:23:49 srv-dell cib: [25852]: info: write_cib_contents: Wrote
version 0.150.0 of the CIB to disk (digest:
6cc9126f6cf67805f5644c202fec21ed)
Jun 3 18:23:49 srv-dell cib: [25852]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.1sksl0 (digest:
/var/lib/heartbeat/crm/cib.tKmMcR)
Jun 3 18:23:50 srv-dell mgmtd: [5673]: info: CIB query: cib
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:50 ERROR: Couldn't unmount
/shared; trying cleanup with SIGTERM
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:50 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:50 INFO: Some processes on /shared
were signalled
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) umount: /shared: device is busy. (In some
cases useful info about processes that use the device is
found by lsof(8) or fuser(1))
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:51 ERROR: Couldn't unmount
/shared; trying cleanup with SIGKILL
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) /shared:
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stdout) 5551
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output: (fs:0:stop:stderr)
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) logd is not running
Jun 3 18:23:51 srv-dell lrmd: [5669]: info: RA output:
(fs:0:stop:stderr) 2009/06/03_18:23:51 INFO: Some processes on /shared
were signalled
Jun 3 18:25:53 srv-dell syslog-ng[3184]: syslog-ng starting up; version='2.0.9'
Jun 3 18:25:53 srv-dell firmware.sh[3183]: Cannot find firmware file
'intel-ucode/06-0f-07'
Jun 3 18:25:53 srv-dell rchal: CPU frequency scaling is not supported
by your processor.
Jun 3 18:25:53 srv-dell rchal: boot with 'CPUFREQ=no' in to avoid this warning.
Jun 3 18:25:53 srv-dell rchal: Cannot load cpufreq governors - No
cpufreq driver available
Jun 3 18:25:54 srv-dell ifup: lo
Jun 3 18:25:54 srv-dell ifup: lo
Jun 3 18:25:54 srv-dell ifup: IP address: 127.0.0.1/8
Jun 3 18:25:54 srv-dell ifup:
Jun 3 18:25:54 srv-dell ifup:
Jun 3 18:25:54 srv-dell ifup: IP address: 127.0.0.2/8
Jun 3 18:25:54 srv-dell ifup:
Jun 3 18:25:54 srv-dell ifup: eth0 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 18:25:54 srv-dell ifup: No configuration found for eth0
Jun 3 18:25:54 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 18:25:55 srv-dell ifup-dhcp: eth1 Starting DHCP4 client
Jun 3 18:25:55 srv-dell dhcpcd[3707]: eth1: dhcpcd 3.2.3 starting
Jun 3 18:25:55 srv-dell dhcpcd[3707]: eth1: hardware address =
00:19:b9:d7:20:d7
Jun 3 18:25:55 srv-dell dhcpcd[3707]: eth1: broadcasting for a lease
Jun 3 18:25:56 srv-dell ifup-dhcp: .
Jun 3 18:25:58 srv-dell ifup-dhcp: .
Jun 3 18:25:58 srv-dell kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067951.754:2):
operation="profile_load" name="/bin/ping" name2="default" pid=2967
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067951.784:3):
operation="profile_load" name="/sbin/klogd" name2="default" pid=2998
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067951.840:4):
operation="profile_load" name="/sbin/syslog-ng" name2="default"
pid=3009
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067951.896:5):
operation="profile_load" name="/sbin/syslogd" name2="default" pid=3023
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067951.956:6):
operation="profile_load" name="/usr/sbin/avahi-daemon" name2="default"
pid=3034
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067952.004:7):
operation="profile_load" name="/usr/sbin/identd" name2="default"
pid=3035
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067952.067:8):
operation="profile_load" name="/usr/sbin/mdnsd" name2="default"
pid=3036
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067952.136:9):
operation="profile_load" name="/usr/sbin/nscd" name2="default"
pid=3037
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067952.240:10):
operation="profile_load" name="/usr/sbin/ntpd" name2="default"
pid=3038
Jun 3 18:25:58 srv-dell kernel: type=1505 audit(1244067952.289:11):
operation="profile_load" name="/usr/sbin/traceroute" name2="default"
pid=3041
Jun 3 18:25:58 srv-dell kernel: IA-32 Microcode Update Driver:
v1.14a-xen <tigran at aivazian.fsnet.co.uk>
Jun 3 18:25:58 srv-dell kernel: firmware: requesting intel-ucode/06-0f-07
Jun 3 18:25:58 srv-dell kernel: bnx2: eth1: using MSI
Jun 3 18:25:58 srv-dell kernel: NET: Registered protocol family 17
Jun 3 18:25:58 srv-dell kernel: bnx2: eth1 NIC Copper Link is Up,
1000 Mbps full duplex, receive & transmit flow control ON
Jun 3 18:26:00 srv-dell ifup-dhcp: .
Jun 3 18:26:01 srv-dell dhcpcd[3707]: eth1: offered 192.168.2.220
from 192.168.2.1
Jun 3 18:26:01 srv-dell dhcpcd[3707]: eth1: checking 192.168.2.220 is
available on attached networks
Jun 3 18:26:02 srv-dell ifup-dhcp: .
Jun 3 18:26:02 srv-dell dhcpcd[3707]: eth1: leased 192.168.2.220 for
86400 seconds
Jun 3 18:26:02 srv-dell dhcpcd[3707]: eth1: no renewal time supplied,
assuming 43200 seconds
Jun 3 18:26:02 srv-dell dhcpcd[3707]: eth1: no rebind time supplied,
assuming 75600 seconds
Jun 3 18:26:02 srv-dell dhcpcd[3707]: eth1: adding IP address 192.168.2.220/24
Jun 3 18:26:02 srv-dell dhcpcd[3707]: eth1: adding default route via
192.168.2.1 metric 0
Jun 3 18:26:02 srv-dell dns-resolver: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 18:26:02 srv-dell dns-resolver: You can find my version in
/etc/resolv.conf.netconfig
Jun 3 18:26:02 srv-dell dhcpcd-hook: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 3 18:26:02 srv-dell dhcpcd-hook: You can find my version in
/etc/resolv.conf.netconfig ...
Jun 3 18:26:02 srv-dell syslog-ng[3184]: Configuration reload request
received, reloading configuration;
Jun 3 18:26:02 srv-dell syslog-ng[3184]: New configuration initialized;
Jun 3 18:26:02 srv-dell ifdown: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 18:26:03 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 3 18:26:03 srv-dell ifup-dhcp:
Jun 3 18:26:03 srv-dell ifup-dhcp: eth1 IP address: 192.168.2.220/24
Jun 3 18:26:03 srv-dell SuSEfirewall2: SuSEfirewall2 not active
Jun 3 18:26:03 srv-dell auditd[4218]: Started dispatcher:
/sbin/audispd pid: 4220
Jun 3 18:26:03 srv-dell audispd: priority_boost_parser called with: 4
Jun 3 18:26:03 srv-dell audispd: af_unix plugin initialized
Jun 3 18:26:03 srv-dell audispd: audispd initialized with q_depth=80
and 1 active plugins
Jun 3 18:26:03 srv-dell auditd[4218]: Init complete, auditd 1.7.7
listening for events (startup state disable)
Jun 3 18:26:03 srv-dell kernel: NET: Registered protocol family 10
Jun 3 18:26:03 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 18:26:03 srv-dell kernel: lo: Disabled Privacy Extensions
Jun 3 18:26:03 srv-dell dhcpcd[3707]: eth1: setting hostname to `srv-dell'
Jun 3 18:26:03 srv-dell dhcpcd[3707]: eth1: exiting
Jun 3 18:26:04 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 18:26:04 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 18:26:04 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 18:26:04 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 18:26:04 srv-dell smartd[4599]: smartd 5.39 2008-10-24 22:33
[x86_64-suse-linux-gnu] (openSUSE RPM) Copyright (C) 2002-8 by Bruce
Allen, http://smartmontools.sourceforge.net
Jun 3 18:26:04 srv-dell smartd[4599]: Opened configuration file
/etc/smartd.conf
Jun 3 18:26:04 srv-dell smartd[4599]: Drive: DEVICESCAN, implied '-a'
Directive on line 26 of file /etc/smartd.conf
Jun 3 18:26:04 srv-dell smartd[4599]: Configuration file
/etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Jun 3 18:26:04 srv-dell smartd[4599]: Device: /dev/sda, opened
Jun 3 18:26:04 srv-dell smartd[4599]: Device: /dev/sda, IE (SMART)
not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on
SMART features
Jun 3 18:26:04 srv-dell smartd[4599]: Unable to monitor any SMART
enabled devices. Try debug (-d) option. Exiting...
Jun 3 18:26:04 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 3 18:26:04 srv-dell kernel: drbd: initialised. Version: 8.2.7
(api:88/proto:86-88)
Jun 3 18:26:04 srv-dell kernel: drbd: GIT-hash:
a1b440e8b3011a1318d8bff1bb7edc763ef995b0 build by lmb at hermes,
2009-02-20 13:35:59
Jun 3 18:26:04 srv-dell kernel: drbd: registered as block device major 147
Jun 3 18:26:04 srv-dell kernel: drbd: minor_table @ 0xffff8801ea7b01c0
Jun 3 18:26:04 srv-dell kernel: iscsi: registered transport (tcp)
Jun 3 18:26:04 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 3 18:26:05 srv-dell sshd[4925]: Server listening on 0.0.0.0 port 22.
Jun 3 18:26:05 srv-dell sshd[4925]: Server listening on :: port 22.
Jun 3 18:26:05 srv-dell kernel: drbd0: conn( StandAlone -> Unconnected )
Jun 3 18:26:05 srv-dell kernel: drbd0: Starting worker thread (from
cqueue [48])
Jun 3 18:26:05 srv-dell kernel: drbd0: Starting receiver thread (from
drbd0_worker [4929])
Jun 3 18:26:05 srv-dell kernel: drbd0: receiver (re)started
Jun 3 18:26:05 srv-dell kernel: drbd0: conn( Unconnected -> WFConnection )
Jun 3 18:26:05 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/Xservers
Jun 3 18:26:05 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/xdm-config
Jun 3 18:26:05 srv-dell kernel: iscsi: registered transport (iser)
Jun 3 18:26:05 srv-dell iscsid: iSCSI logger with pid=5072 started!
Jun 3 18:26:05 srv-dell xenstored: Checking store ...
Jun 3 18:26:05 srv-dell xenstored: Checking store complete.
Jun 3 18:26:05 srv-dell kernel: suspend: event channel 55
Jun 3 18:26:05 srv-dell kernel: vendor=8086 device=244e
Jun 3 18:26:05 srv-dell kernel: pci 0000:10:0d.0: PCI INT A -> GSI 19
(level, low) -> IRQ 19
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:795: blktapctrl: v1.0.0
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [raw image (aio)]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [raw image (sync)]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [vmware image (vmdk)]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [ramdisk image (ram)]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [qcow disk (qcow)]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [qcow2 disk (qcow2)]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [ioemu disk]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl.c:797: Found
driver: [raw image (cdrom)]
Jun 3 18:26:05 srv-dell BLKTAPCTRL[5201]: blktapctrl_linux.c:23:
/dev/xen/blktap0 device already exists
Jun 3 18:26:06 srv-dell kernel: Bridge firewalling registered
Jun 3 18:26:06 srv-dell /usr/sbin/cron[5246]: (CRON) STARTUP (V5.0)
Jun 3 18:26:06 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 3 18:26:06 srv-dell iscsid: iSCSI daemon with pid=5073 started!
Jun 3 18:26:12 srv-dell gdm-simple-greeter[5352]: libglade-WARNING:
Unexpected element <requires-version> inside <glade-interface>.
Jun 3 18:26:12 srv-dell gdm-session-worker[5353]: PAM pam_putenv:
NULL pam handle passed
Jun 3 18:26:14 srv-dell kernel: eth1: no IPv6 routers present
Jun 3 18:26:20 srv-dell pulseaudio[5542]: pid.c: Stale PID file, overwriting.
Jun 3 18:26:20 srv-dell python: hp-systray(init)[5522]: error:
hp-systray cannot be run as root. Exiting.
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] AIS Executive Service
RELEASE 'subrev 1152 version 0.80'
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] AIS Executive Service:
started and ready to provide service.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Token Timeout (5000
ms) retransmit timeout (490 ms)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] token hold (382 ms)
retransmits before loss (10 retrans)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] join (1000 ms)
send_join (45 ms) consensus (2500 ms) merge (200 ms)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] seqno unchanged const
(30 rotations) Maximum network MTU 1500
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] send threads (0 threads)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] RRP token expired
timeout (490 ms)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] RRP token problem
counter (2000 ms)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] RRP threshold (10 problem count)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] RRP mode set to none.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] heartbeat_failures_allowed (0)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] max_network_delay (50 ms)
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] HeartBeat is Disabled.
To enable set heartbeat_failures_allowed > 0
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] The network interface
[192.168.2.220] is now up.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Created or loaded
sequence id 228.192.168.2.220 for this ring.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] entering GATHER state from 15.
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info:
process_ais_conf: Reading configure
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info:
config_find_next: Processing additional logging options...
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: get_config_opt:
Found 'off' for option: debug
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: get_config_opt:
Found 'yes' for option: to_syslog
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: get_config_opt:
Found 'daemon' for option: syslog_facility
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info:
config_find_next: Processing additional service options...
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_logd
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_mgmtd
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info:
pcmk_plugin_init: CRM: Initialized
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] Logging: Initialized
pcmk_plugin_init
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info:
pcmk_plugin_init: Service: 9
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info:
pcmk_plugin_init: Local node id: 1543678144
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info:
pcmk_plugin_init: Local hostname: srv-dell
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: update_member:
Creating entry for node 1543678144 born on 0
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: update_member:
0x772650 Node 1543678144 now known as srv-dell (was: (null))
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: update_member:
Node srv-dell now has 1 quorum votes (was 0)
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: update_member:
Node 1543678144/srv-dell is now: member
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: spawn_child:
Forked child 5617 for process stonithd
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: spawn_child:
Forked child 5618 for process cib
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: spawn_child:
Forked child 5619 for process lrmd
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: spawn_child:
Forked child 5620 for process attrd
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: spawn_child:
Forked child 5621 for process pengine
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: spawn_child:
Forked child 5622 for process crmd
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: spawn_child:
Forked child 5623 for process mgmtd
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info: pcmk_startup:
CRM: Initialized
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] Service initialized
'Pacemaker Cluster Manager'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais extended virtual synchrony service'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais cluster membership service B.01.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais availability management framework B.01.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais checkpoint service B.01.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais event service B.01.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais distributed locking service B.01.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais message service B.01.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais configuration service'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais cluster closed process group service v1.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SERV ] Service initialized
'openais cluster config database access v1.01'
Jun 3 18:26:46 srv-dell openais[5611]: [SYNC ] Not using a virtual
synchrony filter.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Saving state aru 0
high seq received 0
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Storing new sequence
id for ring e8
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] entering COMMIT state.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] entering RECOVERY state.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] previous ring seq 228
rep 192.168.2.220
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] aru 0 high delivered 0
received flag 1
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] Sending initial ORF token
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] New Configuration:
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] Members Left:
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] Members Joined:
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 232: memb=0,
new=0, lost=0
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] New Configuration:
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] Members Left:
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] Members Joined:
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 232: memb=1, new=1,
lost=0
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info:
pcmk_peer_update: NEW: srv-dell 1543678144
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 18:26:46 srv-dell openais[5611]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000053312
(340754)
Jun 3 18:26:46 srv-dell openais[5611]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 18:26:46 srv-dell openais[5611]: [TOTEM] entering OPERATIONAL state.
Jun 3 18:26:46 srv-dell openais[5611]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 18:26:46 srv-dell lrmd: [5619]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:26:46 srv-dell lrmd: [5619]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Jun 3 18:26:46 srv-dell mgmtd: [5623]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:26:46 srv-dell mgmtd: [5623]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 3 18:26:46 srv-dell mgmtd: [5623]: debug: Enabling coredumps
Jun 3 18:26:46 srv-dell mgmtd: [5623]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 18:26:46 srv-dell mgmtd: [5623]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 18:26:46 srv-dell pengine: [5621]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:26:46 srv-dell cib: [5618]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:26:46 srv-dell attrd: [5620]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:26:46 srv-dell mgmtd: [5623]: WARN: lrm_signon: can not
initiate connection
Jun 3 18:26:46 srv-dell mgmtd: [5623]: info: login to lrm: 0, ret:0
Jun 3 18:26:46 srv-dell pengine: [5621]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:26:46 srv-dell stonithd: [5617]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:26:46 srv-dell attrd: [5620]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:26:46 srv-dell cib: [5618]: WARN: Initializing connection to
logging daemon failed. Logging daemon may not be running
Jun 3 18:26:46 srv-dell pengine: [5621]: info: main: Starting pengine
Jun 3 18:26:46 srv-dell crmd: [5622]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:26:46 srv-dell stonithd: [5617]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 18:26:46 srv-dell attrd: [5620]: info: main: Starting up....
Jun 3 18:26:46 srv-dell stonithd: [5617]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 18:26:46 srv-dell attrd: [5620]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:26:46 srv-dell cib: [5618]: info: G_main_add_TriggerHandler:
Added signal manual handler
Jun 3 18:26:46 srv-dell cib: [5618]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 18:26:46 srv-dell crmd: [5622]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:26:46 srv-dell attrd: [5620]: info: init_ais_connection: AIS
connection established
Jun 3 18:26:46 srv-dell crmd: [5622]: info: main: CRM Hg Version:
0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Jun 3 18:26:46 srv-dell lrmd: [5619]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 18:26:46 srv-dell crmd: [5622]: info: crmd_init: Starting crmd
Jun 3 18:26:46 srv-dell lrmd: [5619]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jun 3 18:26:46 srv-dell lrmd: [5619]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jun 3 18:26:46 srv-dell lrmd: [5619]: info: Started.
Jun 3 18:26:46 srv-dell stonithd: [5617]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:26:46 srv-dell crmd: [5622]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 18:26:46 srv-dell attrd: [5620]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 18:26:46 srv-dell cib: [5618]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Jun 3 18:26:46 srv-dell attrd: [5620]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:26:46 srv-dell stonithd: [5617]: info: init_ais_connection:
AIS connection established
Jun 3 18:26:46 srv-dell attrd: [5620]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:26:46 srv-dell stonithd: [5617]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info: pcmk_ipc:
Recorded connection 0x77ccc0 for attrd/5620
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info: pcmk_ipc:
Recorded connection 0x77d010 for stonithd/5617
Jun 3 18:26:46 srv-dell stonithd: [5617]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:26:46 srv-dell stonithd: [5617]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:26:46 srv-dell stonithd: [5617]: notice:
/usr/lib64/heartbeat/stonithd start up successfully.
Jun 3 18:26:46 srv-dell stonithd: [5617]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 18:26:46 srv-dell cib: [5618]: info: startCib: CIB
Initialization completed successfully
Jun 3 18:26:46 srv-dell cib: [5618]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:26:46 srv-dell cib: [5618]: info: init_ais_connection: AIS
connection established
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info: pcmk_ipc:
Recorded connection 0x77cef0 for cib/5618
Jun 3 18:26:46 srv-dell openais[5611]: [crm ] info: pcmk_ipc:
Sending membership update 232 to cib
Jun 3 18:26:46 srv-dell cib: [5618]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 18:26:46 srv-dell cib: [5618]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:26:46 srv-dell cib: [5618]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:26:46 srv-dell cib: [5618]: info: cib_init: Starting cib mainloop
Jun 3 18:26:46 srv-dell cib: [5618]: info: ais_dispatch: Membership
232: quorum still lost
Jun 3 18:26:46 srv-dell cib: [5618]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=232
proc=00000000000000000000000000053312 (new)
Jun 3 18:26:46 srv-dell cib: [5627]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-48.raw
Jun 3 18:26:46 srv-dell cib: [5627]: info: write_cib_contents: Wrote
version 0.150.0 of the CIB to disk (digest:
976f5465383fee4ed1b638c38116ec5b)
Jun 3 18:26:46 srv-dell cib: [5627]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Nttr7h (digest:
/var/lib/heartbeat/crm/cib.YiwD9P)
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] entering GATHER state from 11.
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] Saving state aru a
high seq received a
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] Storing new sequence
id for ring ec
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] entering COMMIT state.
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] entering RECOVERY state.
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] previous ring seq 228
rep 192.168.2.120
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] aru 3b high delivered
3b received flag 1
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] previous ring seq 232
rep 192.168.2.220
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] aru a high delivered a
received flag 1
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] New Configuration:
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] Members Left:
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] Members Joined:
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 236: memb=1,
new=0, lost=0
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] New Configuration:
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] Members Left:
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] Members Joined:
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 236: memb=2, new=1,
lost=0
Jun 3 18:26:47 srv-dell cib: [5618]: info: ais_dispatch: Membership
236: quorum still lost
Jun 3 18:26:47 srv-dell cib: [5618]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 18:26:47 srv-dell cib: [5618]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=236 proc=00000000000000000000000000000000
Jun 3 18:26:47 srv-dell openais[5611]: [MAIN ] info: update_member:
Creating entry for node 2013440192 born on 236
Jun 3 18:26:47 srv-dell openais[5611]: [MAIN ] info: update_member:
Node 2013440192/unknown is now: member
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info:
pcmk_peer_update: NEW: .pending. 2013440192
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info:
pcmk_peer_update: MEMB: .pending. 2013440192
Jun 3 18:26:47 srv-dell cib: [5618]: info: ais_dispatch: Membership
236: quorum still lost
Jun 3 18:26:47 srv-dell cib: [5618]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 18:26:47 srv-dell cib: [5618]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=236 seen=236 proc=00000000000000000000000000053312
(new)
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info:
send_member_notification: Sending membership update 236 to 1 children
Jun 3 18:26:47 srv-dell openais[5611]: [MAIN ] info: update_member:
0x772650 Node 1543678144 ((null)) born on: 236
Jun 3 18:26:47 srv-dell openais[5611]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 18:26:47 srv-dell openais[5611]: [TOTEM] entering OPERATIONAL state.
Jun 3 18:26:47 srv-dell openais[5611]: [MAIN ] info: update_member:
0x7f4870001580 Node 2013440192 (srv-lenovo) born on: 236
Jun 3 18:26:47 srv-dell openais[5611]: [MAIN ] info: update_member:
0x7f4870001580 Node 2013440192 now known as srv-lenovo (was: (null))
Jun 3 18:26:47 srv-dell openais[5611]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 3 18:26:47 srv-dell openais[5611]: [MAIN ] info: update_member:
Node srv-lenovo now has 1 quorum votes (was 0)
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info:
send_member_notification: Sending membership update 236 to 1 children
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 18:26:47 srv-dell openais[5611]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 18:26:47 srv-dell cib: [5618]: info: cib_process_diff: Diff
0.154.2 -> 0.155.1 not applied to 0.150.0: current "epoch" is less
than required
Jun 3 18:26:47 srv-dell cib: [5618]: info: cib_server_process_diff:
Requesting re-sync from peer
Jun 3 18:26:47 srv-dell cib: [5618]: WARN: cib_diff_notify:
Local-only Change (client:crmd, call: 46): 0.0.0 (Application of an
update diff failed, requesting a full refresh)
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 18:26:47 srv-dell cib: [5618]: WARN: cib_server_process_diff:
Not applying diff 0.155.1 -> 0.155.2 (sync in progress)
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 3 18:26:47 srv-dell cib: [5618]: info: cib_replace_notify:
Replaced: 0.150.0 -> 0.155.2 from srv-lenovo
Jun 3 18:26:47 srv-dell cib: [5628]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-49.raw
Jun 3 18:26:47 srv-dell cib: [5628]: info: write_cib_contents: Wrote
version 0.155.0 of the CIB to disk (digest:
a76fcd020145f504e16c1c0e750d288a)
Jun 3 18:26:47 srv-dell cib: [5628]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.7UCAsg (digest:
/var/lib/heartbeat/crm/cib.LjV0uQ)
Jun 3 18:26:47 srv-dell mgmtd: [5623]: info: init_crm
Jun 3 18:26:47 srv-dell crmd: [5622]: info: do_cib_control: CIB
connection established
Jun 3 18:26:47 srv-dell crmd: [5622]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:26:47 srv-dell crmd: [5622]: info: init_ais_connection: AIS
connection established
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f48700011b0 for crmd/5622
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info: pcmk_ipc:
Sending membership update 236 to crmd
Jun 3 18:26:47 srv-dell crmd: [5622]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 18:26:47 srv-dell crmd: [5622]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:26:47 srv-dell crmd: [5622]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:26:47 srv-dell crmd: [5622]: info: do_ha_control: Connected
to the cluster
Jun 3 18:26:47 srv-dell crmd: [5622]: info: do_started: Delaying
start, CCM (0000000000100000) not connected
Jun 3 18:26:47 srv-dell crmd: [5622]: info: crmd_init: Starting crmd's mainloop
Jun 3 18:26:47 srv-dell crmd: [5622]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 18:26:47 srv-dell openais[5611]: [crm ] info:
update_expected_votes: Expected quorum votes 1024 -> 2
Jun 3 18:26:47 srv-dell crmd: [5622]: info: ais_dispatch: Membership
236: quorum still lost
Jun 3 18:26:47 srv-dell crmd: [5622]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 18:26:47 srv-dell crmd: [5622]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 18:26:47 srv-dell crmd: [5622]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=236 seen=236
proc=00000000000000000000000000053312
Jun 3 18:26:47 srv-dell crmd: [5622]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=236 seen=236
proc=00000000000000000000000000053312 (new)
Jun 3 18:26:47 srv-dell crmd: [5622]: info: do_started: The local CRM
is operational
Jun 3 18:26:47 srv-dell crmd: [5622]: info: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Jun 3 18:26:48 srv-dell mgmtd: [5623]: debug: main: run the loop...
Jun 3 18:26:48 srv-dell mgmtd: [5623]: info: Started.
Jun 3 18:26:48 srv-dell crmd: [5622]: notice: ais_dispatch:
Membership 236: quorum aquired
Jun 3 18:26:48 srv-dell crmd: [5622]: info: update_dc: Set DC to
srv-lenovo (3.0.1)
Jun 3 18:26:48 srv-dell crmd: [5622]: info: update_attrd: Connecting
to attrd...
Jun 3 18:26:48 srv-dell crmd: [5622]: info: update_attrd: Updating
terminate=<none> via attrd for srv-dell
Jun 3 18:26:48 srv-dell crmd: [5622]: info: update_attrd: Updating
shutdown=<none> via attrd for srv-dell
Jun 3 18:26:48 srv-dell crmd: [5622]: info: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 3 18:26:48 srv-dell crmd: [5622]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/transient_attributes": ok
(rc=0)
Jun 3 18:26:48 srv-dell crmd: [5622]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 18:26:48 srv-dell crmd: [5622]: info: do_lrm_rsc_op: Performing
key=4:9:7:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 18:26:48 srv-dell lrmd: [5619]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 18:26:48 srv-dell lrmd: [5619]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 18:26:48 srv-dell crmd: [5622]: info: do_lrm_rsc_op: Performing
key=5:9:7:6712d200-ef04-46ab-920d-6e4813d1a596
op=cloneresource_cluster_fencing_null:0_monitor_0 )
Jun 3 18:26:48 srv-dell lrmd: [5619]: info:
rsc:cloneresource_cluster_fencing_null:0: monitor
Jun 3 18:26:48 srv-dell crmd: [5622]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_monitor_0 (call=3,
rc=7, cib-update=8, confirmed=true) complete not running
Jun 3 18:26:48 srv-dell crmd: [5622]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=2, rc=7,
cib-update=9, confirmed=true) complete not running
Jun 3 18:26:56 srv-dell attrd: [5620]: info: main: Sending full refresh
Jun 3 18:26:56 srv-dell attrd: [5620]: info: main: Starting mainloop...
Jun 3 18:26:56 srv-dell attrd: [5620]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 18:26:56 srv-dell attrd: [5620]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 18:26:56 srv-dell attrd: [5620]: info: find_hash_entry:
Creating hash entry for terminate
Jun 3 18:26:56 srv-dell attrd: [5620]: info: find_hash_entry:
Creating hash entry for shutdown
Jun 3 18:26:56 srv-dell attrd: [5620]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 18:26:56 srv-dell attrd: [5620]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 18:26:56 srv-dell attrd: [5620]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 18:27:04 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:27:22 srv-dell crmd: [5622]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 18:27:22 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-dell
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244068042-1" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5623_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5623" >
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive
id="cloneresource_cluster_fencing_null:1" />
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 18:27:22 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 18:27:22 srv-dell crmd: [5622]: info: send_direct_ack: ACK'ing
resource op cloneresource_cluster_fencing_null:1_delete_0 from
mgmtd-5623: lrm_invoke-lrmd-1244068042-5
Jun 3 18:27:22 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:27:24 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-lenovo
Jun 3 18:27:24 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:27:24 srv-dell cib: [5661]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-50.raw
Jun 3 18:27:24 srv-dell cib: [5661]: info: write_cib_contents: Wrote
version 0.156.0 of the CIB to disk (digest:
c66f8c0316cc9f3344290b92f1b152fb)
Jun 3 18:27:24 srv-dell cib: [5661]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.z8mFv2 (digest:
/var/lib/heartbeat/crm/cib.sUr2YV)
Jun 3 18:27:25 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Child 5662 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 18:27:25 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 18:27:25 srv-dell openais[5611]: [MAIN ] Msg[26]
(dest=local:unknown, from=srv-lenovo:crmd.4910, remote=true,
size=922): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 18:27:26 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:27:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:27:26 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:27:26 srv-dell cib: [5663]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-51.raw
Jun 3 18:27:26 srv-dell cib: [5663]: info: write_cib_contents: Wrote
version 0.157.0 of the CIB to disk (digest:
31fbf63ddeba9d0e15c4f29acae25d99)
Jun 3 18:27:26 srv-dell cib: [5663]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Np15O7 (digest:
/var/lib/heartbeat/crm/cib.UBZoK5)
Jun 3 18:27:27 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:27:36 srv-dell cib: [5664]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-52.raw
Jun 3 18:27:36 srv-dell cib: [5664]: info: write_cib_contents: Wrote
version 0.158.0 of the CIB to disk (digest:
b447b43b131782de1fee11bbd52e3051)
Jun 3 18:27:36 srv-dell cib: [5664]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.WiqDeE (digest:
/var/lib/heartbeat/crm/cib.wml8pW)
Jun 3 18:27:36 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:27:36 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:27:36 srv-dell lrmd: [5619]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 18:27:36 srv-dell crmd: [5622]: info: do_lrm_rsc_op: Performing
key=10:12:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=cloneresource_cluster_fencing_null:1_start_0 )
Jun 3 18:27:36 srv-dell lrmd: [5619]: info:
rsc:cloneresource_cluster_fencing_null:1: start
Jun 3 18:27:36 srv-dell lrmd: [5665]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 3 18:27:36 srv-dell stonithd: [5617]: info:
cloneresource_cluster_fencing_null:1 stonith resource started
Jun 3 18:27:36 srv-dell lrmd: [5619]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 18:27:36 srv-dell crmd: [5622]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_start_0 (call=4, rc=0,
cib-update=15, confirmed=true) complete ok
Jun 3 18:27:36 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:27:36 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:36:46 srv-dell cib: [5618]: info: cib_stats: Processed 60
operations (166.00us average, 0% utilization) in the last 10min
Jun 3 18:40:27 srv-dell mgmtd: [5623]: info: CIB replace: resources
Jun 3 18:40:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:27 srv-dell cib: [5822]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-53.raw
Jun 3 18:40:27 srv-dell cib: [5822]: info: write_cib_contents: Wrote
version 0.159.0 of the CIB to disk (digest:
df51240673be28d0372b116382ba588e)
Jun 3 18:40:27 srv-dell cib: [5822]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.zthXVN (digest:
/var/lib/heartbeat/crm/cib.rDKOdb)
Jun 3 18:40:27 srv-dell lrmd: [5619]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 18:40:27 srv-dell crmd: [5622]: info: do_lrm_rsc_op: Performing
key=5:13:7:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitive_cluster_filesystem:0_monitor_0 )
Jun 3 18:40:27 srv-dell lrmd: [5619]: info:
rsc:primitive_cluster_filesystem:0: monitor
Jun 3 18:40:27 srv-dell lrmd: [5619]: debug: RA output [usage:
/etc/ha.d/resource.d//Filesystem <device> <directory> [<fstype>
[<option] didn't match any pattern
Jun 3 18:40:27 srv-dell lrmd: [5619]: CRIT: read_pipe:3662 Attempt to
read from closed file descriptor 10.
Jun 3 18:40:27 srv-dell lrmd: [5619]: notice: read_pipe::3668:
lrmd_op has been freed
Jun 3 18:40:27 srv-dell crmd: [5622]: info: process_lrm_event: LRM
operation primitive_cluster_filesystem:0_monitor_0 (call=5, rc=7,
cib-update=16, confirmed=true) complete not running
Jun 3 18:40:27 srv-dell crmd: [5622]: info: process_lrm_event:
Result: usage: /etc/ha.d/resource.d//Filesystem <device> <directory>
[<fstype> [<option
Jun 3 18:40:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:27 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:40:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:27 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:40:43 srv-dell crm_resource: [5832]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:40:43 srv-dell crm_resource: [5832]: info: Invoked:
crm_resource --meta -r primitive_cluster_filesystem -p target-role -v
started
Jun 3 18:40:43 srv-dell cib: [5833]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-54.raw
Jun 3 18:40:43 srv-dell cib: [5833]: info: write_cib_contents: Wrote
version 0.160.0 of the CIB to disk (digest:
2ceee26649c22033856b99965795d6df)
Jun 3 18:40:43 srv-dell cib: [5833]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.egUaoF (digest:
/var/lib/heartbeat/crm/cib.RnxUZB)
Jun 3 18:40:43 srv-dell crmd: [5622]: info: do_lrm_rsc_op: Performing
key=16:14:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitive_cluster_filesystem:0_start_0 )
Jun 3 18:40:43 srv-dell lrmd: [5619]: info:
rsc:primitive_cluster_filesystem:0: start
Jun 3 18:40:43 srv-dell lrmd: [5619]: info: RA output:
(primitive_cluster_filesystem:0:start:stdout) usage:
/etc/ha.d/resource.d//Filesystem <device> <directory> [<fstype>
[<options>]] (start|stop|status|usage|meta-data)
Jun 3 18:40:43 srv-dell crmd: [5622]: info: process_lrm_event: LRM
operation primitive_cluster_filesystem:0_start_0 (call=6, rc=1,
cib-update=19, confirmed=true) complete unknown error
Jun 3 18:40:44 srv-dell crmd: [5622]: info: do_lrm_rsc_op: Performing
key=3:15:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitive_cluster_filesystem:0_stop_0 )
Jun 3 18:40:44 srv-dell lrmd: [5619]: info:
rsc:primitive_cluster_filesystem:0: stop
Jun 3 18:40:44 srv-dell lrmd: [5619]: info: RA output:
(primitive_cluster_filesystem:0:stop:stdout) usage:
/etc/ha.d/resource.d//Filesystem <device> <directory> [<fstype>
[<options>]] (start|stop|status|usage|meta-data)
Jun 3 18:40:44 srv-dell crmd: [5622]: info: process_lrm_event: LRM
operation primitive_cluster_filesystem:0_stop_0 (call=7, rc=1,
cib-update=20, confirmed=true) complete unknown error
Jun 3 18:40:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:44 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:40:45 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:45 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:45 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:45 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:45 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:45 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:40:45 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:41:12 srv-dell mgmtd: [5623]: info: CIB delete: resources
Jun 3 18:41:12 srv-dell cib: [5851]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-55.raw
Jun 3 18:41:12 srv-dell cib: [5851]: info: write_cib_contents: Wrote
version 0.161.0 of the CIB to disk (digest:
3d5c926283f67a19258afa3bec088271)
Jun 3 18:41:12 srv-dell cib: [5851]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.LaW2ld (digest:
/var/lib/heartbeat/crm/cib.gFvHfa)
Jun 3 18:41:12 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:12 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:41:36 srv-dell crm_attribute: [5853]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:41:36 srv-dell crm_attribute: [5853]: info: Invoked:
crm_attribute -t crm_config -n maintenance-mode -v true
Jun 3 18:41:36 srv-dell cib: [5854]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-56.raw
Jun 3 18:41:36 srv-dell cib: [5854]: info: write_cib_contents: Wrote
version 0.162.0 of the CIB to disk (digest:
423234df0a453a531d04ef9a11c6a7b8)
Jun 3 18:41:36 srv-dell cib: [5854]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.OqrlC1 (digest:
/var/lib/heartbeat/crm/cib.G4XXBQ)
Jun 3 18:41:37 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:38 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:41:40 srv-dell crm_attribute: [5856]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:41:40 srv-dell crm_attribute: [5856]: info: Invoked:
crm_attribute -t crm_config -D -n maintenance-mode
Jun 3 18:41:40 srv-dell cib: [5857]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-57.raw
Jun 3 18:41:40 srv-dell cib: [5857]: info: write_cib_contents: Wrote
version 0.163.0 of the CIB to disk (digest:
4e909356fd618f12db04ab6603966f8f)
Jun 3 18:41:40 srv-dell cib: [5857]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.9ggPJ2 (digest:
/var/lib/heartbeat/crm/cib.6TZJd0)
Jun 3 18:41:41 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:42 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:41:51 srv-dell crmd: [5622]: info: do_lrm_invoke: Removing
resource primitiveresource_cluster_ip from the LRM
Jun 3 18:41:51 srv-dell mgmtd: [5623]: info: Delete fail-count for
primitiveresource_cluster_ip from srv-dell
Jun 3 18:41:51 srv-dell crmd: [5622]: info: send_direct_ack: ACK'ing
resource op primitiveresource_cluster_ip_delete_0 from mgmtd-5623:
lrm_invoke-lrmd-1244068911-8
Jun 3 18:41:51 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:41:51 srv-dell cib: [5858]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-58.raw
Jun 3 18:41:51 srv-dell cib: [5858]: info: write_cib_contents: Wrote
version 0.164.0 of the CIB to disk (digest:
454df021196847ffc93e085016e6d291)
Jun 3 18:41:51 srv-dell cib: [5858]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.9Xcg8x (digest:
/var/lib/heartbeat/crm/cib.RHHCTR)
Jun 3 18:41:53 srv-dell mgmtd: [5623]: info: Delete fail-count for
primitiveresource_cluster_ip from srv-lenovo
Jun 3 18:41:53 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:41:53 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Child 5859 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 18:41:53 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 18:41:53 srv-dell openais[5611]: [MAIN ] Msg[35]
(dest=local:unknown, from=srv-lenovo:crmd.4910, remote=true,
size=906): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 18:41:53 srv-dell cib: [5860]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-59.raw
Jun 3 18:41:53 srv-dell cib: [5860]: info: write_cib_contents: Wrote
version 0.165.0 of the CIB to disk (digest:
f266a832787bb93bd042aed5b00592d2)
Jun 3 18:41:53 srv-dell cib: [5860]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.jBzLdF (digest:
/var/lib/heartbeat/crm/cib.HCk2p3)
Jun 3 18:41:55 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:41:55 srv-dell cib: [5861]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-60.raw
Jun 3 18:41:55 srv-dell cib: [5861]: info: write_cib_contents: Wrote
version 0.166.0 of the CIB to disk (digest:
e9e295409b3549790060f576c19c8c49)
Jun 3 18:41:55 srv-dell cib: [5861]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.K18WMI (digest:
/var/lib/heartbeat/crm/cib.ZOHojb)
Jun 3 18:41:55 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:42:05 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-dell
Jun 3 18:42:05 srv-dell crmd: [5622]: info: do_lrm_invoke: Removing
resource cloneresource_cluster_fencing_null:1 from the LRM
Jun 3 18:42:05 srv-dell crmd: [5622]: info: send_direct_ack: ACK'ing
resource op cloneresource_cluster_fencing_null:1_delete_0 from
mgmtd-5623: lrm_invoke-lrmd-1244068925-11
Jun 3 18:42:05 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:42:05 srv-dell cib: [5862]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-61.raw
Jun 3 18:42:05 srv-dell cib: [5862]: info: write_cib_contents: Wrote
version 0.167.0 of the CIB to disk (digest:
dc0b0d978808a609ffdc5d92b43b484f)
Jun 3 18:42:05 srv-dell cib: [5862]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Fv2kc9 (digest:
/var/lib/heartbeat/crm/cib.dDENDX)
Jun 3 18:42:07 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-lenovo
Jun 3 18:42:07 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:42:07 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Child 5863 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 18:42:07 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 18:42:07 srv-dell openais[5611]: [MAIN ] Msg[39]
(dest=local:unknown, from=srv-lenovo:crmd.4910, remote=true,
size=922): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 18:42:09 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:42:09 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:42:09 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:42:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:42:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:42:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:42:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:42:09 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:42:09 srv-dell cib: [5864]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-62.raw
Jun 3 18:42:09 srv-dell cib: [5864]: info: write_cib_contents: Wrote
version 0.168.0 of the CIB to disk (digest:
4847deb704768418e08b606167627430)
Jun 3 18:42:09 srv-dell cib: [5864]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.lzpCqq (digest:
/var/lib/heartbeat/crm/cib.rjmaUn)
Jun 3 18:42:09 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:42:21 srv-dell pengine: [5865]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/root
Jun 3 18:42:21 srv-dell pengine: [5865]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:42:22 srv-dell crmd: [5866]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/root
Jun 3 18:42:22 srv-dell crmd: [5866]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:42:44 srv-dell mgmtd: [5623]: info: CIB replace: resources
Jun 3 18:42:55 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:0 from srv-dell
Jun 3 18:42:55 srv-dell crmd: [5622]: info: do_lrm_invoke: Removing
resource cloneresource_cluster_fencing_null:0 from the LRM
Jun 3 18:42:55 srv-dell crmd: [5622]: info: send_direct_ack: ACK'ing
resource op cloneresource_cluster_fencing_null:0_delete_0 from
mgmtd-5623: lrm_invoke-lrmd-1244068975-14
Jun 3 18:42:55 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:42:55 srv-dell cib: [5867]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-63.raw
Jun 3 18:42:55 srv-dell cib: [5867]: info: write_cib_contents: Wrote
version 0.169.0 of the CIB to disk (digest:
ed52f85ab6edb9f08c1913e71a943c6d)
Jun 3 18:42:55 srv-dell cib: [5867]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.eFGWfF (digest:
/var/lib/heartbeat/crm/cib.Rk9L2f)
Jun 3 18:42:57 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:0 from srv-lenovo
Jun 3 18:42:57 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:42:57 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Child 5868 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 18:42:57 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 18:42:57 srv-dell openais[5611]: [MAIN ] Msg[42]
(dest=local:unknown, from=srv-lenovo:crmd.4910, remote=true,
size=922): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 18:42:58 srv-dell cib: [5869]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-64.raw
Jun 3 18:42:58 srv-dell cib: [5869]: info: write_cib_contents: Wrote
version 0.170.0 of the CIB to disk (digest:
f40dcea00a22e5137e33cc78d367e8a6)
Jun 3 18:42:58 srv-dell cib: [5869]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.FaeCQ8 (digest:
/var/lib/heartbeat/crm/cib.UhPRWO)
Jun 3 18:42:59 srv-dell crmd: [5622]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244068979-25" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="5623_mgmtd" crm_host_to="srv-dell"
src="srv-dell" >
Jun 3 18:42:59 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-dell
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-5623" >
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive
id="cloneresource_cluster_fencing_null:1" />
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 18:42:59 srv-dell crmd: [5622]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 18:42:59 srv-dell crmd: [5622]: info: send_direct_ack: ACK'ing
resource op cloneresource_cluster_fencing_null:1_delete_0 from
mgmtd-5623: lrm_invoke-lrmd-1244068979-17
Jun 3 18:42:59 srv-dell cib: [5870]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-65.raw
Jun 3 18:42:59 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:42:59 srv-dell cib: [5870]: info: write_cib_contents: Wrote
version 0.171.0 of the CIB to disk (digest:
44d2a2b0c8d81fc6f1b114d9978d5ab7)
Jun 3 18:42:59 srv-dell cib: [5870]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.9CXrRV (digest:
/var/lib/heartbeat/crm/cib.3kuVAF)
Jun 3 18:43:01 srv-dell mgmtd: [5623]: info: Delete fail-count for
cloneresource_cluster_fencing_null:1 from srv-lenovo
Jun 3 18:43:01 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:43:01 srv-dell cib: [5871]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-66.raw
Jun 3 18:43:01 srv-dell cib: [5871]: info: write_cib_contents: Wrote
version 0.172.0 of the CIB to disk (digest:
978780100efc78a3300c82f8c8439c17)
Jun 3 18:43:01 srv-dell cib: [5871]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.2N7cR2 (digest:
/var/lib/heartbeat/crm/cib.Tbhy1Q)
Jun 3 18:43:02 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Child 5872 spawned to record non-fatal assertion
failure line 1299: dest > 0 && dest < SIZEOF(pcmk_children)
Jun 3 18:43:02 srv-dell openais[5611]: [crm ] ERROR:
route_ais_message: Invalid destination: 0
Jun 3 18:43:02 srv-dell openais[5611]: [MAIN ] Msg[46]
(dest=local:unknown, from=srv-lenovo:crmd.4910, remote=true,
size=922): <create_request_adv origin="send_direct_ack" t="crmd"
version="3.0.1" subt="request" refer
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell cib: [5873]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-67.raw
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:43:03 srv-dell cib: [5873]: info: write_cib_contents: Wrote
version 0.173.0 of the CIB to disk (digest:
df9d094e41b7d2b5ad56db92f4de086d)
Jun 3 18:43:03 srv-dell cib: [5873]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.aXpqOa (digest:
/var/lib/heartbeat/crm/cib.v2PWm3)
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:04 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:43:53 srv-dell crm_resource: [5876]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:43:53 srv-dell crm_resource: [5876]: info: Invoked:
crm_resource -R -H srv-dell
Jun 3 18:43:53 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:43:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:54 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:43:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:54 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:43:54 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:44:01 srv-dell crm_resource: [5878]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:44:01 srv-dell crm_resource: [5878]: info: Invoked:
crm_resource -R -H srv-lenovo
Jun 3 18:44:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:02 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:44:07 srv-dell crm_resource: [5880]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:44:07 srv-dell crm_resource: [5880]: info: Invoked: crm_resource -R
Jun 3 18:44:07 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:44:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:44:08 srv-dell mgmtd: [5623]: info: CIB query: cib
Jun 3 18:44:35 srv-dell openais[5611]: [SERV ] Unloading all openais components
Jun 3 18:44:35 srv-dell openais[5611]: [SERV ] Unloading slot 10:
openais cluster config database access v1.01
Jun 3 18:44:35 srv-dell openais[5611]: [SERV ] Unloading openais
component: openais_confdb v0 (16/10)
Jun 3 18:44:35 srv-dell openais[5611]: [SERV ] Unloading slot 9:
Pacemaker Cluster Manager
Jun 3 18:44:35 srv-dell openais[5611]: [SERV ] Unloading openais
component: pacemaker v0 (2/9)
Jun 3 18:44:35 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
Begining shutdown
Jun 3 18:44:35 srv-dell openais[5611]: [MAIN ] notice: stop_child:
Sent -15 to mgmtd: [5623]
Jun 3 18:44:35 srv-dell mgmtd: [5623]: info: mgmtd is shutting down
Jun 3 18:44:35 srv-dell mgmtd: [5623]: debug: [mgmtd] stopped
Jun 3 18:44:35 srv-dell haclient: on_event:None
Jun 3 18:44:36 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
mgmtd (pid=5623) confirmed dead
Jun 3 18:44:36 srv-dell openais[5611]: [MAIN ] notice: stop_child:
Sent -15 to crmd: [5622]
Jun 3 18:44:37 srv-dell crmd: [5622]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 18:44:37 srv-dell crmd: [5622]: info: crm_shutdown: Requesting shutdown
Jun 3 18:44:37 srv-dell crmd: [5622]: info: do_shutdown_req: Sending
shutdown request to DC: srv-lenovo
Jun 3 18:44:37 srv-dell attrd: [5620]: info: attrd_ais_dispatch:
Update relayed from srv-lenovo
Jun 3 18:44:37 srv-dell attrd: [5620]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 18:44:37 srv-dell attrd: [5620]: info: attrd_perform_update:
Sent update 10: shutdown=1244069210
Jun 3 18:44:50 srv-dell cib: [5918]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-68.raw
Jun 3 18:44:50 srv-dell cib: [5918]: info: write_cib_contents: Wrote
version 0.174.0 of the CIB to disk (digest:
c2b958215ca10f1c3920e601ea9e1135)
Jun 3 18:44:50 srv-dell cib: [5918]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.8TkrIL (digest:
/var/lib/heartbeat/crm/cib.o8GsYo)
Jun 3 18:44:56 srv-dell cib: [5931]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-69.raw
Jun 3 18:44:56 srv-dell cib: [5931]: info: write_cib_contents: Wrote
version 0.175.0 of the CIB to disk (digest:
cea8e330ffa0aaaa71f7877b0b4dda47)
Jun 3 18:44:56 srv-dell cib: [5931]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xdQYnv (digest:
/var/lib/heartbeat/crm/cib.CPoEQk)
Jun 3 18:45:00 srv-dell cib: [5940]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-70.raw
Jun 3 18:45:00 srv-dell cib: [5940]: info: write_cib_contents: Wrote
version 0.176.0 of the CIB to disk (digest:
d2f7a52ce6c04ab7340e0ab168065da6)
Jun 3 18:45:00 srv-dell cib: [5940]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.oe4SEY (digest:
/var/lib/heartbeat/crm/cib.C7uHvX)
Jun 3 18:45:05 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
Still waiting for crmd (pid=5622) to terminate...
Jun 3 18:45:08 srv-dell crmd: [5622]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:45:09 srv-dell crmd: [5622]: info: handle_request: Shutting down
Jun 3 18:45:09 srv-dell crmd: [5622]: info: do_state_transition:
State transition S_NOT_DC -> S_STOPPING [ input=I_STOP
cause=C_HA_MESSAGE origin=route_message ]
Jun 3 18:45:09 srv-dell crmd: [5622]: ERROR: verify_stopped: Resource
primitive_cluster_filesystem:0 was active at shutdown. You may ignore
this error if it is unmanaged.
Jun 3 18:45:09 srv-dell crmd: [5622]: info: do_lrm_control:
Disconnected from the LRM
Jun 3 18:45:09 srv-dell crmd: [5622]: info: do_ha_control:
Disconnected from OpenAIS
Jun 3 18:45:09 srv-dell crmd: [5622]: info: do_cib_control: Disconnecting CIB
Jun 3 18:45:09 srv-dell crmd: [5622]: info:
crmd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 18:45:09 srv-dell crmd: [5622]: info: do_exit: Performing
A_EXIT_0 - gracefully exiting the CRMd
Jun 3 18:45:09 srv-dell crmd: [5622]: info: free_mem: Dropping
I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ]
Jun 3 18:45:09 srv-dell crmd: [5622]: info: do_exit: [crmd] stopped (0)
Jun 3 18:45:09 srv-dell cib: [5618]: WARN: send_ipc_message: IPC
Channel to 5622 is not connected
Jun 3 18:45:09 srv-dell cib: [5618]: WARN: send_via_callback_channel:
Delivery of reply to client 5622/913c01cf-3bc1-4b3a-9112-45469854dc14
failed
Jun 3 18:45:09 srv-dell cib: [5618]: WARN: do_local_notify: A-Sync
reply to crmd failed: reply failed
Jun 3 18:45:09 srv-dell openais[5611]: [crm ] info: pcmk_ipc_exit:
Client crmd (conn=0x7f48700011b0, async-conn=0x7f48700011b0) left
Jun 3 18:45:09 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
crmd (pid=5622) confirmed dead
Jun 3 18:45:09 srv-dell openais[5611]: [MAIN ] notice: stop_child:
Sent -15 to pengine: [5621]
Jun 3 18:45:09 srv-dell pengine: [5621]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 18:45:10 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
pengine (pid=5621) confirmed dead
Jun 3 18:45:10 srv-dell openais[5611]: [MAIN ] notice: stop_child:
Sent -15 to attrd: [5620]
Jun 3 18:45:10 srv-dell attrd: [5620]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 18:45:10 srv-dell attrd: [5620]: info: attrd_shutdown: Exiting
Jun 3 18:45:10 srv-dell attrd: [5620]: info: main: Exiting...
Jun 3 18:45:10 srv-dell attrd: [5620]: info:
attrd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 18:45:10 srv-dell openais[5611]: [crm ] info: pcmk_ipc_exit:
Client attrd (conn=0x77ccc0, async-conn=0x77ccc0) left
Jun 3 18:45:11 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
attrd (pid=5620) confirmed dead
Jun 3 18:45:11 srv-dell openais[5611]: [MAIN ] notice: stop_child:
Sent -15 to lrmd: [5619]
Jun 3 18:45:11 srv-dell lrmd: [5619]: info: lrmd is shutting down
Jun 3 18:45:12 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
lrmd (pid=5619) confirmed dead
Jun 3 18:45:12 srv-dell openais[5611]: [MAIN ] notice: stop_child:
Sent -15 to cib: [5618]
Jun 3 18:45:12 srv-dell cib: [5618]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 18:45:12 srv-dell cib: [5618]: info: cib_shutdown: Disconnected 0 clients
Jun 3 18:45:12 srv-dell cib: [5618]: info: cib_process_disconnect:
All clients disconnected...
Jun 3 18:45:12 srv-dell cib: [5618]: info: initiate_exit: Sending
disconnect notification to 2 peers...
Jun 3 18:45:12 srv-dell cib: [5618]: info: cib_process_shutdown_req:
Shutdown ACK from srv-lenovo
Jun 3 18:45:12 srv-dell cib: [5618]: info: cib_ha_connection_destroy:
Heartbeat disconnection complete... exiting
Jun 3 18:45:12 srv-dell cib: [5618]: info: cib_ha_connection_destroy:
Exiting...
Jun 3 18:45:12 srv-dell cib: [5618]: info: cib_process_request:
Operation complete: op cib_shutdown_req for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.0.0): ok (rc=0)
Jun 3 18:45:12 srv-dell cib: [5618]: info: main: Done
Jun 3 18:45:12 srv-dell openais[5611]: [crm ] info: pcmk_ipc_exit:
Client cib (conn=0x77cef0, async-conn=0x77cef0) left
Jun 3 18:45:13 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
cib (pid=5618) confirmed dead
Jun 3 18:45:13 srv-dell openais[5611]: [MAIN ] notice: stop_child:
Sent -15 to stonithd: [5617]
Jun 3 18:45:13 srv-dell stonithd: [5617]: notice:
/usr/lib64/heartbeat/stonithd normally quit.
Jun 3 18:45:13 srv-dell openais[5611]: [crm ] info: pcmk_ipc_exit:
Client stonithd (conn=0x77d010, async-conn=0x77d010) left
Jun 3 18:45:14 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
stonithd (pid=5617) confirmed dead
Jun 3 18:45:14 srv-dell openais[5611]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000000002
(2)
Jun 3 18:45:14 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
Shutdown complete
Jun 3 18:45:14 srv-dell openais[5611]: [crm ] notice: pcmk_shutdown:
Forcing clean exit of OpenAIS
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] AIS Executive Service
RELEASE 'subrev 1152 version 0.80'
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] AIS Executive Service:
started and ready to provide service.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Token Timeout (5000
ms) retransmit timeout (490 ms)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] token hold (382 ms)
retransmits before loss (10 retrans)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] join (1000 ms)
send_join (45 ms) consensus (2500 ms) merge (200 ms)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] seqno unchanged const
(30 rotations) Maximum network MTU 1500
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] send threads (0 threads)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] RRP token expired
timeout (490 ms)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] RRP token problem
counter (2000 ms)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] RRP threshold (10 problem count)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] RRP mode set to none.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] heartbeat_failures_allowed (0)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] max_network_delay (50 ms)
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] HeartBeat is Disabled.
To enable set heartbeat_failures_allowed > 0
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] The network interface
[192.168.2.220] is now up.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Created or loaded
sequence id 236.192.168.2.220 for this ring.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] entering GATHER state from 15.
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info:
process_ais_conf: Reading configure
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info:
config_find_next: Processing additional logging options...
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: get_config_opt:
Found 'off' for option: debug
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: get_config_opt:
Found 'yes' for option: to_syslog
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: get_config_opt:
Found 'daemon' for option: syslog_facility
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info:
config_find_next: Processing additional service options...
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_logd
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_mgmtd
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info:
pcmk_plugin_init: CRM: Initialized
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] Logging: Initialized
pcmk_plugin_init
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info:
pcmk_plugin_init: Service: 9
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info:
pcmk_plugin_init: Local node id: 1543678144
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info:
pcmk_plugin_init: Local hostname: srv-dell
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: update_member:
Creating entry for node 1543678144 born on 0
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: update_member:
0x772220 Node 1543678144 now known as srv-dell (was: (null))
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: update_member:
Node srv-dell now has 1 quorum votes (was 0)
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: update_member:
Node 1543678144/srv-dell is now: member
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: spawn_child:
Forked child 6002 for process stonithd
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: spawn_child:
Forked child 6003 for process cib
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: spawn_child:
Forked child 6004 for process lrmd
Jun 3 18:45:15 srv-dell stonithd: [6002]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:45:15 srv-dell stonithd: [6002]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 18:45:15 srv-dell stonithd: [6002]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 18:45:15 srv-dell lrmd: [6004]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:45:15 srv-dell lrmd: [6004]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Jun 3 18:45:15 srv-dell cib: [6003]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:45:15 srv-dell cib: [6003]: WARN: Initializing connection to
logging daemon failed. Logging daemon may not be running
Jun 3 18:45:15 srv-dell cib: [6003]: info: G_main_add_TriggerHandler:
Added signal manual handler
Jun 3 18:45:15 srv-dell cib: [6003]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 18:45:15 srv-dell cib: [6003]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Jun 3 18:45:15 srv-dell stonithd: [6002]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:45:15 srv-dell stonithd: [6002]: info: init_ais_connection:
Connection to our AIS plugin (9) failed: Try again (6)
Jun 3 18:45:15 srv-dell lrmd: [6004]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 18:45:15 srv-dell attrd: [6005]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:45:15 srv-dell pengine: [6006]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:45:15 srv-dell crmd: [6007]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 18:45:15 srv-dell pengine: [6006]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 18:45:15 srv-dell attrd: [6005]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:45:15 srv-dell mgmtd: [6008]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:45:15 srv-dell mgmtd: [6008]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 3 18:45:15 srv-dell cib: [6003]: info: startCib: CIB
Initialization completed successfully
Jun 3 18:45:15 srv-dell crmd: [6007]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 18:45:15 srv-dell lrmd: [6004]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jun 3 18:45:15 srv-dell mgmtd: [6008]: debug: Enabling coredumps
Jun 3 18:45:15 srv-dell attrd: [6005]: info: main: Starting up....
Jun 3 18:45:15 srv-dell cib: [6003]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:45:15 srv-dell crmd: [6007]: info: main: CRM Hg Version:
0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Jun 3 18:45:15 srv-dell lrmd: [6004]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jun 3 18:45:15 srv-dell lrmd: [6004]: info: Started.
Jun 3 18:45:15 srv-dell pengine: [6006]: info: main: Starting pengine
Jun 3 18:45:15 srv-dell mgmtd: [6008]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 18:45:15 srv-dell attrd: [6005]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:45:15 srv-dell cib: [6003]: info: init_ais_connection: AIS
connection established
Jun 3 18:45:15 srv-dell crmd: [6007]: info: crmd_init: Starting crmd
Jun 3 18:45:15 srv-dell mgmtd: [6008]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 18:45:15 srv-dell crmd: [6007]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 18:45:15 srv-dell mgmtd: [6008]: info: init_crm
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: spawn_child:
Forked child 6005 for process attrd
Jun 3 18:45:15 srv-dell mgmtd: [6008]: info: login to cib: 0, ret:-10
Jun 3 18:45:15 srv-dell attrd: [6005]: info: init_ais_connection: AIS
connection established
Jun 3 18:45:15 srv-dell cib: [6003]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 18:45:15 srv-dell cib: [6003]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:45:15 srv-dell cib: [6003]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: spawn_child:
Forked child 6006 for process pengine
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: spawn_child:
Forked child 6007 for process crmd
Jun 3 18:45:15 srv-dell cib: [6003]: info: cib_init: Starting cib mainloop
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: spawn_child:
Forked child 6008 for process mgmtd
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info: pcmk_startup:
CRM: Initialized
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] Service initialized
'Pacemaker Cluster Manager'
Jun 3 18:45:15 srv-dell cib: [6003]: info: ais_dispatch: Membership
240: quorum still lost
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais extended virtual synchrony service'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais cluster membership service B.01.01'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais availability management framework B.01.01'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais checkpoint service B.01.01'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais event service B.01.01'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais distributed locking service B.01.01'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais message service B.01.01'
Jun 3 18:45:15 srv-dell cib: [6003]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=240
proc=00000000000000000000000000053312 (new)
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais configuration service'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais cluster closed process group service v1.01'
Jun 3 18:45:15 srv-dell openais[5996]: [SERV ] Service initialized
'openais cluster config database access v1.01'
Jun 3 18:45:15 srv-dell openais[5996]: [SYNC ] Not using a virtual
synchrony filter.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Saving state aru 0
high seq received 0
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Storing new sequence
id for ring f0
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] entering COMMIT state.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] entering RECOVERY state.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] previous ring seq 236
rep 192.168.2.220
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] aru 0 high delivered 0
received flag 1
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] Sending initial ORF token
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] New Configuration:
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] Members Left:
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] Members Joined:
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 240: memb=0,
new=0, lost=0
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] New Configuration:
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:45:15 srv-dell attrd: [6005]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] Members Left:
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] Members Joined:
Jun 3 18:45:15 srv-dell attrd: [6005]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:45:15 srv-dell attrd: [6005]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 240: memb=1, new=1,
lost=0
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: NEW: srv-dell 1543678144
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 18:45:15 srv-dell openais[5996]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000053312
(340754)
Jun 3 18:45:15 srv-dell openais[5996]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 18:45:15 srv-dell openais[5996]: [TOTEM] entering OPERATIONAL state.
Jun 3 18:45:15 srv-dell openais[5996]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info: pcmk_ipc:
Recorded connection 0x77da60 for cib/6003
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info: pcmk_ipc:
Sending membership update 240 to cib
Jun 3 18:45:15 srv-dell openais[5996]: [crm ] info: pcmk_ipc:
Recorded connection 0x77dc90 for attrd/6005
Jun 3 18:45:15 srv-dell cib: [6010]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-71.raw
Jun 3 18:45:15 srv-dell cib: [6010]: info: write_cib_contents: Wrote
version 0.176.0 of the CIB to disk (digest:
4004f8f3faff43d449896b1629411b82)
Jun 3 18:45:15 srv-dell cib: [6010]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.7gAFXb (digest:
/var/lib/heartbeat/crm/cib.9CC4TG)
Jun 3 18:45:16 srv-dell stonithd: [6002]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:45:16 srv-dell stonithd: [6002]: info: init_ais_connection:
AIS connection established
Jun 3 18:45:16 srv-dell stonithd: [6002]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 18:45:16 srv-dell stonithd: [6002]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:45:16 srv-dell stonithd: [6002]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info: pcmk_ipc:
Recorded connection 0x77cd50 for stonithd/6002
Jun 3 18:45:16 srv-dell stonithd: [6002]: notice:
/usr/lib64/heartbeat/stonithd start up successfully.
Jun 3 18:45:16 srv-dell stonithd: [6002]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 18:45:16 srv-dell crmd: [6007]: info: do_cib_control: CIB
connection established
Jun 3 18:45:16 srv-dell crmd: [6007]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] entering GATHER state from 11.
Jun 3 18:45:16 srv-dell crmd: [6007]: info: init_ais_connection: AIS
connection established
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info: pcmk_ipc:
Recorded connection 0x77f220 for crmd/6007
Jun 3 18:45:16 srv-dell crmd: [6007]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 18:45:16 srv-dell crmd: [6007]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 18:45:16 srv-dell crmd: [6007]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 18:45:16 srv-dell crmd: [6007]: info: do_ha_control: Connected
to the cluster
Jun 3 18:45:16 srv-dell crmd: [6007]: info: do_started: Delaying
start, CCM (0000000000100000) not connected
Jun 3 18:45:16 srv-dell crmd: [6007]: info: crmd_init: Starting crmd's mainloop
Jun 3 18:45:16 srv-dell crmd: [6007]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 18:45:16 srv-dell cib: [6003]: info: ais_dispatch: Membership
244: quorum still lost
Jun 3 18:45:16 srv-dell cib: [6003]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 18:45:16 srv-dell cib: [6003]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=244 proc=00000000000000000000000000000000
Jun 3 18:45:16 srv-dell cib: [6003]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 18:45:16 srv-dell crmd: [6007]: info: ais_dispatch: Membership
240: quorum still lost
Jun 3 18:45:16 srv-dell crmd: [6007]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=240
proc=00000000000000000000000000053312 (new)
Jun 3 18:45:16 srv-dell cib: [6003]: info: cib_process_diff: Diff
0.176.5 -> 0.176.6 not applied to 0.176.0: current "num_updates" is
less than required
Jun 3 18:45:16 srv-dell cib: [6003]: info: cib_server_process_diff:
Requesting re-sync from peer
Jun 3 18:45:16 srv-dell cib: [6003]: WARN: cib_diff_notify:
Local-only Change (client:crmd, call: 348): 0.0.0 (Application of an
update diff failed, requesting a full refresh)
Jun 3 18:45:16 srv-dell crmd: [6007]: info: do_started: The local CRM
is operational
Jun 3 18:45:16 srv-dell cib: [6003]: info: ais_dispatch: Membership
244: quorum still lost
Jun 3 18:45:16 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Jun 3 18:45:16 srv-dell cib: [6003]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=236 seen=244 proc=00000000000000000000000000053312
(new)
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] Saving state aru a
high seq received a
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info: pcmk_ipc:
Sending membership update 240 to crmd
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] Storing new sequence
id for ring f4
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] entering COMMIT state.
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] entering RECOVERY state.
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] previous ring seq 236
rep 192.168.2.120
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] aru 214 high delivered
214 received flag 1
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] previous ring seq 240
rep 192.168.2.220
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] aru a high delivered a
received flag 1
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] New Configuration:
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] Members Left:
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] Members Joined:
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 244: memb=1,
new=0, lost=0
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] New Configuration:
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] Members Left:
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] Members Joined:
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 244: memb=2, new=1,
lost=0
Jun 3 18:45:16 srv-dell openais[5996]: [MAIN ] info: update_member:
Creating entry for node 2013440192 born on 244
Jun 3 18:45:16 srv-dell openais[5996]: [MAIN ] info: update_member:
Node 2013440192/unknown is now: member
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: NEW: .pending. 2013440192
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: MEMB: .pending. 2013440192
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info:
send_member_notification: Sending membership update 244 to 2 children
Jun 3 18:45:16 srv-dell openais[5996]: [MAIN ] info: update_member:
0x772220 Node 1543678144 ((null)) born on: 244
Jun 3 18:45:16 srv-dell openais[5996]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 18:45:16 srv-dell openais[5996]: [TOTEM] entering OPERATIONAL state.
Jun 3 18:45:16 srv-dell openais[5996]: [MAIN ] info: update_member:
0x77d750 Node 2013440192 (srv-lenovo) born on: 236
Jun 3 18:45:16 srv-dell openais[5996]: [MAIN ] info: update_member:
0x77d750 Node 2013440192 now known as srv-lenovo (was: (null))
Jun 3 18:45:16 srv-dell openais[5996]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 3 18:45:16 srv-dell openais[5996]: [MAIN ] info: update_member:
Node srv-lenovo now has 1 quorum votes (was 0)
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info:
send_member_notification: Sending membership update 244 to 2 children
Jun 3 18:45:16 srv-dell openais[5996]: [crm ] info:
update_expected_votes: Expected quorum votes 1024 -> 2
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 18:45:16 srv-dell openais[5996]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 18:45:16 srv-dell cib: [6003]: WARN: cib_server_process_diff:
Not applying diff 0.176.6 -> 0.176.7 (sync in progress)
Jun 3 18:45:16 srv-dell cib: [6003]: info: cib_replace_notify:
Replaced: 0.176.0 -> 0.176.7 from srv-lenovo
Jun 3 18:45:16 srv-dell cib: [6014]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-72.raw
Jun 3 18:45:16 srv-dell cib: [6014]: info: write_cib_contents: Wrote
version 0.176.0 of the CIB to disk (digest:
5c8ff3058e60984b9f13fd4b1e9758a4)
Jun 3 18:45:16 srv-dell cib: [6014]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.EKWdcl (digest:
/var/lib/heartbeat/crm/cib.UDGSrS)
Jun 3 18:45:17 srv-dell mgmtd: [6008]: debug: main: run the loop...
Jun 3 18:45:17 srv-dell mgmtd: [6008]: info: Started.
Jun 3 18:45:17 srv-dell crmd: [6007]: info: ais_dispatch: Membership
244: quorum still lost
Jun 3 18:45:17 srv-dell crmd: [6007]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 18:45:17 srv-dell crmd: [6007]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=244 proc=00000000000000000000000000000000
Jun 3 18:45:17 srv-dell crmd: [6007]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 18:45:17 srv-dell crmd: [6007]: info: ais_dispatch: Membership
244: quorum still lost
Jun 3 18:45:17 srv-dell crmd: [6007]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=236 seen=244 proc=00000000000000000000000000053312
(new)
Jun 3 18:45:17 srv-dell crmd: [6007]: notice: ais_dispatch:
Membership 244: quorum aquired
Jun 3 18:45:17 srv-dell crmd: [6007]: info: update_dc: Set DC to
srv-lenovo (3.0.1)
Jun 3 18:45:17 srv-dell crmd: [6007]: info: update_attrd: Connecting
to attrd...
Jun 3 18:45:17 srv-dell crmd: [6007]: info: update_attrd: Updating
terminate=<none> via attrd for srv-dell
Jun 3 18:45:17 srv-dell crmd: [6007]: info: update_attrd: Updating
shutdown=<none> via attrd for srv-dell
Jun 3 18:45:17 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 3 18:45:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:17 srv-dell crmd: [6007]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/transient_attributes": ok
(rc=0)
Jun 3 18:45:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:17 srv-dell crmd: [6007]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 18:45:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:17 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:45:18 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=4:54:7:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 18:45:18 srv-dell lrmd: [6004]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 18:45:18 srv-dell lrmd: [6004]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 18:45:18 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=5:54:7:6712d200-ef04-46ab-920d-6e4813d1a596
op=cloneresource_cluster_fencing_null:0_monitor_0 )
Jun 3 18:45:18 srv-dell lrmd: [6004]: info:
rsc:cloneresource_cluster_fencing_null:0: monitor
Jun 3 18:45:18 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_monitor_0 (call=3,
rc=7, cib-update=8, confirmed=true) complete not running
Jun 3 18:45:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:18 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=2, rc=7,
cib-update=9, confirmed=true) complete not running
Jun 3 18:45:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:18 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:45:20 srv-dell attrd: [6005]: info: main: Sending full refresh
Jun 3 18:45:20 srv-dell attrd: [6005]: info: main: Starting mainloop...
Jun 3 18:45:20 srv-dell attrd: [6005]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 18:45:20 srv-dell attrd: [6005]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 18:45:20 srv-dell attrd: [6005]: info: find_hash_entry:
Creating hash entry for terminate
Jun 3 18:45:20 srv-dell attrd: [6005]: info: find_hash_entry:
Creating hash entry for shutdown
Jun 3 18:45:20 srv-dell attrd: [6005]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 18:45:20 srv-dell attrd: [6005]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 18:45:20 srv-dell attrd: [6005]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 18:45:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:33 srv-dell cib: [6036]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-73.raw
Jun 3 18:45:33 srv-dell cib: [6036]: info: write_cib_contents: Wrote
version 0.177.0 of the CIB to disk (digest:
4221fbd3bfd7613850aaed48f57d842f)
Jun 3 18:45:33 srv-dell cib: [6036]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.37Q2qz (digest:
/var/lib/heartbeat/crm/cib.FMHJ2G)
Jun 3 18:45:33 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:45:33 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:34 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:34 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:45:44 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=4:57:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitiveresource_cluster_ip_start_0 )
Jun 3 18:45:44 srv-dell lrmd: [6004]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 18:45:44 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=6:57:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=cloneresource_cluster_fencing_null:0_start_0 )
Jun 3 18:45:44 srv-dell lrmd: [6004]: info:
rsc:cloneresource_cluster_fencing_null:0: start
Jun 3 18:45:44 srv-dell lrmd: [6038]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 18:45:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:44 srv-dell stonithd: [6002]: info:
cloneresource_cluster_fencing_null:0 stonith resource started
Jun 3 18:45:44 srv-dell lrmd: [6004]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 18:45:44 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_start_0 (call=5, rc=0,
cib-update=13, confirmed=true) complete ok
Jun 3 18:45:44 srv-dell cib: [6040]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-74.raw
Jun 3 18:45:44 srv-dell cib: [6040]: info: write_cib_contents: Wrote
version 0.178.0 of the CIB to disk (digest:
9577c8ae6074e9e93601cf4a48ed2bad)
Jun 3 18:45:44 srv-dell cib: [6040]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.k8F252 (digest:
/var/lib/heartbeat/crm/cib.UfObPy)
Jun 3 18:45:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:44 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 18:45:44 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_18:45:44 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 18:45:44 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 18:45:44 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_18:45:44 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 18:45:44 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 18:45:44 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_18:45:44 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 18:45:44 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:45:44 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=4, rc=0,
cib-update=14, confirmed=true) complete ok
Jun 3 18:45:44 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=5:57:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 18:45:44 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:44 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:45:45 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=6, rc=0,
cib-update=15, confirmed=false) complete ok
Jun 3 18:45:45 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:45 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:45:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:45:50 srv-dell cib: [6181]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-75.raw
Jun 3 18:45:50 srv-dell cib: [6181]: info: write_cib_contents: Wrote
version 0.179.0 of the CIB to disk (digest:
c4d3ef1f8ce626c284026fe59eefcb5a)
Jun 3 18:45:50 srv-dell cib: [6181]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.l9dMeW (digest:
/var/lib/heartbeat/crm/cib.GgNulG)
Jun 3 18:45:51 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:02 srv-dell crmd: [6007]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:46:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:02 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:02 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:02 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:15 srv-dell crmd: [6007]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244069308-1" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="4911_mgmtd" crm_host_to="srv-dell"
src="srv-lenovo" seq="66" >
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-4911" >
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive
id="cloneresource_cluster_fencing_null:1" />
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 18:46:15 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 18:46:15 srv-dell crmd: [6007]: info: send_direct_ack: ACK'ing
resource op cloneresource_cluster_fencing_null:1_delete_0 from
mgmtd-4911: lrm_invoke-lrmd-1244069175-6
Jun 3 18:46:15 srv-dell crmd: [6007]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:46:15 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:15 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:15 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:16 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:16 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:17 srv-dell crmd: [6007]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:46:17 srv-dell cib: [6272]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-76.raw
Jun 3 18:46:17 srv-dell cib: [6272]: info: write_cib_contents: Wrote
version 0.180.0 of the CIB to disk (digest:
6697ec86ee59e330053775900aa9f1a2)
Jun 3 18:46:17 srv-dell cib: [6272]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xsMwRs (digest:
/var/lib/heartbeat/crm/cib.5ELZY8)
Jun 3 18:46:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:17 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:17 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:18 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:18 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:19 srv-dell cib: [6273]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-77.raw
Jun 3 18:46:19 srv-dell cib: [6273]: info: write_cib_contents: Wrote
version 0.181.0 of the CIB to disk (digest:
40865ff4e1d164b98f4e0365ede4f61b)
Jun 3 18:46:19 srv-dell cib: [6273]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.U4ioS1 (digest:
/var/lib/heartbeat/crm/cib.RrgAgN)
Jun 3 18:46:20 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:41 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:41 srv-dell cib: [6364]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-78.raw
Jun 3 18:46:41 srv-dell cib: [6364]: info: write_cib_contents: Wrote
version 0.182.0 of the CIB to disk (digest:
5722b13a9bc070410a247732c035b056)
Jun 3 18:46:41 srv-dell cib: [6364]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.FDiLs6 (digest:
/var/lib/heartbeat/crm/cib.VzomtC)
Jun 3 18:46:41 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:58 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:46:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:46:59 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:19 srv-dell crmd: [6007]: info: send_direct_ack: ACK'ing
resource op primitiveresource_cluster_ip_monitor_5000 from
1:70:0:6712d200-ef04-46ab-920d-6e4813d1a596:
lrm_invoke-lrmd-1244069239-7
Jun 3 18:47:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:19 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=6, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 18:47:19 srv-dell cib: [6491]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-79.raw
Jun 3 18:47:19 srv-dell cib: [6491]: info: write_cib_contents: Wrote
version 0.183.0 of the CIB to disk (digest:
b2426174dcaf574f52667054a40008d7)
Jun 3 18:47:19 srv-dell cib: [6491]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ID4wSK (digest:
/var/lib/heartbeat/crm/cib.TybMdC)
Jun 3 18:47:19 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:19 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:27 srv-dell crmd: [6007]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:47:27 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:27 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:47 srv-dell crmd: [6007]: ERROR: do_lrm_invoke: Not
creating resource for a delete event: (null)
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <create_request_adv origin="delete_lrm_rsc"
t="crmd" version="3.0.1" subt="request"
reference="lrm_delete-mgmtd-1244069400-7" crm_task="lrm_delete"
crm_sys_to="lrmd" crm_sys_from="4911_mgmtd" crm_host_to="srv-dell"
src="srv-lenovo" seq="74" >
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <crm_xml >
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <rsc_op transition-key="mgmtd-4911" >
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <primitive
id="primitive_cluster_filesystem:1" />
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input <attributes crm_feature_set="3.0.1" />
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input </rsc_op>
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input </crm_xml>
Jun 3 18:47:47 srv-dell crmd: [6007]: WARN: log_data_element:
do_lrm_invoke: bad input </create_request_adv>
Jun 3 18:47:47 srv-dell crmd: [6007]: info: send_direct_ack: ACK'ing
resource op primitive_cluster_filesystem:1_delete_0 from mgmtd-4911:
lrm_invoke-lrmd-1244069267-8
Jun 3 18:47:47 srv-dell crmd: [6007]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:47:49 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:49 srv-dell crmd: [6007]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 3 18:47:49 srv-dell cib: [6492]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-80.raw
Jun 3 18:47:49 srv-dell cib: [6492]: info: write_cib_contents: Wrote
version 0.184.0 of the CIB to disk (digest:
632010cb2a57ac4586fc0f97eb328d39)
Jun 3 18:47:49 srv-dell cib: [6492]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Tgldhb (digest:
/var/lib/heartbeat/crm/cib.DDUdz6)
Jun 3 18:47:50 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:50 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:50 srv-dell cib: [6493]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-81.raw
Jun 3 18:47:50 srv-dell cib: [6493]: info: write_cib_contents: Wrote
version 0.185.0 of the CIB to disk (digest:
95f019ec1d29e431c8be868d49479106)
Jun 3 18:47:50 srv-dell cib: [6493]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.hrPR0y (digest:
/var/lib/heartbeat/crm/cib.0Gafmv)
Jun 3 18:47:50 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 18:47:50 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:50 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:56 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:56 srv-dell cib: [6494]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-82.raw
Jun 3 18:47:56 srv-dell cib: [6494]: info: write_cib_contents: Wrote
version 0.186.0 of the CIB to disk (digest:
9faedbcba482e4d2ed4822840ecb11e6)
Jun 3 18:47:56 srv-dell cib: [6494]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.8W74wa (digest:
/var/lib/heartbeat/crm/cib.QcNjFk)
Jun 3 18:47:56 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:56 srv-dell mgmtd: [6008]: ERROR: native_add_running:
Resource ocf::IPaddr:primitiveresource_cluster_ip appears to be active
on 2 nodes.
Jun 3 18:47:56 srv-dell mgmtd: [6008]: WARN: See
http://clusterlabs.org/wiki/FAQ#Resource_is_Too_Active for more
information.
Jun 3 18:47:57 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=6:75:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 18:47:57 srv-dell lrmd: [6004]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 18:47:57 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 18:47:57 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 18:47:57 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 18:47:57 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_18:47:57 INFO:
ifconfig eth1:0 down
Jun 3 18:47:57 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=7, rc=0,
cib-update=23, confirmed=true) complete ok
Jun 3 18:47:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:57 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:57 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:47:58 srv-dell haclient: on_event:evt:cib_changed
Jun 3 18:47:58 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 18:55:15 srv-dell cib: [6003]: info: cib_stats: Processed 138
operations (289.00us average, 0% utilization) in the last 10min
Jun 3 19:00:10 srv-dell kernel: RPC: Registered udp transport module.
Jun 3 19:00:10 srv-dell kernel: RPC: Registered tcp transport module.
Jun 3 19:00:27 srv-dell kernel: st: Version 20080504, fixed bufsize
32768, s/g segs 256
Jun 3 19:00:45 srv-dell kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jun 3 19:06:15 srv-dell kernel: No iBFT detected.
Jun 3 19:06:30 srv-dell kernel: scsi3 : iSCSI Initiator over TCP/IP
Jun 3 19:06:30 srv-dell kernel: scsi 3:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] 41929650 512-byte
hardware sectors: (21.4GB/19.9GiB)
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:06:30 srv-dell kernel: sdb: sdb1
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: [sdb] Attached SCSI disk
Jun 3 19:06:30 srv-dell kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Jun 3 19:06:30 srv-dell iscsid: connection1:0 is operational now
Jun 3 19:07:58 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 19:07:58 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 19:07:58 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 19:07:58 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 19:08:13 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 19:08:13 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 19:08:13 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 19:08:13 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 19:11:29 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 19:11:29 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 19:11:29 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 19:11:29 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 19:12:13 srv-dell kernel: scsi4 : iSCSI Initiator over TCP/IP
Jun 3 19:12:13 srv-dell kernel: scsi 4:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] Write Protect is off
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] Write Protect is off
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:12:13 srv-dell kernel: sdb: sdb1
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: [sdb] Attached SCSI disk
Jun 3 19:12:13 srv-dell kernel: sd 4:0:0:0: Attached scsi generic sg3 type 0
Jun 3 19:12:14 srv-dell iscsid: connection2:0 is operational now
Jun 3 19:13:19 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 19:13:19 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 19:13:19 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 19:13:19 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 19:13:23 srv-dell kernel: connection2:0: detected conn error (1011)
Jun 3 19:13:24 srv-dell iscsid: Kernel reported iSCSI connection 2:0
error (1011) state (3)
Jun 3 19:13:27 srv-dell iscsid: conn 0 login rejected: initiator
error - target not found (02/03)
Jun 3 19:13:30 srv-dell iscsid: conn 0 login rejected: initiator
error - target not found (02/03)
Jun 3 19:13:33 srv-dell iscsid: conn 0 login rejected: initiator
error - target not found (02/03)
Jun 3 19:13:36 srv-dell iscsid: conn 0 login rejected: initiator
error - target not found (02/03)
Jun 3 19:13:39 srv-dell iscsid: conn 0 login rejected: initiator
error - target not found (02/03)
Jun 3 19:13:46 srv-dell kernel: scsi5 : iSCSI Initiator over TCP/IP
Jun 3 19:13:47 srv-dell iscsid: conn 0 login rejected: initiator
error - target not found (02/03)
Jun 3 19:13:52 srv-dell kernel: scsi6 : iSCSI Initiator over TCP/IP
Jun 3 19:13:52 srv-dell iscsid: conn 0 login rejected: initiator
error - target not found (02/03)
Jun 3 19:14:11 srv-dell kernel: scsi7 : iSCSI Initiator over TCP/IP
Jun 3 19:14:12 srv-dell kernel: scsi 7:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] Write Protect is off
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] Write Protect is off
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:14:12 srv-dell kernel: sdb: sdb1
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: [sdb] Attached SCSI disk
Jun 3 19:14:12 srv-dell kernel: sd 7:0:0:0: Attached scsi generic sg3 type 0
Jun 3 19:14:12 srv-dell iscsid: connection5:0 is operational now
Jun 3 19:14:42 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 19:14:42 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 19:14:42 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 19:14:42 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 19:17:39 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 19:17:39 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 19:17:39 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 19:17:39 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 19:18:04 srv-dell kernel: scsi8 : iSCSI Initiator over TCP/IP
Jun 3 19:18:04 srv-dell kernel: scsi 8:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] Write Protect is off
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] Write Protect is off
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 3 19:18:04 srv-dell kernel: sdb: unknown partition table
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: [sdb] Attached SCSI disk
Jun 3 19:18:04 srv-dell kernel: sd 8:0:0:0: Attached scsi generic sg3 type 0
Jun 3 19:18:04 srv-dell kernel: kjournald starting. Commit interval 5 seconds
Jun 3 19:18:04 srv-dell kernel: EXT3 FS on sdb, internal journal
Jun 3 19:18:04 srv-dell kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Jun 3 19:18:04 srv-dell hald: mounted /dev/sdb on behalf of uid 0
Jun 3 19:18:04 srv-dell iscsid: connection6:0 is operational now
Jun 3 19:18:21 srv-dell hald: unmounted /dev/sdb from '/media/disk-1'
on behalf of uid 0
Jun 3 19:23:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:23:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:23:50 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 19:23:50 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=7:78:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitiveresource_cluster_ip_start_0 )
Jun 3 19:23:50 srv-dell lrmd: [6004]: info:
rsc:primitiveresource_cluster_ip: start
Jun 3 19:23:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:23:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:23:50 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 19:23:50 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_19:23:50 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 3 19:23:50 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 19:23:50 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_19:23:50 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 3 19:23:50 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 3 19:23:51 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/03_19:23:51 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 3 19:23:51 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=8, rc=0,
cib-update=24, confirmed=true) complete ok
Jun 3 19:23:51 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=8:78:0:6712d200-ef04-46ab-920d-6e4813d1a596
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 3 19:23:51 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:23:51 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 19:23:52 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=9, rc=0,
cib-update=25, confirmed=false) complete ok
Jun 3 19:23:52 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:23:52 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 19:23:55 srv-dell cib: [6003]: info: cib_process_shutdown_req:
Shutdown REQ from srv-lenovo
Jun 3 19:23:55 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_shutdown_req for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.186.11): ok (rc=0)
Jun 3 19:23:57 srv-dell cib: [6003]: notice: ais_dispatch: Membership
244: quorum aquired
Jun 3 19:23:57 srv-dell crmd: [6007]: info: ais_dispatch: Membership
244: quorum retained
Jun 3 19:23:57 srv-dell crmd: [6007]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=236 seen=244 proc=00000000000000000000000000000002 (new)
Jun 3 19:23:57 srv-dell cib: [6003]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=236 seen=244 proc=00000000000000000000000000000002 (new)
Jun 3 19:23:57 srv-dell openais[5996]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000000002
(2)
Jun 3 19:23:57 srv-dell openais[5996]: [crm ] info:
send_member_notification: Sending membership update 244 to 2 children
Jun 3 19:24:02 srv-dell openais[5996]: [TOTEM] The token was lost in
the OPERATIONAL state.
Jun 3 19:24:02 srv-dell openais[5996]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 19:24:02 srv-dell openais[5996]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 19:24:02 srv-dell openais[5996]: [TOTEM] entering GATHER state from 2.
Jun 3 19:24:05 srv-dell cib: [6003]: notice: ais_dispatch: Membership
248: quorum lost
Jun 3 19:24:05 srv-dell cib: [6003]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=236 seen=244 proc=00000000000000000000000000000002
Jun 3 19:24:05 srv-dell crmd: [6007]: notice: ais_dispatch:
Membership 248: quorum lost
Jun 3 19:24:05 srv-dell crmd: [6007]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=236 seen=244 proc=00000000000000000000000000000002
Jun 3 19:24:05 srv-dell crmd: [6007]: WARN: check_dead_member: Our DC
node (srv-lenovo) left the cluster
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=check_dead_member ]
Jun 3 19:24:05 srv-dell crmd: [6007]: info: update_dc: Unset DC srv-lenovo
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] entering GATHER state from 0.
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_te_control:
Registering TE UUID: 434dde7d-044a-4525-90a4-cce17f9131fc
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 19:24:05 srv-dell crmd: [6007]: WARN:
cib_client_add_notify_callback: Callback already present
Jun 3 19:24:05 srv-dell crmd: [6007]: info: set_graph_functions:
Setting custom graph functions
Jun 3 19:24:05 srv-dell crmd: [6007]: info: unpack_graph: Unpacked
transition -1: 0 actions in 0 synapses
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_dc_takeover: Taking
over DC status for this partition
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] Saving state aru 11e
high seq received 11e
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] Storing new sequence
id for ring f8
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] entering COMMIT state.
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] entering RECOVERY state.
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] previous ring seq 244
rep 192.168.2.120
Jun 3 19:24:05 srv-dell cib: [6003]: info: cib_process_readwrite: We
are now in R/W mode
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] aru 11e high delivered
11e received flag 1
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] Sending initial ORF token
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] New Configuration:
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] Members Left:
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] Members Joined:
Jun 3 19:24:05 srv-dell openais[5996]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 248: memb=1,
new=0, lost=1
Jun 3 19:24:05 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 19:24:05 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: lost: srv-lenovo 2013440192
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] New Configuration:
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] Members Left:
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] Members Joined:
Jun 3 19:24:05 srv-dell openais[5996]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 248: memb=1, new=0,
lost=0
Jun 3 19:24:05 srv-dell openais[5996]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 19:24:05 srv-dell openais[5996]: [crm ] info:
ais_mark_unseen_peer_dead: Node srv-lenovo was not seen in the
previous transition
Jun 3 19:24:05 srv-dell openais[5996]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: lost
Jun 3 19:24:05 srv-dell openais[5996]: [crm ] info:
send_member_notification: Sending membership update 248 to 2 children
Jun 3 19:24:05 srv-dell openais[5996]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 19:24:05 srv-dell openais[5996]: [TOTEM] entering OPERATIONAL state.
Jun 3 19:24:05 srv-dell openais[5996]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 19:24:05 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_master for section 'all'
(origin=local/crmd/27, version=0.186.11): ok (rc=0)
Jun 3 19:24:05 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/28, version=0.186.11): ok (rc=0)
Jun 3 19:24:05 srv-dell crmd: [6007]: info: join_make_offer: Making
join offers based on membership 248
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_dc_join_offer_all:
join-1: Waiting on 1 outstanding join acks
Jun 3 19:24:05 srv-dell crmd: [6007]: info: ais_dispatch: Membership
248: quorum still lost
Jun 3 19:24:05 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/30, version=0.186.11): ok (rc=0)
Jun 3 19:24:05 srv-dell crmd: [6007]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 19:24:05 srv-dell crmd: [6007]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 3 19:24:05 srv-dell crmd: [6007]: info: ais_dispatch: Membership
248: quorum still lost
Jun 3 19:24:05 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/33, version=0.186.11): ok (rc=0)
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_state_transition: All
1 cluster nodes responded to the join offer.
Jun 3 19:24:05 srv-dell crmd: [6007]: info: do_dc_join_finalize:
join-1: Syncing the CIB from srv-dell to the rest of the cluster
Jun 3 19:24:05 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/36, version=0.186.11): ok (rc=0)
Jun 3 19:24:05 srv-dell crmd: [6007]: info: te_connect_stonith:
Attempting connection to fencing daemon...
Jun 3 19:24:05 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/37, version=0.186.11): ok (rc=0)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: te_connect_stonith: Connected
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_dc_join_ack: join-1:
Updating node state to member for srv-dell
Jun 3 19:24:06 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/38, version=0.186.11): ok (rc=0)
Jun 3 19:24:06 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/39,
version=0.186.12): ok (rc=0)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 19:24:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 3 19:24:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:24:06 srv-dell crmd: [6007]: info: crm_update_quorum:
Updating quorum status to false (call=43)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 3 19:24:06 srv-dell attrd: [6005]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_pe_invoke: Query 44:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 19:24:06 srv-dell attrd: [6005]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 19:24:06 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/41, version=0.186.13): ok (rc=0)
Jun 3 19:24:06 srv-dell crmd: [6007]: WARN: match_down_event: No
match for shutdown action on srv-lenovo
Jun 3 19:24:06 srv-dell crmd: [6007]: info: te_update_diff:
Stonith/shutdown of srv-lenovo not matched
Jun 3 19:24:06 srv-dell crmd: [6007]: info: abort_transition_graph:
te_update_diff:191 - Triggered transition abort (complete=1,
tag=node_state, id=srv-lenovo, magic=NA) : Node failure
Jun 3 19:24:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_pe_invoke: Query 45:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 19:24:06 srv-dell attrd: [6005]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 19:24:06 srv-dell cib: [6003]: info: log_data_element:
cib:diff: - <cib have-quorum="1" dc-uuid="srv-lenovo" admin_epoch="0"
epoch="186" num_updates="14" />
Jun 3 19:24:06 srv-dell cib: [6003]: info: log_data_element:
cib:diff: + <cib have-quorum="0" dc-uuid="srv-dell" admin_epoch="0"
epoch="187" num_updates="1" />
Jun 3 19:24:06 srv-dell cib: [6003]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/43, version=0.187.1): ok (rc=0)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 3 19:24:06 srv-dell crmd: [6007]: info: need_abort: Aborting on
change to have-quorum
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_pe_invoke: Query 46:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 19:24:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244071446-16, seq=248, quorate=0
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 19:24:06 srv-dell pengine: [6006]: info:
determine_online_status_fencing: Node srv-lenovo is down
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource o2cb:0 running
on srv-lenovo
Jun 3 19:24:06 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="o2cb:0"
type="o2cb" class="ocf" provider="ocfs2" />
Jun 3 19:24:06 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan o2cb:0 is stopped
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource dlm:0 running on
srv-lenovo
Jun 3 19:24:06 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="dlm:0"
type="controld" class="ocf" provider="pacemaker" />
Jun 3 19:24:06 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan dlm:0 is stopped
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource
primitive_cluster_filesystem:0 running on srv-lenovo
Jun 3 19:24:06 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="primitive_cluster_filesystem:0" type="Filesystem"
class="heartbeat" />
Jun 3 19:24:06 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan
primitive_cluster_filesystem:0 is stopped
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource fs:0 running on
srv-lenovo
Jun 3 19:24:06 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="fs:0"
type="Filesystem" class="ocf" provider="heartbeat" />
Jun 3 19:24:06 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan fs:0 is stopped
Jun 3 19:24:06 srv-dell pengine: [6006]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-lenovo returned
0 (ok) instead of the expected value: 7 (not running)
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-lenovo
Jun 3 19:24:06 srv-dell pengine: [6006]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-lenovo returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-lenovo
Jun 3 19:24:06 srv-dell pengine: [6006]: info:
determine_online_status: Node srv-dell is online
Jun 3 19:24:06 srv-dell pengine: [6006]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 3 19:24:06 srv-dell pengine: [6006]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 19:24:06 srv-dell pengine: [6006]: notice: print_list:
Started: [ srv-dell ]
Jun 3 19:24:06 srv-dell cib: [11868]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-83.raw
Jun 3 19:24:06 srv-dell pengine: [6006]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:1 ]
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 19:24:06 srv-dell pengine: [6006]: notice: LogActions: Stop
resource primitiveresource_cluster_ip (srv-dell)
Jun 3 19:24:06 srv-dell pengine: [6006]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-dell)
Jun 3 19:24:06 srv-dell pengine: [6006]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Stopped)
Jun 3 19:24:06 srv-dell cib: [11868]: info: write_cib_contents: Wrote
version 0.187.0 of the CIB to disk (digest:
596a7670fb2405135f23709a29ec0d40)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 19:24:06 srv-dell crmd: [6007]: info: unpack_graph: Unpacked
transition 0: 2 actions in 2 synapses
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1244071446-16) derived from
/var/lib/pengine/pe-warn-66.bz2
Jun 3 19:24:06 srv-dell cib: [11868]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ejFDmM (digest:
/var/lib/heartbeat/crm/cib.ERFwL8)
Jun 3 19:24:06 srv-dell pengine: [6006]: WARN: process_pe_message:
Transition 0: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-66.bz2
Jun 3 19:24:06 srv-dell pengine: [6006]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 19:24:06 srv-dell crmd: [6007]: info: te_rsc_command:
Initiating action 5: stop primitiveresource_cluster_ip_stop_0 on
srv-dell (local)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=5:0:0:434dde7d-044a-4525-90a4-cce17f9131fc
op=primitiveresource_cluster_ip_stop_0 )
Jun 3 19:24:06 srv-dell lrmd: [6004]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 3 19:24:06 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=9, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 3 19:24:06 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 3 19:24:06 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 3 19:24:06 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 3 19:24:06 srv-dell lrmd: [6004]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/03_19:24:06 INFO:
ifconfig eth1:0 down
Jun 3 19:24:06 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=10, rc=0,
cib-update=47, confirmed=true) complete ok
Jun 3 19:24:06 srv-dell crmd: [6007]: info: match_graph_event: Action
primitiveresource_cluster_ip_stop_0 (5) confirmed on srv-dell (rc=0)
Jun 3 19:24:06 srv-dell crmd: [6007]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 3 19:24:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:24:06 srv-dell crmd: [6007]: info: run_graph:
====================================================
Jun 3 19:24:06 srv-dell crmd: [6007]: notice: run_graph: Transition 0
(Complete=2, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-66.bz2): Complete
Jun 3 19:24:06 srv-dell crmd: [6007]: info: te_graph_trigger:
Transition 0 is now complete
Jun 3 19:24:06 srv-dell crmd: [6007]: info: notify_crmd: Transition 0
status: done - <null>
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 19:24:06 srv-dell crmd: [6007]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 3 19:24:06 srv-dell mgmtd: [6008]: info: CIB query: cib
Jun 3 19:24:19 srv-dell mgmtd: [6008]: info: mgmtd is shutting down
Jun 3 19:24:19 srv-dell mgmtd: [6008]: debug: [mgmtd] stopped
Jun 3 19:24:19 srv-dell openais[5996]: [SERV ] Unloading all openais components
Jun 3 19:24:19 srv-dell openais[5996]: [SERV ] Unloading slot 10:
openais cluster config database access v1.01
Jun 3 19:24:19 srv-dell openais[5996]: [SERV ] Unloading openais
component: openais_confdb v0 (16/10)
Jun 3 19:24:19 srv-dell openais[5996]: [SERV ] Unloading slot 9:
Pacemaker Cluster Manager
Jun 3 19:24:19 srv-dell openais[5996]: [SERV ] Unloading openais
component: pacemaker v0 (2/9)
Jun 3 19:24:19 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
Begining shutdown
Jun 3 19:24:19 srv-dell openais[5996]: [MAIN ] notice: stop_child:
Sent -15 to mgmtd: [6008]
Jun 3 19:24:19 srv-dell haclient: on_event:None
Jun 3 19:24:20 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
mgmtd (pid=6008) confirmed dead
Jun 3 19:24:20 srv-dell openais[5996]: [MAIN ] notice: stop_child:
Sent -15 to crmd: [6007]
Jun 3 19:24:20 srv-dell crmd: [6007]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 19:24:20 srv-dell crmd: [6007]: info: crm_shutdown: Requesting shutdown
Jun 3 19:24:20 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_SHUTDOWN
cause=C_SHUTDOWN origin=crm_shutdown ]
Jun 3 19:24:20 srv-dell crmd: [6007]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 3 19:24:20 srv-dell crmd: [6007]: info: do_shutdown_req: Sending
shutdown request to DC: srv-dell
Jun 3 19:24:20 srv-dell crmd: [6007]: info: handle_shutdown_request:
Creating shutdown request for srv-dell (state=S_POLICY_ENGINE)
Jun 3 19:24:20 srv-dell crmd: [6007]: info: update_attrd: Updating
shutdown=1244071460 via attrd for srv-dell
Jun 3 19:24:20 srv-dell attrd: [6005]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 19:24:20 srv-dell attrd: [6005]: info: attrd_perform_update:
Sent update 15: shutdown=1244071460
Jun 3 19:24:20 srv-dell crmd: [6007]: info: abort_transition_graph:
te_update_diff:146 - Triggered transition abort (complete=1,
tag=transient_attributes, id=srv-dell, magic=NA) : Transient
attribute: update
Jun 3 19:24:20 srv-dell crmd: [6007]: info: do_pe_invoke: Query 49:
Requesting the current CIB: S_POLICY_ENGINE
Jun 3 19:24:20 srv-dell crmd: [6007]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244071460-19, seq=248, quorate=0
Jun 3 19:24:20 srv-dell pengine: [6006]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 3 19:24:20 srv-dell pengine: [6006]: info:
determine_online_status_fencing: Node srv-lenovo is down
Jun 3 19:24:20 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource o2cb:0 running
on srv-lenovo
Jun 3 19:24:20 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="o2cb:0"
type="o2cb" class="ocf" provider="ocfs2" />
Jun 3 19:24:20 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan o2cb:0 is stopped
Jun 3 19:24:20 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource dlm:0 running on
srv-lenovo
Jun 3 19:24:20 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="dlm:0"
type="controld" class="ocf" provider="pacemaker" />
Jun 3 19:24:20 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan dlm:0 is stopped
Jun 3 19:24:20 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource
primitive_cluster_filesystem:0 running on srv-lenovo
Jun 3 19:24:20 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="primitive_cluster_filesystem:0" type="Filesystem"
class="heartbeat" />
Jun 3 19:24:20 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan
primitive_cluster_filesystem:0 is stopped
Jun 3 19:24:20 srv-dell pengine: [6006]: WARN:
process_orphan_resource: Nothing known about resource fs:0 running on
srv-lenovo
Jun 3 19:24:20 srv-dell pengine: [6006]: info: log_data_element:
create_fake_resource: Orphan resource <primitive id="fs:0"
type="Filesystem" class="ocf" provider="heartbeat" />
Jun 3 19:24:20 srv-dell pengine: [6006]: info:
process_orphan_resource: Making sure orphan fs:0 is stopped
Jun 3 19:24:20 srv-dell pengine: [6006]: info: unpack_rsc_op:
cloneresource_cluster_fencing_null:0_monitor_0 on srv-lenovo returned
0 (ok) instead of the expected value: 7 (not running)
Jun 3 19:24:20 srv-dell pengine: [6006]: WARN: unpack_rsc_op:
Operation cloneresource_cluster_fencing_null:0_monitor_0 found
resource cloneresource_cluster_fencing_null:0 active on srv-lenovo
Jun 3 19:24:20 srv-dell pengine: [6006]: info: unpack_rsc_op:
primitiveresource_cluster_ip_monitor_0 on srv-lenovo returned 0 (ok)
instead of the expected value: 7 (not running)
Jun 3 19:24:20 srv-dell pengine: [6006]: WARN: unpack_rsc_op:
Operation primitiveresource_cluster_ip_monitor_0 found resource
primitiveresource_cluster_ip active on srv-lenovo
Jun 3 19:24:20 srv-dell pengine: [6006]: info:
determine_online_status: Node srv-dell is shutting down
Jun 3 19:24:20 srv-dell pengine: [6006]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 3 19:24:20 srv-dell pengine: [6006]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 3 19:24:20 srv-dell pengine: [6006]: notice: print_list:
Started: [ srv-dell ]
Jun 3 19:24:23 srv-dell pengine: [6006]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:1 ]
Jun 3 19:24:23 srv-dell pengine: [6006]: WARN: native_color: Resource
primitiveresource_cluster_ip cannot run anywhere
Jun 3 19:24:23 srv-dell pengine: [6006]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 3 19:24:23 srv-dell pengine: [6006]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:1 cannot run anywhere
Jun 3 19:24:23 srv-dell pengine: [6006]: info: stage6: Scheduling
Node srv-dell for shutdown
Jun 3 19:24:23 srv-dell pengine: [6006]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Stopped)
Jun 3 19:24:23 srv-dell pengine: [6006]: notice: LogActions: Stop
resource cloneresource_cluster_fencing_null:0 (srv-dell)
Jun 3 19:24:23 srv-dell pengine: [6006]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Stopped)
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 3 19:24:23 srv-dell crmd: [6007]: info: unpack_graph: Unpacked
transition 1: 5 actions in 5 synapses
Jun 3 19:24:23 srv-dell pengine: [6006]: WARN: process_pe_message:
Transition 1: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-67.bz2
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_te_invoke: Processing
graph 1 (ref=pe_calc-dc-1244071460-19) derived from
/var/lib/pengine/pe-warn-67.bz2
Jun 3 19:24:23 srv-dell pengine: [6006]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 3 19:24:23 srv-dell crmd: [6007]: info: te_pseudo_action: Pseudo
action 7 fired and confirmed
Jun 3 19:24:23 srv-dell crmd: [6007]: info: te_pseudo_action: Pseudo
action 1 fired and confirmed
Jun 3 19:24:23 srv-dell crmd: [6007]: info: te_rsc_command:
Initiating action 4: stop cloneresource_cluster_fencing_null:0_stop_0
on srv-dell (local)
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_lrm_rsc_op: Performing
key=4:1:0:434dde7d-044a-4525-90a4-cce17f9131fc
op=cloneresource_cluster_fencing_null:0_stop_0 )
Jun 3 19:24:23 srv-dell lrmd: [6004]: info:
rsc:cloneresource_cluster_fencing_null:0: stop
Jun 3 19:24:23 srv-dell lrmd: [11908]: info: Try to stop STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:0> : Device=null
Jun 3 19:24:23 srv-dell crmd: [6007]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:0_stop_0 (call=11, rc=0,
cib-update=50, confirmed=true) complete ok
Jun 3 19:24:23 srv-dell crmd: [6007]: info: match_graph_event: Action
cloneresource_cluster_fencing_null:0_stop_0 (4) confirmed on srv-dell
(rc=0)
Jun 3 19:24:23 srv-dell crmd: [6007]: info: te_pseudo_action: Pseudo
action 8 fired and confirmed
Jun 3 19:24:23 srv-dell crmd: [6007]: info: te_crm_command: Executing
crm-event (10): do_shutdown on srv-dell
Jun 3 19:24:23 srv-dell crmd: [6007]: info: te_crm_command: crm-event
(10) is a local shutdown
Jun 3 19:24:23 srv-dell crmd: [6007]: info: run_graph:
====================================================
Jun 3 19:24:23 srv-dell crmd: [6007]: notice: run_graph: Transition 1
(Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-67.bz2): Complete
Jun 3 19:24:23 srv-dell crmd: [6007]: info: te_graph_trigger:
Transition 1 is now complete
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_STOPPING [ input=I_STOP
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_dc_release: DC role released
Jun 3 19:24:23 srv-dell crmd: [6007]: info: pe_connection_destroy:
Connection to the Policy Engine released
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_te_control:
Transitioner is now inactive
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_te_control:
Disconnecting STONITH...
Jun 3 19:24:23 srv-dell crmd: [6007]: info:
tengine_stonith_connection_destroy: Fencing daemon disconnected
Jun 3 19:24:23 srv-dell crmd: [6007]: notice: Not currently connected.
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_lrm_control:
Disconnected from the LRM
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_ha_control:
Disconnected from OpenAIS
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_cib_control: Disconnecting CIB
Jun 3 19:24:23 srv-dell crmd: [6007]: info:
crmd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_exit: Performing
A_EXIT_0 - gracefully exiting the CRMd
Jun 3 19:24:23 srv-dell crmd: [6007]: info: free_mem: Dropping
I_RELEASE_SUCCESS: [ state=S_STOPPING cause=C_FSA_INTERNAL
origin=do_dc_release ]
Jun 3 19:24:23 srv-dell crmd: [6007]: info: free_mem: Dropping
I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ]
Jun 3 19:24:23 srv-dell cib: [6003]: info: cib_process_readwrite: We
are now in R/O mode
Jun 3 19:24:23 srv-dell cib: [6003]: WARN: send_ipc_message: IPC
Channel to 6007 is not connected
Jun 3 19:24:23 srv-dell crmd: [6007]: info: do_exit: [crmd] stopped (0)
Jun 3 19:24:23 srv-dell cib: [6003]: WARN: send_via_callback_channel:
Delivery of reply to client 6007/53df5099-1684-4632-a77d-5cc2b5943a71
failed
Jun 3 19:24:23 srv-dell cib: [6003]: WARN: do_local_notify: A-Sync
reply to crmd failed: reply failed
Jun 3 19:24:23 srv-dell openais[5996]: [crm ] info: pcmk_ipc_exit:
Client crmd (conn=0x77f220, async-conn=0x77f220) left
Jun 3 19:24:23 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
crmd (pid=6007) confirmed dead
Jun 3 19:24:23 srv-dell openais[5996]: [MAIN ] notice: stop_child:
Sent -15 to pengine: [6006]
Jun 3 19:24:23 srv-dell pengine: [6006]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 19:24:24 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
pengine (pid=6006) confirmed dead
Jun 3 19:24:24 srv-dell openais[5996]: [MAIN ] notice: stop_child:
Sent -15 to attrd: [6005]
Jun 3 19:24:24 srv-dell attrd: [6005]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 19:24:24 srv-dell attrd: [6005]: info: attrd_shutdown: Exiting
Jun 3 19:24:24 srv-dell attrd: [6005]: info: main: Exiting...
Jun 3 19:24:24 srv-dell attrd: [6005]: info:
attrd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 19:24:24 srv-dell openais[5996]: [crm ] info: pcmk_ipc_exit:
Client attrd (conn=0x77dc90, async-conn=0x77dc90) left
Jun 3 19:24:25 srv-dell lrmd: [6004]: info: lrmd is shutting down
Jun 3 19:24:25 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
attrd (pid=6005) confirmed dead
Jun 3 19:24:25 srv-dell openais[5996]: [MAIN ] notice: stop_child:
Sent -15 to lrmd: [6004]
Jun 3 19:24:26 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
lrmd (pid=6004) confirmed dead
Jun 3 19:24:26 srv-dell openais[5996]: [MAIN ] notice: stop_child:
Sent -15 to cib: [6003]
Jun 3 19:24:26 srv-dell cib: [6003]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 19:24:26 srv-dell cib: [6003]: info: cib_shutdown: Disconnected 0 clients
Jun 3 19:24:26 srv-dell cib: [6003]: info: cib_process_disconnect:
All clients disconnected...
Jun 3 19:24:26 srv-dell cib: [6003]: info: cib_ha_connection_destroy:
Heartbeat disconnection complete... exiting
Jun 3 19:24:26 srv-dell cib: [6003]: info: cib_ha_connection_destroy:
Exiting...
Jun 3 19:24:26 srv-dell cib: [6003]: info: main: Done
Jun 3 19:24:26 srv-dell openais[5996]: [crm ] info: pcmk_ipc_exit:
Client cib (conn=0x77da60, async-conn=0x77da60) left
Jun 3 19:24:27 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
cib (pid=6003) confirmed dead
Jun 3 19:24:27 srv-dell stonithd: [6002]: notice:
/usr/lib64/heartbeat/stonithd normally quit.
Jun 3 19:24:27 srv-dell openais[5996]: [MAIN ] notice: stop_child:
Sent -15 to stonithd: [6002]
Jun 3 19:24:27 srv-dell openais[5996]: [crm ] info: pcmk_ipc_exit:
Client stonithd (conn=0x77cd50, async-conn=0x77cd50) left
Jun 3 19:24:28 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
stonithd (pid=6002) confirmed dead
Jun 3 19:24:28 srv-dell openais[5996]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000000002
(2)
Jun 3 19:24:28 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
Shutdown complete
Jun 3 19:24:28 srv-dell openais[5996]: [crm ] notice: pcmk_shutdown:
Forcing clean exit of OpenAIS
Jun 3 19:26:02 srv-dell syslog-ng[3184]: Log statistics;
dropped='pipe(/dev/xconsole)=0', dropped='pipe(/dev/tty10)=0',
processed='center(queued)=1713', processed='center(received)=1487',
processed='destination(newsnotice)=0',
processed='destination(acpid)=1', processed='destination(firewall)=0',
processed='destination(null)=1', processed='destination(mail)=2',
processed='destination(mailinfo)=2',
processed='destination(console)=40',
processed='destination(newserr)=0',
processed='destination(newscrit)=0',
processed='destination(messages)=1483',
processed='destination(mailwarn)=0',
processed='destination(localmessages)=2',
processed='destination(netmgm)=0', processed='destination(mailerr)=0',
processed='destination(xconsole)=40',
processed='destination(warn)=142', processed='source(src)=1487'
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] AIS Executive Service
RELEASE 'subrev 1152 version 0.80'
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] Copyright (C) 2006
Red Hat, Inc.
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] AIS Executive
Service: started and ready to provide service.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Token Timeout (5000
ms) retransmit timeout (490 ms)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] token hold (382 ms)
retransmits before loss (10 retrans)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] join (1000 ms)
send_join (45 ms) consensus (2500 ms) merge (200 ms)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] seqno unchanged const
(30 rotations) Maximum network MTU 1500
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] send threads (0 threads)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] RRP token expired
timeout (490 ms)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] RRP token problem
counter (2000 ms)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] RRP threshold (10
problem count)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] RRP mode set to none.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] heartbeat_failures_allowed (0)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] max_network_delay (50 ms)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] HeartBeat is
Disabled. To enable set heartbeat_failures_allowed > 0
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] The network interface
[192.168.2.220] is now up.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Created or loaded
sequence id 248.192.168.2.220 for this ring.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] entering GATHER state from 15.
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info:
process_ais_conf: Reading configure
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info:
config_find_next: Processing additional logging options...
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: get_config_opt:
Found 'off' for option: debug
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: get_config_opt:
Found 'yes' for option: to_syslog
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: get_config_opt:
Found 'daemon' for option: syslog_facility
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info:
config_find_next: Processing additional service options...
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_logd
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_mgmtd
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info:
pcmk_plugin_init: CRM: Initialized
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] Logging: Initialized
pcmk_plugin_init
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info:
pcmk_plugin_init: Service: 9
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info:
pcmk_plugin_init: Local node id: 1543678144
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info:
pcmk_plugin_init: Local hostname: srv-dell
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: update_member:
Creating entry for node 1543678144 born on 0
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: update_member:
0x772110 Node 1543678144 now known as srv-dell (was: (null))
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: update_member:
Node srv-dell now has 1 quorum votes (was 0)
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: update_member:
Node 1543678144/srv-dell is now: member
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: spawn_child:
Forked child 11968 for process stonithd
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: spawn_child:
Forked child 11969 for process cib
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: spawn_child:
Forked child 11970 for process lrmd
Jun 3 19:34:03 srv-dell stonithd: [11968]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 19:34:03 srv-dell stonithd: [11968]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 19:34:03 srv-dell stonithd: [11968]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 19:34:03 srv-dell lrmd: [11970]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 19:34:03 srv-dell lrmd: [11970]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 3 19:34:03 srv-dell cib: [11969]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 19:34:03 srv-dell cib: [11969]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 19:34:03 srv-dell cib: [11969]: info:
G_main_add_TriggerHandler: Added signal manual handler
Jun 3 19:34:03 srv-dell cib: [11969]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 3 19:34:03 srv-dell cib: [11969]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Jun 3 19:34:03 srv-dell lrmd: [11970]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: spawn_child:
Forked child 11971 for process attrd
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: spawn_child:
Forked child 11972 for process pengine
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: spawn_child:
Forked child 11973 for process crmd
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: spawn_child:
Forked child 11974 for process mgmtd
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info: pcmk_startup:
CRM: Initialized
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] Service initialized
'Pacemaker Cluster Manager'
Jun 3 19:34:03 srv-dell lrmd: [11970]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 19:34:03 srv-dell lrmd: [11970]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 19:34:03 srv-dell attrd: [11971]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 19:34:03 srv-dell pengine: [11972]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 19:34:03 srv-dell crmd: [11973]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 3 19:34:03 srv-dell mgmtd: [11974]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 19:34:03 srv-dell mgmtd: [11974]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 3 19:34:03 srv-dell cib: [11969]: info: startCib: CIB
Initialization completed successfully
Jun 3 19:34:03 srv-dell lrmd: [11970]: info: Started.
Jun 3 19:34:03 srv-dell crmd: [11973]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 19:34:03 srv-dell mgmtd: [11974]: debug: Enabling coredumps
Jun 3 19:34:03 srv-dell attrd: [11971]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 3 19:34:03 srv-dell cib: [11969]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 19:34:03 srv-dell pengine: [11972]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 19:34:03 srv-dell crmd: [11973]: info: main: CRM Hg Version:
0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Jun 3 19:34:03 srv-dell mgmtd: [11974]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 3 19:34:03 srv-dell stonithd: [11968]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 19:34:03 srv-dell attrd: [11971]: info: main: Starting up....
Jun 3 19:34:03 srv-dell cib: [11969]: info: init_ais_connection: AIS
connection established
Jun 3 19:34:03 srv-dell attrd: [11971]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 19:34:03 srv-dell mgmtd: [11974]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 3 19:34:03 srv-dell crmd: [11973]: info: crmd_init: Starting crmd
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais extended virtual synchrony service'
Jun 3 19:34:03 srv-dell pengine: [11972]: info: main: Starting pengine
Jun 3 19:34:03 srv-dell crmd: [11973]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 19:34:03 srv-dell mgmtd: [11974]: info: init_crm
Jun 3 19:34:03 srv-dell mgmtd: [11974]: info: login to cib: 0, ret:-10
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais cluster membership service B.01.01'
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais availability management framework B.01.01'
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais checkpoint service B.01.01'
Jun 3 19:34:03 srv-dell attrd: [11971]: info: init_ais_connection:
AIS connection established
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais event service B.01.01'
Jun 3 19:34:03 srv-dell cib: [11969]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 19:34:03 srv-dell cib: [11969]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 19:34:03 srv-dell cib: [11969]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais distributed locking service B.01.01'
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais message service B.01.01'
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais configuration service'
Jun 3 19:34:03 srv-dell cib: [11969]: info: cib_init: Starting cib mainloop
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais cluster closed process group service v1.01'
Jun 3 19:34:03 srv-dell openais[11962]: [SERV ] Service initialized
'openais cluster config database access v1.01'
Jun 3 19:34:03 srv-dell openais[11962]: [SYNC ] Not using a virtual
synchrony filter.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Creating commit token
because I am the rep.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Saving state aru 0
high seq received 0
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Storing new sequence
id for ring fc
Jun 3 19:34:03 srv-dell cib: [11969]: info: ais_dispatch: Membership
252: quorum still lost
Jun 3 19:34:03 srv-dell cib: [11969]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=252
proc=00000000000000000000000000053312 (new)
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] entering COMMIT state.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] entering RECOVERY state.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] position [0] member
192.168.2.220:
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] previous ring seq 248
rep 192.168.2.220
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] aru 0 high delivered
0 received flag 1
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] Sending initial ORF token
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] New Configuration:
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] Members Left:
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] Members Joined:
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 252: memb=0,
new=0, lost=0
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] New Configuration:
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 19:34:03 srv-dell attrd: [11971]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 19:34:03 srv-dell attrd: [11971]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 19:34:03 srv-dell attrd: [11971]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] Members Left:
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] Members Joined:
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 252: memb=1, new=1,
lost=0
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info:
pcmk_peer_update: NEW: srv-dell 1543678144
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 19:34:03 srv-dell openais[11962]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000053312
(340754)
Jun 3 19:34:03 srv-dell openais[11962]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 19:34:03 srv-dell openais[11962]: [TOTEM] entering OPERATIONAL state.
Jun 3 19:34:03 srv-dell openais[11962]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info: pcmk_ipc:
Recorded connection 0x77c8f0 for cib/11969
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info: pcmk_ipc:
Sending membership update 252 to cib
Jun 3 19:34:03 srv-dell stonithd: [11968]: info: init_ais_connection:
AIS connection established
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info: pcmk_ipc:
Recorded connection 0x77cca0 for attrd/11971
Jun 3 19:34:03 srv-dell stonithd: [11968]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 19:34:03 srv-dell openais[11962]: [crm ] info: pcmk_ipc:
Recorded connection 0x77cb20 for stonithd/11968
Jun 3 19:34:03 srv-dell stonithd: [11968]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 19:34:03 srv-dell stonithd: [11968]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 19:34:03 srv-dell stonithd: [11968]: notice:
/usr/lib64/heartbeat/stonithd start up successfully.
Jun 3 19:34:03 srv-dell stonithd: [11968]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 3 19:34:03 srv-dell cib: [11976]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-84.raw
Jun 3 19:34:03 srv-dell cib: [11976]: info: write_cib_contents: Wrote
version 0.187.0 of the CIB to disk (digest:
2ba63cc8a5dfea3d0ad0da4266e796fb)
Jun 3 19:34:03 srv-dell cib: [11976]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.Vlc1GH (digest:
/var/lib/heartbeat/crm/cib.b5fM31)
Jun 3 19:34:04 srv-dell crmd: [11973]: info: do_cib_control: CIB
connection established
Jun 3 19:34:04 srv-dell crmd: [11973]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 3 19:34:04 srv-dell crmd: [11973]: info: init_ais_connection: AIS
connection established
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info: pcmk_ipc:
Recorded connection 0x77c4b0 for crmd/11973
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info: pcmk_ipc:
Sending membership update 252 to crmd
Jun 3 19:34:04 srv-dell crmd: [11973]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 3 19:34:04 srv-dell crmd: [11973]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 3 19:34:04 srv-dell crmd: [11973]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 3 19:34:04 srv-dell crmd: [11973]: info: do_ha_control: Connected
to the cluster
Jun 3 19:34:04 srv-dell crmd: [11973]: info: do_started: Delaying
start, CCM (0000000000100000) not connected
Jun 3 19:34:04 srv-dell crmd: [11973]: info: crmd_init: Starting
crmd's mainloop
Jun 3 19:34:04 srv-dell crmd: [11973]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info:
update_expected_votes: Expected quorum votes 1024 -> 2
Jun 3 19:34:04 srv-dell crmd: [11973]: info: ais_dispatch: Membership
252: quorum still lost
Jun 3 19:34:04 srv-dell crmd: [11973]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=252
proc=00000000000000000000000000053312 (new)
Jun 3 19:34:04 srv-dell crmd: [11973]: info: do_started: The local
CRM is operational
Jun 3 19:34:04 srv-dell crmd: [11973]: info: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] entering GATHER state from 11.
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] Saving state aru b
high seq received b
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] Storing new sequence
id for ring 100
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] entering COMMIT state.
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] entering RECOVERY state.
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] position [0] member
192.168.2.120:
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] previous ring seq 248
rep 192.168.2.120
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] aru 22 high delivered
22 received flag 1
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] position [1] member
192.168.2.220:
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] previous ring seq 252
rep 192.168.2.220
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] aru b high delivered
b received flag 1
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] New Configuration:
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] Members Left:
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] Members Joined:
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 256: memb=1,
new=0, lost=0
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] CLM CONFIGURATION CHANGE
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] New Configuration:
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] r(0) ip(192.168.2.220)
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] Members Left:
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] Members Joined:
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] r(0) ip(192.168.2.120)
Jun 3 19:34:04 srv-dell cib: [11969]: notice: ais_dispatch:
Membership 256: quorum aquired
Jun 3 19:34:04 srv-dell cib: [11969]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 3 19:34:04 srv-dell cib: [11969]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=256 proc=00000000000000000000000000000000
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 256: memb=2, new=1,
lost=0
Jun 3 19:34:04 srv-dell openais[11962]: [MAIN ] info: update_member:
Creating entry for node 2013440192 born on 256
Jun 3 19:34:04 srv-dell openais[11962]: [MAIN ] info: update_member:
Node 2013440192/unknown is now: member
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info:
pcmk_peer_update: NEW: .pending. 2013440192
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info:
pcmk_peer_update: MEMB: .pending. 2013440192
Jun 3 19:34:04 srv-dell cib: [11969]: info: ais_dispatch: Membership
256: quorum retained
Jun 3 19:34:04 srv-dell cib: [11969]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 19:34:04 srv-dell cib: [11969]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=256 seen=256 proc=00000000000000000000000000053312
(new)
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info:
send_member_notification: Sending membership update 256 to 2 children
Jun 3 19:34:04 srv-dell openais[11962]: [MAIN ] info: update_member:
0x772110 Node 1543678144 ((null)) born on: 256
Jun 3 19:34:04 srv-dell openais[11962]: [SYNC ] This node is within
the primary component and will provide service.
Jun 3 19:34:04 srv-dell openais[11962]: [TOTEM] entering OPERATIONAL state.
Jun 3 19:34:04 srv-dell openais[11962]: [MAIN ] info: update_member:
0x778440 Node 2013440192 (srv-lenovo) born on: 256
Jun 3 19:34:04 srv-dell openais[11962]: [MAIN ] info: update_member:
0x778440 Node 2013440192 now known as srv-lenovo (was: (null))
Jun 3 19:34:04 srv-dell openais[11962]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 3 19:34:04 srv-dell openais[11962]: [MAIN ] info: update_member:
Node srv-lenovo now has 1 quorum votes (was 0)
Jun 3 19:34:04 srv-dell openais[11962]: [crm ] info:
send_member_notification: Sending membership update 256 to 2 children
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] got nodejoin message
192.168.2.120
Jun 3 19:34:04 srv-dell openais[11962]: [CLM ] got nodejoin message
192.168.2.220
Jun 3 19:34:04 srv-dell cib: [11969]: info: cib_process_diff: Diff
0.187.5 -> 0.188.1 not applied to 0.187.0: current "num_updates" is
less than required
Jun 3 19:34:04 srv-dell cib: [11969]: info: cib_server_process_diff:
Requesting re-sync from peer
Jun 3 19:34:04 srv-dell cib: [11969]: WARN: cib_diff_notify:
Local-only Change (client:crmd, call: 35): 0.0.0 (Application of an
update diff failed, requesting a full refresh)
Jun 3 19:34:04 srv-dell cib: [11969]: WARN: cib_server_process_diff:
Not applying diff 0.188.1 -> 0.188.2 (sync in progress)
Jun 3 19:34:04 srv-dell cib: [11969]: info: cib_replace_notify:
Replaced: 0.187.0 -> 0.188.2 from srv-lenovo
Jun 3 19:34:04 srv-dell cib: [11980]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-85.raw
Jun 3 19:34:04 srv-dell cib: [11980]: info: write_cib_contents: Wrote
version 0.188.0 of the CIB to disk (digest:
273b9bd6d28267b74487b7b8f10fdcb8)
Jun 3 19:34:04 srv-dell cib: [11980]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.KBCMiT (digest:
/var/lib/heartbeat/crm/cib.CATm8f)
Jun 3 19:34:04 srv-dell mgmtd: [11974]: debug: main: run the loop...
Jun 3 19:34:04 srv-dell mgmtd: [11974]: info: Started.
Jun 3 19:34:05 srv-dell crmd: [11973]: info: ais_dispatch: Membership
252: quorum still lost
Jun 3 19:34:05 srv-dell crmd: [11973]: notice: ais_dispatch:
Membership 256: quorum aquired
Jun 3 19:34:05 srv-dell crmd: [11973]: info: crm_new_peer: Node
<null> now has id: 2013440192
Jun 3 19:34:05 srv-dell crmd: [11973]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=256 proc=00000000000000000000000000000000
Jun 3 19:34:05 srv-dell crmd: [11973]: info: ais_dispatch: Membership
256: quorum retained
Jun 3 19:34:05 srv-dell crmd: [11973]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 19:34:05 srv-dell crmd: [11973]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=256 seen=256 proc=00000000000000000000000000053312
(new)
Jun 3 19:34:05 srv-dell crmd: [11973]: info: update_dc: Set DC to
srv-lenovo (3.0.1)
Jun 3 19:34:05 srv-dell crmd: [11973]: info: update_attrd: Connecting
to attrd...
Jun 3 19:34:05 srv-dell crmd: [11973]: info: update_attrd: Updating
terminate=<none> via attrd for srv-dell
Jun 3 19:34:05 srv-dell crmd: [11973]: info: update_attrd: Updating
shutdown=<none> via attrd for srv-dell
Jun 3 19:34:05 srv-dell crmd: [11973]: info: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 3 19:34:05 srv-dell crmd: [11973]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/transient_attributes": ok
(rc=0)
Jun 3 19:34:05 srv-dell crmd: [11973]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 3 19:34:05 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=4:1:7:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_monitor_0 )
Jun 3 19:34:05 srv-dell lrmd: [11970]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 3 19:34:05 srv-dell lrmd: [11970]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 3 19:34:05 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=5:1:7:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_monitor_0 )
Jun 3 19:34:05 srv-dell lrmd: [11970]: info:
rsc:cloneresource_cluster_fencing_null:1: monitor
Jun 3 19:34:05 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_monitor_0 (call=3,
rc=7, cib-update=9, confirmed=true) complete not running
Jun 3 19:34:05 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=2, rc=7,
cib-update=10, confirmed=true) complete not running
Jun 3 19:34:05 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=11:1:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_start_0 )
Jun 3 19:34:05 srv-dell lrmd: [11970]: info:
rsc:cloneresource_cluster_fencing_null:1: start
Jun 3 19:34:05 srv-dell lrmd: [12002]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 3 19:34:05 srv-dell stonithd: [11968]: info:
cloneresource_cluster_fencing_null:1 stonith resource started
Jun 3 19:34:05 srv-dell lrmd: [11970]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 3 19:34:05 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_start_0 (call=4, rc=0,
cib-update=14, confirmed=true) complete ok
Jun 3 19:34:05 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:34:05 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:34:06 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:34:06 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:34:08 srv-dell attrd: [11971]: info: main: Sending full refresh
Jun 3 19:34:08 srv-dell attrd: [11971]: info: main: Starting mainloop...
Jun 3 19:34:08 srv-dell attrd: [11971]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 3 19:34:08 srv-dell attrd: [11971]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 3 19:34:08 srv-dell attrd: [11971]: info: find_hash_entry:
Creating hash entry for terminate
Jun 3 19:34:08 srv-dell attrd: [11971]: info: find_hash_entry:
Creating hash entry for shutdown
Jun 3 19:34:08 srv-dell attrd: [11971]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 3 19:34:08 srv-dell attrd: [11971]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 3 19:34:08 srv-dell attrd: [11971]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 19:36:03 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=5:2:7:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitive_cluster_dlm_monitor_0 )
Jun 3 19:36:03 srv-dell lrmd: [11970]: info: rsc:primitive_cluster_dlm: monitor
Jun 3 19:36:03 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:36:03 srv-dell cib: [12007]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-86.raw
Jun 3 19:36:03 srv-dell cib: [12007]: info: write_cib_contents: Wrote
version 0.189.0 of the CIB to disk (digest:
206baac0735ebd1ef5fe674b89f590b0)
Jun 3 19:36:03 srv-dell cib: [12007]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.wBBj26 (digest:
/var/lib/heartbeat/crm/cib.tNKO4E)
Jun 3 19:36:03 srv-dell lrmd: [11970]: info: RA output:
(primitive_cluster_dlm:monitor:stderr) dlm_controld.pcmk: no process
killed
Jun 3 19:36:03 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation primitive_cluster_dlm_monitor_0 (call=5, rc=7,
cib-update=15, confirmed=true) complete not running
Jun 3 19:36:03 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:36:03 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:36:03 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:37:40 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:37:40 srv-dell cib: [12020]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-87.raw
Jun 3 19:37:40 srv-dell cib: [12020]: info: write_cib_contents: Wrote
version 0.190.0 of the CIB to disk (digest:
80049daba24f990b458bb9198494d7d1)
Jun 3 19:37:40 srv-dell cib: [12020]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.WD8T42 (digest:
/var/lib/heartbeat/crm/cib.EnWso1)
Jun 3 19:37:40 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=5:3:7:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=dlm:0_monitor_0 )
Jun 3 19:37:40 srv-dell lrmd: [11970]: info: rsc:dlm:0: monitor
Jun 3 19:37:40 srv-dell lrmd: [11970]: info: RA output:
(dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Jun 3 19:37:40 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation dlm:0_monitor_0 (call=6, rc=7, cib-update=18,
confirmed=true) complete not running
Jun 3 19:37:40 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:37:40 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:37:41 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:37:48 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:37:48 srv-dell cib: [12035]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-88.raw
Jun 3 19:37:48 srv-dell cib: [12035]: info: write_cib_contents: Wrote
version 0.191.0 of the CIB to disk (digest:
4aa8bfb77dc5eb810e1f586977bb5073)
Jun 3 19:37:48 srv-dell cib: [12035]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.JOlkXD (digest:
/var/lib/heartbeat/crm/cib.sOTnmS)
Jun 3 19:37:48 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:39:05 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:39:05 srv-dell cib: [12036]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-89.raw
Jun 3 19:39:05 srv-dell cib: [12036]: info: write_cib_contents: Wrote
version 0.192.0 of the CIB to disk (digest:
92a2eacf014c2962f2eded2d5dbb7539)
Jun 3 19:39:05 srv-dell cib: [12036]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.ZTktAm (digest:
/var/lib/heartbeat/crm/cib.tBY8Gi)
Jun 3 19:39:05 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=5:5:7:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_0 )
Jun 3 19:39:05 srv-dell lrmd: [11970]: info: rsc:ocb2:0: monitor
Jun 3 19:39:05 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:monitor:stderr) logd is not running
Jun 3 19:39:05 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:monitor:stderr) 2009/06/03_19:39:05 INFO: configfs not laoded
Jun 3 19:39:05 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation ocb2:0_monitor_0 (call=7, rc=7, cib-update=21,
confirmed=true) complete not running
Jun 3 19:39:05 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:39:05 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:39:05 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:40:50 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:40:50 srv-dell cib: [12052]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-90.raw
Jun 3 19:40:50 srv-dell cib: [12052]: info: write_cib_contents: Wrote
version 0.193.0 of the CIB to disk (digest:
126e34a9907772dc04ee7f3387f5693b)
Jun 3 19:40:50 srv-dell cib: [12052]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.iJJORS (digest:
/var/lib/heartbeat/crm/cib.K7tgex)
Jun 3 19:40:51 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:42:26 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:42:26 srv-dell cib: [12053]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-91.raw
Jun 3 19:42:26 srv-dell cib: [12053]: info: write_cib_contents: Wrote
version 0.194.0 of the CIB to disk (digest:
7c467ebbc5e9ed9878b32e71d8137562)
Jun 3 19:42:26 srv-dell cib: [12053]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.d5R0jf (digest:
/var/lib/heartbeat/crm/cib.YZRCIg)
Jun 3 19:42:27 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:42:59 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:42:59 srv-dell cib: [12055]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-92.raw
Jun 3 19:42:59 srv-dell cib: [12055]: info: write_cib_contents: Wrote
version 0.195.0 of the CIB to disk (digest:
6590de1d1d55e2305a116eca5dabef60)
Jun 3 19:42:59 srv-dell cib: [12055]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.WcwRMj (digest:
/var/lib/heartbeat/crm/cib.xrskOu)
Jun 3 19:42:59 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=16:8:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=dlm:0_start_0 )
Jun 3 19:42:59 srv-dell lrmd: [11970]: info: rsc:dlm:0: start
Jun 3 19:42:59 srv-dell lrmd: [11970]: info: RA output:
(dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Jun 3 19:42:59 srv-dell kernel: DLM (built Feb 28 2009 07:25:25) installed
Jun 3 19:43:00 srv-dell cluster-dlm[12095]: main: dlm_controld 1239267206
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info:
init_ais_connection: AIS connection established
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 19:43:00 srv-dell openais[11962]: [crm ] info: pcmk_notify:
Enabling node notifications for child 12095 (0x77c280)
Jun 3 19:43:00 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 3 19:43:00 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 3 19:43:00 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 3 19:43:00 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 3 19:43:00 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 3 19:43:00 srv-dell cluster-dlm: set_protocol: set_protocol
member_count 1 propose daemon 1.1.1 kernel 1.1.1
Jun 3 19:43:00 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 1543678144
Jun 3 19:43:00 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 3 19:43:00 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 3 19:43:00 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: notice: ais_dispatch:
Membership 256: quorum aquired
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=256
proc=00000000000000000000000000053312
Jun 3 19:43:00 srv-dell cluster-dlm: [12095]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=256 seen=256
proc=00000000000000000000000000053312 (new)
Jun 3 19:43:00 srv-dell cluster-dlm: update_cluster: Processing membership 256
Jun 3 19:43:00 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 3 19:43:00 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 3 19:43:00 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=256, last-seen=256, this-event=256,
last-event=0
Jun 3 19:43:00 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 3 19:43:00 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 3 19:43:00 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=256, last-seen=256, this-event=256,
last-event=0
Jun 3 19:43:00 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:43:01 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation dlm:0_start_0 (call=8, rc=0, cib-update=24, confirmed=true)
complete ok
Jun 3 19:43:01 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=17:8:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=dlm:0_monitor_120000 )
Jun 3 19:43:01 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:01 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=9, rc=0, cib-update=25,
confirmed=false) complete ok
Jun 3 19:43:01 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:01 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:43:01 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:01 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 3 19:43:01 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:43:05 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:05 srv-dell cib: [12125]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-93.raw
Jun 3 19:43:05 srv-dell cib: [12125]: info: write_cib_contents: Wrote
version 0.196.0 of the CIB to disk (digest:
bf421f0b3c3c1a7d9185fe98fde12c1f)
Jun 3 19:43:05 srv-dell cib: [12125]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.AjRyie (digest:
/var/lib/heartbeat/crm/cib.z45WUB)
Jun 3 19:43:05 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=26:9:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_start_0 )
Jun 3 19:43:05 srv-dell lrmd: [11970]: info: rsc:ocb2:0: start
Jun 3 19:43:05 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 3 19:43:05 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:start:stderr) 2009/06/03_19:43:05 INFO: Stack glue driver not
loaded
Jun 3 19:43:05 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 3 19:43:05 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:start:stderr) 2009/06/03_19:43:05 INFO: Starting ocb2:0
Jun 3 19:43:05 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 3 19:43:05 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 3 19:43:05 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:43:05 srv-dell kernel: OCFS2 1.5.0
Jun 3 19:43:05 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info:
init_ais_connection: AIS connection established
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info:
get_ais_nodeid: Server details: id=1543678144 uname=srv-dell
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 3 19:43:05 srv-dell openais[11962]: [crm ] info: pcmk_notify:
Enabling node notifications for child 12218 (0x77e8f0)
Jun 3 19:43:05 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 3 19:43:05 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 3 19:43:05 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 3 19:43:05 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 19:43:05 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 3 19:43:05 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 3 19:43:05 srv-dell ocfs2_controld: cpg_join succeeded
Jun 3 19:43:05 srv-dell ocfs2_controld: setup done
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: notice:
ais_dispatch: Membership 256: quorum aquired
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=256
proc=00000000000000000000000000053312
Jun 3 19:43:05 srv-dell ocfs2_controld: [12218]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=256 seen=256
proc=00000000000000000000000000053312 (new)
Jun 3 19:43:05 srv-dell ocfs2_controld: confchg called
Jun 3 19:43:05 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 3 19:43:05 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 3 19:43:05 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 3 19:43:05 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Creating section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Created section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 3 19:43:05 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Creating section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 3 19:43:05 srv-dell ocfs2_controld: Created section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 3 19:43:05 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 3 19:43:05 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 3 19:43:05 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 3 19:43:05 srv-dell ocfs2_controld: Opening control device
Jun 3 19:43:05 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 3 19:43:05 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 3 19:43:05 srv-dell ocfs2_controld: new listening connection 4
Jun 3 19:43:06 srv-dell ocfs2_controld: confchg called
Jun 3 19:43:06 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 3 19:43:07 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation ocb2:0_start_0 (call=10, rc=0, cib-update=26,
confirmed=true) complete ok
Jun 3 19:43:07 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=27:9:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_120000 )
Jun 3 19:43:07 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:07 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=11, rc=0, cib-update=27,
confirmed=false) complete ok
Jun 3 19:43:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:08 srv-dell haclient: on_event:evt:cib_changed
Jun 3 19:43:08 srv-dell mgmtd: [11974]: info: CIB query: cib
Jun 3 19:44:03 srv-dell cib: [11969]: info: cib_stats: Processed 99
operations (707.00us average, 0% utilization) in the last 10min
Jun 3 19:45:17 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 3 19:45:17 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 3 19:45:17 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 3 19:45:17 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 3 20:05:17 srv-dell -- MARK --
Jun 3 20:21:48 srv-dell init: Switching to runlevel: 0
Jun 3 20:21:50 srv-dell libvirtd: Shutting down on signal 15
Jun 3 20:21:50 srv-dell auditd[4218]: Error sending signal_info
request (Operation not supported)
Jun 3 20:21:50 srv-dell auditd[4218]: The audit daemon is exiting.
Jun 3 20:21:50 srv-dell bonobo-activation-server (root-14458): could
not associate with desktop session: Failed to connect to socket
/tmp/dbus-Kj5yx4dbVW: Connection refused
Jun 3 20:21:50 srv-dell openais[11962]: [SERV ] Unloading all openais
components
Jun 3 20:21:50 srv-dell openais[11962]: [SERV ] Unloading slot 10:
openais cluster config database access v1.01
Jun 3 20:21:50 srv-dell openais[11962]: [SERV ] Unloading openais
component: openais_confdb v0 (16/10)
Jun 3 20:21:50 srv-dell openais[11962]: [SERV ] Unloading slot 9:
Pacemaker Cluster Manager
Jun 3 20:21:50 srv-dell openais[11962]: [SERV ] Unloading openais
component: pacemaker v0 (2/9)
Jun 3 20:21:50 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: Begining shutdown
Jun 3 20:21:50 srv-dell openais[11962]: [MAIN ] notice: stop_child:
Sent -15 to mgmtd: [11974]
Jun 3 20:21:50 srv-dell mgmtd: [11974]: info: mgmtd is shutting down
Jun 3 20:21:50 srv-dell mgmtd: [11974]: debug: [mgmtd] stopped
Jun 3 20:21:51 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: mgmtd (pid=11974) confirmed dead
Jun 3 20:21:51 srv-dell openais[11962]: [MAIN ] notice: stop_child:
Sent -15 to crmd: [11973]
Jun 3 20:21:51 srv-dell crmd: [11973]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 20:21:51 srv-dell crmd: [11973]: info: crm_shutdown: Requesting shutdown
Jun 3 20:21:51 srv-dell crmd: [11973]: info: do_shutdown_req: Sending
shutdown request to DC: srv-lenovo
Jun 3 20:21:51 srv-dell attrd: [11971]: info: attrd_ais_dispatch:
Update relayed from srv-lenovo
Jun 3 20:21:51 srv-dell attrd: [11971]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 3 20:21:51 srv-dell attrd: [11971]: info: attrd_perform_update:
Sent update 10: shutdown=1244075044
Jun 3 20:21:51 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=26:12:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_stop_0 )
Jun 3 20:21:51 srv-dell lrmd: [11970]: info: rsc:ocb2:0: stop
Jun 3 20:21:51 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=11, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 20:21:51 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 3 20:21:51 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/03_20:21:51 INFO: Stopping ocb2:0
Jun 3 20:21:51 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 3 20:21:51 srv-dell lrmd: [11970]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/03_20:21:51 INFO: Stopping
ocfs2_controld.pcmk
Jun 3 20:21:51 srv-dell ocfs2_controld: Caught signal 15
Jun 3 20:21:51 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 3 20:21:51 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 3 20:21:51 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 3 20:21:51 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 3 20:21:51 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 3 20:21:51 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 3 20:21:51 srv-dell ocfs2_controld: closing cpg connection
Jun 3 20:21:51 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 3 20:21:51 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 3 20:21:51 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 3 20:21:51 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 3 20:21:51 srv-dell ocfs2_controld: closing pacemaker connection
Jun 3 20:21:51 srv-dell ocfs2_controld: [12218]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 3 20:21:52 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 3 20:21:52 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation ocb2:0_stop_0 (call=12, rc=0, cib-update=28, confirmed=true)
complete ok
Jun 3 20:21:52 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=19:12:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=dlm:0_stop_0 )
Jun 3 20:21:52 srv-dell lrmd: [11970]: info: rsc:dlm:0: stop
Jun 3 20:21:52 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=9, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 3 20:21:52 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/1543678144"
Jun 3 20:21:52 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 3 20:21:52 srv-dell dlm_controld.pcmk: [12095]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 3 20:21:52 srv-dell kernel: dlm: closing connection to node 1543678144
Jun 3 20:21:52 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 3 20:21:52 srv-dell lrmd: [11970]: info: RA output:
(dlm:0:stop:stderr) dlm_controld.pcmk: no process killed
Jun 3 20:21:53 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation dlm:0_stop_0 (call=13, rc=0, cib-update=29, confirmed=true)
complete ok
Jun 3 20:21:53 srv-dell crmd: [11973]: info: do_lrm_rsc_op:
Performing key=14:12:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_stop_0 )
Jun 3 20:21:53 srv-dell lrmd: [11970]: info:
rsc:cloneresource_cluster_fencing_null:1: stop
Jun 3 20:21:53 srv-dell lrmd: [15135]: info: Try to stop STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 3 20:21:53 srv-dell crmd: [11973]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_stop_0 (call=14, rc=0,
cib-update=30, confirmed=true) complete ok
Jun 3 20:21:53 srv-dell crmd: [11973]: info: handle_request: Shutting down
Jun 3 20:21:53 srv-dell crmd: [11973]: info: do_state_transition:
State transition S_NOT_DC -> S_STOPPING [ input=I_STOP
cause=C_HA_MESSAGE origin=route_message ]
Jun 3 20:21:53 srv-dell crmd: [11973]: info: do_lrm_control:
Disconnected from the LRM
Jun 3 20:21:53 srv-dell crmd: [11973]: info: do_ha_control:
Disconnected from OpenAIS
Jun 3 20:21:53 srv-dell crmd: [11973]: info: do_cib_control: Disconnecting CIB
Jun 3 20:21:53 srv-dell crmd: [11973]: info:
crmd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 20:21:53 srv-dell crmd: [11973]: info: do_exit: Performing
A_EXIT_0 - gracefully exiting the CRMd
Jun 3 20:21:53 srv-dell crmd: [11973]: info: free_mem: Dropping
I_TERMINATE: [ state=S_STOPPING cause=C_FSA_INTERNAL origin=do_stop ]
Jun 3 20:21:53 srv-dell crmd: [11973]: info: do_exit: [crmd] stopped (0)
Jun 3 20:21:53 srv-dell openais[11962]: [crm ] info: pcmk_ipc_exit:
Client crmd (conn=0x77c4b0, async-conn=0x77c4b0) left
Jun 3 20:21:53 srv-dell cib: [11969]: WARN: send_ipc_message: IPC
Channel to 11973 is not connected
Jun 3 20:21:53 srv-dell cib: [11969]: WARN:
send_via_callback_channel: Delivery of reply to client
11973/aa42f630-a252-43db-b805-68f680294566 failed
Jun 3 20:21:53 srv-dell cib: [11969]: WARN: do_local_notify: A-Sync
reply to crmd failed: reply failed
Jun 3 20:21:53 srv-dell cib: [11969]: WARN: send_ipc_message: IPC
Channel to 11973 is not connected
Jun 3 20:21:53 srv-dell cib: [11969]: WARN:
send_via_callback_channel: Delivery of reply to client
11973/aa42f630-a252-43db-b805-68f680294566 failed
Jun 3 20:21:53 srv-dell cib: [11969]: WARN: do_local_notify: A-Sync
reply to crmd failed: reply failed
Jun 3 20:21:53 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: crmd (pid=11973) confirmed dead
Jun 3 20:21:53 srv-dell openais[11962]: [MAIN ] notice: stop_child:
Sent -15 to pengine: [11972]
Jun 3 20:21:53 srv-dell pengine: [11972]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 20:21:54 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: pengine (pid=11972) confirmed dead
Jun 3 20:21:54 srv-dell openais[11962]: [MAIN ] notice: stop_child:
Sent -15 to attrd: [11971]
Jun 3 20:21:54 srv-dell attrd: [11971]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 20:21:54 srv-dell attrd: [11971]: info: attrd_shutdown: Exiting
Jun 3 20:21:54 srv-dell attrd: [11971]: info: main: Exiting...
Jun 3 20:21:54 srv-dell attrd: [11971]: info:
attrd_cib_connection_destroy: Connection to the CIB terminated...
Jun 3 20:21:54 srv-dell openais[11962]: [crm ] info: pcmk_ipc_exit:
Client attrd (conn=0x77cca0, async-conn=0x77cca0) left
Jun 3 20:21:55 srv-dell lrmd: [11970]: info: lrmd is shutting down
Jun 3 20:21:55 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: attrd (pid=11971) confirmed dead
Jun 3 20:21:55 srv-dell openais[11962]: [MAIN ] notice: stop_child:
Sent -15 to lrmd: [11970]
Jun 3 20:21:56 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: lrmd (pid=11970) confirmed dead
Jun 3 20:21:56 srv-dell openais[11962]: [MAIN ] notice: stop_child:
Sent -15 to cib: [11969]
Jun 3 20:21:56 srv-dell cib: [11969]: info: crm_signal_dispatch:
Invoking handler for signal 15: Terminated
Jun 3 20:21:56 srv-dell cib: [11969]: info: cib_shutdown:
Disconnected 0 clients
Jun 3 20:21:56 srv-dell cib: [11969]: info: cib_process_disconnect:
All clients disconnected...
Jun 3 20:21:56 srv-dell cib: [11969]: info: initiate_exit: Sending
disconnect notification to 2 peers...
Jun 3 20:21:56 srv-dell cib: [11969]: info: cib_process_shutdown_req:
Shutdown ACK from srv-lenovo
Jun 3 20:21:56 srv-dell cib: [11969]: info:
cib_ha_connection_destroy: Heartbeat disconnection complete... exiting
Jun 3 20:21:56 srv-dell cib: [11969]: info:
cib_ha_connection_destroy: Exiting...
Jun 3 20:21:56 srv-dell cib: [11969]: info: cib_process_request:
Operation complete: op cib_shutdown_req for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.0.0): ok (rc=0)
Jun 3 20:21:56 srv-dell cib: [11969]: info: main: Done
Jun 3 20:21:56 srv-dell openais[11962]: [crm ] info: pcmk_ipc_exit:
Client cib (conn=0x77c8f0, async-conn=0x77c8f0) left
Jun 3 20:21:57 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: cib (pid=11969) confirmed dead
Jun 3 20:21:57 srv-dell openais[11962]: [MAIN ] notice: stop_child:
Sent -15 to stonithd: [11968]
Jun 3 20:21:57 srv-dell stonithd: [11968]: notice:
/usr/lib64/heartbeat/stonithd normally quit.
Jun 3 20:21:57 srv-dell openais[11962]: [crm ] info: pcmk_ipc_exit:
Client stonithd (conn=0x77cb20, async-conn=0x77cb20) left
Jun 3 20:21:58 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: stonithd (pid=11968) confirmed dead
Jun 3 20:21:58 srv-dell openais[11962]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000000002
(2)
Jun 3 20:21:58 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: Shutdown complete
Jun 3 20:21:58 srv-dell openais[11962]: [crm ] notice:
pcmk_shutdown: Forcing clean exit of OpenAIS
Jun 3 20:21:59 srv-dell kernel: drbd0: conn( WFConnection -> Disconnecting )
Jun 3 20:21:59 srv-dell kernel: drbd0: Discarding network configuration.
Jun 3 20:21:59 srv-dell kernel: drbd0: Connection closed
Jun 3 20:21:59 srv-dell kernel: drbd0: conn( Disconnecting -> StandAlone )
Jun 3 20:21:59 srv-dell kernel: drbd0: receiver terminated
Jun 3 20:21:59 srv-dell kernel: drbd0: Terminating receiver thread
Jun 3 20:21:59 srv-dell kernel: drbd0: drbd_bm_resize called with capacity == 0
Jun 3 20:21:59 srv-dell kernel: drbd0: worker terminated
Jun 3 20:21:59 srv-dell kernel: drbd0: Terminating worker thread
Jun 3 20:21:59 srv-dell kernel: drbd: module cleanup done.
Jun 3 20:21:59 srv-dell sshd[4925]: Received signal 15; terminating.
Jun 3 20:21:59 srv-dell rpcbind: rpcbind terminating on signal.
Restart with "rpcbind -w"
Jun 3 20:21:59 srv-dell kernel: Kernel logging (proc) stopped.
Jun 3 20:21:59 srv-dell kernel: Kernel log daemon terminating.
Jun 3 20:21:59 srv-dell syslog-ng[3184]: Termination requested via
signal, terminating;
Jun 3 20:21:59 srv-dell syslog-ng[3184]: syslog-ng shutting down;
version='2.0.9'
Jun 4 11:02:30 srv-dell syslog-ng[3095]: syslog-ng starting up; version='2.0.9'
Jun 4 11:02:30 srv-dell firmware.sh[3094]: Cannot find firmware file
'intel-ucode/06-0f-07'
Jun 4 11:02:31 srv-dell rchal: CPU frequency scaling is not supported
by your processor.
Jun 4 11:02:31 srv-dell rchal: boot with 'CPUFREQ=no' in to avoid this warning.
Jun 4 11:02:31 srv-dell rchal: Cannot load cpufreq governors - No
cpufreq driver available
Jun 4 11:02:31 srv-dell ifup: lo
Jun 4 11:02:31 srv-dell ifup: lo
Jun 4 11:02:31 srv-dell ifup: IP address: 127.0.0.1/8
Jun 4 11:02:31 srv-dell ifup:
Jun 4 11:02:31 srv-dell ifup:
Jun 4 11:02:31 srv-dell ifup: IP address: 127.0.0.2/8
Jun 4 11:02:31 srv-dell ifup:
Jun 4 11:02:32 srv-dell ifup: eth0 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 4 11:02:32 srv-dell ifup: No configuration found for eth0
Jun 4 11:02:32 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 4 11:02:32 srv-dell ifup-dhcp: eth1 Starting DHCP4 client
Jun 4 11:02:32 srv-dell dhcpcd[3620]: eth1: dhcpcd 3.2.3 starting
Jun 4 11:02:32 srv-dell dhcpcd[3620]: eth1: hardware address =
00:19:b9:d7:20:d7
Jun 4 11:02:32 srv-dell dhcpcd[3620]: eth1: broadcasting for a lease
Jun 4 11:02:33 srv-dell ifup-dhcp: .
Jun 4 11:02:35 srv-dell ifup-dhcp: .
Jun 4 11:02:35 srv-dell kernel: klogd 1.4.1, log source = /proc/kmsg started.
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127748.963:2):
operation="profile_load" name="/bin/ping" name2="default" pid=2876
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127748.992:3):
operation="profile_load" name="/sbin/klogd" name2="default" pid=2885
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.048:4):
operation="profile_load" name="/sbin/syslog-ng" name2="default"
pid=2886
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.106:5):
operation="profile_load" name="/sbin/syslogd" name2="default" pid=2892
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.168:6):
operation="profile_load" name="/usr/sbin/avahi-daemon" name2="default"
pid=2917
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.218:7):
operation="profile_load" name="/usr/sbin/identd" name2="default"
pid=2934
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.274:8):
operation="profile_load" name="/usr/sbin/mdnsd" name2="default"
pid=2946
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.338:9):
operation="profile_load" name="/usr/sbin/nscd" name2="default"
pid=2949
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.431:10):
operation="profile_load" name="/usr/sbin/ntpd" name2="default"
pid=2950
Jun 4 11:02:35 srv-dell kernel: type=1505 audit(1244127749.490:11):
operation="profile_load" name="/usr/sbin/traceroute" name2="default"
pid=2951
Jun 4 11:02:35 srv-dell kernel: IA-32 Microcode Update Driver:
v1.14a-xen <tigran at aivazian.fsnet.co.uk>
Jun 4 11:02:35 srv-dell kernel: firmware: requesting intel-ucode/06-0f-07
Jun 4 11:02:35 srv-dell kernel: bnx2: eth1: using MSI
Jun 4 11:02:35 srv-dell kernel: NET: Registered protocol family 17
Jun 4 11:02:35 srv-dell kernel: bnx2: eth1 NIC Copper Link is Up,
1000 Mbps full duplex, receive & transmit flow control ON
Jun 4 11:02:37 srv-dell ifup-dhcp: .
Jun 4 11:02:38 srv-dell dhcpcd[3620]: eth1: offered 192.168.2.220
from 192.168.2.1
Jun 4 11:02:38 srv-dell dhcpcd[3620]: eth1: checking 192.168.2.220 is
available on attached networks
Jun 4 11:02:39 srv-dell ifup-dhcp: .
Jun 4 11:02:39 srv-dell dhcpcd[3620]: eth1: leased 192.168.2.220 for
86400 seconds
Jun 4 11:02:39 srv-dell dhcpcd[3620]: eth1: no renewal time supplied,
assuming 43200 seconds
Jun 4 11:02:39 srv-dell dhcpcd[3620]: eth1: no rebind time supplied,
assuming 75600 seconds
Jun 4 11:02:39 srv-dell dhcpcd[3620]: eth1: adding IP address 192.168.2.220/24
Jun 4 11:02:39 srv-dell dhcpcd[3620]: eth1: adding default route via
192.168.2.1 metric 0
Jun 4 11:02:39 srv-dell dns-resolver: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 4 11:02:39 srv-dell dns-resolver: You can find my version in
/etc/resolv.conf.netconfig
Jun 4 11:02:39 srv-dell dhcpcd-hook: ATTENTION: You have modified
/etc/resolv.conf. Leaving it untouched...
Jun 4 11:02:39 srv-dell dhcpcd-hook: You can find my version in
/etc/resolv.conf.netconfig ...
Jun 4 11:02:39 srv-dell syslog-ng[3095]: Configuration reload request
received, reloading configuration;
Jun 4 11:02:39 srv-dell syslog-ng[3095]: New configuration initialized;
Jun 4 11:02:40 srv-dell ifdown: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 4 11:02:40 srv-dell ifup: eth1 device: Broadcom
Corporation NetXtreme II BCM5708 Gigabit Ethernet (rev 12)
Jun 4 11:02:40 srv-dell ifup-dhcp:
Jun 4 11:02:40 srv-dell ifup-dhcp: eth1 IP address: 192.168.2.220/24
Jun 4 11:02:40 srv-dell SuSEfirewall2: SuSEfirewall2 not active
Jun 4 11:02:40 srv-dell auditd[4130]: Started dispatcher:
/sbin/audispd pid: 4132
Jun 4 11:02:41 srv-dell audispd: priority_boost_parser called with: 4
Jun 4 11:02:41 srv-dell audispd: af_unix plugin initialized
Jun 4 11:02:41 srv-dell audispd: audispd initialized with q_depth=80
and 1 active plugins
Jun 4 11:02:41 srv-dell auditd[4130]: Init complete, auditd 1.7.7
listening for events (startup state disable)
Jun 4 11:02:41 srv-dell kernel: NET: Registered protocol family 10
Jun 4 11:02:41 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 4 11:02:41 srv-dell kernel: lo: Disabled Privacy Extensions
Jun 4 11:02:41 srv-dell dhcpcd[3620]: eth1: setting hostname to `srv-dell'
Jun 4 11:02:41 srv-dell dhcpcd[3620]: eth1: exiting
Jun 4 11:02:42 srv-dell smartd[4520]: smartd 5.39 2008-10-24 22:33
[x86_64-suse-linux-gnu] (openSUSE RPM) Copyright (C) 2002-8 by Bruce
Allen, http://smartmontools.sourceforge.net
Jun 4 11:02:42 srv-dell smartd[4520]: Opened configuration file
/etc/smartd.conf
Jun 4 11:02:42 srv-dell smartd[4520]: Drive: DEVICESCAN, implied '-a'
Directive on line 26 of file /etc/smartd.conf
Jun 4 11:02:42 srv-dell smartd[4520]: Configuration file
/etc/smartd.conf was parsed, found DEVICESCAN, scanning devices
Jun 4 11:02:42 srv-dell smartd[4520]: Device: /dev/sda, opened
Jun 4 11:02:42 srv-dell smartd[4520]: Device: /dev/sda, IE (SMART)
not enabled, skip device Try 'smartctl -s on /dev/sda' to turn on
SMART features
Jun 4 11:02:42 srv-dell smartd[4520]: Unable to monitor any SMART
enabled devices. Try debug (-d) option. Exiting...
Jun 4 11:02:42 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 4 11:02:42 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 4 11:02:42 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 4 11:02:42 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 4 11:02:42 srv-dell sshd[4773]: Server listening on 0.0.0.0 port 22.
Jun 4 11:02:42 srv-dell sshd[4773]: Server listening on :: port 22.
Jun 4 11:02:42 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/Xservers
Jun 4 11:02:42 srv-dell logger: /etc/init.d/xdm: No changes for
/etc/X11/xdm/xdm-config
Jun 4 11:02:42 srv-dell kernel: drbd: initialised. Version: 8.2.7
(api:88/proto:86-88)
Jun 4 11:02:42 srv-dell kernel: drbd: GIT-hash:
a1b440e8b3011a1318d8bff1bb7edc763ef995b0 build by lmb at hermes,
2009-02-20 13:35:59
Jun 4 11:02:42 srv-dell kernel: drbd: registered as block device major 147
Jun 4 11:02:42 srv-dell kernel: drbd: minor_table @ 0xffff8801ea658ec0
Jun 4 11:02:42 srv-dell kernel: drbd0: conn( StandAlone -> Unconnected )
Jun 4 11:02:42 srv-dell kernel: drbd0: Starting worker thread (from
cqueue [48])
Jun 4 11:02:42 srv-dell kernel: klogd 1.4.1, ---------- state change ----------
Jun 4 11:02:42 srv-dell kernel: drbd0: Starting receiver thread (from
drbd0_worker [4841])
Jun 4 11:02:42 srv-dell kernel: drbd0: receiver (re)started
Jun 4 11:02:42 srv-dell kernel: drbd0: conn( Unconnected -> WFConnection )
Jun 4 11:02:43 srv-dell kernel: vendor=8086 device=244e
Jun 4 11:02:43 srv-dell kernel: pci 0000:10:0d.0: PCI INT A -> GSI 19
(level, low) -> IRQ 19
Jun 4 11:02:43 srv-dell /usr/sbin/cron[4917]: (CRON) STARTUP (V5.0)
Jun 4 11:02:43 srv-dell kernel: Loading iSCSI transport class v2.0-870.
Jun 4 11:02:43 srv-dell kernel: iscsi: registered transport (tcp)
Jun 4 11:02:43 srv-dell kernel: iscsi: registered transport (iser)
Jun 4 11:02:43 srv-dell iscsid: iSCSI logger with pid=5044 started!
Jun 4 11:02:43 srv-dell xenstored: Checking store ...
Jun 4 11:02:43 srv-dell xenstored: Checking store complete.
Jun 4 11:02:43 srv-dell kernel: suspend: event channel 55
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:795: blktapctrl: v1.0.0
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [raw image (aio)]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [raw image (sync)]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [vmware image (vmdk)]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [ramdisk image (ram)]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [qcow disk (qcow)]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [qcow2 disk (qcow2)]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [ioemu disk]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl.c:797: Found
driver: [raw image (cdrom)]
Jun 4 11:02:43 srv-dell BLKTAPCTRL[5120]: blktapctrl_linux.c:23:
/dev/xen/blktap0 device already exists
Jun 4 11:02:44 srv-dell kernel: Bridge firewalling registered
Jun 4 11:02:44 srv-dell iscsid: transport class version 2.0-870.
iscsid version 2.0-870
Jun 4 11:02:44 srv-dell iscsid: iSCSI daemon with pid=5045 started!
Jun 4 11:02:49 srv-dell gdm-simple-greeter[5263]: libglade-WARNING:
Unexpected element <requires-version> inside <glade-interface>.
Jun 4 11:02:50 srv-dell gdm-session-worker[5265]: PAM pam_putenv:
NULL pam handle passed
Jun 4 11:02:51 srv-dell kernel: eth1: no IPv6 routers present
Jun 4 11:03:08 srv-dell python: hp-systray(init)[5429]: error:
hp-systray cannot be run as root. Exiting.
Jun 4 11:03:09 srv-dell kernel: kjournald starting. Commit interval 5 seconds
Jun 4 11:03:09 srv-dell kernel: EXT3 FS on sda4, internal journal
Jun 4 11:03:09 srv-dell kernel: EXT3-fs: mounted filesystem with
ordered data mode.
Jun 4 11:03:09 srv-dell hald: mounted /dev/sda4 on behalf of uid 0
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] AIS Executive Service
RELEASE 'subrev 1152 version 0.80'
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] Copyright (C)
2002-2006 MontaVista Software, Inc and contributors.
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] Copyright (C) 2006 Red Hat, Inc.
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] AIS Executive Service:
started and ready to provide service.
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] Token Timeout (5000
ms) retransmit timeout (490 ms)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] token hold (382 ms)
retransmits before loss (10 retrans)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] join (1000 ms)
send_join (45 ms) consensus (2500 ms) merge (200 ms)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] downcheck (1000 ms)
fail to recv const (50 msgs)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] seqno unchanged const
(30 rotations) Maximum network MTU 1500
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] window size per
rotation (50 messages) maximum messages per rotation (20 messages)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] send threads (0 threads)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] RRP token expired
timeout (490 ms)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] RRP token problem
counter (2000 ms)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] RRP threshold (10 problem count)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] RRP mode set to none.
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] heartbeat_failures_allowed (0)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] max_network_delay (50 ms)
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] HeartBeat is Disabled.
To enable set heartbeat_failures_allowed > 0
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] The network interface
[192.168.2.220] is now up.
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] Created or loaded
sequence id 256.192.168.2.220 for this ring.
Jun 4 11:03:36 srv-dell openais[5529]: [TOTEM] entering GATHER state from 15.
Jun 4 11:03:36 srv-dell openais[5529]: [crm ] info:
process_ais_conf: Reading configure
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info:
config_find_next: Processing additional logging options...
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: get_config_opt:
Found 'off' for option: debug
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: get_config_opt:
Found 'yes' for option: to_syslog
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: get_config_opt:
Found 'daemon' for option: syslog_facility
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info:
config_find_next: Processing additional service options...
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_logd
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: get_config_opt:
Found 'yes' for option: use_mgmtd
Jun 4 11:03:36 srv-dell openais[5529]: [crm ] info:
pcmk_plugin_init: CRM: Initialized
Jun 4 11:03:36 srv-dell openais[5529]: [crm ] Logging: Initialized
pcmk_plugin_init
Jun 4 11:03:36 srv-dell openais[5529]: [crm ] info:
pcmk_plugin_init: Service: 9
Jun 4 11:03:36 srv-dell openais[5529]: [crm ] info:
pcmk_plugin_init: Local node id: 1543678144
Jun 4 11:03:36 srv-dell openais[5529]: [crm ] info:
pcmk_plugin_init: Local hostname: srv-dell
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: update_member:
Creating entry for node 1543678144 born on 0
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: update_member:
0x740380 Node 1543678144 now known as srv-dell (was: (null))
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: update_member:
Node srv-dell now has 1 quorum votes (was 0)
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: update_member:
Node 1543678144/srv-dell is now: member
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: spawn_child:
Forked child 5535 for process stonithd
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: spawn_child:
Forked child 5536 for process cib
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: spawn_child:
Forked child 5537 for process lrmd
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: spawn_child:
Forked child 5538 for process attrd
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: spawn_child:
Forked child 5539 for process pengine
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: spawn_child:
Forked child 5540 for process crmd
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] info: spawn_child:
Forked child 5541 for process mgmtd
Jun 4 11:03:36 srv-dell openais[5529]: [crm ] info: pcmk_startup:
CRM: Initialized
Jun 4 11:03:36 srv-dell openais[5529]: [MAIN ] Service initialized
'Pacemaker Cluster Manager'
Jun 4 11:03:36 srv-dell openais[5529]: [SERV ] Service initialized
'openais extended virtual synchrony service'
Jun 4 11:03:36 srv-dell openais[5529]: [SERV ] Service initialized
'openais cluster membership service B.01.01'
Jun 4 11:03:36 srv-dell openais[5529]: [SERV ] Service initialized
'openais availability management framework B.01.01'
Jun 4 11:03:36 srv-dell openais[5529]: [SERV ] Service initialized
'openais checkpoint service B.01.01'
Jun 4 11:03:36 srv-dell openais[5529]: [SERV ] Service initialized
'openais event service B.01.01'
Jun 4 11:03:36 srv-dell openais[5529]: [SERV ] Service initialized
'openais distributed locking service B.01.01'
Jun 4 11:03:37 srv-dell openais[5529]: [SERV ] Service initialized
'openais message service B.01.01'
Jun 4 11:03:37 srv-dell openais[5529]: [SERV ] Service initialized
'openais configuration service'
Jun 4 11:03:37 srv-dell openais[5529]: [SERV ] Service initialized
'openais cluster closed process group service v1.01'
Jun 4 11:03:37 srv-dell openais[5529]: [SERV ] Service initialized
'openais cluster config database access v1.01'
Jun 4 11:03:37 srv-dell openais[5529]: [SYNC ] Not using a virtual
synchrony filter.
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] Creating commit token
because I am the rep.
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] Saving state aru 0
high seq received 0
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] Storing new sequence
id for ring 104
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] entering COMMIT state.
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] entering RECOVERY state.
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] position [0] member
192.168.2.220:
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] previous ring seq 256
rep 192.168.2.220
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] aru 0 high delivered 0
received flag 1
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] Sending initial ORF token
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 260: memb=0,
new=0, lost=0
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 260: memb=1, new=1,
lost=0
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: NEW: srv-dell 1543678144
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 4 11:03:37 srv-dell openais[5529]: [MAIN ] info: update_member:
Node srv-dell now has process list: 00000000000000000000000000053312
(340754)
Jun 4 11:03:37 srv-dell openais[5529]: [SYNC ] This node is within
the primary component and will provide service.
Jun 4 11:03:37 srv-dell openais[5529]: [TOTEM] entering OPERATIONAL state.
Jun 4 11:03:37 srv-dell openais[5529]: [CLM ] got nodejoin message
192.168.2.220
Jun 4 11:03:37 srv-dell lrmd: [5537]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:37 srv-dell lrmd: [5537]: info: G_main_add_SignalHandler:
Added signal handler for signal 15
Jun 4 11:03:37 srv-dell lrmd: [5537]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 4 11:03:37 srv-dell mgmtd: [5541]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:37 srv-dell mgmtd: [5541]: info:
G_main_add_SignalHandler: Added signal handler for signal 15
Jun 4 11:03:37 srv-dell mgmtd: [5541]: debug: Enabling coredumps
Jun 4 11:03:37 srv-dell lrmd: [5537]: info: G_main_add_SignalHandler:
Added signal handler for signal 10
Jun 4 11:03:37 srv-dell lrmd: [5537]: info: G_main_add_SignalHandler:
Added signal handler for signal 12
Jun 4 11:03:37 srv-dell mgmtd: [5541]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 4 11:03:37 srv-dell lrmd: [5537]: info: Started.
Jun 4 11:03:37 srv-dell mgmtd: [5541]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 4 11:03:37 srv-dell attrd: [5538]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 4 11:03:37 srv-dell pengine: [5539]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 4 11:03:37 srv-dell cib: [5536]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 4 11:03:37 srv-dell attrd: [5538]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:37 srv-dell pengine: [5539]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:37 srv-dell stonithd: [5535]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:37 srv-dell attrd: [5538]: info: main: Starting up....
Jun 4 11:03:37 srv-dell cib: [5536]: WARN: Initializing connection to
logging daemon failed. Logging daemon may not be running
Jun 4 11:03:37 srv-dell stonithd: [5535]: info:
G_main_add_SignalHandler: Added signal handler for signal 10
Jun 4 11:03:37 srv-dell mgmtd: [5541]: info: init_crm
Jun 4 11:03:37 srv-dell stonithd: [5535]: info:
G_main_add_SignalHandler: Added signal handler for signal 12
Jun 4 11:03:37 srv-dell attrd: [5538]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 4 11:03:37 srv-dell crmd: [5540]: info: crm_log_init: Changed
active directory to /var/lib/heartbeat/cores/hacluster
Jun 4 11:03:37 srv-dell cib: [5536]: info: G_main_add_TriggerHandler:
Added signal manual handler
Jun 4 11:03:37 srv-dell attrd: [5538]: info: init_ais_connection: AIS
connection established
Jun 4 11:03:37 srv-dell mgmtd: [5541]: info: login to cib: 0, ret:-10
Jun 4 11:03:37 srv-dell crmd: [5540]: WARN: Initializing connection
to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:37 srv-dell pengine: [5539]: info: main: Starting pengine
Jun 4 11:03:37 srv-dell cib: [5536]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f5fa00340b0 for attrd/5538
Jun 4 11:03:37 srv-dell crmd: [5540]: info: main: CRM Hg Version:
0080ec086ae9c20ad5c4c3562000c0ad68374f0a
Jun 4 11:03:37 srv-dell stonithd: [5535]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 4 11:03:37 srv-dell attrd: [5538]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 4 11:03:37 srv-dell crmd: [5540]: info: crmd_init: Starting crmd
Jun 4 11:03:37 srv-dell cib: [5536]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
/var/lib/heartbeat/crm/cib.xml.sig)
Jun 4 11:03:37 srv-dell attrd: [5538]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:03:37 srv-dell attrd: [5538]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:03:37 srv-dell crmd: [5540]: info: G_main_add_SignalHandler:
Added signal handler for signal 17
Jun 4 11:03:37 srv-dell stonithd: [5535]: info: init_ais_connection:
AIS connection established
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f5fa0034380 for stonithd/5535
Jun 4 11:03:37 srv-dell stonithd: [5535]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:03:37 srv-dell stonithd: [5535]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:03:37 srv-dell stonithd: [5535]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:03:37 srv-dell stonithd: [5535]: notice:
/usr/lib64/heartbeat/stonithd start up successfully.
Jun 4 11:03:37 srv-dell stonithd: [5535]: info:
G_main_add_SignalHandler: Added signal handler for signal 17
Jun 4 11:03:37 srv-dell cib: [5536]: info: startCib: CIB
Initialization completed successfully
Jun 4 11:03:37 srv-dell cib: [5536]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 4 11:03:37 srv-dell cib: [5536]: info: init_ais_connection: AIS
connection established
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f5fa00346d0 for cib/5536
Jun 4 11:03:37 srv-dell openais[5529]: [crm ] info: pcmk_ipc:
Sending membership update 260 to cib
Jun 4 11:03:37 srv-dell cib: [5536]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 4 11:03:37 srv-dell cib: [5536]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:03:37 srv-dell cib: [5536]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:03:37 srv-dell cib: [5536]: info: cib_init: Starting cib mainloop
Jun 4 11:03:37 srv-dell cib: [5536]: info: ais_dispatch: Membership
260: quorum still lost
Jun 4 11:03:37 srv-dell cib: [5536]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=0 seen=260
proc=00000000000000000000000000053312 (new)
Jun 4 11:03:37 srv-dell cib: [5545]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-94.raw
Jun 4 11:03:37 srv-dell cib: [5545]: info: write_cib_contents: Wrote
version 0.196.0 of the CIB to disk (digest:
39615822b53b6896686c371045ae331f)
Jun 4 11:03:37 srv-dell cib: [5545]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.CR1JTf (digest:
/var/lib/heartbeat/crm/cib.t5XE08)
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] entering GATHER state from 11.
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] Saving state aru a
high seq received a
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] Storing new sequence
id for ring 108
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] entering COMMIT state.
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] entering RECOVERY state.
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] position [0] member
192.168.2.120:
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] previous ring seq 260
rep 192.168.2.120
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] aru 55 high delivered
55 received flag 1
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] position [1] member
192.168.2.220:
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] previous ring seq 260
rep 192.168.2.220
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] aru a high delivered a
received flag 1
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 264: memb=1,
new=0, lost=0
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.120)
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:03:38 srv-dell cib: [5536]: info: ais_dispatch: Membership
264: quorum still lost
Jun 4 11:03:38 srv-dell cib: [5536]: info: crm_new_peer: Node <null>
now has id: 2013440192
Jun 4 11:03:38 srv-dell cib: [5536]: info: crm_update_peer: Node
(null): id=2013440192 state=member (new) addr=r(0) ip(192.168.2.120)
votes=0 born=0 seen=264 proc=00000000000000000000000000000000
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.120)
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 264: memb=2, new=1,
lost=0
Jun 4 11:03:38 srv-dell openais[5529]: [MAIN ] info: update_member:
Creating entry for node 2013440192 born on 264
Jun 4 11:03:38 srv-dell openais[5529]: [MAIN ] info: update_member:
Node 2013440192/unknown is now: member
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: NEW: .pending. 2013440192
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: MEMB: .pending. 2013440192
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info:
send_member_notification: Sending membership update 264 to 1 children
Jun 4 11:03:38 srv-dell openais[5529]: [MAIN ] info: update_member:
0x740380 Node 1543678144 ((null)) born on: 264
Jun 4 11:03:38 srv-dell cib: [5536]: info: ais_dispatch: Membership
264: quorum still lost
Jun 4 11:03:38 srv-dell cib: [5536]: info: crm_get_peer: Node
2013440192 is now known as srv-lenovo
Jun 4 11:03:38 srv-dell cib: [5536]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 (new) born=256 seen=264 proc=00000000000000000000000000053312
(new)
Jun 4 11:03:38 srv-dell openais[5529]: [SYNC ] This node is within
the primary component and will provide service.
Jun 4 11:03:38 srv-dell openais[5529]: [TOTEM] entering OPERATIONAL state.
Jun 4 11:03:38 srv-dell openais[5529]: [MAIN ] info: update_member:
0x7f5fa0034f60 Node 2013440192 (srv-lenovo) born on: 256
Jun 4 11:03:38 srv-dell openais[5529]: [MAIN ] info: update_member:
0x7f5fa0034f60 Node 2013440192 now known as srv-lenovo (was: (null))
Jun 4 11:03:38 srv-dell openais[5529]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 4 11:03:38 srv-dell openais[5529]: [MAIN ] info: update_member:
Node srv-lenovo now has 1 quorum votes (was 0)
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info:
send_member_notification: Sending membership update 264 to 1 children
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] got nodejoin message
192.168.2.120
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 4 11:03:38 srv-dell openais[5529]: [CLM ] got nodejoin message
192.168.2.220
Jun 4 11:03:38 srv-dell cib: [5536]: info: cib_process_diff: Diff
0.207.2 -> 0.207.3 not applied to 0.196.0: current "epoch" is less
than required
Jun 4 11:03:38 srv-dell cib: [5536]: info: cib_server_process_diff:
Requesting re-sync from peer
Jun 4 11:03:38 srv-dell cib: [5536]: WARN: cib_diff_notify:
Local-only Change (client:crmd, call: 212): 0.0.0 (Application of an
update diff failed, requesting a full refresh)
Jun 4 11:03:38 srv-dell cib: [5536]: WARN: cib_server_process_diff:
Not applying diff 0.207.3 -> 0.207.4 (sync in progress)
Jun 4 11:03:38 srv-dell cib: [5536]: WARN: cib_server_process_diff:
Not applying diff 0.207.4 -> 0.207.5 (sync in progress)
Jun 4 11:03:38 srv-dell cib: [5536]: WARN: cib_server_process_diff:
Not applying diff 0.207.5 -> 0.208.1 (sync in progress)
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 4 11:03:38 srv-dell cib: [5536]: info: cib_replace_notify:
Replaced: 0.196.0 -> 0.208.1 from srv-lenovo
Jun 4 11:03:38 srv-dell cib: [5546]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-95.raw
Jun 4 11:03:38 srv-dell cib: [5546]: info: write_cib_contents: Wrote
version 0.208.0 of the CIB to disk (digest:
b6452e87f8273261389c042752fe2b7e)
Jun 4 11:03:38 srv-dell cib: [5546]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.g3UOyb (digest:
/var/lib/heartbeat/crm/cib.C8nZz6)
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] WARN:
route_ais_message: Sending message to local.crmd failed: unknown
(rc=-2)
Jun 4 11:03:38 srv-dell crmd: [5540]: info: do_cib_control: CIB
connection established
Jun 4 11:03:38 srv-dell crmd: [5540]: info: init_ais_connection:
Creating connection to our AIS plugin
Jun 4 11:03:38 srv-dell crmd: [5540]: info: init_ais_connection: AIS
connection established
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info: pcmk_ipc:
Recorded connection 0x7f5fa0034e50 for crmd/5540
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info: pcmk_ipc:
Sending membership update 264 to crmd
Jun 4 11:03:38 srv-dell crmd: [5540]: info: get_ais_nodeid: Server
details: id=1543678144 uname=srv-dell
Jun 4 11:03:38 srv-dell crmd: [5540]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:03:38 srv-dell crmd: [5540]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:03:38 srv-dell crmd: [5540]: info: do_ha_control: Connected
to the cluster
Jun 4 11:03:38 srv-dell crmd: [5540]: info: do_started: Delaying
start, CCM (0000000000100000) not connected
Jun 4 11:03:38 srv-dell crmd: [5540]: info: crmd_init: Starting crmd's mainloop
Jun 4 11:03:38 srv-dell crmd: [5540]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 4 11:03:38 srv-dell openais[5529]: [crm ] info:
update_expected_votes: Expected quorum votes 1024 -> 2
Jun 4 11:03:38 srv-dell crmd: [5540]: info: ais_dispatch: Membership
264: quorum still lost
Jun 4 11:03:38 srv-dell crmd: [5540]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 4 11:03:38 srv-dell crmd: [5540]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 4 11:03:38 srv-dell crmd: [5540]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:03:38 srv-dell crmd: [5540]: info: crm_update_peer: Node
srv-dell: id=1543678144 state=member (new) addr=r(0) ip(192.168.2.220)
(new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:03:38 srv-dell crmd: [5540]: info: do_started: The local CRM
is operational
Jun 4 11:03:38 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_STARTING -> S_PENDING [ input=I_PENDING
cause=C_FSA_INTERNAL origin=do_started ]
Jun 4 11:03:38 srv-dell mgmtd: [5541]: debug: main: run the loop...
Jun 4 11:03:38 srv-dell mgmtd: [5541]: info: Started.
Jun 4 11:03:39 srv-dell crmd: [5540]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:03:39 srv-dell crmd: [5540]: info: update_dc: Set DC to
srv-lenovo (3.0.1)
Jun 4 11:03:39 srv-dell crmd: [5540]: info: update_attrd: Connecting
to attrd...
Jun 4 11:03:39 srv-dell crmd: [5540]: info: update_attrd: Updating
terminate=<none> via attrd for srv-dell
Jun 4 11:03:39 srv-dell crmd: [5540]: info: update_attrd: Updating
shutdown=<none> via attrd for srv-dell
Jun 4 11:03:39 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC
cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
Jun 4 11:03:39 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/transient_attributes": ok
(rc=0)
Jun 4 11:03:39 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 4 11:03:39 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=4:81:7:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_monitor_0 )
Jun 4 11:03:39 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 4 11:03:39 srv-dell lrmd: [5537]: notice: lrmd_rsc_new(): No
lrm_rprovider field in message
Jun 4 11:03:39 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=5:81:7:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_monitor_0 )
Jun 4 11:03:39 srv-dell lrmd: [5537]: info:
rsc:cloneresource_cluster_fencing_null:1: monitor
Jun 4 11:03:39 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=6:81:7:caf53829-4bd1-4932-989e-f155cf0ecf6e op=dlm:0_monitor_0 )
Jun 4 11:03:39 srv-dell lrmd: [5537]: info: rsc:dlm:0: monitor
Jun 4 11:03:39 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=7:81:7:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_monitor_0 )
Jun 4 11:03:39 srv-dell lrmd: [5537]: info: rsc:ocb2:0: monitor
Jun 4 11:03:39 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_monitor_0 (call=3,
rc=7, cib-update=8, confirmed=true) complete not running
Jun 4 11:03:39 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:monitor:stderr) logd is not running
Jun 4 11:03:39 srv-dell lrmd: [5537]: info: RA output:
(dlm:0:monitor:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:03:39 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_monitor_0 (call=4, rc=7, cib-update=9, confirmed=true)
complete not running
Jun 4 11:03:39 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=2, rc=7,
cib-update=10, confirmed=true) complete not running
Jun 4 11:03:39 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:monitor:stderr) 2009/06/04_11:03:39 INFO: configfs not laoded
Jun 4 11:03:39 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_0 (call=5, rc=7, cib-update=11,
confirmed=true) complete not running
Jun 4 11:03:39 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=13:81:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_start_0 )
Jun 4 11:03:39 srv-dell lrmd: [5537]: info:
rsc:cloneresource_cluster_fencing_null:1: start
Jun 4 11:03:39 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=18:81:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=dlm:0_start_0 )
Jun 4 11:03:39 srv-dell lrmd: [5537]: info: rsc:dlm:0: start
Jun 4 11:03:39 srv-dell lrmd: [5597]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 4 11:03:39 srv-dell stonithd: [5535]: info:
cloneresource_cluster_fencing_null:1 stonith resource started
Jun 4 11:03:39 srv-dell lrmd: [5537]: debug: stonithRA plugin:
provider attribute is not needed and will be ignored.
Jun 4 11:03:39 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_start_0 (call=6, rc=0,
cib-update=15, confirmed=true) complete ok
Jun 4 11:03:39 srv-dell lrmd: [5537]: info: RA output:
(dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:03:39 srv-dell kernel: DLM (built Feb 28 2009 07:25:25) installed
Jun 4 11:03:39 srv-dell cluster-dlm[5638]: main: dlm_controld 1239267206
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info:
init_ais_connection: AIS connection established
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:03:39 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 5638 (0x7f5fa0035160)
Jun 4 11:03:39 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 4 11:03:39 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 4 11:03:39 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 4 11:03:39 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 4 11:03:39 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 4 11:03:39 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 2013440192
Jun 4 11:03:39 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 4 11:03:39 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 4 11:03:39 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:03:39 srv-dell cluster-dlm: [5638]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:03:39 srv-dell cluster-dlm: update_cluster: Processing membership 264
Jun 4 11:03:39 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 4 11:03:39 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 4 11:03:39 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=256, last-seen=264, this-event=264,
last-event=0
Jun 4 11:03:39 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 4 11:03:39 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 4 11:03:39 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=264, last-seen=264, this-event=264,
last-event=0
Jun 4 11:03:40 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_start_0 (call=7, rc=0, cib-update=16, confirmed=true)
complete ok
Jun 4 11:03:40 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=19:81:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=dlm:0_monitor_120000 )
Jun 4 11:03:40 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=26:81:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_start_0 )
Jun 4 11:03:40 srv-dell lrmd: [5537]: info: rsc:ocb2:0: start
Jun 4 11:03:40 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=8, rc=0, cib-update=17,
confirmed=false) complete ok
Jun 4 11:03:40 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:03:40 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:03:40 INFO: Stack glue driver not
loaded
Jun 4 11:03:40 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:03:40 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:03:40 INFO: Starting ocb2:0
Jun 4 11:03:40 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 4 11:03:40 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 4 11:03:40 srv-dell kernel: OCFS2 1.5.0
Jun 4 11:03:40 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info:
init_ais_connection: AIS connection established
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 4 11:03:40 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 4 11:03:40 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 5764 (0x7f5fa00359f0)
Jun 4 11:03:40 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 4 11:03:40 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 4 11:03:40 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:03:40 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:03:40 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 4 11:03:40 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:03:40 srv-dell ocfs2_controld: setup done
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:03:40 srv-dell ocfs2_controld: [5764]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:03:40 srv-dell ocfs2_controld: confchg called
Jun 4 11:03:40 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 4 11:03:40 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 4 11:03:40 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 4 11:03:40 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Creating section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Created section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:03:40 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Creating section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:03:40 srv-dell ocfs2_controld: Created section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:03:40 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 4 11:03:40 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 4 11:03:40 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 4 11:03:40 srv-dell ocfs2_controld: Opening control device
Jun 4 11:03:40 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 4 11:03:40 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 4 11:03:40 srv-dell ocfs2_controld: new listening connection 4
Jun 4 11:03:40 srv-dell ocfs2_controld: confchg called
Jun 4 11:03:40 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 4 11:03:42 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_start_0 (call=9, rc=0, cib-update=18, confirmed=true)
complete ok
Jun 4 11:03:42 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=27:81:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_120000 )
Jun 4 11:03:42 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=10, rc=0, cib-update=19,
confirmed=false) complete ok
Jun 4 11:03:47 srv-dell attrd: [5538]: info: main: Sending full refresh
Jun 4 11:03:47 srv-dell attrd: [5538]: info: main: Starting mainloop...
Jun 4 11:03:47 srv-dell attrd: [5538]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 4 11:03:47 srv-dell attrd: [5538]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 4 11:03:47 srv-dell attrd: [5538]: info: find_hash_entry:
Creating hash entry for terminate
Jun 4 11:03:47 srv-dell attrd: [5538]: info: find_hash_entry:
Creating hash entry for shutdown
Jun 4 11:03:47 srv-dell attrd: [5538]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 4 11:03:47 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 4 11:03:47 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 4 11:04:03 srv-dell cib: [5800]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-96.raw
Jun 4 11:04:03 srv-dell cib: [5800]: info: write_cib_contents: Wrote
version 0.209.0 of the CIB to disk (digest:
3b77b9d2a2d64c0871d9fdfdf6869bc9)
Jun 4 11:04:03 srv-dell cib: [5800]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.TZgx8K (digest:
/var/lib/heartbeat/crm/cib.doaURz)
Jun 4 11:04:04 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=26:82:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_stop_0 )
Jun 4 11:04:04 srv-dell lrmd: [5537]: info: rsc:ocb2:0: stop
Jun 4 11:04:04 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=10, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:04:04 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:04:04 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:04:04 INFO: Stopping ocb2:0
Jun 4 11:04:04 srv-dell ocfs2_controld: confchg called
Jun 4 11:04:04 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 1, joined 0
Jun 4 11:04:04 srv-dell ocfs2_controld: node daemon left 2013440192
Jun 4 11:04:04 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:04:04 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:04:04 INFO: Stopping
ocfs2_controld.pcmk
Jun 4 11:04:04 srv-dell ocfs2_controld: Caught signal 15
Jun 4 11:04:04 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:04:04 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:04:04 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 4 11:04:04 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 4 11:04:04 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 4 11:04:04 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 4 11:04:04 srv-dell ocfs2_controld: closing cpg connection
Jun 4 11:04:04 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:04 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 4 11:04:04 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 4 11:04:04 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 4 11:04:04 srv-dell ocfs2_controld: closing pacemaker connection
Jun 4 11:04:04 srv-dell ocfs2_controld: [5764]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:04:04 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 4 11:04:04 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_stop_0 (call=11, rc=0, cib-update=20, confirmed=true)
complete ok
Jun 4 11:04:04 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=19:82:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=dlm:0_stop_0 )
Jun 4 11:04:04 srv-dell lrmd: [5537]: info: rsc:dlm:0: stop
Jun 4 11:04:04 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=8, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:04:04 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/1543678144"
Jun 4 11:04:04 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 4 11:04:04 srv-dell kernel: dlm: closing connection to node 1543678144
Jun 4 11:04:04 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 4 11:04:04 srv-dell dlm_controld.pcmk: [5638]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:04:04 srv-dell lrmd: [5537]: info: RA output:
(dlm:0:stop:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:04:05 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_stop_0 (call=12, rc=0, cib-update=21, confirmed=true)
complete ok
Jun 4 11:04:05 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=14:82:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_stop_0 )
Jun 4 11:04:05 srv-dell lrmd: [5537]: info:
rsc:cloneresource_cluster_fencing_null:1: stop
Jun 4 11:04:05 srv-dell lrmd: [5959]: info: Try to stop STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 4 11:04:05 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_stop_0 (call=13, rc=0,
cib-update=22, confirmed=true) complete ok
Jun 4 11:04:18 srv-dell cib: [5961]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-97.raw
Jun 4 11:04:18 srv-dell cib: [5961]: info: write_cib_contents: Wrote
version 0.210.0 of the CIB to disk (digest:
bece00e8490842cb12ff34f514ed9582)
Jun 4 11:04:18 srv-dell cib: [5961]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.puAhki (digest:
/var/lib/heartbeat/crm/cib.CzStKD)
Jun 4 11:04:18 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=12:83:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_start_0 )
Jun 4 11:04:18 srv-dell lrmd: [5537]: info:
rsc:cloneresource_cluster_fencing_null:1: start
Jun 4 11:04:18 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=17:83:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=dlm:0_start_0 )
Jun 4 11:04:18 srv-dell lrmd: [5537]: info: rsc:dlm:0: start
Jun 4 11:04:18 srv-dell lrmd: [5962]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 4 11:04:18 srv-dell stonithd: [5535]: info:
cloneresource_cluster_fencing_null:1 stonith resource started
Jun 4 11:04:18 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_start_0 (call=14, rc=0,
cib-update=23, confirmed=true) complete ok
Jun 4 11:04:18 srv-dell lrmd: [5537]: info: RA output:
(dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:04:18 srv-dell cluster-dlm[5980]: main: dlm_controld 1239267206
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info:
init_ais_connection: AIS connection established
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:04:18 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 5980 (0x7f5fa0033f60)
Jun 4 11:04:18 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 4 11:04:18 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 4 11:04:18 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 4 11:04:18 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 4 11:04:18 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 4 11:04:18 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 2013440192
Jun 4 11:04:18 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 4 11:04:18 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 4 11:04:18 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:04:18 srv-dell cluster-dlm: [5980]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:04:18 srv-dell cluster-dlm: update_cluster: Processing membership 264
Jun 4 11:04:18 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 4 11:04:18 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 4 11:04:18 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=256, last-seen=264, this-event=264,
last-event=0
Jun 4 11:04:18 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 4 11:04:18 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 4 11:04:18 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=264, last-seen=264, this-event=264,
last-event=0
Jun 4 11:04:19 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_start_0 (call=15, rc=0, cib-update=24, confirmed=true)
complete ok
Jun 4 11:04:19 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=18:83:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=dlm:0_monitor_120000 )
Jun 4 11:04:19 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=25:83:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_start_0 )
Jun 4 11:04:19 srv-dell lrmd: [5537]: info: rsc:ocb2:0: start
Jun 4 11:04:19 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=16, rc=0, cib-update=25,
confirmed=false) complete ok
Jun 4 11:04:19 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:04:19 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:04:19 INFO: Stack glue driver not
loaded
Jun 4 11:04:19 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:04:19 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:04:19 INFO: Starting ocb2:0
Jun 4 11:04:19 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 4 11:04:20 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 4 11:04:20 srv-dell kernel: OCFS2 1.5.0
Jun 4 11:04:20 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info:
init_ais_connection: AIS connection established
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 4 11:04:20 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 4 11:04:20 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 6092 (0x7f5fa0036840)
Jun 4 11:04:20 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 4 11:04:20 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 4 11:04:20 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:04:20 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:04:20 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 4 11:04:20 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:04:20 srv-dell ocfs2_controld: setup done
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:04:20 srv-dell ocfs2_controld: [6092]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:04:20 srv-dell ocfs2_controld: confchg called
Jun 4 11:04:20 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 4 11:04:20 srv-dell ocfs2_controld: CPG is live, we are not the
first daemon
Jun 4 11:04:20 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 4 11:04:20 srv-dell ocfs2_controld: Reading from section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Read section
"daemon_protocol" from checkpoint "ocfs2:controld"
Jun 4 11:04:20 srv-dell ocfs2_controld: Reading from section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:04:20 srv-dell ocfs2_controld: Read section "ocfs2_protocol"
from checkpoint "ocfs2:controld"
Jun 4 11:04:20 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 4 11:04:20 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 4 11:04:20 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 4 11:04:20 srv-dell ocfs2_controld: Opening control device
Jun 4 11:04:20 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 4 11:04:20 srv-dell ocfs2_controld: new listening connection 4
Jun 4 11:04:20 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 4 11:04:22 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_start_0 (call=17, rc=0, cib-update=26,
confirmed=true) complete ok
Jun 4 11:04:22 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=26:83:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_120000 )
Jun 4 11:04:22 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=18, rc=0, cib-update=27,
confirmed=false) complete ok
Jun 4 11:04:28 srv-dell cib: [6119]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-98.raw
Jun 4 11:04:28 srv-dell cib: [6119]: info: write_cib_contents: Wrote
version 0.211.0 of the CIB to disk (digest:
1eed6d667053b6ed19b43284e6656df1)
Jun 4 11:04:28 srv-dell cib: [6119]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.fFAXBs (digest:
/var/lib/heartbeat/crm/cib.QOlxx9)
Jun 4 11:04:29 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=27:84:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_stop_0 )
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: rsc:ocb2:0: stop
Jun 4 11:04:29 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=18, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:04:29 INFO: Stopping ocb2:0
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:04:29 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=11:84:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_start_0 )
Jun 4 11:04:29 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: start
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:04:29 INFO: Stopping
ocfs2_controld.pcmk
Jun 4 11:04:29 srv-dell ocfs2_controld: Caught signal 15
Jun 4 11:04:29 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:04:29 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:04:29 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 4 11:04:29 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 4 11:04:29 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 4 11:04:29 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 4 11:04:29 srv-dell ocfs2_controld: closing cpg connection
Jun 4 11:04:29 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 4 11:04:29 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 4 11:04:29 srv-dell ocfs2_controld: closing pacemaker connection
Jun 4 11:04:29 srv-dell ocfs2_controld: [6092]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:04:29 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:04:29 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:04:29 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 4 11:04:29 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 4 11:04:29 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=20, rc=0,
cib-update=28, confirmed=true) complete ok
Jun 4 11:04:29 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=12:84:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 4 11:04:29 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_stop_0 (call=19, rc=0, cib-update=29, confirmed=true)
complete ok
Jun 4 11:04:29 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=28:84:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_start_0 )
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: rsc:ocb2:0: start
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:04:29 INFO: Stack glue driver not
loaded
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:04:29 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:04:29 INFO: Starting ocb2:0
Jun 4 11:04:29 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 4 11:04:29 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 4 11:04:29 srv-dell kernel: OCFS2 1.5.0
Jun 4 11:04:29 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info:
init_ais_connection: AIS connection established
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 4 11:04:29 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 4 11:04:29 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 6463 (0x7f5fa0035460)
Jun 4 11:04:29 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 4 11:04:29 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 4 11:04:29 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:04:29 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:04:29 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 4 11:04:29 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:04:29 srv-dell ocfs2_controld: setup done
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:04:29 srv-dell ocfs2_controld: [6463]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:04:29 srv-dell ocfs2_controld: confchg called
Jun 4 11:04:29 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 4 11:04:29 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 4 11:04:29 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 4 11:04:29 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Creating section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Created section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:04:29 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Creating section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:04:29 srv-dell ocfs2_controld: Created section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:04:29 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 4 11:04:29 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 4 11:04:29 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 4 11:04:29 srv-dell ocfs2_controld: Opening control device
Jun 4 11:04:29 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 4 11:04:29 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 4 11:04:29 srv-dell ocfs2_controld: new listening connection 4
Jun 4 11:04:30 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=21, rc=0,
cib-update=30, confirmed=false) complete ok
Jun 4 11:04:31 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_start_0 (call=22, rc=0, cib-update=31,
confirmed=true) complete ok
Jun 4 11:04:31 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=5:84:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_120000 )
Jun 4 11:04:31 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=23, rc=0, cib-update=32,
confirmed=false) complete ok
Jun 4 11:04:46 srv-dell cib: [6562]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-99.raw
Jun 4 11:04:46 srv-dell cib: [6562]: info: write_cib_contents: Wrote
version 0.212.0 of the CIB to disk (digest:
54178c99379b4aa63253a2409f4376b1)
Jun 4 11:04:46 srv-dell cib: [6562]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.OVCxg2 (digest:
/var/lib/heartbeat/crm/cib.Aaf2cm)
Jun 4 11:04:46 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=8:85:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_stop_0 )
Jun 4 11:04:46 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 4 11:04:46 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=21, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 4 11:04:46 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 4 11:04:46 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 4 11:04:46 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 4 11:04:46 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/04_11:04:46 INFO:
ifconfig eth1:0 down
Jun 4 11:04:46 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=24, rc=0,
cib-update=33, confirmed=true) complete ok
Jun 4 11:04:48 srv-dell ocfs2_controld: confchg called
Jun 4 11:04:48 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 4 11:06:48 srv-dell kernel: RPC: Registered udp transport module.
Jun 4 11:06:48 srv-dell kernel: RPC: Registered tcp transport module.
Jun 4 11:07:31 srv-dell kernel: ISO 9660 Extensions: Microsoft Joliet Level 3
Jun 4 11:07:31 srv-dell kernel: ISO 9660 Extensions: RRIP_1991A
Jun 4 11:08:00 srv-dell cib: [7195]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-100.raw
Jun 4 11:08:00 srv-dell cib: [7195]: info: write_cib_contents: Wrote
version 0.213.0 of the CIB to disk (digest:
e36320b9bfda1f981531b72bd8217358)
Jun 4 11:08:00 srv-dell cib: [7195]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.7CvEnT (digest:
/var/lib/heartbeat/crm/cib.Ckn5R1)
Jun 4 11:08:00 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=26:86:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_stop_0 )
Jun 4 11:08:00 srv-dell lrmd: [5537]: info: rsc:ocb2:0: stop
Jun 4 11:08:00 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=23, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:08:00 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:08:00 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:08:00 INFO: Stopping ocb2:0
Jun 4 11:08:00 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:08:00 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:08:00 INFO: Stopping
ocfs2_controld.pcmk
Jun 4 11:08:00 srv-dell ocfs2_controld: Caught signal 15
Jun 4 11:08:00 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:08:00 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:08:00 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 4 11:08:00 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 4 11:08:00 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 4 11:08:00 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 4 11:08:00 srv-dell ocfs2_controld: closing cpg connection
Jun 4 11:08:00 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:08:00 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 4 11:08:00 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 4 11:08:00 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 4 11:08:00 srv-dell ocfs2_controld: closing pacemaker connection
Jun 4 11:08:00 srv-dell ocfs2_controld: [6463]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:08:00 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 4 11:08:00 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_stop_0 (call=25, rc=0, cib-update=34, confirmed=true)
complete ok
Jun 4 11:08:01 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=19:86:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=dlm:0_stop_0 )
Jun 4 11:08:01 srv-dell lrmd: [5537]: info: rsc:dlm:0: stop
Jun 4 11:08:01 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=16, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:08:01 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/1543678144"
Jun 4 11:08:01 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 4 11:08:01 srv-dell dlm_controld.pcmk: [5980]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:08:01 srv-dell kernel: dlm: closing connection to node 1543678144
Jun 4 11:08:01 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 4 11:08:01 srv-dell lrmd: [5537]: info: RA output:
(dlm:0:stop:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:08:02 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_stop_0 (call=26, rc=0, cib-update=35, confirmed=true)
complete ok
Jun 4 11:08:02 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=14:86:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_stop_0 )
Jun 4 11:08:02 srv-dell lrmd: [5537]: info:
rsc:cloneresource_cluster_fencing_null:1: stop
Jun 4 11:08:02 srv-dell lrmd: [7354]: info: Try to stop STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 4 11:08:02 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_stop_0 (call=27, rc=0,
cib-update=36, confirmed=true) complete ok
Jun 4 11:08:11 srv-dell cib: [7356]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-1.raw
Jun 4 11:08:11 srv-dell cib: [7356]: info: write_cib_contents: Wrote
version 0.214.0 of the CIB to disk (digest:
9a65b1290d770cb70349f229e6596228)
Jun 4 11:08:11 srv-dell cib: [7356]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.zhcUF9 (digest:
/var/lib/heartbeat/crm/cib.3dDdXF)
Jun 4 11:08:11 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=12:87:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=cloneresource_cluster_fencing_null:1_start_0 )
Jun 4 11:08:11 srv-dell lrmd: [5537]: info:
rsc:cloneresource_cluster_fencing_null:1: start
Jun 4 11:08:11 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=17:87:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=dlm:0_start_0 )
Jun 4 11:08:11 srv-dell lrmd: [5537]: info: rsc:dlm:0: start
Jun 4 11:08:11 srv-dell lrmd: [7357]: info: Try to start STONITH
resource <rsc_id=cloneresource_cluster_fencing_null:1> : Device=null
Jun 4 11:08:11 srv-dell stonithd: [5535]: info:
cloneresource_cluster_fencing_null:1 stonith resource started
Jun 4 11:08:11 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation cloneresource_cluster_fencing_null:1_start_0 (call=28, rc=0,
cib-update=37, confirmed=true) complete ok
Jun 4 11:08:11 srv-dell lrmd: [5537]: info: RA output:
(dlm:0:start:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:08:11 srv-dell cluster-dlm[7375]: main: dlm_controld 1239267206
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info:
init_ais_connection: AIS connection established
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:08:11 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 7375 (0x7f5fa0035160)
Jun 4 11:08:11 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 4 11:08:11 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 4 11:08:11 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 4 11:08:11 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 4 11:08:11 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 4 11:08:11 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 2013440192
Jun 4 11:08:11 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 4 11:08:11 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 4 11:08:11 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:08:11 srv-dell cluster-dlm: [7375]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:08:11 srv-dell cluster-dlm: update_cluster: Processing membership 264
Jun 4 11:08:11 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 4 11:08:11 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 4 11:08:11 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=256, last-seen=264, this-event=264,
last-event=0
Jun 4 11:08:11 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 4 11:08:11 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 4 11:08:11 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=264, last-seen=264, this-event=264,
last-event=0
Jun 4 11:08:12 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_start_0 (call=29, rc=0, cib-update=38, confirmed=true)
complete ok
Jun 4 11:08:12 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=18:87:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=dlm:0_monitor_120000 )
Jun 4 11:08:12 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=25:87:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_start_0 )
Jun 4 11:08:12 srv-dell lrmd: [5537]: info: rsc:ocb2:0: start
Jun 4 11:08:12 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=30, rc=0, cib-update=39,
confirmed=false) complete ok
Jun 4 11:08:12 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:08:12 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:08:12 INFO: Stack glue driver not
loaded
Jun 4 11:08:12 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:08:12 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:08:12 INFO: Starting ocb2:0
Jun 4 11:08:12 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 4 11:08:12 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 4 11:08:12 srv-dell kernel: OCFS2 1.5.0
Jun 4 11:08:12 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info:
init_ais_connection: AIS connection established
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 4 11:08:12 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 4 11:08:12 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 7487 (0x7f5fa00358a0)
Jun 4 11:08:12 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 4 11:08:12 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 4 11:08:12 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:08:12 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:08:12 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 4 11:08:12 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:08:12 srv-dell ocfs2_controld: setup done
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:08:12 srv-dell ocfs2_controld: [7487]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:08:12 srv-dell ocfs2_controld: confchg called
Jun 4 11:08:12 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 4 11:08:12 srv-dell ocfs2_controld: CPG is live, we are not the
first daemon
Jun 4 11:08:12 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 4 11:08:12 srv-dell ocfs2_controld: Reading from section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Read section
"daemon_protocol" from checkpoint "ocfs2:controld"
Jun 4 11:08:12 srv-dell ocfs2_controld: Reading from section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:08:12 srv-dell ocfs2_controld: Read section "ocfs2_protocol"
from checkpoint "ocfs2:controld"
Jun 4 11:08:12 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 4 11:08:12 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 4 11:08:12 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 4 11:08:12 srv-dell ocfs2_controld: Opening control device
Jun 4 11:08:12 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 4 11:08:12 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 4 11:08:12 srv-dell ocfs2_controld: new listening connection 4
Jun 4 11:08:14 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_start_0 (call=31, rc=0, cib-update=40,
confirmed=true) complete ok
Jun 4 11:08:14 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=26:87:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_120000 )
Jun 4 11:08:14 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=32, rc=0, cib-update=41,
confirmed=false) complete ok
Jun 4 11:13:37 srv-dell cib: [5536]: info: cib_stats: Processed 120
operations (83.00us average, 0% utilization) in the last 10min
Jun 4 11:18:18 srv-dell kernel: st: Version 20080504, fixed bufsize
32768, s/g segs 256
Jun 4 11:18:29 srv-dell hald: unmounted /dev/sda4 from
'/media/disk-1' on behalf of uid 0
Jun 4 11:18:41 srv-dell kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jun 4 11:19:08 srv-dell kernel: iSCSI Enterprise Target Software -
version 0.4.16
Jun 4 11:19:08 srv-dell kernel: iscsi_trgt: Registered io type fileio
Jun 4 11:19:08 srv-dell kernel: iscsi_trgt: Registered io type blockio
Jun 4 11:19:08 srv-dell kernel: iscsi_trgt: Registered io type nullio
Jun 4 11:20:10 srv-dell cib: [8173]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-2.raw
Jun 4 11:20:10 srv-dell cib: [8173]: info: write_cib_contents: Wrote
version 0.215.0 of the CIB to disk (digest:
bc5d9f2d3b471bf7e030baf824ef1fe5)
Jun 4 11:20:10 srv-dell cib: [8173]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.rnoXKX (digest:
/var/lib/heartbeat/crm/cib.Qo6uWI)
Jun 4 11:20:10 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=27:88:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_stop_0 )
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: rsc:ocb2:0: stop
Jun 4 11:20:10 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=32, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:20:10 INFO: Stopping ocb2:0
Jun 4 11:20:10 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=11:88:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_start_0 )
Jun 4 11:20:10 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: start
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:20:10 INFO: Stopping
ocfs2_controld.pcmk
Jun 4 11:20:10 srv-dell ocfs2_controld: Caught signal 15
Jun 4 11:20:10 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:20:10 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:20:10 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 4 11:20:10 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 4 11:20:10 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 4 11:20:10 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 4 11:20:10 srv-dell ocfs2_controld: closing cpg connection
Jun 4 11:20:10 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:20:10 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 4 11:20:10 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 4 11:20:10 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 4 11:20:10 srv-dell ocfs2_controld: closing pacemaker connection
Jun 4 11:20:10 srv-dell ocfs2_controld: [7487]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:20:10 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:20:10 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:20:10 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 4 11:20:10 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=34, rc=0,
cib-update=42, confirmed=true) complete ok
Jun 4 11:20:10 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=12:88:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 4 11:20:10 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 4 11:20:10 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_stop_0 (call=33, rc=0, cib-update=43, confirmed=true)
complete ok
Jun 4 11:20:10 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=28:88:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_start_0 )
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: rsc:ocb2:0: start
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:20:10 INFO: Stack glue driver not
loaded
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:20:10 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:20:10 INFO: Starting ocb2:0
Jun 4 11:20:10 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 4 11:20:10 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 4 11:20:10 srv-dell kernel: OCFS2 1.5.0
Jun 4 11:20:10 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 4 11:20:10 srv-dell ocfs2_controld: [8502]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:20:10 srv-dell ocfs2_controld: [8502]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:20:10 srv-dell ocfs2_controld: [8502]: info:
init_ais_connection: AIS connection established
Jun 4 11:20:10 srv-dell ocfs2_controld: [8502]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:20:10 srv-dell ocfs2_controld: [8502]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 4 11:20:10 srv-dell ocfs2_controld: [8502]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 4 11:20:10 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 4 11:20:10 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 8502 (0x7f5fa00358a0)
Jun 4 11:20:10 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 4 11:20:10 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 4 11:20:10 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 4 11:20:11 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:20:11 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:20:11 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 4 11:20:11 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:20:11 srv-dell ocfs2_controld: setup done
Jun 4 11:20:11 srv-dell ocfs2_controld: [8502]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:20:11 srv-dell ocfs2_controld: [8502]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 4 11:20:11 srv-dell ocfs2_controld: [8502]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 4 11:20:11 srv-dell ocfs2_controld: [8502]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:20:11 srv-dell ocfs2_controld: [8502]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:20:11 srv-dell ocfs2_controld: confchg called
Jun 4 11:20:11 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 4 11:20:11 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 4 11:20:11 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 4 11:20:11 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Creating section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Created section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:20:11 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Creating section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:20:11 srv-dell ocfs2_controld: Created section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:20:11 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 4 11:20:11 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 4 11:20:11 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 4 11:20:11 srv-dell ocfs2_controld: Opening control device
Jun 4 11:20:11 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 4 11:20:11 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 4 11:20:11 srv-dell ocfs2_controld: new listening connection 4
Jun 4 11:20:11 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=35, rc=0,
cib-update=44, confirmed=false) complete ok
Jun 4 11:20:13 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_start_0 (call=36, rc=0, cib-update=45,
confirmed=true) complete ok
Jun 4 11:20:13 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=5:88:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_120000 )
Jun 4 11:20:13 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=37, rc=0, cib-update=46,
confirmed=false) complete ok
Jun 4 11:20:19 srv-dell cib: [8578]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-3.raw
Jun 4 11:20:19 srv-dell cib: [8578]: info: write_cib_contents: Wrote
version 0.216.0 of the CIB to disk (digest:
dc34f95a07c3eb85417785f2359dc77d)
Jun 4 11:20:19 srv-dell cib: [8578]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.3XzFri (digest:
/var/lib/heartbeat/crm/cib.TLwacn)
Jun 4 11:20:19 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=8:89:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_stop_0 )
Jun 4 11:20:19 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 4 11:20:19 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=35, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 4 11:20:19 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 4 11:20:19 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 4 11:20:19 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 4 11:20:19 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/04_11:20:19 INFO:
ifconfig eth1:0 down
Jun 4 11:20:19 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=38, rc=0,
cib-update=47, confirmed=true) complete ok
Jun 4 11:20:20 srv-dell ocfs2_controld: confchg called
Jun 4 11:20:20 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 4 11:20:49 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:16 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:16 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:16 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:16 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=27:90:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_stop_0 )
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: rsc:ocb2:0: stop
Jun 4 11:21:16 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=37, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:21:16 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=11:90:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_start_0 )
Jun 4 11:21:16 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: start
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:21:16 INFO: Stopping ocb2:0
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:21:16 INFO: Stopping
ocfs2_controld.pcmk
Jun 4 11:21:16 srv-dell ocfs2_controld: Caught signal 15
Jun 4 11:21:16 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:21:16 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:21:16 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 4 11:21:16 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 4 11:21:16 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 4 11:21:16 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 4 11:21:16 srv-dell ocfs2_controld: closing cpg connection
Jun 4 11:21:16 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 4 11:21:16 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 4 11:21:16 srv-dell ocfs2_controld: closing pacemaker connection
Jun 4 11:21:16 srv-dell ocfs2_controld: [8502]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:21:16 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:21:16 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:21:16 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 4 11:21:16 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 4 11:21:16 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=40, rc=0,
cib-update=48, confirmed=true) complete ok
Jun 4 11:21:16 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=12:90:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 4 11:21:16 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:16 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:16 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 4 11:21:16 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_stop_0 (call=39, rc=0, cib-update=49, confirmed=true)
complete ok
Jun 4 11:21:16 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=28:90:0:caf53829-4bd1-4932-989e-f155cf0ecf6e op=ocb2:0_start_0 )
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: rsc:ocb2:0: start
Jun 4 11:21:16 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:21:16 INFO: Stack glue driver not
loaded
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) logd is not running
Jun 4 11:21:16 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:start:stderr) 2009/06/04_11:21:16 INFO: Starting ocb2:0
Jun 4 11:21:16 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 4 11:21:16 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 4 11:21:16 srv-dell kernel: OCFS2 1.5.0
Jun 4 11:21:16 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info:
init_ais_connection: AIS connection established
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 4 11:21:16 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 8943 (0x7f5fa00358a0)
Jun 4 11:21:16 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 4 11:21:16 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 4 11:21:16 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 4 11:21:16 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:21:16 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:21:16 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 4 11:21:16 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:21:16 srv-dell ocfs2_controld: setup done
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: notice: ais_dispatch:
Membership 264: quorum aquired
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000053312
Jun 4 11:21:16 srv-dell ocfs2_controld: [8943]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=264 seen=264
proc=00000000000000000000000000053312 (new)
Jun 4 11:21:16 srv-dell ocfs2_controld: confchg called
Jun 4 11:21:16 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 4 11:21:16 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 4 11:21:16 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 4 11:21:16 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Creating section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Created section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:21:16 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Creating section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:21:16 srv-dell ocfs2_controld: Created section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:21:16 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 4 11:21:16 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 4 11:21:16 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 4 11:21:16 srv-dell ocfs2_controld: Opening control device
Jun 4 11:21:16 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 4 11:21:16 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 4 11:21:16 srv-dell ocfs2_controld: new listening connection 4
Jun 4 11:21:16 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:17 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=41, rc=0,
cib-update=50, confirmed=false) complete ok
Jun 4 11:21:17 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:17 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:18 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:18 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:18 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_start_0 (call=42, rc=0, cib-update=51,
confirmed=true) complete ok
Jun 4 11:21:18 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=5:90:0:caf53829-4bd1-4932-989e-f155cf0ecf6e
op=ocb2:0_monitor_120000 )
Jun 4 11:21:18 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:18 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=43, rc=0, cib-update=52,
confirmed=false) complete ok
Jun 4 11:21:18 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:19 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:22 srv-dell cib: [5536]: info: cib_process_shutdown_req:
Shutdown REQ from srv-lenovo
Jun 4 11:21:22 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_shutdown_req for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.216.20): ok (rc=0)
Jun 4 11:21:24 srv-dell openais[5529]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000000002
(2)
Jun 4 11:21:24 srv-dell cib: [5536]: notice: ais_dispatch: Membership
264: quorum aquired
Jun 4 11:21:24 srv-dell cib: [5536]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=256 seen=264 proc=00000000000000000000000000000002 (new)
Jun 4 11:21:24 srv-dell crmd: [5540]: info: ais_dispatch: Membership
264: quorum retained
Jun 4 11:21:24 srv-dell crmd: [5540]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=256 seen=264 proc=00000000000000000000000000000002 (new)
Jun 4 11:21:24 srv-dell cluster-dlm: [7375]: info: ais_dispatch:
Membership 264: quorum retained
Jun 4 11:21:24 srv-dell ocfs2_controld: [8943]: info: ais_dispatch:
Membership 264: quorum retained
Jun 4 11:21:24 srv-dell cluster-dlm: [7375]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000000002 (new)
Jun 4 11:21:24 srv-dell ocfs2_controld: [8943]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000000002 (new)
Jun 4 11:21:24 srv-dell openais[5529]: [crm ] info:
send_member_notification: Sending membership update 264 to 4 children
Jun 4 11:21:29 srv-dell openais[5529]: [TOTEM] The token was lost in
the OPERATIONAL state.
Jun 4 11:21:29 srv-dell openais[5529]: [TOTEM] Receive multicast
socket recv buffer size (288000 bytes).
Jun 4 11:21:29 srv-dell openais[5529]: [TOTEM] Transmit multicast
socket send buffer size (288000 bytes).
Jun 4 11:21:29 srv-dell openais[5529]: [TOTEM] entering GATHER state from 2.
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] entering GATHER state from 0.
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] Creating commit token
because I am the rep.
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] Saving state aru 25e
high seq received 25e
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] Storing new sequence
id for ring 10c
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] entering COMMIT state.
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] entering RECOVERY state.
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] position [0] member
192.168.2.220:
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] previous ring seq 264
rep 192.168.2.120
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] aru 25e high delivered
25e received flag 1
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] Sending initial ORF token
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:21:31 srv-dell crmd: [5540]: notice: ais_dispatch:
Membership 268: quorum lost
Jun 4 11:21:31 srv-dell crmd: [5540]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=256 seen=264 proc=00000000000000000000000000000002
Jun 4 11:21:31 srv-dell crmd: [5540]: WARN: check_dead_member: Our DC
node (srv-lenovo) left the cluster
Jun 4 11:21:31 srv-dell cluster-dlm: [7375]: notice: ais_dispatch:
Membership 268: quorum lost
Jun 4 11:21:31 srv-dell ocfs2_controld: [8943]: notice: ais_dispatch:
Membership 268: quorum lost
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION
cause=C_FSA_INTERNAL origin=check_dead_member ]
Jun 4 11:21:31 srv-dell cib: [5536]: notice: ais_dispatch: Membership
268: quorum lost
Jun 4 11:21:31 srv-dell cluster-dlm: [7375]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=lost (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000000002
Jun 4 11:21:31 srv-dell cluster-dlm: update_cluster: Processing membership 268
Jun 4 11:21:31 srv-dell crmd: [5540]: info: update_dc: Unset DC srv-lenovo
Jun 4 11:21:31 srv-dell cib: [5536]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=lost (new) addr=r(0) ip(192.168.2.120)
votes=1 born=256 seen=264 proc=00000000000000000000000000000002
Jun 4 11:21:31 srv-dell ocfs2_controld: [8943]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=lost (new)
addr=r(0) ip(192.168.2.120) votes=1 born=256 seen=264
proc=00000000000000000000000000000002
Jun 4 11:21:31 srv-dell cluster-dlm: del_configfs_node:
del_configfs_node rmdir
"/sys/kernel/config/dlm/cluster/comms/2013440192"
Jun 4 11:21:31 srv-dell cluster-dlm: dlm_process_node: Removed
inactive node 2013440192: born-on=256, last-seen=264, this-event=268,
last-event=264
Jun 4 11:21:31 srv-dell cluster-dlm: dlm_process_node: Skipped active
node 1543678144: born-on=264, last-seen=268, this-event=268,
last-event=264
Jun 4 11:21:31 srv-dell kernel: dlm: closing connection to node 2013440192
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.120)
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:21:31 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 268: memb=1,
new=0, lost=1
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_te_control:
Registering TE UUID: 398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a
Jun 4 11:21:31 srv-dell crmd: [5540]: WARN:
cib_client_add_notify_callback: Callback already present
Jun 4 11:21:31 srv-dell crmd: [5540]: info: set_graph_functions:
Setting custom graph functions
Jun 4 11:21:31 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition -1: 0 actions in 0 synapses
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_dc_takeover: Taking
over DC status for this partition
Jun 4 11:21:31 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 4 11:21:31 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: lost: srv-lenovo 2013440192
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:21:31 srv-dell cib: [5536]: info: cib_process_readwrite: We
are now in R/W mode
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:21:31 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 268: memb=1, new=0,
lost=0
Jun 4 11:21:31 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 4 11:21:31 srv-dell openais[5529]: [crm ] info:
ais_mark_unseen_peer_dead: Node srv-lenovo was not seen in the
previous transition
Jun 4 11:21:31 srv-dell openais[5529]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: lost
Jun 4 11:21:31 srv-dell openais[5529]: [crm ] info:
send_member_notification: Sending membership update 268 to 4 children
Jun 4 11:21:31 srv-dell openais[5529]: [SYNC ] This node is within
the primary component and will provide service.
Jun 4 11:21:31 srv-dell openais[5529]: [TOTEM] entering OPERATIONAL state.
Jun 4 11:21:31 srv-dell openais[5529]: [CLM ] got nodejoin message
192.168.2.220
Jun 4 11:21:31 srv-dell openais[5529]: [CPG ] got joinlist message
from node 1543678144
Jun 4 11:21:31 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_master for section 'all'
(origin=local/crmd/54, version=0.216.20): ok (rc=0)
Jun 4 11:21:31 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/55, version=0.216.20): ok (rc=0)
Jun 4 11:21:31 srv-dell crmd: [5540]: info: join_make_offer: Making
join offers based on membership 268
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_dc_join_offer_all:
join-1: Waiting on 1 outstanding join acks
Jun 4 11:21:31 srv-dell crmd: [5540]: info: ais_dispatch: Membership
268: quorum still lost
Jun 4 11:21:31 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/57, version=0.216.20): ok (rc=0)
Jun 4 11:21:31 srv-dell crmd: [5540]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 4 11:21:31 srv-dell crmd: [5540]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 4 11:21:31 srv-dell crmd: [5540]: info: ais_dispatch: Membership
268: quorum still lost
Jun 4 11:21:31 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/60, version=0.216.20): ok (rc=0)
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_state_transition: All
1 cluster nodes responded to the join offer.
Jun 4 11:21:31 srv-dell crmd: [5540]: info: do_dc_join_finalize:
join-1: Syncing the CIB from srv-dell to the rest of the cluster
Jun 4 11:21:31 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/63, version=0.216.20): ok (rc=0)
Jun 4 11:21:31 srv-dell crmd: [5540]: info: te_connect_stonith:
Attempting connection to fencing daemon...
Jun 4 11:21:31 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/64, version=0.216.20): ok (rc=0)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: te_connect_stonith: Connected
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_dc_join_ack: join-1:
Updating node state to member for srv-dell
Jun 4 11:21:32 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/65, version=0.216.20): ok (rc=0)
Jun 4 11:21:32 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/66,
version=0.216.21): ok (rc=0)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 4 11:21:32 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 4 11:21:32 srv-dell crmd: [5540]: info: crm_update_quorum:
Updating quorum status to false (call=70)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 4 11:21:32 srv-dell attrd: [5538]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_pe_invoke: Query 71:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:21:32 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 4 11:21:32 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:32 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/68, version=0.216.22): ok (rc=0)
Jun 4 11:21:32 srv-dell crmd: [5540]: WARN: match_down_event: No
match for shutdown action on srv-lenovo
Jun 4 11:21:32 srv-dell crmd: [5540]: info: te_update_diff:
Stonith/shutdown of srv-lenovo not matched
Jun 4 11:21:32 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <cib have-quorum="1" dc-uuid="srv-lenovo" admin_epoch="0"
epoch="216" num_updates="23" />
Jun 4 11:21:32 srv-dell crmd: [5540]: info: abort_transition_graph:
te_update_diff:191 - Triggered transition abort (complete=1,
tag=node_state, id=srv-lenovo, magic=NA) : Node failure
Jun 4 11:21:32 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <cib have-quorum="0" dc-uuid="srv-dell" admin_epoch="0"
epoch="217" num_updates="1" />
Jun 4 11:21:32 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/70, version=0.217.1): ok (rc=0)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 4 11:21:32 srv-dell crmd: [5540]: info: need_abort: Aborting on
change to have-quorum
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_pe_invoke: Query 72:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:21:32 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_pe_invoke: Query 73:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:21:32 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:32 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:32 srv-dell cib: [9042]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-4.raw
Jun 4 11:21:32 srv-dell cib: [9042]: info: write_cib_contents: Wrote
version 0.217.0 of the CIB to disk (digest:
f62cbc3daad39ea4caf329812ae4db70)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244128892-11, seq=268, quorate=0
Jun 4 11:21:32 srv-dell cib: [9042]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.iLxNZK (digest:
/var/lib/heartbeat/crm/cib.abGGpq)
Jun 4 11:21:32 srv-dell pengine: [5539]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 4 11:21:32 srv-dell pengine: [5539]: info:
determine_online_status_fencing: Node srv-lenovo is down
Jun 4 11:21:32 srv-dell pengine: [5539]: WARN:
process_orphan_resource: Nothing known about resource
primitive_cluster_dlm running on srv-lenovo
Jun 4 11:21:32 srv-dell pengine: [5539]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="primitive_cluster_dlm" type="controld" class="ocf"
provider="pacemaker" />
Jun 4 11:21:32 srv-dell pengine: [5539]: info:
process_orphan_resource: Making sure orphan primitive_cluster_dlm is
stopped
Jun 4 11:21:32 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-dell is online
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-dell ]
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0 ]
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: clone_dlm
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-dell ]
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ dlm:1 ]
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: ocb2_clone
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-dell ]
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ ocb2:1 ]
Jun 4 11:21:32 srv-dell pengine: [5539]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 4 11:21:32 srv-dell pengine: [5539]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 4 11:21:32 srv-dell pengine: [5539]: WARN: native_color: Resource
ocb2:1 cannot run anywhere
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: LogActions: Stop
resource primitiveresource_cluster_ip (srv-dell)
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Stopped)
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-dell)
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: LogActions: Stop
resource dlm:0 (srv-dell)
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:1 (Stopped)
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: LogActions: Stop
resource ocb2:0 (srv-dell)
Jun 4 11:21:32 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:1 (Stopped)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 4 11:21:32 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition 0: 12 actions in 12 synapses
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1244128892-11) derived from
/var/lib/pengine/pe-warn-68.bz2
Jun 4 11:21:32 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 7: stop primitiveresource_cluster_ip_stop_0 on
srv-dell (local)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=7:0:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a
op=primitiveresource_cluster_ip_stop_0 )
Jun 4 11:21:32 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: stop
Jun 4 11:21:32 srv-dell pengine: [5539]: WARN: process_pe_message:
Transition 0: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-68.bz2
Jun 4 11:21:32 srv-dell pengine: [5539]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 4 11:21:32 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 25 fired and confirmed
Jun 4 11:21:32 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=41, rc=-2,
cib-update=0, confirmed=true) Cancelled unknown exec error
Jun 4 11:21:32 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 21: stop ocb2:0_stop_0 on srv-dell (local)
Jun 4 11:21:32 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=21:0:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a op=ocb2:0_stop_0 )
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: rsc:ocb2:0: stop
Jun 4 11:21:32 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_monitor_120000 (call=43, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stdout) In IP Stop
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) SIOCDELRT: No such process
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) logd is not running
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:stop:stderr) 2009/06/04_11:21:32 INFO:
ifconfig eth1:0 down
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:21:32 INFO: Stopping ocb2:0
Jun 4 11:21:32 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_stop_0 (call=44, rc=0,
cib-update=74, confirmed=true) complete ok
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) logd is not running
Jun 4 11:21:32 srv-dell lrmd: [5537]: info: RA output:
(ocb2:0:stop:stderr) 2009/06/04_11:21:32 INFO: Stopping
ocfs2_controld.pcmk
Jun 4 11:21:32 srv-dell crmd: [5540]: info: match_graph_event: Action
primitiveresource_cluster_ip_stop_0 (7) confirmed on srv-dell (rc=0)
Jun 4 11:21:32 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:32 srv-dell ocfs2_controld: Caught signal 15
Jun 4 11:21:32 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:21:32 srv-dell ocfs2_controld: Caught signal 15, exiting
Jun 4 11:21:32 srv-dell ocfs2_controld: Closing dlm_controld connection
Jun 4 11:21:32 srv-dell ocfs2_controld: leaving group "ocfs2:controld"
Jun 4 11:21:32 srv-dell cluster-dlm: process_connection: connection 5
read error -1
Jun 4 11:21:32 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 4 11:21:32 srv-dell ocfs2_controld: closing cpg connection
Jun 4 11:21:32 srv-dell ocfs2_controld: Closing checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:21:32 srv-dell ocfs2_controld: Closed checkpoint
"ocfs2:controld:5c02a8c0"
Jun 4 11:21:32 srv-dell ocfs2_controld: Disconnecting from CKPT service (try 1)
Jun 4 11:21:32 srv-dell ocfs2_controld: Disconnected from CKPT service
Jun 4 11:21:32 srv-dell ocfs2_controld: closing pacemaker connection
Jun 4 11:21:32 srv-dell ocfs2_controld: [8943]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:21:33 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:33 srv-dell kernel: ocfs2: Unregistered cluster interface user
Jun 4 11:21:33 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:0_stop_0 (call=45, rc=0, cib-update=75, confirmed=true)
complete ok
Jun 4 11:21:33 srv-dell crmd: [5540]: info: match_graph_event: Action
ocb2:0_stop_0 (21) confirmed on srv-dell (rc=0)
Jun 4 11:21:33 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 26 fired and confirmed
Jun 4 11:21:33 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 19 fired and confirmed
Jun 4 11:21:33 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 15: stop dlm:0_stop_0 on srv-dell (local)
Jun 4 11:21:33 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=15:0:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a op=dlm:0_stop_0 )
Jun 4 11:21:33 srv-dell lrmd: [5537]: info: rsc:dlm:0: stop
Jun 4 11:21:33 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_monitor_120000 (call=30, rc=-2, cib-update=0,
confirmed=true) Cancelled unknown exec error
Jun 4 11:21:33 srv-dell cluster-dlm: clear_configfs_comms:
clear_configfs_nodes rmdir
"/sys/kernel/config/dlm/cluster/comms/1543678144"
Jun 4 11:21:33 srv-dell dlm_controld.pcmk: [7375]: notice:
terminate_ais_connection: Disconnected from AIS
Jun 4 11:21:33 srv-dell kernel: dlm: closing connection to node 1543678144
Jun 4 11:21:33 srv-dell lrmd: [5537]: info: RA output:
(dlm:0:stop:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:21:33 srv-dell haclient: on_event: from message queue: evt:cib_changed
Jun 4 11:21:33 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:34 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:0_stop_0 (call=46, rc=0, cib-update=76, confirmed=true)
complete ok
Jun 4 11:21:34 srv-dell crmd: [5540]: info: match_graph_event: Action
dlm:0_stop_0 (15) confirmed on srv-dell (rc=0)
Jun 4 11:21:34 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 20 fired and confirmed
Jun 4 11:21:34 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 4 fired and confirmed
Jun 4 11:21:34 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 17 fired and confirmed
Jun 4 11:21:34 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 18 fired and confirmed
Jun 4 11:21:34 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 23 fired and confirmed
Jun 4 11:21:34 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 24 fired and confirmed
Jun 4 11:21:34 srv-dell crmd: [5540]: info: run_graph:
====================================================
Jun 4 11:21:34 srv-dell crmd: [5540]: notice: run_graph: Transition 0
(Complete=12, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-68.bz2): Complete
Jun 4 11:21:34 srv-dell crmd: [5540]: info: te_graph_trigger:
Transition 0 is now complete
Jun 4 11:21:34 srv-dell crmd: [5540]: info: notify_crmd: Transition 0
status: done - <null>
Jun 4 11:21:34 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 4 11:21:34 srv-dell crmd: [5540]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 4 11:21:34 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:21:34 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:21:48 srv-dell crm_resource: [9228]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:21:48 srv-dell crm_resource: [9228]: info: Invoked:
crm_resource --meta -r primitiveresource_cluster_ip -p target-role -v
started
Jun 4 11:21:48 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section resources
(origin=local/crm_resource/4, version=0.217.4): ok (rc=0)
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_lrm_invoke: Removing
resource primitiveresource_cluster_ip from the LRM
Jun 4 11:22:06 srv-dell mgmtd: [5541]: info: Delete fail-count for
primitiveresource_cluster_ip from srv-dell
Jun 4 11:22:06 srv-dell crmd: [5540]: info: send_direct_ack: ACK'ing
resource op primitiveresource_cluster_ip_delete_0 from mgmtd-5541:
lrm_invoke-lrmd-1244128926-16
Jun 4 11:22:06 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']//lrm_resource[@id='primitiveresource_cluster_ip']
(origin=local/crmd/77, version=0.217.5): ok (rc=0)
Jun 4 11:22:06 srv-dell crmd: [5540]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=primitiveresource_cluster_ip_monitor_0,
magic=0:7;4:81:7:caf53829-4bd1-4932-989e-f155cf0ecf6e) : Resource op
removal
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_pe_invoke: Query 80:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_lrm_invoke: Forcing a
local LRM refresh
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="217" num_updates="5" >
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <configuration >
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <crm_config >
Jun 4 11:22:06 srv-dell crmd: [5540]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <cluster_property_set id="cib-bootstrap-options" >
Jun 4 11:22:06 srv-dell crmd: [5540]: info: need_abort: Aborting on
change to admin_epoch
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <nvpair value="1244081328"
id="cib-bootstrap-options-last-lrm-refresh" />
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - </cluster_property_set>
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - </crm_config>
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_pe_invoke: Query 82:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - </configuration>
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element: cib:diff: - </cib>
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="218" num_updates="1" >
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <configuration >
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <crm_config >
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <cluster_property_set id="cib-bootstrap-options" >
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <nvpair value="1244128926"
id="cib-bootstrap-options-last-lrm-refresh" />
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + </cluster_property_set>
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + </crm_config>
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + </configuration>
Jun 4 11:22:06 srv-dell cib: [5536]: info: log_data_element: cib:diff: + </cib>
Jun 4 11:22:06 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/79, version=0.218.1): ok (rc=0)
Jun 4 11:22:06 srv-dell cib: [9229]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-5.raw
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244128926-17, seq=268, quorate=0
Jun 4 11:22:06 srv-dell crmd: [5540]: info: config_query_callback:
Checking for expired actions every 900000ms
Jun 4 11:22:06 srv-dell crmd: [5540]: info: ais_dispatch: Membership
268: quorum still lost
Jun 4 11:22:06 srv-dell cib: [9229]: info: write_cib_contents: Wrote
version 0.218.0 of the CIB to disk (digest:
50ded783cbc1340f59fcbbee58b50175)
Jun 4 11:22:06 srv-dell pengine: [5539]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 4 11:22:06 srv-dell pengine: [5539]: info:
determine_online_status_fencing: Node srv-lenovo is down
Jun 4 11:22:06 srv-dell pengine: [5539]: WARN:
process_orphan_resource: Nothing known about resource
primitive_cluster_dlm running on srv-lenovo
Jun 4 11:22:06 srv-dell pengine: [5539]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="primitive_cluster_dlm" type="controld" class="ocf"
provider="pacemaker" />
Jun 4 11:22:06 srv-dell pengine: [5539]: info:
process_orphan_resource: Making sure orphan primitive_cluster_dlm is
stopped
Jun 4 11:22:06 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-dell is online
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-dell ]
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0 ]
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: clone_dlm
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: ocb2_clone
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ ocb2:0 ocb2:1 ]
Jun 4 11:22:06 srv-dell pengine: [5539]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 4 11:22:06 srv-dell pengine: [5539]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 4 11:22:06 srv-dell pengine: [5539]: WARN: native_color: Resource
ocb2:1 cannot run anywhere
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Stopped)
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-dell)
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: LogActions: Start
dlm:0 (srv-dell)
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:1 (Stopped)
Jun 4 11:22:06 srv-dell cib: [9229]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.PZGnO0 (digest:
/var/lib/heartbeat/crm/cib.owdboS)
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: LogActions: Start
ocb2:0 (srv-dell)
Jun 4 11:22:06 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:1 (Stopped)
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 4 11:22:06 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition 1: 3 actions in 3 synapses
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_te_invoke: Processing
graph 1 (ref=pe_calc-dc-1244128926-17) derived from
/var/lib/pengine/pe-warn-69.bz2
Jun 4 11:22:06 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 4: monitor primitiveresource_cluster_ip_monitor_0 on
srv-dell (local)
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=4:1:7:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a
op=primitiveresource_cluster_ip_monitor_0 )
Jun 4 11:22:06 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/85, version=0.218.1): ok (rc=0)
Jun 4 11:22:06 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: monitor
Jun 4 11:22:06 srv-dell pengine: [5539]: WARN: process_pe_message:
Transition 1: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-69.bz2
Jun 4 11:22:06 srv-dell pengine: [5539]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 4 11:22:06 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_0 (call=47, rc=7,
cib-update=86, confirmed=true) complete not running
Jun 4 11:22:06 srv-dell crmd: [5540]: info: match_graph_event: Action
primitiveresource_cluster_ip_monitor_0 (4) confirmed on srv-dell
(rc=0)
Jun 4 11:22:06 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 3: probe_complete probe_complete on srv-dell (local)
- no waiting
Jun 4 11:22:06 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 4 11:22:06 srv-dell crmd: [5540]: info: run_graph:
====================================================
Jun 4 11:22:06 srv-dell crmd: [5540]: notice: run_graph: Transition 1
(Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-69.bz2): Complete
Jun 4 11:22:06 srv-dell crmd: [5540]: info: te_graph_trigger:
Transition 1 is now complete
Jun 4 11:22:06 srv-dell crmd: [5540]: info: notify_crmd: Transition 1
status: done - <null>
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 4 11:22:06 srv-dell crmd: [5540]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 4 11:22:08 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/mgmtd/21, version=0.218.2): ok (rc=0)
Jun 4 11:22:08 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:22:08 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:22:08 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:22:09 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:22:25 srv-dell crm_resource: [9249]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:22:25 srv-dell crm_resource: [9249]: info: Invoked:
crm_resource -M -r primitiveresource_cluster_ip -H srv-dell
Jun 4 11:22:25 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section constraints
(origin=local/crm_resource/3, version=0.218.2): ok (rc=0)
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <cib admin_epoch="0" epoch="218" num_updates="2" >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <configuration >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <constraints >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <rsc_location
id="cli-prefer-primitiveresource_cluster_ip" >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <rule
id="cli-prefer-rule-primitiveresource_cluster_ip" >
Jun 4 11:22:25 srv-dell crmd: [5540]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <expression value="srv-lenovo"
id="cli-prefer-expr-primitiveresource_cluster_ip" />
Jun 4 11:22:25 srv-dell crmd: [5540]: info: need_abort: Aborting on
change to admin_epoch
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - </rule>
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - </rsc_location>
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - </constraints>
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - </configuration>
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_state_transition: All
1 cluster nodes are eligible to run resources.
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element: cib:diff: - </cib>
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_pe_invoke: Query 89:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <cib admin_epoch="0" epoch="219" num_updates="1" >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <configuration >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <constraints >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <rsc_location
id="cli-prefer-primitiveresource_cluster_ip" >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <rule
id="cli-prefer-rule-primitiveresource_cluster_ip" >
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <expression value="srv-dell"
id="cli-prefer-expr-primitiveresource_cluster_ip" />
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + </rule>
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + </rsc_location>
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + </constraints>
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + </configuration>
Jun 4 11:22:25 srv-dell cib: [5536]: info: log_data_element: cib:diff: + </cib>
Jun 4 11:22:25 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section constraints
(origin=local/crm_resource/4, version=0.219.1): ok (rc=0)
Jun 4 11:22:25 srv-dell cib: [9250]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-6.raw
Jun 4 11:22:25 srv-dell cib: [9250]: info: write_cib_contents: Wrote
version 0.219.0 of the CIB to disk (digest:
c7fb7c4e3af7a13a07e6e4906b100611)
Jun 4 11:22:25 srv-dell cib: [9250]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.tjDYdD (digest:
/var/lib/heartbeat/crm/cib.nCeL09)
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244128945-20, seq=268, quorate=0
Jun 4 11:22:25 srv-dell pengine: [5539]: WARN: cluster_status: We do
not have quorum - fencing and resource management disabled
Jun 4 11:22:25 srv-dell pengine: [5539]: info:
determine_online_status_fencing: Node srv-lenovo is down
Jun 4 11:22:25 srv-dell pengine: [5539]: WARN:
process_orphan_resource: Nothing known about resource
primitive_cluster_dlm running on srv-lenovo
Jun 4 11:22:25 srv-dell pengine: [5539]: info: log_data_element:
create_fake_resource: Orphan resource <primitive
id="primitive_cluster_dlm" type="controld" class="ocf"
provider="pacemaker" />
Jun 4 11:22:25 srv-dell pengine: [5539]: info:
process_orphan_resource: Making sure orphan primitive_cluster_dlm is
stopped
Jun 4 11:22:25 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-dell is online
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-dell ]
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0 ]
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: clone_dlm
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: ocb2_clone
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ ocb2:0 ocb2:1 ]
Jun 4 11:22:25 srv-dell pengine: [5539]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 4 11:22:25 srv-dell pengine: [5539]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 4 11:22:25 srv-dell pengine: [5539]: WARN: native_color: Resource
ocb2:1 cannot run anywhere
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Stopped)
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-dell)
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: LogActions: Start
dlm:0 (srv-dell)
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:1 (Stopped)
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: LogActions: Start
ocb2:0 (srv-dell)
Jun 4 11:22:25 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:1 (Stopped)
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 4 11:22:25 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition 2: 0 actions in 0 synapses
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_te_invoke: Processing
graph 2 (ref=pe_calc-dc-1244128945-20) derived from
/var/lib/pengine/pe-warn-70.bz2
Jun 4 11:22:25 srv-dell crmd: [5540]: info: run_graph:
====================================================
Jun 4 11:22:25 srv-dell crmd: [5540]: notice: run_graph: Transition 2
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-warn-70.bz2): Complete
Jun 4 11:22:25 srv-dell crmd: [5540]: info: te_graph_trigger:
Transition 2 is now complete
Jun 4 11:22:25 srv-dell crmd: [5540]: info: notify_crmd: Transition 2
status: done - <null>
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 4 11:22:25 srv-dell crmd: [5540]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 4 11:22:25 srv-dell pengine: [5539]: WARN: process_pe_message:
Transition 2: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-70.bz2
Jun 4 11:22:25 srv-dell pengine: [5539]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 4 11:22:26 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:22:26 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:22:59 srv-dell crm_resource: [9252]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:22:59 srv-dell crm_resource: [9252]: info: Invoked:
crm_resource -M -r primitiveresource_cluster_ip -H srv-dell -f
Jun 4 11:22:59 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section constraints
(origin=local/crm_resource/3, version=0.219.1): ok (rc=0)
Jun 4 11:22:59 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section constraints
(origin=local/crm_resource/4, version=0.219.1): ok (rc=0)
Jun 4 11:23:37 srv-dell cib: [5536]: info: cib_stats: Processed 117
operations (4273.00us average, 0% utilization) in the last 10min
Jun 4 11:23:55 srv-dell openais[5529]: [TOTEM] entering GATHER state from 11.
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] Saving state aru 4d
high seq received 4d
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] Storing new sequence
id for ring 110
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] entering COMMIT state.
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] entering RECOVERY state.
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] position [0] member
192.168.2.120:
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] previous ring seq 268
rep 192.168.2.120
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] aru a high delivered a
received flag 1
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] position [1] member
192.168.2.220:
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] previous ring seq 268
rep 192.168.2.220
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] aru 4d high delivered
4d received flag 1
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] Did not need to
originate any messages in recovery.
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Transitional membership event on ring 272: memb=1,
new=0, lost=0
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: memb: srv-dell 1543678144
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] CLM CONFIGURATION CHANGE
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] New Configuration:
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.120)
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.220)
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] Members Left:
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] Members Joined:
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] r(0) ip(192.168.2.120)
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] notice:
pcmk_peer_update: Stable membership event on ring 272: memb=2, new=1,
lost=0
Jun 4 11:23:57 srv-dell openais[5529]: [MAIN ] info: update_member:
Node 2013440192/srv-lenovo is now: member
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: NEW: srv-lenovo 2013440192
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: MEMB: srv-lenovo 2013440192
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] info:
pcmk_peer_update: MEMB: srv-dell 1543678144
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] info:
send_member_notification: Sending membership update 272 to 2 children
Jun 4 11:23:57 srv-dell openais[5529]: [SYNC ] This node is within
the primary component and will provide service.
Jun 4 11:23:57 srv-dell openais[5529]: [TOTEM] entering OPERATIONAL state.
Jun 4 11:23:57 srv-dell crmd: [5540]: notice: ais_dispatch:
Membership 272: quorum aquired
Jun 4 11:23:57 srv-dell crmd: [5540]: info: ais_status_callback:
status: srv-lenovo is now member (was lost)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=272
proc=00000000000000000000000000000002
Jun 4 11:23:57 srv-dell cib: [5536]: notice: ais_dispatch: Membership
272: quorum aquired
Jun 4 11:23:57 srv-dell cib: [5536]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=256 seen=272
proc=00000000000000000000000000000002
Jun 4 11:23:57 srv-dell crmd: [5540]: info: crm_update_quorum:
Updating quorum status to true (call=94)
Jun 4 11:23:57 srv-dell openais[5529]: [MAIN ] info: update_member:
0x7f5fa0034f60 Node 2013440192 (srv-lenovo) born on: 272
Jun 4 11:23:57 srv-dell openais[5529]: [MAIN ] info: update_member:
Node srv-lenovo now has process list: 00000000000000000000000000053312
(340754)
Jun 4 11:23:57 srv-dell openais[5529]: [crm ] info:
send_member_notification: Sending membership update 272 to 2 children
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] got nodejoin message
192.168.2.120
Jun 4 11:23:57 srv-dell openais[5529]: [CLM ] got nodejoin message
192.168.2.220
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/90,
version=0.219.2): ok (rc=0)
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=local/crmd/91, version=0.219.3): ok (rc=0)
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/92, version=0.219.3): ok (rc=0)
Jun 4 11:23:57 srv-dell cib: [5536]: info: log_data_element:
cib:diff: - <cib have-quorum="0" admin_epoch="0" epoch="219"
num_updates="4" />
Jun 4 11:23:57 srv-dell cib: [5536]: info: log_data_element:
cib:diff: + <cib have-quorum="1" admin_epoch="0" epoch="220"
num_updates="1" />
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/94, version=0.220.1): ok (rc=0)
Jun 4 11:23:57 srv-dell cib: [5536]: info: ais_dispatch: Membership
272: quorum retained
Jun 4 11:23:57 srv-dell cib: [5536]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=272 seen=272 proc=00000000000000000000000000053312 (new)
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_sync_one for section 'all'
(origin=srv-lenovo/srv-lenovo/(null), version=0.220.1): ok (rc=0)
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell crmd: [5540]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=dlm:0_monitor_0,
magic=0:7;7:3:7:caf53829-4bd1-4932-989e-f155cf0ecf6e) : Resource op
removal
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell cib: [9253]: info: write_cib_contents:
Archived previous version as /var/lib/heartbeat/crm/cib-7.raw
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/96, version=0.220.1): ok (rc=0)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: abort_transition_graph:
te_update_diff:157 - Triggered transition abort (complete=1,
tag=transient_attributes, id=srv-lenovo, magic=NA) : Transient
attribute: removal
Jun 4 11:23:57 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/transient_attributes":
ok (rc=0)
Jun 4 11:23:57 srv-dell cib: [9253]: info: write_cib_contents: Wrote
version 0.220.0 of the CIB to disk (digest:
b41ec493b0bd04316b66658fa480456d)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: abort_transition_graph:
need_abort:59 - Triggered transition abort (complete=1) : Non-status
change
Jun 4 11:23:57 srv-dell crmd: [5540]: info: need_abort: Aborting on
change to have-quorum
Jun 4 11:23:57 srv-dell cib: [9253]: info: retrieveCib: Reading
cluster configuration from: /var/lib/heartbeat/crm/cib.sFjB5k (digest:
/var/lib/heartbeat/crm/cib.Pru4k5)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
Membership changed: 268 -> 272 - join restart
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_pe_invoke: Query 97:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_FSA_INTERNAL origin=do_state_transition ]
Jun 4 11:23:57 srv-dell crmd: [5540]: info: update_dc: Unset DC srv-dell
Jun 4 11:23:57 srv-dell crmd: [5540]: info: join_make_offer: Making
join offers based on membership 272
Jun 4 11:23:57 srv-dell crmd: [5540]: info: join_make_offer: Peer
process on srv-lenovo is not active (yet?): 00000002 2
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_dc_join_offer_all:
join-2: Waiting on 1 outstanding join acks
Jun 4 11:23:57 srv-dell crmd: [5540]: info: ais_dispatch: Membership
272: quorum retained
Jun 4 11:23:57 srv-dell crmd: [5540]: info: crm_update_peer: Node
srv-lenovo: id=2013440192 state=member addr=r(0) ip(192.168.2.120)
votes=1 born=272 seen=272 proc=00000000000000000000000000053312 (new)
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/98, version=0.220.1): ok (rc=0)
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell crmd: [5540]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section crm_config
(origin=local/crmd/101, version=0.220.2): ok (rc=0)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition: All
1 cluster nodes responded to the join offer.
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_dc_join_finalize:
join-2: Syncing the CIB from srv-dell to the rest of the cluster
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/103, version=0.220.2): ok (rc=0)
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/104, version=0.220.2): ok (rc=0)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_dc_join_ack: join-2:
Updating node state to member for srv-dell
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/105,
version=0.220.3): ok (rc=0)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: do_state_transition: Only
1 of 2 cluster nodes are eligible to run resources - continue 0
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 4 11:23:57 srv-dell crmd: [5540]: info: crm_update_quorum:
Updating quorum status to true (call=109)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_pe_invoke: Query 110:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:23:57 srv-dell attrd: [5538]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 4 11:23:57 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/107, version=0.220.4): ok (rc=0)
Jun 4 11:23:57 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:57 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/109, version=0.220.5): ok (rc=0)
Jun 4 11:23:57 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244129037-27, seq=272, quorate=1
Jun 4 11:23:57 srv-dell pengine: [5539]: info:
determine_online_status_fencing: Node srv-lenovo is not ready to run
resources
Jun 4 11:23:57 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-lenovo is pending
Jun 4 11:23:57 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-dell is online
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-dell ]
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0 ]
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: clone_dlm
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: ocb2_clone
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ ocb2:0 ocb2:1 ]
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: native_color: Resource
cloneresource_cluster_fencing_null:0 cannot run anywhere
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: native_color: Resource
dlm:1 cannot run anywhere
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: native_color: Resource
ocb2:1 cannot run anywhere
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: custom_action: Action
primitiveresource_cluster_ip_monitor_0 on srv-lenovo is unrunnable
(pending)
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: custom_action: Action
cloneresource_cluster_fencing_null:0_monitor_0 on srv-lenovo is
unrunnable (pending)
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: custom_action: Action
dlm:0_monitor_0 on srv-lenovo is unrunnable (pending)
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: custom_action: Action
ocb2:0_monitor_0 on srv-lenovo is unrunnable (pending)
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (5s) for primitiveresource_cluster_ip on srv-dell
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (120s) for dlm:0 on srv-dell
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (120s) for ocb2:0 on srv-dell
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Stopped)
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-dell)
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: LogActions: Start
dlm:0 (srv-dell)
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:1 (Stopped)
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: LogActions: Start
ocb2:0 (srv-dell)
Jun 4 11:23:57 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:1 (Stopped)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 4 11:23:57 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition 3: 15 actions in 15 synapses
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_te_invoke: Processing
graph 3 (ref=pe_calc-dc-1244129037-27) derived from
/var/lib/pengine/pe-warn-71.bz2
Jun 4 11:23:57 srv-dell crmd: [5540]: notice: run_graph:
====================================================
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: run_graph: Transition 3
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=15,
Source=/var/lib/pengine/pe-warn-71.bz2): Terminated
Jun 4 11:23:57 srv-dell crmd: [5540]: ERROR: te_graph_trigger:
Transition failed: terminated
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Graph 3 (15
actions in 15 synapses): batch-limit=30 jobs, network-delay=60000ms
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 0 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
9]: Pending (id: primitiveresource_cluster_ip_start_0, loc: srv-dell,
priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 1 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
10]: Pending (id: primitiveresource_cluster_ip_monitor_5000, loc:
srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
9]: Pending (id: primitiveresource_cluster_ip_start_0, loc: srv-dell,
priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 2 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
17]: Pending (id: dlm:0_start_0, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
19]: Pending (id: clone_dlm_start_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 3 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
18]: Pending (id: dlm:0_monitor_120000, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
17]: Pending (id: dlm:0_start_0, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 4 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
19]: Pending (id: clone_dlm_start_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell pengine: [5539]: WARN: process_pe_message:
Transition 3: WARNINGs found during PE processing. PEngine Input
stored in: /var/lib/pengine/pe-warn-71.bz2
Jun 4 11:23:57 srv-dell pengine: [5539]: info: process_pe_message:
Configuration WARNINGs found during PE processing. Please run
"crm_verify -L" to identify issues.
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
21]: Pending (id: clone_dlm_stop_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 5 is
pending (priority: 1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
20]: Pending (id: clone_dlm_running_0, type: pseduo, priority:
1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
17]: Pending (id: dlm:0_start_0, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
19]: Pending (id: clone_dlm_start_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 6 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
21]: Pending (id: clone_dlm_stop_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
28]: Pending (id: ocb2_clone_stopped_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 7 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
23]: Pending (id: ocb2:0_start_0, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
25]: Pending (id: ocb2_clone_start_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 8 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
24]: Pending (id: ocb2:0_monitor_120000, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
23]: Pending (id: ocb2:0_start_0, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 9 is
pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
25]: Pending (id: ocb2_clone_start_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
20]: Pending (id: clone_dlm_running_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
27]: Pending (id: ocb2_clone_stop_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
28]: Pending (id: ocb2_clone_stopped_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 10
is pending (priority: 1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
26]: Pending (id: ocb2_clone_running_0, type: pseduo, priority:
1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
23]: Pending (id: ocb2:0_start_0, loc: srv-dell, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
25]: Pending (id: ocb2_clone_start_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 11
is pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
27]: Pending (id: ocb2_clone_stop_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 12
is pending (priority: 1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
28]: Pending (id: ocb2_clone_stopped_0, type: pseduo, priority:
1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
27]: Pending (id: ocb2_clone_stop_0, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 13
is pending (priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
2]: Pending (id: probe_complete, type: pseduo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
4]: Pending (id: probe_complete, loc: srv-lenovo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_graph: Synapse 14
is pending (priority: 1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: [Action
4]: Pending (id: probe_complete, loc: srv-lenovo, priority: 1000000)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
5]: Pending (id: primitiveresource_cluster_ip_monitor_0, loc:
srv-lenovo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
6]: Pending (id: cloneresource_cluster_fencing_null:0_monitor_0, loc:
srv-lenovo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
7]: Pending (id: dlm:0_monitor_0, loc: srv-lenovo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: WARN: print_elem: * [Input
8]: Pending (id: ocb2:0_monitor_0, loc: srv-lenovo, priority: 0)
Jun 4 11:23:57 srv-dell crmd: [5540]: info: te_graph_trigger:
Transition 3 is now complete
Jun 4 11:23:57 srv-dell crmd: [5540]: info: notify_crmd: Transition 3
status: done - <null>
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 4 11:23:57 srv-dell crmd: [5540]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 4 11:23:57 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_HA_MESSAGE origin=route_message ]
Jun 4 11:23:58 srv-dell crmd: [5540]: info: abort_transition_graph:
do_te_invoke:193 - Triggered transition abort (complete=1) : Peer Halt
Jun 4 11:23:58 srv-dell crmd: [5540]: info: update_dc: Unset DC srv-dell
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_dc_join_offer_all: A
new node joined the cluster
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_dc_join_offer_all:
join-3: Waiting on 2 outstanding join acks
Jun 4 11:23:58 srv-dell crmd: [5540]: info: update_dc: Set DC to
srv-dell (3.0.1)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_state_transition: All
2 cluster nodes responded to the join offer.
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_dc_join_finalize:
join-3: Syncing the CIB from srv-dell to the rest of the cluster
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_sync for section 'all'
(origin=local/crmd/112, version=0.220.5): ok (rc=0)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_dc_join_ack: join-3:
Updating node state to member for srv-lenovo
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_dc_join_ack: join-3:
Updating node state to member for srv-dell
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/113, version=0.220.5): ok (rc=0)
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/114, version=0.220.5): ok (rc=0)
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=local/crmd/115,
version=0.220.5): ok (rc=0)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-lenovo']/lrm": ok (rc=0)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [
input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_dc_join_final:
Ensuring DC, quorum and node attributes are up-to-date
Jun 4 11:23:58 srv-dell crmd: [5540]: info: crm_update_quorum:
Updating quorum status to true (call=121)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: abort_transition_graph:
do_te_invoke:190 - Triggered transition abort (complete=1) : Peer
Cancelled
Jun 4 11:23:58 srv-dell attrd: [5538]: info: attrd_local_callback:
Sending full refresh (origin=crmd)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_pe_invoke: Query 122:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: terminate
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-dell']/lrm (origin=local/crmd/117,
version=0.220.7): ok (rc=0)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: abort_transition_graph:
te_update_diff:267 - Triggered transition abort (complete=1,
tag=lrm_rsc_op, id=dlm:0_monitor_0,
magic=0:7;6:81:7:caf53829-4bd1-4932-989e-f155cf0ecf6e) : Resource op
removal
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: erase_xpath_callback:
Deletion of "//node_state[@uname='srv-dell']/lrm": ok (rc=0)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_pe_invoke: Query 123:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/transient_attributes
(origin=srv-lenovo/crmd/6, version=0.220.7): ok (rc=0)
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_delete for section
//node_state[@uname='srv-lenovo']/lrm (origin=srv-lenovo/crmd/7,
version=0.220.8): ok (rc=0)
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_update_diff: Detected
LRM refresh - 4 resources updated: Skipping all resource events
Jun 4 11:23:58 srv-dell crmd: [5540]: info: abort_transition_graph:
te_update_diff:227 - Triggered transition abort (complete=1, tag=diff,
id=(null), magic=NA) : LRM Refresh
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_pe_invoke: Query 124:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section nodes
(origin=local/crmd/119, version=0.220.9): ok (rc=0)
Jun 4 11:23:58 srv-dell cib: [5536]: info: cib_process_request:
Operation complete: op cib_modify for section cib
(origin=local/crmd/121, version=0.220.9): ok (rc=0)
Jun 4 11:23:58 srv-dell attrd: [5538]: info: attrd_trigger_update:
Sending flush op to all hosts for: shutdown
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244129038-34, seq=272, quorate=1
Jun 4 11:23:58 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-lenovo is online
Jun 4 11:23:58 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-dell is online
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Stopped
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-dell ]
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ cloneresource_cluster_fencing_null:0 ]
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: clone_dlm
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ dlm:0 dlm:1 ]
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: ocb2_clone
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: print_list:
Stopped: [ ocb2:0 ocb2:1 ]
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (5s) for primitiveresource_cluster_ip on srv-dell
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (120s) for dlm:0 on srv-lenovo
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (120s) for dlm:1 on srv-dell
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (120s) for ocb2:0 on srv-lenovo
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: RecurringOp: Start
recurring monitor (120s) for ocb2:1 on srv-dell
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: LogActions: Start
primitiveresource_cluster_ip (srv-dell)
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: LogActions: Start
cloneresource_cluster_fencing_null:0 (srv-lenovo)
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-dell)
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: LogActions: Start
dlm:0 (srv-lenovo)
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: LogActions: Start
dlm:1 (srv-dell)
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: LogActions: Start
ocb2:0 (srv-lenovo)
Jun 4 11:23:58 srv-dell pengine: [5539]: notice: LogActions: Start
ocb2:1 (srv-dell)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 4 11:23:58 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition 4: 26 actions in 26 synapses
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_te_invoke: Processing
graph 4 (ref=pe_calc-dc-1244129038-34) derived from
/var/lib/pengine/pe-input-23.bz2
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 5: monitor primitiveresource_cluster_ip_monitor_0 on
srv-lenovo
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 6: monitor
cloneresource_cluster_fencing_null:0_monitor_0 on srv-lenovo
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 7: monitor dlm:0_monitor_0 on srv-lenovo
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 8: monitor ocb2:0_monitor_0 on srv-lenovo
Jun 4 11:23:58 srv-dell pengine: [5539]: info: process_pe_message:
Transition 4: PEngine Input stored in:
/var/lib/pengine/pe-input-23.bz2
Jun 4 11:23:58 srv-dell crmd: [5540]: info: match_graph_event: Action
ocb2:0_monitor_0 (8) confirmed on srv-lenovo (rc=0)
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: match_graph_event: Action
primitiveresource_cluster_ip_monitor_0 (5) confirmed on srv-lenovo
(rc=0)
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: match_graph_event: Action
dlm:0_monitor_0 (7) confirmed on srv-lenovo (rc=0)
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: match_graph_event: Action
cloneresource_cluster_fencing_null:0_monitor_0 (6) confirmed on
srv-lenovo (rc=0)
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 4: probe_complete probe_complete on srv-lenovo - no
waiting
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 2 fired and confirmed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 9: start primitiveresource_cluster_ip_start_0 on
srv-dell (local)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=9:4:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a
op=primitiveresource_cluster_ip_start_0 )
Jun 4 11:23:58 srv-dell lrmd: [5537]: info:
rsc:primitiveresource_cluster_ip: start
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 14 fired and confirmed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 32 fired and confirmed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 33 fired and confirmed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 11: start
cloneresource_cluster_fencing_null:0_start_0 on srv-lenovo
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 24 fired and confirmed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 22 fired and confirmed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 18: start dlm:0_start_0 on srv-lenovo
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 20: start dlm:1_start_0 on srv-dell (local)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=20:4:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a op=dlm:1_start_0 )
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: rsc:dlm:1: start
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: RA output:
(dlm:1:start:stderr) dlm_controld.pcmk: no process killed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: match_graph_event: Action
cloneresource_cluster_fencing_null:0_start_0 (11) confirmed on
srv-lenovo (rc=0)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 15 fired and confirmed
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell cluster-dlm[9295]: main: dlm_controld 1239267206
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info:
init_ais_connection: AIS connection established
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info: crm_new_peer: Node
srv-dell now has id: 1543678144
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info: crm_new_peer: Node
1543678144 is now known as srv-dell
Jun 4 11:23:58 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 9295 (0x7f5fa0035160)
Jun 4 11:23:58 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm-control minor 58
Jun 4 11:23:58 srv-dell cluster-dlm: setup_misc_devices: found
/dev/misc/dlm_plock minor 57
Jun 4 11:23:58 srv-dell cluster-dlm: update_comms_nodes:
/sys/kernel/config/dlm/cluster/comms: opendir failed: 2
Jun 4 11:23:58 srv-dell cluster-dlm: clear_configfs_spaces:
/sys/kernel/config/dlm/cluster/spaces: opendir failed: 2
Jun 4 11:23:58 srv-dell cluster-dlm: setup_cpg: setup_cpg 10
Jun 4 11:23:58 srv-dell cluster-dlm: receive_protocol: run protocol
from nodeid 2013440192
Jun 4 11:23:58 srv-dell cluster-dlm: set_protocol: daemon run 1.1.1
max 1.1.1 kernel run 1.1.1 max 1.1.1
Jun 4 11:23:58 srv-dell cluster-dlm: setup_plocks: plocks 12
Jun 4 11:23:58 srv-dell cluster-dlm: setup_plocks: plock cpg message
size: 104 bytes
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: notice: ais_dispatch:
Membership 272: quorum aquired
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info: crm_new_peer: Node
srv-lenovo now has id: 2013440192
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info: crm_new_peer: Node
2013440192 is now known as srv-lenovo
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info: crm_update_peer:
Node srv-lenovo: id=2013440192 state=member (new) addr=r(0)
ip(192.168.2.120) votes=1 born=272 seen=272
proc=00000000000000000000000000053312
Jun 4 11:23:58 srv-dell cluster-dlm: [9295]: info: crm_update_peer:
Node srv-dell: id=1543678144 state=member (new) addr=r(0)
ip(192.168.2.220) (new) votes=1 (new) born=264 seen=272
proc=00000000000000000000000000053312 (new)
Jun 4 11:23:58 srv-dell cluster-dlm: update_cluster: Processing membership 272
Jun 4 11:23:58 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.120) to configfs for node 2013440192
Jun 4 11:23:58 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 2013440192 192.168.2.120 local 0
Jun 4 11:23:58 srv-dell cluster-dlm: dlm_process_node: Added active
node 2013440192: born-on=272, last-seen=272, this-event=272,
last-event=0
Jun 4 11:23:58 srv-dell cluster-dlm: dlm_process_node: Adding address
ip(192.168.2.220) to configfs for node 1543678144
Jun 4 11:23:58 srv-dell cluster-dlm: add_configfs_node:
set_configfs_node 1543678144 192.168.2.220 local 1
Jun 4 11:23:58 srv-dell cluster-dlm: dlm_process_node: Added active
node 1543678144: born-on=264, last-seen=272, this-event=272,
last-event=0
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:23:58 INFO:
Using calculated nic for 192.168.2.10: eth1
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:23:58 INFO:
Using calculated netmask for 192.168.2.10: 255.255.255.0
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) logd is not running
Jun 4 11:23:58 srv-dell lrmd: [5537]: info: RA output:
(primitiveresource_cluster_ip:start:stderr) 2009/06/04_11:23:58 INFO:
eval ifconfig eth1:0 192.168.2.10 netmask 255.255.255.0 broadcast
192.168.2.255
Jun 4 11:23:58 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_start_0 (call=48, rc=0,
cib-update=125, confirmed=true) complete ok
Jun 4 11:23:58 srv-dell crmd: [5540]: info: match_graph_event: Action
primitiveresource_cluster_ip_start_0 (9) confirmed on srv-dell (rc=0)
Jun 4 11:23:58 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 10: monitor
primitiveresource_cluster_ip_monitor_5000 on srv-dell (local)
Jun 4 11:23:58 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:58 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=10:4:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a
op=primitiveresource_cluster_ip_monitor_5000 )
Jun 4 11:23:58 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:23:59 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:1_start_0 (call=49, rc=0, cib-update=126,
confirmed=true) complete ok
Jun 4 11:23:59 srv-dell crmd: [5540]: info: match_graph_event: Action
dlm:1_start_0 (20) confirmed on srv-dell (rc=0)
Jun 4 11:23:59 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 21: monitor dlm:1_monitor_120000 on srv-dell (local)
Jun 4 11:23:59 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=21:4:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a
op=dlm:1_monitor_120000 )
Jun 4 11:23:59 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:59 srv-dell crmd: [5540]: info: match_graph_event: Action
dlm:0_start_0 (18) confirmed on srv-lenovo (rc=0)
Jun 4 11:23:59 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 19: monitor dlm:0_monitor_120000 on srv-lenovo
Jun 4 11:23:59 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:59 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 23 fired and confirmed
Jun 4 11:23:59 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 30 fired and confirmed
Jun 4 11:23:59 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 26: start ocb2:0_start_0 on srv-lenovo
Jun 4 11:23:59 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 28: start ocb2:1_start_0 on srv-dell (local)
Jun 4 11:23:59 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=28:4:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a op=ocb2:1_start_0 )
Jun 4 11:23:59 srv-dell lrmd: [5537]: info: rsc:ocb2:1: start
Jun 4 11:23:59 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation dlm:1_monitor_120000 (call=51, rc=0, cib-update=127,
confirmed=false) complete ok
Jun 4 11:23:59 srv-dell crmd: [5540]: info: match_graph_event: Action
dlm:1_monitor_120000 (21) confirmed on srv-dell (rc=0)
Jun 4 11:23:59 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:59 srv-dell lrmd: [5537]: info: RA output:
(ocb2:1:start:stderr) logd is not running
Jun 4 11:23:59 srv-dell lrmd: [5537]: info: RA output:
(ocb2:1:start:stderr) 2009/06/04_11:23:59 INFO: Stack glue driver not
loaded
Jun 4 11:23:59 srv-dell lrmd: [5537]: info: RA output:
(ocb2:1:start:stderr) logd is not running
Jun 4 11:23:59 srv-dell crmd: [5540]: info: match_graph_event: Action
dlm:0_monitor_120000 (19) confirmed on srv-lenovo (rc=0)
Jun 4 11:23:59 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:59 srv-dell lrmd: [5537]: info: RA output:
(ocb2:1:start:stderr) 2009/06/04_11:23:59 INFO: Starting ocb2:1
Jun 4 11:23:59 srv-dell kernel: ocfs2: Registered cluster interface user
Jun 4 11:23:59 srv-dell kernel: OCFS2 Node Manager 1.5.0
Jun 4 11:23:59 srv-dell kernel: OCFS2 1.5.0
Jun 4 11:23:59 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation primitiveresource_cluster_ip_monitor_5000 (call=50, rc=0,
cib-update=128, confirmed=false) complete ok
Jun 4 11:23:59 srv-dell ocfs2_controld.pcmk: Core dumps enabled:
/var/lib/openais
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: WARN: Initializing
connection to logging daemon failed. Logging daemon may not be running
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info:
init_ais_connection: Creating connection to our AIS plugin
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info:
init_ais_connection: AIS connection established
Jun 4 11:23:59 srv-dell crmd: [5540]: info: match_graph_event: Action
primitiveresource_cluster_ip_monitor_5000 (10) confirmed on srv-dell
(rc=0)
Jun 4 11:23:59 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info: get_ais_nodeid:
Server details: id=1543678144 uname=srv-dell
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info: crm_new_peer:
Node srv-dell now has id: 1543678144
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info: crm_new_peer:
Node 1543678144 is now known as srv-dell
Jun 4 11:23:59 srv-dell openais[5529]: [crm ] info: pcmk_notify:
Enabling node notifications for child 9490 (0x7f5fa0035460)
Jun 4 11:23:59 srv-dell ocfs2_controld: Added Pacemaker as client 1 with fd 3
Jun 4 11:23:59 srv-dell ocfs2_controld: Initializing CKPT service (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Connected to CKPT service
with handle 0x625558ec00000000
Jun 4 11:23:59 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld:5c02a8c0" with handle 0x3d1b58ba00000000
Jun 4 11:23:59 srv-dell ocfs2_controld: Writing to section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Creating section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Created section
"daemon_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:23:59 srv-dell ocfs2_controld: Writing to section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Creating section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Created section
"ocfs2_max_protocol" on checkpoint "ocfs2:controld:5c02a8c0"
Jun 4 11:23:59 srv-dell ocfs2_controld: Starting join for group
"ocfs2:controld"
Jun 4 11:23:59 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:23:59 srv-dell ocfs2_controld: setup done
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: notice: ais_dispatch:
Membership 272: quorum aquired
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info: crm_new_peer:
Node srv-lenovo now has id: 2013440192
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info: crm_new_peer:
Node 2013440192 is now known as srv-lenovo
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info:
crm_update_peer: Node srv-lenovo: id=2013440192 state=member (new)
addr=r(0) ip(192.168.2.120) votes=1 born=272 seen=272
proc=00000000000000000000000000053312
Jun 4 11:23:59 srv-dell ocfs2_controld: [9490]: info:
crm_update_peer: Node srv-dell: id=1543678144 state=member (new)
addr=r(0) ip(192.168.2.220) (new) votes=1 (new) born=264 seen=272
proc=00000000000000000000000000053312 (new)
Jun 4 11:23:59 srv-dell ocfs2_controld: confchg called
Jun 4 11:23:59 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 1, left 0, joined 1
Jun 4 11:23:59 srv-dell ocfs2_controld: CPG is live, we are the first daemon
Jun 4 11:23:59 srv-dell ocfs2_controld: Opening checkpoint
"ocfs2:controld" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Opened checkpoint
"ocfs2:controld" with handle 0x79e2a9e300000001
Jun 4 11:23:59 srv-dell ocfs2_controld: Writing to section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Creating section
"daemon_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Created section
"daemon_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:23:59 srv-dell ocfs2_controld: Writing to section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Creating section
"ocfs2_protocol" on checkpoint "ocfs2:controld" (try 1)
Jun 4 11:23:59 srv-dell ocfs2_controld: Created section
"ocfs2_protocol" on checkpoint "ocfs2:controld"
Jun 4 11:23:59 srv-dell ocfs2_controld: Daemon protocol is 1.0
Jun 4 11:23:59 srv-dell ocfs2_controld: fs protocol is 1.0
Jun 4 11:23:59 srv-dell ocfs2_controld: Connecting to dlm_controld
Jun 4 11:23:59 srv-dell cluster-dlm: process_listener: client
connection 5 fd 13
Jun 4 11:23:59 srv-dell ocfs2_controld: Opening control device
Jun 4 11:23:59 srv-dell ocfs2_controld: Starting to listen for mounters
Jun 4 11:23:59 srv-dell ocfs2_controld: new listening connection 4
Jun 4 11:23:59 srv-dell ocfs2_controld: confchg called
Jun 4 11:23:59 srv-dell ocfs2_controld: ocfs2_controld (group
"ocfs2:controld") confchg: members 2, left 0, joined 1
Jun 4 11:23:59 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:24:01 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:1_start_0 (call=52, rc=0, cib-update=129,
confirmed=true) complete ok
Jun 4 11:24:01 srv-dell crmd: [5540]: info: match_graph_event: Action
ocb2:1_start_0 (28) confirmed on srv-dell (rc=0)
Jun 4 11:24:01 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 29: monitor ocb2:1_monitor_120000 on srv-dell
(local)
Jun 4 11:24:01 srv-dell crmd: [5540]: info: do_lrm_rsc_op: Performing
key=29:4:0:398e0ea6-cd74-4c24-9f3c-f6b0e65baf0a
op=ocb2:1_monitor_120000 )
Jun 4 11:24:01 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:24:01 srv-dell crmd: [5540]: info: match_graph_event: Action
ocb2:0_start_0 (26) confirmed on srv-lenovo (rc=0)
Jun 4 11:24:01 srv-dell crmd: [5540]: info: te_rsc_command:
Initiating action 27: monitor ocb2:0_monitor_120000 on srv-lenovo
Jun 4 11:24:01 srv-dell crmd: [5540]: info: te_pseudo_action: Pseudo
action 31 fired and confirmed
Jun 4 11:24:01 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:24:01 srv-dell crmd: [5540]: info: process_lrm_event: LRM
operation ocb2:1_monitor_120000 (call=53, rc=0, cib-update=130,
confirmed=false) complete ok
Jun 4 11:24:01 srv-dell crmd: [5540]: info: match_graph_event: Action
ocb2:1_monitor_120000 (29) confirmed on srv-dell (rc=0)
Jun 4 11:24:01 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:24:01 srv-dell crmd: [5540]: info: match_graph_event: Action
ocb2:0_monitor_120000 (27) confirmed on srv-lenovo (rc=0)
Jun 4 11:24:01 srv-dell haclient: on_event:evt:cib_changed
Jun 4 11:24:01 srv-dell crmd: [5540]: info: run_graph:
====================================================
Jun 4 11:24:01 srv-dell crmd: [5540]: notice: run_graph: Transition 4
(Complete=26, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-23.bz2): Complete
Jun 4 11:24:01 srv-dell crmd: [5540]: info: te_graph_trigger:
Transition 4 is now complete
Jun 4 11:24:01 srv-dell crmd: [5540]: info: notify_crmd: Transition 4
status: done - <null>
Jun 4 11:24:01 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 4 11:24:01 srv-dell crmd: [5540]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 4 11:24:02 srv-dell mgmtd: [5541]: info: CIB query: cib
Jun 4 11:28:26 srv-dell kernel: No iBFT detected.
Jun 4 11:28:49 srv-dell kernel: scsi3 : iSCSI Initiator over TCP/IP
Jun 4 11:28:49 srv-dell kernel: scsi 3:0:0:0: Direct-Access IET
VIRTUAL-DISK 0 PQ: 0 ANSI: 4
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] 46379655 512-byte
hardware sectors: (23.7GB/22.1GiB)
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] Write Protect is off
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] Mode Sense: 77 00 00 08
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] Write cache:
disabled, read cache: enabled, doesn't support DPO or FUA
Jun 4 11:28:49 srv-dell kernel: sdb: sdb1
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: [sdb] Attached SCSI disk
Jun 4 11:28:49 srv-dell kernel: sd 3:0:0:0: Attached scsi generic sg3 type 0
Jun 4 11:28:49 srv-dell ocfs2_controld: new client connection 5
Jun 4 11:28:49 srv-dell ocfs2_controld: client msg
Jun 4 11:28:49 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 4 11:28:49 srv-dell ocfs2_controld: start_mount: uuid
"E79BFDBA89F049ADB8E0895DAF6CE990", device "/dev/sdb1", service
"ocfs2"
Jun 4 11:28:49 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell ocfs2_controld: Starting join for group
"ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990"
Jun 4 11:28:49 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:28:49 srv-dell ocfs2_controld: start_mount returns 0
Jun 4 11:28:49 srv-dell ocfs2_controld: confchg called
Jun 4 11:28:49 srv-dell ocfs2_controld: group
"ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990" confchg: members 1, left 0,
joined 1
Jun 4 11:28:49 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell ocfs2_controld: This node joins group
ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell ocfs2_controld: Registering mountgroup
E79BFDBA89F049ADB8E0895DAF6CE990 with dlm_controld
Jun 4 11:28:49 srv-dell ocfs2_controld: Registering
"E79BFDBA89F049ADB8E0895DAF6CE990" with dlm_controld
Jun 4 11:28:49 srv-dell ocfs2_controld: message from dlmcontrol
Jun 4 11:28:49 srv-dell ocfs2_controld: Registration of
"E79BFDBA89F049ADB8E0895DAF6CE990" complete
Jun 4 11:28:49 srv-dell ocfs2_controld: Mountgroup
E79BFDBA89F049ADB8E0895DAF6CE990 successfully registered with
dlm_controld
Jun 4 11:28:49 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 4 11:28:49 srv-dell ocfs2_controld: Notified client: 1
Jun 4 11:28:49 srv-dell kernel: dlm: Using TCP for communications
Jun 4 11:28:49 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell cluster-dlm: process_uevent: kernel: add@
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell cluster-dlm: process_uevent: kernel: online@
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:28:49 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 4 11:28:49 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 4 11:28:49 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 1 joined 1 remove 0 failed 0
Jun 4 11:28:49 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 4 11:28:49 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 4 11:28:49 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 4 11:28:49 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 1 1 0 0
Jun 4 11:28:49 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 76
Jun 4 11:28:49 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 4 11:28:49 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 1
Jun 4 11:28:49 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 1
Jun 4 11:28:49 srv-dell cluster-dlm: do_sysfs: write "2248110482" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/id"
Jun 4 11:28:49 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/E79BFDBA89F049ADB8E0895DAF6CE990/nodes/1543678144"
Jun 4 11:28:49 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/control"
Jun 4 11:28:49 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/event_done"
Jun 4 11:28:49 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 4 11:28:49 srv-dell kernel: ocfs2: Mounting device (8,17) on
(node 1543678, slot 0) with ordered data mode.
Jun 4 11:28:49 srv-dell ocfs2_controld: client msg
Jun 4 11:28:49 srv-dell ocfs2_controld: client message 1 from 5: MRESULT
Jun 4 11:28:49 srv-dell ocfs2_controld: complete_mount: uuid
"E79BFDBA89F049ADB8E0895DAF6CE990", errcode "0", service "ocfs2"
Jun 4 11:28:49 srv-dell ocfs2_controld: client msg
Jun 4 11:28:49 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 4 11:28:49 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 4 11:28:49 srv-dell ocfs2_hb_ctl[10871]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -P -d /dev/sdb1
Jun 4 11:28:49 srv-dell ocfs2_controld: new client connection 5
Jun 4 11:28:49 srv-dell ocfs2_controld: client msg
Jun 4 11:28:49 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 4 11:28:49 srv-dell ocfs2_controld: client msg
Jun 4 11:28:49 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 4 11:28:49 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 4 11:28:49 srv-dell hald: mounted /dev/sdb1 on behalf of uid 0
Jun 4 11:28:50 srv-dell iscsid: connection1:0 is operational now
Jun 4 11:29:06 srv-dell cluster-dlm: process_uevent: uevent:
offline@/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell cluster-dlm: process_uevent: kernel: offline@
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell cluster-dlm: confchg_cb: confchg for our leave
Jun 4 11:29:06 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 0
Jun 4 11:29:06 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/control"
Jun 4 11:29:06 srv-dell cluster-dlm: update_dir_members: dir_member 1543678144
Jun 4 11:29:06 srv-dell cluster-dlm: set_configfs_members:
set_members rmdir
"/sys/kernel/config/dlm/cluster/spaces/E79BFDBA89F049ADB8E0895DAF6CE990/nodes/1543678144"
Jun 4 11:29:06 srv-dell cluster-dlm: set_configfs_members:
set_members lockspace rmdir
"/sys/kernel/config/dlm/cluster/spaces/E79BFDBA89F049ADB8E0895DAF6CE990"
Jun 4 11:29:06 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/event_done"
Jun 4 11:29:06 srv-dell cluster-dlm: purge_plocks: purged 0 plocks
for 1543678144
Jun 4 11:29:06 srv-dell cluster-dlm: process_uevent: uevent:
remove@/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell cluster-dlm: process_uevent: kernel: remove@
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell ocfs2_hb_ctl[10955]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -K -u E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell ocfs2_controld: new client connection 5
Jun 4 11:29:06 srv-dell ocfs2_controld: client msg
Jun 4 11:29:06 srv-dell ocfs2_controld: client message 2 from 5: UNMOUNT
Jun 4 11:29:06 srv-dell ocfs2_controld: remove_mount: uuid
"E79BFDBA89F049ADB8E0895DAF6CE990", service "ocfs2"
Jun 4 11:29:06 srv-dell ocfs2_controld: Unregistering mountgroup
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell ocfs2_controld: Unregistering
"E79BFDBA89F049ADB8E0895DAF6CE990" from dlm_controld
Jun 4 11:29:06 srv-dell ocfs2_controld: time to leave group
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell ocfs2_controld: calling LEAVE for group
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell ocfs2_controld: leaving group
"ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990"
Jun 4 11:29:06 srv-dell ocfs2_controld: cpg_leave succeeded
Jun 4 11:29:06 srv-dell ocfs2_controld: confchg called
Jun 4 11:29:06 srv-dell ocfs2_controld: group
"ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990" confchg: members 0, left 1,
joined 0
Jun 4 11:29:06 srv-dell ocfs2_controld: Node 1543678144 leaves group
ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:06 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 4 11:29:06 srv-dell ocfs2_controld: Notified client: 1
Jun 4 11:29:06 srv-dell ocfs2_controld: client 6 fd 14 dead
Jun 4 11:29:06 srv-dell ocfs2_controld: client msg
Jun 4 11:29:06 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 4 11:29:06 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 4 11:29:06 srv-dell kernel: ocfs2: Unmounting device (8,17) on
(node 1543678)
Jun 4 11:29:06 srv-dell hald: unmounted /dev/sdb1 from
'/media/disk-1' on behalf of uid 0
Jun 4 11:29:59 srv-dell ocfs2_controld: new client connection 5
Jun 4 11:29:59 srv-dell ocfs2_controld: client msg
Jun 4 11:29:59 srv-dell ocfs2_controld: client message 0 from 5: MOUNT
Jun 4 11:29:59 srv-dell ocfs2_controld: start_mount: uuid
"E79BFDBA89F049ADB8E0895DAF6CE990", device "/dev/sdb1", service
"ocfs2"
Jun 4 11:29:59 srv-dell ocfs2_controld: Adding service ocfs2 to
device /dev/sdb1 uuid E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell ocfs2_controld: Starting join for group
"ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990"
Jun 4 11:29:59 srv-dell ocfs2_controld: cpg_join succeeded
Jun 4 11:29:59 srv-dell ocfs2_controld: start_mount returns 0
Jun 4 11:29:59 srv-dell ocfs2_controld: confchg called
Jun 4 11:29:59 srv-dell ocfs2_controld: group
"ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990" confchg: members 1, left 0,
joined 1
Jun 4 11:29:59 srv-dell ocfs2_controld: Node 1543678144 joins group
ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell ocfs2_controld: This node joins group
ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell ocfs2_controld: Filling node 1543678144 to
group ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell ocfs2_controld: Registering mountgroup
E79BFDBA89F049ADB8E0895DAF6CE990 with dlm_controld
Jun 4 11:29:59 srv-dell ocfs2_controld: Registering
"E79BFDBA89F049ADB8E0895DAF6CE990" with dlm_controld
Jun 4 11:29:59 srv-dell ocfs2_controld: message from dlmcontrol
Jun 4 11:29:59 srv-dell ocfs2_controld: Registration of
"E79BFDBA89F049ADB8E0895DAF6CE990" complete
Jun 4 11:29:59 srv-dell ocfs2_controld: Mountgroup
E79BFDBA89F049ADB8E0895DAF6CE990 successfully registered with
dlm_controld
Jun 4 11:29:59 srv-dell ocfs2_controld: notify_mount_client sending 0 "OK"
Jun 4 11:29:59 srv-dell ocfs2_controld: Notified client: 1
Jun 4 11:29:59 srv-dell kernel: dlm: Using TCP for communications
Jun 4 11:29:59 srv-dell cluster-dlm: process_uevent: uevent:
add@/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell cluster-dlm: process_uevent: kernel: add@
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell cluster-dlm: process_uevent: uevent:
online@/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell cluster-dlm: process_uevent: kernel: online@
E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:29:59 srv-dell cluster-dlm: add_change: add_change cg 1
joined nodeid 1543678144
Jun 4 11:29:59 srv-dell cluster-dlm: add_change: add_change cg 1 we joined
Jun 4 11:29:59 srv-dell cluster-dlm: add_change: add_change cg 1
counts member 1 joined 1 remove 0 failed 0
Jun 4 11:29:59 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 4 11:29:59 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 4 11:29:59 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 4 11:29:59 srv-dell cluster-dlm: send_info: send_start cg 1 flags
1 counts 0 1 1 0 0
Jun 4 11:29:59 srv-dell cluster-dlm: receive_start: receive_start
1543678144:1 len 76
Jun 4 11:29:59 srv-dell cluster-dlm: match_change: match_change
1543678144:1 matches cg 1
Jun 4 11:29:59 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 1 got all 1
Jun 4 11:29:59 srv-dell cluster-dlm: start_kernel: start_kernel cg 1
member_count 1
Jun 4 11:29:59 srv-dell cluster-dlm: do_sysfs: write "2248110482" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/id"
Jun 4 11:29:59 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/E79BFDBA89F049ADB8E0895DAF6CE990/nodes/1543678144"
Jun 4 11:29:59 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/control"
Jun 4 11:29:59 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/event_done"
Jun 4 11:29:59 srv-dell kernel: kjournald2 starting. Commit interval 5 seconds
Jun 4 11:29:59 srv-dell kernel: ocfs2: Mounting device (8,17) on
(node 1543678, slot 0) with ordered data mode.
Jun 4 11:29:59 srv-dell ocfs2_controld: client msg
Jun 4 11:29:59 srv-dell ocfs2_controld: client message 1 from 5: MRESULT
Jun 4 11:29:59 srv-dell ocfs2_controld: complete_mount: uuid
"E79BFDBA89F049ADB8E0895DAF6CE990", errcode "0", service "ocfs2"
Jun 4 11:29:59 srv-dell ocfs2_controld: client msg
Jun 4 11:29:59 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 4 11:29:59 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 4 11:29:59 srv-dell ocfs2_hb_ctl[11330]: ocfs2_hb_ctl
/sbin/ocfs2_hb_ctl -P -d /dev/sdb1
Jun 4 11:29:59 srv-dell ocfs2_controld: new client connection 5
Jun 4 11:29:59 srv-dell ocfs2_controld: client msg
Jun 4 11:29:59 srv-dell ocfs2_controld: client message 6 from 5: LISTCLUSTERS
Jun 4 11:29:59 srv-dell ocfs2_controld: client msg
Jun 4 11:29:59 srv-dell ocfs2_controld: client 5 fd 13 dead
Jun 4 11:29:59 srv-dell ocfs2_controld: client 5 fd -1 dead
Jun 4 11:30:15 srv-dell ocfs2_controld: confchg called
Jun 4 11:30:15 srv-dell ocfs2_controld: group
"ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990" confchg: members 2, left 0,
joined 1
Jun 4 11:30:15 srv-dell ocfs2_controld: Node 2013440192 joins group
ocfs2:E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:30:15 srv-dell cluster-dlm: add_change: add_change cg 2
joined nodeid 2013440192
Jun 4 11:30:15 srv-dell cluster-dlm: add_change: add_change cg 2
counts member 2 joined 1 remove 0 failed 0
Jun 4 11:30:15 srv-dell cluster-dlm: stop_kernel: stop_kernel cg 2
Jun 4 11:30:15 srv-dell cluster-dlm: do_sysfs: write "0" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/control"
Jun 4 11:30:15 srv-dell cluster-dlm: check_fencing_done: check_fencing done
Jun 4 11:30:15 srv-dell cluster-dlm: check_quorum_done: check_quorum disabled
Jun 4 11:30:15 srv-dell cluster-dlm: check_fs_done: check_fs done
Jun 4 11:30:15 srv-dell cluster-dlm: send_info: send_start cg 2 flags
2 counts 1 2 1 0 0
Jun 4 11:30:15 srv-dell cluster-dlm: receive_start: receive_start
2013440192:1 len 80
Jun 4 11:30:15 srv-dell cluster-dlm: match_change: match_change
2013440192:1 matches cg 2
Jun 4 11:30:15 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 2 need 1 of 2
Jun 4 11:30:15 srv-dell cluster-dlm: receive_start: receive_start
1543678144:2 len 80
Jun 4 11:30:15 srv-dell cluster-dlm: match_change: match_change
1543678144:2 matches cg 2
Jun 4 11:30:15 srv-dell cluster-dlm: wait_messages_done:
wait_messages cg 2 got all 2
Jun 4 11:30:15 srv-dell cluster-dlm: start_kernel: start_kernel cg 2
member_count 2
Jun 4 11:30:15 srv-dell cluster-dlm: update_dir_members: dir_member 1543678144
Jun 4 11:30:15 srv-dell cluster-dlm: set_configfs_members:
set_members mkdir
"/sys/kernel/config/dlm/cluster/spaces/E79BFDBA89F049ADB8E0895DAF6CE990/nodes/2013440192"
Jun 4 11:30:15 srv-dell cluster-dlm: do_sysfs: write "1" to
"/sys/kernel/dlm/E79BFDBA89F049ADB8E0895DAF6CE990/control"
Jun 4 11:30:15 srv-dell cluster-dlm: set_plock_ckpt_node:
set_plock_ckpt_node from 1543678144 to 1543678144
Jun 4 11:30:15 srv-dell cluster-dlm: _unlink_checkpoint: unlink ckpt 0
Jun 4 11:30:15 srv-dell kernel: dlm: connecting to 2013440192
Jun 4 11:30:15 srv-dell cluster-dlm: _unlink_checkpoint: unlink ckpt
error 12 E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:30:15 srv-dell cluster-dlm: _unlink_checkpoint: unlink ckpt
status error 9 E79BFDBA89F049ADB8E0895DAF6CE990
Jun 4 11:30:15 srv-dell cluster-dlm: store_plocks: store_plocks:
r_count 0, lock_count 0, pp 40 bytes
Jun 4 11:30:15 srv-dell cluster-dlm: store_plocks: store_plocks:
total 0 bytes, max_section 0 bytes
Jun 4 11:30:15 srv-dell cluster-dlm: store_plocks: store_plocks: open
ckpt handle 1220085400000000
Jun 4 11:30:15 srv-dell cluster-dlm: send_info: send_plocks_stored cg
2 flags 2 counts 1 2 1 0 0
Jun 4 11:30:15 srv-dell cluster-dlm: receive_plocks_stored:
receive_plocks_stored 1543678144:2 need_plocks 0
Jun 4 11:33:37 srv-dell cib: [5536]: info: cib_stats: Processed 78
operations (5128.00us average, 0% utilization) in the last 10min
Jun 4 11:39:01 srv-dell crmd: [5540]: info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped!
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 4 11:39:01 srv-dell crmd: [5540]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_pe_invoke: Query 131:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244129941-51, seq=272, quorate=1
Jun 4 11:39:01 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-lenovo is online
Jun 4 11:39:01 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-dell is online
Jun 4 11:39:01 srv-dell pengine: [5539]: info: find_clone: Internally
renamed dlm:0 on srv-dell to dlm:1
Jun 4 11:39:01 srv-dell pengine: [5539]: info: find_clone: Internally
renamed ocb2:0 on srv-dell to ocb2:1
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-lenovo srv-dell ]
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: clone_dlm
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-lenovo srv-dell ]
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: ocb2_clone
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-lenovo srv-dell ]
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-lenovo)
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-dell)
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:0 (Started srv-lenovo)
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:1 (Started srv-dell)
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:0 (Started srv-lenovo)
Jun 4 11:39:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:1 (Started srv-dell)
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 4 11:39:01 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition 5: 0 actions in 0 synapses
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_te_invoke: Processing
graph 5 (ref=pe_calc-dc-1244129941-51) derived from
/var/lib/pengine/pe-input-24.bz2
Jun 4 11:39:01 srv-dell crmd: [5540]: info: run_graph:
====================================================
Jun 4 11:39:01 srv-dell crmd: [5540]: notice: run_graph: Transition 5
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-24.bz2): Complete
Jun 4 11:39:01 srv-dell crmd: [5540]: info: te_graph_trigger:
Transition 5 is now complete
Jun 4 11:39:01 srv-dell crmd: [5540]: info: notify_crmd: Transition 5
status: done - <null>
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 4 11:39:01 srv-dell crmd: [5540]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 4 11:39:01 srv-dell pengine: [5539]: info: process_pe_message:
Transition 5: PEngine Input stored in:
/var/lib/pengine/pe-input-24.bz2
Jun 4 11:43:37 srv-dell cib: [5536]: info: cib_stats: Processed 1
operations (10000.00us average, 0% utilization) in the last 10min
Jun 4 11:54:01 srv-dell crmd: [5540]: info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped!
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_TIMER_POPPED origin=crm_timer_popped ]
Jun 4 11:54:01 srv-dell crmd: [5540]: WARN: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_state_transition: All
2 cluster nodes are eligible to run resources.
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_pe_invoke: Query 132:
Requesting the current CIB: S_POLICY_ENGINE
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_pe_invoke_callback:
Invoking the PE: ref=pe_calc-dc-1244130841-52, seq=272, quorate=1
Jun 4 11:54:01 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-lenovo is online
Jun 4 11:54:01 srv-dell pengine: [5539]: info:
determine_online_status: Node srv-dell is online
Jun 4 11:54:01 srv-dell pengine: [5539]: info: find_clone: Internally
renamed dlm:0 on srv-dell to dlm:1
Jun 4 11:54:01 srv-dell pengine: [5539]: info: find_clone: Internally
renamed ocb2:0 on srv-dell to ocb2:1
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: native_print:
primitiveresource_cluster_ip (ocf::heartbeat:IPaddr): Started srv-dell
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: cloneresource_cluster_fencing
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-lenovo srv-dell ]
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: clone_dlm
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-lenovo srv-dell ]
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: clone_print: Clone
Set: ocb2_clone
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: print_list:
Started: [ srv-lenovo srv-dell ]
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource primitiveresource_cluster_ip (Started srv-dell)
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:0 (Started srv-lenovo)
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource cloneresource_cluster_fencing_null:1 (Started srv-dell)
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:0 (Started srv-lenovo)
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource dlm:1 (Started srv-dell)
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:0 (Started srv-lenovo)
Jun 4 11:54:01 srv-dell pengine: [5539]: notice: LogActions: Leave
resource ocb2:1 (Started srv-dell)
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [
input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Jun 4 11:54:01 srv-dell crmd: [5540]: info: unpack_graph: Unpacked
transition 6: 0 actions in 0 synapses
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_te_invoke: Processing
graph 6 (ref=pe_calc-dc-1244130841-52) derived from
/var/lib/pengine/pe-input-25.bz2
Jun 4 11:54:01 srv-dell crmd: [5540]: info: run_graph:
====================================================
Jun 4 11:54:01 srv-dell crmd: [5540]: notice: run_graph: Transition 6
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-25.bz2): Complete
Jun 4 11:54:01 srv-dell crmd: [5540]: info: te_graph_trigger:
Transition 6 is now complete
Jun 4 11:54:01 srv-dell crmd: [5540]: info: notify_crmd: Transition 6
status: done - <null>
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_state_transition:
State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Jun 4 11:54:01 srv-dell crmd: [5540]: info: do_state_transition:
Starting PEngine Recheck Timer
Jun 4 11:54:01 srv-dell pengine: [5539]: info: process_pe_message:
Transition 6: PEngine Input stored in:
/var/lib/pengine/pe-input-25.bz2
Jun 4 11:58:35 srv-dell cibadmin: [18045]: info: Invoked: cibadmin -Q
More information about the Ocfs2-users
mailing list