<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Previous history : This cluster has been stable etc for months . </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">On two occasions this week , we have experienced one server lock up resulting in many D state processes - primarily nfs - requiring a reboot . </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><pre style="margin-top:0px;margin-bottom:0px;padding:5px 10px;font-family:Consolas,Menlo,'Liberation Mono',Courier,monospace;font-size:12px;line-height:1.4;border:1px solid rgb(204,204,204);border-radius:3px;word-wrap:normal;color:rgb(51,51,51);background:rgb(245,245,245)"><span class="" style="color:rgb(0,153,153)">26337</span> D lookup_slow smbd
<span class="" style="color:rgb(0,153,153)">26372</span> D lookup_slow smbd
<span class="" style="color:rgb(0,153,153)">26381</span> D dlm_wait_for_lock_mastery smbd
<span class="" style="color:rgb(0,153,153)">26406</span> D iterate_dir smbd
<span class="" style="color:rgb(0,153,153)">26417</span> D iterate_dir bash
<span class="" style="color:rgb(0,153,153)">26530</span> D iterate_dir smbd
<span class="" style="color:rgb(0,153,153)">26557</span> D iterate_dir smbd
<span class="" style="color:rgb(0,153,153)">26761</span> D iterate_dir smbd</pre></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style=""><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:05:01 server1 CRON[60021]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:27 server1 kernel: [1633038.930488] o2net: Connection to node server2 (num 0) at <a href="http://10.0.0.11:7777">10.0.0.11:7777</a> has been idle for 30.5 secs, shutting it down.</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:27 server1 kernel: [1633038.930579] o2net: No longer connected to node server2 (num 0) at <a href="http://10.0.0.11:7777">10.0.0.11:7777</a></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:27 server1 kernel: [1633039.018762] o2net: Connection to node server3 (num 2) at <a href="http://10.0.0.30:7777">10.0.0.30:7777</a> shutdown, state 8</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:27 server1 kernel: [1633039.018846] o2net: No longer connected to node server3 (num 2) at <a href="http://10.0.0.30:7777">10.0.0.30:7777</a></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:27 server1 kernel: [1633039.018987] (kworker/u192:2,59600,0):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32 instead of 24</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:32 server1 kernel: [1633043.622980] o2net: Accepted connection from node server3 (num 2) at <a href="http://10.0.0.30:7777">10.0.0.30:7777</a></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:32 server1 kernel: [1633043.623052] o2net: Connected to node server2 (num 0) at <a href="http://10.0.0.11:7777">10.0.0.11:7777</a></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><div class="gmail_default">Oct 28 12:01:35 server2 kernel: [158857.315968] (kworker/u64:0,25700,8):dlm_send_remote_convert_request:392 ERROR: Error -107 when sending message 504 (key 0x276c0b15) to node 1</div><div class="gmail_default">Oct 28 12:01:35 server2 kernel: [158857.317060] o2dlm: Waiting on the death of node 1 in domain 27C73AA97B7F4D9398D884DB8DA467DB</div><div class="gmail_default">Oct 28 12:01:40 server2 kernel: [158862.288279] (nfsd,7107,7):dlm_send_remote_convert_request:392 ERROR: Error -107 when sending message 504 (key 0xbd7d045e) to node 1</div><div class="gmail_default">Oct 28 12:01:40 server2 kernel: [158862.289730] o2dlm: Waiting on the death of node 1 in domain 71DEDAB8A10A4F849C60F454E41418FB</div><div class="gmail_default">Oct 28 12:01:40 server2 kernel: [158862.420276] (kworker/u64:0,25700,10):dlm_send_remote_convert_request:392 ERROR: Error -107 when sending message 504 (key 0x276c0b15) to node 1</div><div class="gmail_default">Oct 28 12:01:40 server2 kernel: [158862.421855] o2dlm: Waiting on the death of node 1 in domain 27C73AA97B7F4D9398D884DB8DA467DB</div><div class="gmail_default">Oct 28 12:01:45 server2 kernel: [158866.795872] (smbd,6126,8):dlm_do_master_request:1344 ERROR: link to 1 went down!</div><div class="gmail_default">Oct 28 12:01:45 server2 kernel: [158866.796643] (smbd,6126,8):dlm_get_lock_resource:929 ERROR: status = -107</div><div class="gmail_default">Oct 28 12:01:45 server2 kernel: [158867.392484] (nfsd,7107,7):dlm_send_remote_convert_request:392 ERROR: Er</div></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:46 server2 kernel: [441237.624180] (smbd,9618,7):dlm_do_master_request:1344 ERROR: link to 1 went down!</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:46 server2 kernel: [441237.624181] (smbd,9618,7):dlm_get_lock_resource:929 ERROR: status = -107</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:46 server2 kernel: [441237.624832] (smbd,9618,7):dlm_do_master_request:1344 ERROR: link to 1 went down!</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:46 server2 kernel: [441237.624833] (smbd,9618,7):dlm_get_lock_resource:929 ERROR: status = -107</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:46 server2 kernel: [441237.914234] (smbd,16974,11):dlm_get_lock_resource:929 ERROR: status = -112</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:12:50 server2 kernel: [441242.308190] o2net: Accepted connection from node server1 (num 1) at <a href="http://10.0.0.80:7777">10.0.0.80:7777</a></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 26 12:15:01 server2 CRON[17178]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.124152] o2net: Connection to node server2 (num 0) at <a href="http://10.0.0.11:7777">10.0.0.11:7777</a> shutdown, state 8</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.124252] o2net: No longer connected to node server2 (num 0) at <a href="http://10.0.0.11:7777">10.0.0.11:7777</a></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.124285] (python,7983,0):dlm_do_master_request:1344 ERROR: link to 0 went down!</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.124563] (kworker/u192:1,9377,4):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32 instead of 24</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.145675] (python,7983,0):dlm_get_lock_resource:929 ERROR: status = -112</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.237950] o2net: Connection to node server3 (num 2) at <a href="http://10.0.0.30:7777">10.0.0.30:7777</a> shutdown, state 8</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.237993] o2net: No longer connected to node server3 (num 2) at <a href="http://10.0.0.30:7777">10.0.0.30:7777</a></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.238067] (python,7983,2):dlm_do_master_request:1344 ERROR: link to 2 went down!</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.238078] (kworker/u192:1,9377,12):o2net_send_tcp_msg:960 ERROR: sendmsg returned -32 instead of 24</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.238118] (dlm_thread,4881,6):dlm_send_proxy_ast_msg:482 ERROR: 71DEDAB8A10A4F849C60F454E41418FB: res P000000000000000000000000000000, error -112 send AST to node 2</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Oct 28 12:01:04 server1 kernel: [1805223.238121] (dlm_thread,4881,6):dlm_flush_asts:596 ERROR: status = -112</font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">I am investigating a perc controller as we had a battery fail message on server1 during reboot , but wanted some more input on these errors as well. </font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Thank you for your time , </font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default" style=""><font face="arial, helvetica, sans-serif">Jon </font></div></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div>