<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"><html><head><meta content="text/html;charset=UTF-8" http-equiv="Content-Type"></head><body ><div style='font-size:10pt;'><div>Hi Joseph,<br></div><div><br></div><div> We have total 7 nodes and this problem occurs in multiple nodes simultaneously not in particular one node. we checked network and its fine.<br></div><div>When we remount the ocfs2 partition, this problem is get fixed temporarily and same problem reoccurs after some time. <br></div><div><br></div><div>Even we do have problem while unmountinng. umount process goes to "D" stat, then i need to restart server itself. Is there any solution for this issue.<br></div><div><br></div><div> I have tried running fsck.ocfs2 in problematic machine but its throwing error.<br></div><div><br></div><div>fsck.ocfs2 1.8.0<br></div><div>fsck.ocfs2: I/O error on channel while opening "/zoho/build/downloads"<br></div><div><br></div><div><br></div><div>Please refer the latest logs from one node.<br></div><div><br></div><div>[258418.054204] o2cb: o2dlm has evicted node 7 from domain A895BC216BE641A8A7E20AA89D57E051<br></div><div>[258418.957738] o2cb: o2dlm has evicted node 7 from domain A895BC216BE641A8A7E20AA89D57E051<br></div><div>[264056.408719] o2dlm: Node 7 joins domain A895BC216BE641A8A7E20AA89D57E051 ( 1 2 3 4 7 ) 5 nodes<br></div><div>[264464.605542] o2dlm: Node 7 leaves domain A895BC216BE641A8A7E20AA89D57E051 ( 1 2 3 4 ) 4 nodes<br></div><div>[275619.497198] o2dlm: Node 7 joins domain A895BC216BE641A8A7E20AA89D57E051 ( 1 2 3 4 7 ) 5 nodes<br></div><div>[426628.076148] o2cb: o2dlm has evicted node 1 from domain A895BC216BE641A8A7E20AA89D57E051<br></div><div>[426628.885084] o2dlm: Begin recovery on domain A895BC216BE641A8A7E20AA89D57E051 for node 1<br></div><div>[426628.891170] o2dlm: Node 3 (me) is the Recovery Master for the dead node 1 in domain A895BC216BE641A8A7E20AA89D57E051<br></div><div>[426634.182384] o2dlm: End recovery on domain A895BC216BE641A8A7E20AA89D57E051<br></div><div>[427001.383315] o2dlm: Node 1 joins domain A895BC216BE641A8A7E20AA89D57E051 ( 1 2 3 4 7 ) 5 nodes<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div id=""><div><span class="colour" style="color:rgb(0, 0, 0)">Regards</span><br></div><div><span class="colour" style="color:rgb(0, 0, 0)">G.J</span><br></div><div style="color: rgb(255, 0, 0);"><b></b><br></div><div><span class="size" style="font-size:16px"><span class="colour" style="color:rgb(192, 192, 192)"><span class="font" style="font-family:arial, helvetica, sans-serif"><span class="size" style="font-size:24px"><span class="colour" style="color:rgb(0, 0, 255)"></span></span><span></span></span></span></span><br></div></div><div><br></div><div class="zmail_extra"><div id="1"><div><br></div><div>---- On Fri, 25 Sep 2015 06:26:57 +0530 <b>Joseph Qi <joseph.qi@huawei.com></b> wrote ----<br></div></div><div><br></div><blockquote style="border-left: 1px solid #cccccc; padding-left: 6px; margin:0 0 0 5px"><div><div>On 2015/9/24 18:30, gjprabu wrote:
<br></div><div>> Hi All,
<br></div><div>>
<br></div><div>> Can someone tell me what kind of is this.
<br></div><div>>
<br></div><div>> Regards
<br></div><div>> Prabu GJ
<br></div><div>>
<br></div><div>>
<br></div><div>> ---- On Wed, 23 Sep 2015 18:26:13 +0530 *gjprabu <<a href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a>>* wrote ----
<br></div><div>>
<br></div><div>> Hi All,
<br></div><div>>
<br></div><div>> This issue we faced in locally machine also. but it is not in all the client only two ocfs2 client we facing this issue.
<br></div><div>>
<br></div><div>> Regards
<br></div><div>> Prabu GJ
<br></div><div>>
<br></div><div>>
<br></div><div>>
<br></div><div>> ---- On Wed, 23 Sep 2015 17:49:51 +0530 *gjprabu <<a href="mailto:gjprabu@zohocorp.com" target="_blank">gjprabu@zohocorp.com</a> <mailto:gjprabu@zohocorp.com>>* wrote ----
<br></div><div>>
<br></div><div>>
<br></div><div>>
<br></div><div>> Hi All,
<br></div><div>>
<br></div><div>> We are using ocfs2 for RBD mounting and everything works fine, but while writing or moving the data via the scripts after written it shows below error. Please anybody help on this issue.
<br></div><div>>
<br></div><div>>
<br></div><div>>
<br></div><div>> # ls -althr
<br></div><div>> ls: cannot access MICKEYLITE_3_0_M4_1_TEST: Input/output error
<br></div><div>> ls: cannot access MICKEYLITE_3_0_M4_1_OLD: Input/output error
<br></div><div>> total 0
<br></div><div>> d????????? ? ? ? ? ? MICKEYLITE_3_0_M4_1_TEST
<br></div><div>> d????????? ? ? ? ? ? MICKEYLITE_3_0_M4_1_OLD
<br></div><div>>
<br></div><div>> _*Partition details.*_
<br></div><div>>
<br></div><div>> /dev/rbd0 ocfs2 9.6T 140G 9.5T 2% /zoho/build/downloads
<br></div><div>>
<br></div><div>> /etc/ocfs2/cluster.conf
<br></div><div>> cluster:
<br></div><div>> node_count=7
<br></div><div>> heartbeat_mode = local
<br></div><div>> name=ocfs2
<br></div><div>>
<br></div><div>> node:
<br></div><div>> ip_port = 7777
<br></div><div>> ip_address = 10.1.1.50
<br></div><div>> number = 1
<br></div><div>> name = integ-hm5
<br></div><div>> cluster = ocfs2
<br></div><div>>
<br></div><div>> node:
<br></div><div>> ip_port = 7777
<br></div><div>> ip_address = 10.1.1.51
<br></div><div>> number = 2
<br></div><div>> name = integ-hm9
<br></div><div>> cluster = ocfs2
<br></div><div>>
<br></div><div>> node:
<br></div><div>> ip_port = 7777
<br></div><div>> ip_address = 10.1.1.52
<br></div><div>> number = 3
<br></div><div>> name = integ-hm2
<br></div><div>> cluster = ocfs2
<br></div><div>>
<br></div><div>> node:
<br></div><div>> ip_port = 7777
<br></div><div>> ip_address = 10.1.1.53
<br></div><div>> number = 4
<br></div><div>> name = integ-ci-1
<br></div><div>> cluster = ocfs2
<br></div><div>> node:
<br></div><div>> ip_port = 7777
<br></div><div>> ip_address = 10.1.1.54
<br></div><div>> number = 5
<br></div><div>> name = integ-cm2
<br></div><div>> cluster = ocfs2
<br></div><div>> node:
<br></div><div>> ip_port = 7777
<br></div><div>> ip_address = 10.1.1.55
<br></div><div>> number = 6
<br></div><div>> name = integ-cm1
<br></div><div>> cluster = ocfs2
<br></div><div>> node:
<br></div><div>> ip_port = 7777
<br></div><div>> ip_address = 10.1.1.56
<br></div><div>> number = 7
<br></div><div>> name = integ-hm8
<br></div><div>> cluster = ocfs2
<br></div><div>>
<br></div><div>>
<br></div><div>> *_Error on dmesg_*
<br></div><div>>
<br></div><div>>
<br></div><div>> [516421.342393] (dlm_thread,51005,25):dlm_flush_asts:599 ERROR: status = -112
<br></div><div>> [517119.689992] (httpd,64399,31):dlm_do_master_request:1383 ERROR: link to 1 went down!
<br></div><div>> [517119.690003] (dlm_thread,51005,25):dlm_send_proxy_ast_msg:482 ERROR: A895BC216BE641A8A7E20AA89D57E051: res S000000000000000000000200000000, error -112 send AST to node 1
<br></div><div>> [517119.690028] (dlm_thread,51005,25):dlm_flush_asts:599 ERROR: status = -112
<br></div><div>> [517119.690034] (dlm_thread,51005,25):dlm_send_proxy_ast_msg:482 ERROR: A895BC216BE641A8A7E20AA89D57E051: res S000000000000000000000200000000, error -107 send AST to node 1
<br></div><div>> [517119.690036] (dlm_thread,51005,25):dlm_flush_asts:599 ERROR: status = -107
<br></div><div>> [517119.700425] (httpd,64399,31):dlm_get_lock_resource:968 ERROR: status = -112
<br></div><div>> [517517.894949] (dlm_thread,51005,25):dlm_send_proxy_ast_msg:482 ERROR: A895BC216BE641A8A7E20AA89D57E051: res S000000000000000000000200000000, error -112 send AST to node 1
<br></div><div>> [517517.899640] (dlm_thread,51005,25):dlm_flush_asts:599 ERROR: status = -112
<br></div><div>>
<br></div><div>The error messages means the connection between this node and node 1 has problem.
<br></div><div>You have to check the network.
<br></div><div><br></div><div>>
<br></div><div>> Regards
<br></div><div>> Prabu GJ
<br></div><div>>
<br></div><div>>
<br></div><div>>
<br></div><div>> _______________________________________________
<br></div><div>> Ocfs2-users mailing list
<br></div><div>> <a href="mailto:Ocfs2-users@oss.oracle.com" target="_blank">Ocfs2-users@oss.oracle.com</a> <mailto:Ocfs2-users@oss.oracle.com>
<br></div><div>> <a href="https://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">https://oss.oracle.com/mailman/listinfo/ocfs2-users</a> <br></div><div>>
<br></div><div>>
<br></div><div>>
<br></div><div>>
<br></div><div>> _______________________________________________
<br></div><div>> Ocfs2-users mailing list
<br></div><div>> <a href="mailto:Ocfs2-users@oss.oracle.com" target="_blank">Ocfs2-users@oss.oracle.com</a> <br></div><div>> <a href="https://oss.oracle.com/mailman/listinfo/ocfs2-users" target="_blank">https://oss.oracle.com/mailman/listinfo/ocfs2-users</a> <br></div><div>>
<br></div><div><br></div><div><br></div></div></blockquote></div><div><br></div></div></body></html>