[Ocfs2-devel] A o2cb DLM problem

[Gang He]

Hello Junxiao,


Thank for quick reply, the information is very helpful.

-Gang


>>> 
> On 10/12/2017 02:37 PM, Gang He wrote:
>> Hello list,
>> 
>> We got a o2cb DLM problem from the customer, which is using o2cb stack for 
> OCFS2 file system on SLES12SP1(3.12.49-11-default).
>> The problem description is as below,
>> 
>> Customer has three node oracle rack cluster
>> gal7gblr2084
>> gal7gblr2085
>> gal7gblr2086
>> 
>> On each node they have configured two ocfs resources as a filesystem. The 
> two node gal7gblr2085 and gal7gblr2086 got hung and went into loop to kill 
> each other and they want root cause analysis.
>> Anyway, all I see in logs is those messages flooding /var/log/messages
>> 
>> 2017-10-05T06:50:25.980773+01:00 gal7gblr2085 kernel: [16874541.314199] o2net: 
> Connection to node gal7gblr2086 (num 2) at 10.233.217.12:7777 has been idle 
> for 30.5 secs, shutting it down.
> Looks it is an old kernel. Shutting down connection when idle timeout
> will cause losing dlm message which may cause hung. Please apply the
> following 3 patches.
> 
> 8c7b638cece1 ocfs2: quorum: add a log for node not fenced
> 8e9801dfe37c ocfs2: o2net: set tcp user timeout to max value
> c43c363def04 ocfs2: o2net: don't shutdown connection when idle timeout
> 
> Thanks,
> Junxiao.
>> 2017-10-05T06:50:37.456786+01:00 gal7gblr2085 kernel: [16874552.778726] o2net: 
> No longer connected to node gal7gblr2086 (num 2) at 10.233.217.12:7777
>> 2017-10-05T06:50:45.176798+01:00 gal7gblr2085 kernel: [16874560.487834] 
> (kworker/u64:1,13245,10):dlm_send_remote_convert_request:392 ERROR: Error -107 
> when sending message 504 (key 0x4a68dd81) to node 2
>> 2017-10-05T06:50:45.176812+01:00 gal7gblr2085 kernel: [16874560.487838] o2dlm: 
> Waiting on the death of node 2 in domain 18AE08328428452BA610E7BDE26F5246
>> 2017-10-05T06:50:50.284796+01:00 gal7gblr2085 kernel: [16874565.589996] 
> (kworker/u64:1,13245,10):dlm_send_remote_convert_request:392 ERROR: Error -107 
> when sending message 504 (key 0x4a68dd81) to node 2
>> 2017-10-05T06:50:50.284811+01:00 gal7gblr2085 kernel: [16874565.590000] o2dlm: 
> Waiting on the death of node 2 in domain 18AE08328428452BA610E7BDE26F5246
>> 2017-10-05T06:50:55.400808+01:00 gal7gblr2085 kernel: [16874570.700448] 
> (kworker/u64:1,13245,10):dlm_send_remote_convert_request:392 ERROR: Error -107 
> when sending message 504 (key 0x4a68dd81) to node 2
>> 2017-10-05T06:50:55.400824+01:00 gal7gblr2085 kernel: [16874570.700452] o2dlm: 
> Waiting on the death of node 2 in domain 18AE08328428452BA610E7BDE26F5246
>> 2017-10-05T06:51:00.512766+01:00 gal7gblr2085 kernel: [16874575.808944] 
> (kworker/u64:1,13245,26):dlm_send_remote_convert_request:392 ERROR: Error -107 
> when sending message 504 (key 0x4a68dd81) to node 2
>> 2017-10-05T06:51:00.512783+01:00 gal7gblr2085 kernel: [16874575.808948] o2dlm: 
> Waiting on the death of node 2 in domain 18AE08328428452BA610E7BDE26F5246
>> 2017-10-05T06:51:02.456785+01:00 gal7gblr2085 kernel: [16874577.749286] 
> (ora_diag_rcp2,24339,0):dlm_do_master_request:1344 ERROR: link to 2 went 
> down!
>> 2017-10-05T06:51:02.456797+01:00 gal7gblr2085 kernel: [16874577.749289] 
> (ora_diag_rcp2,24339,0):dlm_get_lock_resource:929 ERROR: status = -107
>> 2017-10-05T06:51:05.632955+01:00 gal7gblr2085 kernel: [16874580.920124] 
> (kworker/u64:1,13245,26):dlm_send_remote_convert_request:392 ERROR: Error -107 
> when sending message 504 (key 0x4a68dd81) to node 2
>> 2017-10-05T06:51:05.632973+01:00 gal7gblr2085 kernel: [16874580.920132] o2dlm: 
> Waiting on the death of node 2 in domain 18AE08328428452BA610E7BDE26F5246
>> 2017-10-05T06:51:07.976787+01:00 gal7gblr2085 kernel: [16874583.262561] o2net: 
> No connection established with node 2 after 30.0 seconds, giving up.
>> 2017-10-05T10:03:38.439542+01:00 gal7gblr2084 kernel: [1911889.097543] 
> (mdb_psp0_-mgmtd,21126,0):dlm_send_remote_unlock_request:358 ERROR: Error -107 
> when sending message 506 (key 0x4a68dd81) to node 1
>> 2017-10-05T10:03:38.439543+01:00 gal7gblr2084 kernel: [1911889.097547] 
> (mdb_psp0_-mgmtd,21126,0):dlm_send_remote_unlock_request:358 ERROR: Error -107 
> when sending message 506 (key 0x4a68dd81) to node 1
>> 
>> 
>> Did you guys encounter such problem when using o2cb stack? since we mainly 
> focus on pmck stack, but I still want to help this customer to know the root 
> cause.
>> 
>> 
>> Thanks
>> Gang
>> 
>> 
>> 
>> 
>> 
>>