[Ocfs2-devel] [RFC] make ocfs2/o2net reliable

[Changwei Ge]

On 2017/11/17 13:51, jiangyiwen wrote:
> On 2017/11/17 11:53, Changwei Ge wrote:
>> Hi Yiwen,
>>
>> On 2017/11/17 11:06, jiangyiwen wrote:
>>> On 2017/11/16 17:49, Changwei Ge wrote:
>>>> Hi all,
>>>> As far as we know, ocfs2/o2net is not a reliable message mechanism.
>>>> Messages might get lost due to a sudden TCP socket connection shutdown.
>>> Hi Changwei,
>>>
>>> Junxiao has already solved the situation about you mentioned.
>>> in commit(c43c363def04cdaed0d9e26dae846081f55714e7), it don't shutdown
>>> connection until node is fenced, so I don't understand the scenario
>>> what you mentioned about TCP socket connection shutdown, can you give
>>> a specific description? thank you.
>>
>> I'm afraid Juxiao's patch can't cover all scenarios. It addresses o2net
>> timeout scenario but not tcp socket resetting case.
>>
>>>
>>> In addition, as far as I know, TCP is reliable and trustworthy, TCP
>>> will resend messages in a certain retransmit time. So as long as
>>> o2net didn't active shutdown socket, TCP will resend message for
>>> us.
>>>
>>> Thanks,
>>> Yiwen Jiang.
>>
>> Actually, TCP event doesn't begin to send packets from its send buffer
>> but closed due to underlying unknown reason. So we lose them.
>>
>>
>> Thanks,
>> Changwei
>>
> 
> I think firstly we should find the reason why tcp socket is reset/closed,
> that is the underlying unknown reason you mentioned above, maybe it is
> TCP bug. After analyzing, it is normal that tcp is closed in certain
> condition, then we discuss the solution.

Um, I am a little confused. You mean we have to find out the root cause 
why TCP has to shutdown existed connection?
I think should enhance o2net reliability making it like other reliable 
message mechanism.

Thanks,
Changwei

> 
> Thanks,
> Yiwen Jiang.
> 
>>>> And the only customer of o2net is ocfs2/dlm, so this may cause ocfs2/dlm
>>>> hang(missing AST and ASSERT MASTER). Sometimes it also causes
>>>> ocfs2/dlm's infinite wait for accomplishment of DLM recovery. But that
>>>> won't happen since target node is still heartbeating and no dlm recovery
>>>> procedure will be launched.
>>>>
>>>> So I think above cases drive us to improve current ocfs2/o2net making it
>>>> more reliable. I already have a draft design for it. And we indeed need
>>>> to change o2net behavior.
>>>>
>>>> To accomplish this goal, we tag each o2net message with a sequence
>>>> ::msg_seq to let receiver tell if the newly coming message is a
>>>> duplicated one or not and ::msg_seq will work as a key value for
>>>> searching a following key structure in a red-black tree.
>>>>
>>>> A brandy new structure is added to o2net named as *Message Holder*, it
>>>> is responsible for _handle_status_ storing.
>>>>
>>>> When TCP has to shutdown or reset due to unknown reason, although we
>>>> lose the packets in send or receive buffer, o2net still manages those
>>>> messages. This gives a chance to o2net to re-send the messages once TCP
>>>> connection is established again.
>>>>
>>>> Below diagram demonstrates how it works:
>>>>
>>>> SEND					RECV
>>>> send message				
>>>> tag message header with ::msg_seq	
>>>> 					search for Message Holder with
>>>> 					  ::msg_seq
>>>> 					NOT FOUND - insert one
>>>> 					(FOUND - means a duplicated one)
>>>> 					handle message
>>>> 					store status into Message Holder
>>>> 					send back status
>>>> instruct RECV to remove MH
>>>> 					notify SEND that MH is already
>>>> 					  removed
>>>> return to caller
>>>>
>>>> I am expecting your comments especially from @Mark, @Joseph and @Junxiao.
>>>>
>>>> Thanks,
>>>> Changwei.
>>>>
>>>> _______________________________________________
>>>> Ocfs2-devel mailing list
>>>> Ocfs2-devel at oss.oracle.com
>>>> https://oss.oracle.com/mailman/listinfo/ocfs2-devel
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>> .
>>
> 
> 
>