[Ocfs2-devel] avoid being purged when queued for assert_master

[Wengang Wang]

I will try to reproduce this in house by hacking codes(adding sleeps).
So before the reproduction in house, please put this aside ;)

thanks,
wengang.
On 11-10-14 08:19, Wengang Wang wrote:
> 2.6.18-128.xxxx
> 
> thanks,
> wengang.
> On 11-10-13 16:37, Sunil Mushran wrote:
> > which kernel?
> > 
> > On 10/13/2011 04:35 PM, Wengang Wang wrote:
> > >On 11-10-13 09:09, Sunil Mushran wrote:
> > >>The last email you said it reproduced. Now you say it did not.
> > >>I'm confused.
> > >Oh? Did I. If I did, I meant it had reproductions in different customers's ENV,
> > >I had no reproduction in house.
> > >
> > >Sorry for confusion :P
> > >
> > >thanks,
> > >wengang.
> > >>On 10/12/2011 07:13 PM, Wengang Wang wrote:
> > >>>On 11-10-12 19:11, Sunil Mushran wrote:
> > >>>>That's what ovm does. Have you reproduced it with ovm3 kernel?
> > >>>>
> > >>>No, I have no reproductions.
> > >>>
> > >>>thanks,
> > >>>wengang.
> > >>>>On 10/12/2011 07:07 PM, Wengang Wang wrote:
> > >>>>>On 11-10-13 09:51, Wengang Wang wrote:
> > >>>>>>On 11-10-12 18:47, Sunil Mushran wrote:
> > >>>>>>>I meant master_request (not query). We set refmap _before_
> > >>>>>>>asserting. So that should not happen.
> > >>>>>>Why can't the remote node requested deref (DLM_DEREF_LOCKRES_MSG)?
> > >>>>>The problem can easily happen on this dlmfs useage:
> > >>>>>
> > >>>>>reopen:
> > >>>>>	open(create) /dlm/dirxx/filexx
> > >>>>>	close	     /dlm/dirxx/filexx
> > >>>>>	sleep 60
> > >>>>>	goto reopen
> > >>>>>
> > >>>>>>thanks,
> > >>>>>>wengang.
> > >>>>>>>On 10/12/2011 06:02 PM, Wengang Wang wrote:
> > >>>>>>>>Hi Sunil,
> > >>>>>>>>
> > >>>>>>>>On 11-10-12 17:32, Sunil Mushran wrote:
> > >>>>>>>>>So you are saying a lockres can get purged before the node is asserting
> > >>>>>>>>>master to other nodes?
> > >>>>>>>>>
> > >>>>>>>>>The main place where we dispatch assert is during master_query.
> > >>>>>>>>>There we set refmap before dispatching. Meaning refmap will protect
> > >>>>>>>>>us from purging.
> > >>>>>>>>>
> > >>>>>>>>>But I think it could happen in master_requery, which only comes into
> > >>>>>>>>>play if a node dies during migration.
> > >>>>>>>>>
> > >>>>>>>>>Is that the case here?
> > >>>>>>>>I think this can mainly include the response for a master_request.
> > >>>>>>>>in dlm_master_request_handler(), the master node quques assert_master.
> > >>>>>>>>The node which requested a master_request knows the master by receving
> > >>>>>>>>response values. It doesn't need to wait until the assert_master come.
> > >>>>>>>>As you know, the asserting master work is done in a workqueue. And the
> > >>>>>>>>work item in it can be heavily delayed. So in the duriation from the
> > >>>>>>>>(old) master responding with "Yes, I am master" to it sending assert_master,
> > >>>>>>>>Anything can heppan, the worse case is the lockres on the (old) master
> > >>>>>>>>get purged and is remasted by another node. So in this case,
> > >>>>>>>>apparently, the old master shouldn't send the assert_master any longer.
> > >>>>>>>>To prevent that from happening, we should keep the lockres un-purged as
> > >>>>>>>>long as it's queued for master_request.
> > >>>>>>>>
> > >>>>>>>>#the problem is what my flush_workqueue patch tries to fix.
> > >>>>>>>>
> > >>>>>>>>thanks,
> > >>>>>>>>wengang.
> > >>>>>>>>
> > >>>>>>>>>On 10/12/2011 12:04 AM, Wengang Wang wrote:
> > >>>>>>>>>>Hi Sunil/Joel/Mark and anyone who has interest,
> > >>>>>>>>>>
> > >>>>>>>>>>This is not a patch but a discuss.
> > >>>>>>>>>>
> > >>>>>>>>>>Currently we have a problem:
> > >>>>>>>>>>When a lockres is still queued(in dlm->work_list) for sending an
> > >>>>>>>>>>assert_master(or in processing of sending), the lockres can't be
> > >>>>>>>>>>purged(removed from hash). there is no flag/state,on lockres its self,dinotes
> > >>>>>>>>>>this situation.
> > >>>>>>>>>>
> > >>>>>>>>>>The badness is that if the lockres is purged(surely not the owner at the
> > >>>>>>>>>>moment), and the assert_master is after the purge. it can confuse other
> > >>>>>>>>>>nodes. On another node, the owner now can be any other nodes, thus on
> > >>>>>>>>>>receiving the assert_master, it can trigger a BUG() because 'owner'
> > >>>>>>>>>>doesn't match.
> > >>>>>>>>>>
> > >>>>>>>>>>So we'd better to prevent the lockres from be purged when it's queued
> > >>>>>>>>>>for something(assert_master).
> > >>>>>>>>>>
> > >>>>>>>>>>Srini and I discussed some possible fixes:
> > >>>>>>>>>>1) adding a flag to lockres->state.
> > >>>>>>>>>>    this does not work. A lockres can have multiple instances in the queue list.
> > >>>>>>>>>>    A simple flag is not safe. And the instances are not nested, so even
> > >>>>>>>>>>    saving a previous flags doesn't work. Neither can we merge the instances
> > >>>>>>>>>>    because they can be for different purposes.
> > >>>>>>>>>>
> > >>>>>>>>>>2) checking if the lockres if queued before purging it.
> > >>>>>>>>>>   this works, but doesn't sounds good. it needs changes of current behaviour
> > >>>>>>>>>>   on the queue list.   Also, we have no idea on the performance of the checking
> > >>>>>>>>>>   (searching list).
> > >>>>>>>>>>
> > >>>>>>>>>>3) making use of lockres->inflight_locks.
> > >>>>>>>>>>   this works, but seems to be a mis-use of inflight_locks.
> > >>>>>>>>>>
> > >>>>>>>>>>4) adding a new member to lockres counting the queued time.
> > >>>>>>>>>>    this works and simple. but needs extra memory.
> > >>>>>>>>>>
> > >>>>>>>>>>I prefer to the 4).
> > >>>>>>>>>>
> > >>>>>>>>>>What's your idea?
> > >>>>>>>>>>
> > >>>>>>>>>>thanks,
> > >>>>>>>>>>wengang.
> > >>>>>>>>>>
> > >>>>>>>>>>_______________________________________________
> > >>>>>>>>>>Ocfs2-devel mailing list
> > >>>>>>>>>>Ocfs2-devel at oss.oracle.com
> > >>>>>>>>>>http://oss.oracle.com/mailman/listinfo/ocfs2-devel
> >