[Ocfs2-users] umount hang + high CPU

Thu Jul 9 09:00:11 PDT 2009

 I checked the logs now in the hanged machine, this is what was written there priror to the hang:

Jul  7 23:35:14 ocfs2Server kernel: [159179.624911] ocfs2_dlm: Nodes in domain ("6A468A219FF141429D2BAFF54FA8D514"): 1
Jul  7 23:35:14 ocfs2Server kernel: [159179.625090] (9464,0):ocfs2_find_slot:502 slot 1 is already allocated to this node!
Jul  7 23:35:14 ocfs2Server kernel: [159179.630086] (9464,3):ocfs2_check_volume:2270 File system was not unmounted cleanly, recovering volume.
Jul  7 23:35:14 ocfs2Server kernel: [159179.686060] kjournald2 starting: pid 9471, dev dm-5:25, commit interval 5 seconds
Jul  7 23:35:14 ocfs2Server kernel: [159179.687710] ocfs2: Mounting device (253,5) on (node 1, slot 1) with ordered data mode.
Jul  7 23:35:14 ocfs2Server kernel: [159179.688139] (9473,1):ocfs2_replay_journal:1593 Recovering node 0 from slot 0 on device (253,5)
Jul  7 23:35:17 ocfs2Server kernel: [159182.532458] kjournald2 starting: pid 9485, dev dm-5:24, commit interval 5 seconds
Jul  7 23:35:17 ocfs2Server kernel: [159182.591988] (9473,2):ocfs2_begin_quota_recovery:374 Beginning quota recovery in slot 0
Jul  7 23:35:18 ocfs2Server kernel: [159183.125091] (4797,2):ocfs2_finish_quota_recovery:564 Finishing quota recovery in slot 0
Jul  8 20:39:13 ocfs2Server syslogd 1.5.0#5: restart.

I am going to install netconsole and try again.

-----Original Message-----
From: Sunil Mushran <sunil.mushran at oracle.com>
To: sylarrrrrrr at aim.com
Cc: ocfs2-users at oss.oracle.com
Sent: Wed, Jul 8, 2009 1:25 am
Subject: Re: [Ocfs2-users] umount hang + high CPU

Well
, that means the hung node exited the dlm domain successfully. 

This is not a dlm issue. 

Run alt-sysrq-t on the hung node. If you have netconsole setup you 

should see a log. 

sylarrrrrrr at aim.com wrote: 

> Aha, ok, I don't see the oops, or anything about the hang in the logs. 
> The hanged machine still reply to pings. 

> 

> The story now is , that I thought that I can use the : 

> 

>  tunefs.ocfs2  --cloned-volume /dev/mylvmsnapshot 

> 

> in order to mount the snapshot... (big mistake)...well I did manage to 
> mount the snapshot, but as soon as 

> I umounted it, the umount process hanged, and then the whole machine 
> hanged, except that it responds to pings. 

> 

> 

> Now, I have downloaded the ocfs2-1.4-userguide.pdf , and went to 
> section 'f) DLM Debuging', and tried the commands 

> there on the still working node, but only 'cat 
> /sys/kernel/debug/o2dlm/*/dlm_state' worked and produced the following 
> output: 

> 

> Domain: 1ACAFCEE7ACA47C089069117560F5C91  Key: 0xb9d649ba 

> Thread Pid: 5664  Node: 0  State: JOINED 

> Number of Joins: 1  Joining Node: 255 

> Domain Map: 0 

> Live Map: 0 

> Lock Resources: 51168 (180512) 

> MLEs: 0 (291689) 

>   Blocking: 0 (139713) 

>   Mastery: 0 (151976) 

>   Migration: 0 (0) 

> Lists: Dirty=Empty  Purge=InUse  PendingASTs=Empty  PendingBASTs=EmptyC2

> Purge Count: 8  Refs: 51169 

> Dead Node: 255 

> Recovery Pid: 5665  Master: 255  State: INACTIVE 

> Recovery Map: 

> Recovery Node State: 

> 

> the other commands: 

> debugfs.ocfs2 –R “fs_locks –B” /dev/drbd0 

> debugfs.ocfs2 –R “fs_locks –B” /dev/vg/lv 

> debugfs.ocfs2 –R “dlm_locks M000000000000000022d63c00000000” /dev/drbd0 

> 

> produced the error: 

> open: Device name specified was not found while opening context for 
> device –R 

> debugfs.ocfs2 1.4.2 

> debugfs: 

> 

> and: 

> 

> ps -e -o pid,stat,comm,wchan=WIDE-WCHAN-COLUMN 

> 

> procuded no D state process. 

> 

> 

> I am sorry I write it in the mailing list, but I am a noob, so I don't 
> even know if it is a bug, or a misconfiguration, or a misunderstanding. 

> 

> PS. Is nodiratime option supported for mounts? I used it, but I don't 
> see it in the user-guide. 

> 

> -----Original Message----- 

> From: Sunil Mushran <sunil.mushran at oracle.com> 

> To: sylarrrrrrr at aim.com 

> Cc: tao.ma at oracle.com; ocfs2-users at oss.oracle.com 

> Sent: Tue, Jul 7, 2009 8:46 pm 

> Subject: Re: [Ocfs2-users] umount hang + high CPU 

> 

> The fix was for the oops you saw. 
>  
> The hang is a different issue. We have no info on that. 
>  
> For that, if you would like20to diagnose the problem, read up the dlm 
> notes 
> in the 1.4 user's guide. It explains a debugging process vis-a-vis hangs. 
>  
> If the issue is dlm related, then we would like to have the tcpdumps. 
>  
> Lastly, emails are no t an efficient vehicle for handling such issues. 
> Use 
> the bugzilla as it allows us to collect information in one place. 
>  
> Sunil 
>  
> sylarrrrrrr at aim.com <mailto:sylarrrrrrr at aim.com> wrote: 
> > So this bug is not over yet :( 
> > 
> > I have checked my kernel source and indeed it have this patch but I 
> > still get the hang. 
> > 
> > PS. my linux-2.6-2.6.30/fs/ocfs2/dcache.c kernel source has: 
> > 
> > 290 else 
> > 291 mlog_errno(ret); 
> > 292 
> > 293 /* 
> > 294 * In case of error, manually free the allocation and > do the 
> iput(). 
> > 295 * We need to do this because error here means no > d_instantiate(), 
> > 296 * which means iput() will not be called during > dput(dentry). 
> > 297 */ 
> > 298 if (ret < 0 && !alias) { 
> > 299 ocfs2_lock_res_free(&dl->dl_lockres); 
> > 300 BUG_ON(dl->dl_count != 1); 
> > 301 spin_lock(&dentry_attach_lock); 
> > =2 0302 dentry->d_fsdata = NULL; 
> > 303 spin_unlock(&dentry_attach_lock); 
> > 304 kfree(dl); 
> > 305 iput(inode); 
> > 306 } 
> > 307 
> > 308 dput(alias); 
> > 309 
> > 310 return ret; 
> > 311 } 
> > 
> > 
>  
> 

> --------------------------------------------------------------------
---- 

> *A Good Credit Score is 700 or Above. See yours in just 2 easy steps! 
> <http://pr.atwola.com/promoclk/100126575x1222585089x1201462806/aol?redir=http://www.freecreditreport.com/pm/default.aspx?sc=668072%26hmpgID=62%26bcd=JulystepsfooterNO62>* 
> 

> ------------------------------------------------------------------------ 

> 

> _______________________________________________ 

> Ocfs2-users mailing list 

> Ocfs2-users at oss.oracle.com 

> http://oss.oracle.com/mailman/listinfo/ocfs2-users 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090709/f90d8799/attachment.html 