[Ocfs2-users] RE: [Ocfs2-devel] systems hang when accessing parts of the OCFS2 filesystem

Fri Jan 11 10:26:44 PST 2008

If ls is hanging, invariably means the dlm is waiting for a node to respond.

Do:
$ ps -e -o pid,stat,comm,wchan=WIDE-WCHAN-COLUMN

If the ls process is in ocfs2_wait_for_status_completion() it means that
that is the case.

You can find more info on debugging here.
http://oss.oracle.com/osswiki/OCFS2/Debugging

One thing to check is:

$ cat /proc/fs/ocfs2_nodemanager/sock_containers

If you notice that a connect between two nodes is missing, you are
encountering a issue we are currently fixing. That is, a connect between
two live nodes breaks and a reconnect attempt is not made. This will
lead to a hang too.

As far as other issues:

1. Yes, debugfs.ocfs2 will always be able to read the device as it
does dirty reads. The output for mounted vols may be stale.

2. No, fsck.ocfs2 is not showing real errors. If you run fsck when the
device is mounted, fsck is not seeing the full picture as the current
image of the block(s) may be cached by other nodes in the cluster.

3. sdf was formatted with ocfs2 before it was partitioned. While
mounted.ocfs2 can successfully read the superblock it errors because
it is unable to see more of the device (and that is correct.) We will
fix mounted to ignore such cases.

Sunil

bob findlay (TOC) wrote:
>
> is having both sdf & sdf1 cause for concern? especially as the 
> mounted.ocfs2 -f complains about a bad magic number on sdf. it doesn't 
> seem right that both sdf and sdf1 have oracle as the label? we're 
> mounting by label, and it's sdf1 that gets mounted.
>
> [root at jic55124 bin]# mounted.ocfs2 -d
> Device FS UUID Label
> /dev/sdf ocfs2 e9b6b495-a72d-4792-9b51-b294702b7ed4 oracle
> /dev/sdf1 ocfs2 e418cb00-242f-4df2-96b4-6f3b0ae9b2e8 oracle
> /dev/sdg ocfs2 79a4a600-4f9c-4be0-b983-fbadf44a35d7 temp
> [root at jic55124 bin]# mounted.ocfs2 -f
> Device FS Nodes
> /dev/sdf ocfs2 *Unknown: Bad magic number in inode*
> /dev/sdf1 ocfs2 jic55124, jic55123, node3, node8, node4, node1, node5, 
> node6, node7
> /dev/sdg ocfs2 jic55123
>
>
> Thanks
>
> Bob.
>
> =====================================================
> Bob Findlay
> The Operations Centre – Norwich BioScience Institutes
> Tel: 01603 450474 (2474 internal)
> Fax: 01603 450045
> =====================================================
>
>
> -----Original Message-----
> From: ocfs2-devel-bounces at oss.oracle.com 
> [mailto:ocfs2-devel-bounces at oss.oracle.com] On Behalf Of bob findlay (TOC)
> Sent: 11 January 2008 11:17
> To: ocfs2-devel at oss.oracle.com; ocfs2-users at oss.oracle.com
> Subject: [Ocfs2-devel] systems hang when accessing parts of the OCFS2 
> filesystem
>
> Hi everyone
>
> Firstly, apologies for the cross post, I am not sure which list is 
> most appropriate for this question. I should also point out, that I 
> did not install OCFS2 and I am not the person that normally looks 
> after these kind of things, so please can you bear that in mind when 
> you make any suggestions (I will need a lot of detail!)
>
> The problem: accessing certain directories within the cluster file 
> system e.g. with "ls" cause the process to hang permanently. I cannot 
> cancel the request, I have to terminate the session. This is happening 
> across multiple nodes, so I am assuming that OCFS2 is the root cause 
> of the problem.
>
> Accessing the directory in debug mode seems to work fine eg this 
> command will hang my session
>
> [root at jic55124 databases]# ls -l /common/users/cbu/vigourom
>
> Whereas this works fine
>
> [root at jic55124 databases]# echo "ls -l /users/cbu/vigourom" | 
> debugfs.ocfs2 -n /dev/sdf1
> 25447960 drwxr-xr-x 33 2522 2004 4096 10-Jan-2008 16:30 .
> 25447672 drwxr-xr-x 5 3773 2004 4096 30-Nov-2007 14:27 ..
> 25447961 drwx------ 2 2522 2004 4096 1-Aug-2007 12:06 .ssh
> 25447963 -rw-r--r-- 1 2522 2004 3814 1-Aug-2007 17:04 addgi_new3.pl
> 25447964 -rw-r--r-- 1 0 0 0 1-Aug-2007 17:05 allmaize.out
> 25447965 -rw------- 1 2522 2004 1741 15-Aug-2007 11:13 .viminfo
> 25447966 drwxr-xr-x 3 2522 2004 4096 4-Sep-2007 12:07 .mcop
> 25447970 drwxr-xr-x 2 2522 2004 4096 4-Sep-2007 15:43 forUNIGENE
> 25447971 -rw-r--r-- 1 0 0 325655 1-Aug-2007 15:02 maize.out
> 25447972 -rw-r--r-- 1 0 0 264 1-Aug-2007 15:42 README
> 25447973 -rwxr--r-- 1 2522 2004 7209696 8-Aug-2007 14:53 
> bioperl-1.5.2_102.zip
> 25447974 drwxrwsr-x 9 2522 2004 4096 13-Aug-2007 14:59 bioperl-1.5.2_102
> 22610705 drwxr-xr-x 2 2522 2004 4096 14-Aug-2007 17:10 perl5lib
> 22610706 drwxr-xr-x 3 2522 2004 4096 14-Aug-2007 17:11 .cpan
> 22610709 drwx------ 4 2522 2004 4096 4-Sep-2007 11:39 .gnome
> 22610713 drwx------ 4 2522 2004 4096 4-Sep-2007 14:58 .gnome2
> 22610719 drwx------ 2 2522 2004 4096 4-Sep-2007 11:39 .gnome2_private
> 22610720 drwx------ 4 2522 2004 4096 4-Sep-2007 11:40 .kde
> 229702011 -rw------- 1 2522 2004 771 10-Jan-2008 09:40 .Xauthority
> 22610820 drwx------ 4 2522 2004 4096 9-Jan-2008 14:08 .gconf
> 22610835 drwx------ 2 2522 2004 4096 10-Jan-2008 13:41 .gconfd
> 22610837 drwxr-xr-x 3 2522 2004 4096 4-Sep-2007 11:39 .nautilus
> 22610842 drwxr-xr-x 4 2522 2004 4096 4-Sep-2007 15:27 Desktop
> 28545914 drwxr-xr-x 2 2522 2004 4096 4-Sep-2007 11:40 .qt
> 28545917 drwxr-xr-x 2 2522 2004 4096 4-Sep-2007 11:42 .fonts
> 28545922 drwx------ 3 2522 2004 4096 4-Sep-2007 12:13 .mozilla
> 4567882 -rw-r--r-- 1 2522 2004 53 9-Jan-2008 14:08 .fonts.cache-1
> 28545956 -rw------- 1 2522 2004 0 6-Sep-2007 15:30 .ICEauthority
> 28545957 -rw-r--r-- 1 2522 2004 110 4-Sep-2007 11:42 .fonts.conf
> 28545958 -rw------- 1 2522 2004 31 4-Sep-2007 12:07 .mcoprc
> 28545959 drwxr-xr-x 2 2522 2004 4096 4-Sep-2007 12:17 .wp
> 28545962 drwxr-xr-x 2 2522 2004 4096 4-Sep-2007 15:04 .seqlab-node7
> 28545967 -rw-r--r-- 1 2522 2004 707 4-Sep-2007 16:16 .seqlab-history
> 28545968 drwxr-xr-x 5 2522 2004 4096 4-Sep-2007 15:05 GCGSeqmergeTests
> etc
>
> stat gives
>
> [root at jic55124 databases]# echo "stat /users/cbu/vigourom" | 
> debugfs.ocfs2 -n /dev/sdf1
> Inode: 25447960 Mode: 0755 Generation: 1766836575 (0x694fc95f)
> FS Generation: 3856768928 (0xe5e19fa0)
> Type: Directory Attr: 0x0 Flags: Valid
> User: 2522 (vigourom) Group: 2004 (cbu) Size: 4096
> Links: 33 Clusters: 1
> ctime: 0x4786481b -- Thu Jan 10 16:30:19 2008
> atime: 0x46a9a7dc -- Fri Jul 27 09:07:56 2007
> mtime: 0x4786481b -- Thu Jan 10 16:30:19 2008
> dtime: 0x0 -- Thu Jan 1 01:00:00 1970
> ctime_nsec: 0x33de5143 -- 870207811
> atime_nsec: 0x0ba52bb0 -- 195374000
> mtime_nsec: 0x33de5143 -- 870207811
> Last Extblk: 0
> Sub Alloc Slot: 4 Sub Alloc Bit: 544
> Tree Depth: 0 Count: 243 Next Free Rec: 1
> ## Offset Clusters Block#
> 0 0 1 20289216
>
> fsck.ocfs2 gives internal logic failures (or faliures ;) amongst other 
> things, which sounds pretty bad. Is it?
>
> [root at jic55124 ~]# fsck.ocfs2 -fn /dev/sdf1
> Checking OCFS2 filesystem in /dev/sdf1:
> label: oracle
> uuid: e4 18 cb 00 24 2f 4d f2 96 b4 6f 3b 0a e9 b2 e8
> number of blocks: 243930952
> bytes per block: 4096
> number of clusters: 30491369
> bytes per cluster: 32768
> max slots: 24
>
> ** Skipping journal replay because -n was given. There may be spurious 
> errors that journal replay would fix. **
> /dev/sdf1 was run with -f, check forced.
> Pass 0a: Checking cluster allocation chains
> [GROUP_FREE_BITS] Group descriptor at block 177020928 claims to have 2 
> free bits which is more than 0 bits indicated by the bitmap.n
> Pass 0b: Checking inode allocation chains
> Pass 0c: Checking extent block allocation chains
> Pass 1: Checking inodes and blocks.
> o2fsck_mark_cluster_allocated: Internal logic faliure !! duplicate 
> cluster 22151173
> [DIR_ZERO] Inode 149371341 is a zero length directory, clear it? n
> [CLUSTER_ALLOC_BIT] Cluster 11553628 is marked in the global cluster 
> bitmap but it isn't in use. Clear its bit in the bitmap? n
> [CLUSTER_ALLOC_BIT] Cluster 16917926 is marked in the global cluster 
> bitmap but it isn't in use. Clear its bit in the bitmap? n
> Pass 2: Checking directory entries.
> [DIRENT_INODE_FREE] Directory entry '#74502784' refers to inode number 
> 74502784 which isn't allocated, clear the entry? n
> Pass 3: Checking directory connectivity.
> [DIR_NOT_CONNECTED] Directory inode 149371341 isn't connected to the 
> filesystem. Move it to lost+found? n
> Pass 4a: checking for orphaned inodes
> ** Skipping orphan dir replay because -n was given.
> Pass 4b: Checking inodes link counts.
> [INODE_COUNT] Inode 74502784 has a link count of 0 on disk but 
> directory entry references come to 1. Update the count on disk to man
> [INODE_COUNT] Inode 142698567 has a link count of 1 on disk but 
> directory entry references come to 2. Update the count on disk to mn
> pass4: Internal logic faliure fsck's thinks inode 149371307 has a link 
> count of 1 but on disk it is 0
> [INODE_COUNT] Inode 149371307 has a link count of 1 on disk but 
> directory entry references come to 2. Update the count on disk to mn
> [INODE_NOT_CONNECTED] Inode 149371307 isn't referenced by any 
> directory entries. Move it to lost+found? n
> [INODE_COUNT] Inode 149371341 has a link count of 2 on disk but 
> directory entry references come to 0. Update the count on disk to mn
> All passes succeeded.
>
>
> This has happened before and was "resolved" by shutting down the 
> cluster and performing a fsck.ocfs2, but that doesn't help us prevent 
> it in the future, so I would really like to resolve it properly.
>
> any suggestions as to how I can narrow down the cause of this problem 
> please? (or how to fix it would be even better! ;-)
>
> Thanks
>
> Bob.
>
> =====================================================
> Bob Findlay
> The Operations Centre – Norwich BioScience Institutes
> Tel: 01603 450474 (2474 internal)
> Fax: 01603 450045
> =====================================================
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users