[Ocfs2-users] ocfs2, redhat 2.6.18-238.x kernel and async i/o hang

[Matthew Huff]

I migrated a Oracle 11gR2 database from one HP RHEL5 box to another. The only difference I can tell is that the older box was running 2.16.18-194 kernel, and the newer box is running 2.6.18.238 kernel (I've tried 2.6.18.238.el5, 238.5.1.el5 and 239.19.1.el5). On the new box, any Oracle instance hangs on startup mounting the files if async io is turned on. Direct I/O works file. The files are all on ocfs2 partitions.

Anyone seen this issue? The databases run fine if the spfile has:

*.disk_asynch_io=FALSE
*.filesystemio_options='directio'

But hang 100% of the time (even after a few hours of waiting) if the parameters are set to:
*.disk_asynch_io=TRUE
*.filesystemio_options='setall'


Other than having libaio rpm loaded, I don't think there are any configuration or knobs I need to turn. 


Here are some details:


[oracle at hubble lib64]$ ldd $ORACLE_HOME/bin/oracle
        linux-vdso.so.1 =>  (0x00007fff9b7fc000)
        libodm11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libodm11.so (0x00002b487959a000)
        libcell11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libcell11.so (0x00002b487969c000)
        libskgxp11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libskgxp11.so (0x00002b4879800000)
        librt.so.1 => /lib64/librt.so.1 (0x00000032fa600000)
        libnnz11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libnnz11.so (0x00002b48799ea000)
        libclsra11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libclsra11.so (0x00002b4879db7000)
        libdbcfg11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libdbcfg11.so (0x00002b4879ecf000)
        libhasgen11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libhasgen11.so (0x00002b4879ff2000)
        libskgxn2.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libskgxn2.so (0x00002b487a7c6000)
        libocr11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocr11.so (0x00002b487a8c8000)
        libocrb11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocrb11.so (0x00002b487aa6e000)
        libocrutl11.so => /opt/oracle/product/11.2.0/dbhome_1/lib/libocrutl11.so (0x00002b487ac0e000)
        libaio.so.1 => /usr/lib64/libaio.so.1 (0x00002b487ad19000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00000032f9a00000)
        libm.so.6 => /lib64/libm.so.6 (0x00000032f9600000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00000032f9e00000)
        libnsl.so.1 => /lib64/libnsl.so.1 (0x00000032fc600000)
        libc.so.6 => /lib64/libc.so.6 (0x00000032f9200000)
        /lib64/ld-linux-x86-64.so.2 (0x00000032f8e00000)

[oracle at hubble lib64]$ rpm -qa | grep aio
libaio-0.3.106-5
libaio-devel-0.3.106-5
libsane-hpaio-1.6.7-6.el5_6.1
libaio-0.3.106-5
libaio-devel-0.3.106-5

[oracle at hubble dbs]$ cd /usr/lib64
[oracle at hubble lib64]$ ls -lF *aio*
-rw-r--r-- 1 root root 15870 Dec 21  2009 libaio.a
lrwxrwxrwx 1 root root    15 Jul 23 15:21 libaio.so -> libaio.so.1.0.1*
lrwxrwxrwx 1 root root    15 Jul 22 21:59 libaio.so.1 -> libaio.so.1.0.1*
-rwxr-xr-x 1 root root  3768 Dec 21  2009 libaio.so.1.0.0*
-rwxr-xr-x 1 root root  3768 Dec 21  2009 libaio.so.1.0.1*
-rw-r--r-- 1 root root 92866 Apr  8 09:14 librtkaio.a

[root at hubble ~]# grep ocfs2 /etc/fstab
LABEL=oracle_logfiles   /u01                    ocfs2   _netdev,datavolume,nointr
LABEL=oracle_datafiles  /u02                    ocfs2   _netdev,datavolume,nointr

[root at hubble ~]# grep ocfs2 /etc/mtab
ocfs2_dlmfs /dlm ocfs2_dlmfs rw 0 0
/dev/cciss/c0d1 /u01 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0
/dev/cciss/c0d2 /u02 ocfs2 rw,_netdev,datavolume,nointr,heartbeat=local 0 0

[oracle at hubble lib64]$ rpm -qa | grep ocfs
ocfs2console-1.4.4-1.el5
ocfs2-2.6.18-238.19.1.el5-1.4.7-1.el5
ocfs2-tools-1.4.4-1.el5 

[root at hubble ~]# uname -a
Linux hubble.ox.com 2.6.18-238.19.1.el5 #1 SMP Sun Jul 10 08:43:41 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux


----
Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039
aim: matthewbhuff        | Fax:   914-460-4139