[Ocfs2-users] Encountered disk I/O error 19502
Diane Petersen
diane_petersen at yahoo.com
Mon Apr 6 12:47:39 PDT 2009
Hi Luis,
We are using ASM diskgroups +DATA1 and +REDO1 for datafiles and redo logs respectively. We have two separate OCFS2 partitions, /u02 is for RMAN backups and /u03 is for the archive logs for both nodes. I think what you're referring to are the redo logs in ASM which eaxh instance is attempting to write out to the OCFS2 partition during the archive process. Here's a copy of the /etc/fstab:
LABEL=/ / ext3 defaults 1 1
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
LABEL=SWAP-sda5 swap swap defaults 0 0
/dev/mapper/disk1p1 /u02 ocfs2 _netdev,datavolume,nointr 0 0
/dev/mapper/disk4p1 /u03 ocfs2 _netdev,datavolume,nointr 0 0
I believe this is a problem writing to the OCFS2 partition not reading from ASM, but I don't know what's causing it.
Thanks,
Diane Petersen
ServerCare, Inc.
________________________________
From: Luis Freitas <lfreitas34 at yahoo.com>
To: ocfs2-users at oss.oracle.com; Diane Petersen <diane_petersen at yahoo.com>
Sent: Monday, April 6, 2009 10:44:52 AM
Subject: Re: [Ocfs2-users] Encountered disk I/O error 19502
Diane,
Are you using ASM and OCFS2? Some of the log messages point to a disk group.
Can you post a copy of your /etc/fstab with the mount options?
Regards,
Luis
--- On Mon, 4/6/09, Diane Petersen <diane_petersen at yahoo.com> wrote:
> From: Diane Petersen <diane_petersen at yahoo.com>
> Subject: Re: [Ocfs2-users] Encountered disk I/O error 19502
> To: "Karim Alkhayer" <kkhayer at gmail.com>, ocfs2-users at oss.oracle.com
> Date: Monday, April 6, 2009, 1:42 PM
> Hi,
>
> We already have TAF implemented, unfortunately that
> doesn't help. It suppose TAF might help if the instance
> was terminated, but that's not happening instead it
> terminates these individual sessions directly.
>
> This happens on both nodes during writes to the OCFS2
> partition at random times but never at the same time. There
> is nothing else in the db alert log or crs logs other than
> what I've included below.
>
> Thanks,
> Diane Petersen
> ServerCare, Inc.
>
>
>
>
> ________________________________
> From: Karim Alkhayer <kkhayer at gmail.com>
> To: Diane Petersen <diane_petersen at yahoo.com>;
> ocfs2-users at oss.oracle.com
> Sent: Monday, April 6, 2009 9:11:06 AM
> Subject: RE: [Ocfs2-users] Encountered disk I/O error 19502
>
>
> Hello Diane,
>
> I believe that implementing TAF could help a bit in this
> case, at
> least to become transparent to the end users, unless of
> course, the following
> points are blocking in your case:
>
> 1. ALTER SESSION statements are lost:
> Statements such as "ALTER
> SESSION ..." are not automatically re-issued to the
> server following a
> failover. This can have a significant effect on application
> behavior. For
> example:
> ALTER SESSION
> SET NLS_DATE_FORMAT='YYYY-MM-DD';
> select sysdate
> from dual;
> Result>
> 2009-01-31
> << Fail
> over the connection >>
> select sysdate
> from dual;
> Result>
> 31-JAN-09
> 2. In-progress transactions must be rolled back
> 3. Continuing work on existing cursors may raise an
> error (eg:
> ORA-25401 "cannot continue fetches")
> 4. Failed over selects may take time to re-position
> (when FAILOVER_TYPE=SELECT)
> 5. Client awareness of a Failover
>
> Can we have an overview of the database setup, nature of
> transactions, and parameters?
>
> It would also help to examine the troublesome node behavior
> and
> recovery measures.
>
> Best regards,
> Karim Alkhayer
>
> From:ocfs2-users-bounces at oss.oracle.com
> [mailto:ocfs2-users-bounces at oss.oracle.com] On
> Behalf Of Diane Petersen
> Sent: Monday, April 06, 2009 4:06 PM
> To: ocfs2-users at oss.oracle.com
> Subject: [Ocfs2-users] Encountered disk I/O error 19502
>
> Hi,
>
> We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5
> with Linux kernel
> 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing
> errors on both nodes almost
> ever other day. The system administrator has checked the
> SAN array and said
> there are no issues being reported.
>
> Another part of the problem, it appears the instances alter
> the service_names
> parameter not allowing new connections to the node with the
> reported error, but
> also terminate sessions already connected using the RAC
> service. The errors all
> start with - Encountered disk I/O error 19502 - and contain
> the following:
> ARC2: Encountered disk I/O error 19502
>
>
> (ifxdb2)
>
>
> Errors in file
> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
>
>
> ORA-19502: write error on file
> "/u03/arch/2_1917_656008464.dbf", block number
> 155649 (block size=512)
>
>
> ORA-27072: File I/O error
>
>
> Linux-x86_64 Error: 5: Input/output error
>
>
> Additional information: 4
>
>
> Additional information: 155649
>
>
> Additional information: -1
>
>
> ORA-19502: write error on file
> "/u03/arch/2_1917_656008464.dbf", block number
> 155649 (block size=512)
>
>
> Errors in file
> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
>
>
> ORA-19502: write error on file
> "/u03/arch/2_1917_656008464.dbf", block number
> 155649 (block size=512)
>
>
> ORA-27072: File I/O error
>
>
> Linux-x86_64 Error: 5: Input/output error
>
>
> Additional information: 4
>
>
> Additional information: 155649
>
>
> Additional information: -1
>
>
> ORA-19502: write error on file
> "/u03/arch/2_1917_656008464.dbf", block number
> 155649 (block size=512)
>
>
> ARC2: I/O error 19502 archiving log 10 to
> '/u03/arch/2_1917_656008464.dbf'
>
>
> ARCH: Archival stopped, error occurred. Will continue
> retrying
>
>
> ORACLE
> Instance ifxdb2 - Archival Error
>
>
> ORA-16038: log 10 sequence# 1917 cannot be archived
>
>
> ORA-19502: write error on file "", block number
> (block size=)
>
>
> ORA-00312: online log 10 thread 2:
> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
>
>
> Errors in file
> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
>
>
> ORA-16038: log 10 sequence# 1917 cannot be archived
>
>
> ORA-19502: write error on file "", block number
> (block size=)
>
>
> ORA-00312: online log 10 thread 2:
> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
>
>
> Sun Apr 05 15:05:16 2009
>
>
> ALTER SYSTEM SET
> service_names='ifxdb.gointranet.com' SCOPE=MEMORY
> SID='ifxdb2';
>
>
> Immediate Kill Session#: 185, Serial#: 40263
>
>
> Immediate Kill Session: sess: 0x1274fabc8 OS pid: 13270
>
>
> Immediate Kill Session#: 187, Serial#: 41391
>
>
> Immediate Kill Session: sess: 0x1274fd710 OS pid: 27697
>
>
> Immediate Kill Session#: 191, Serial#: 40464
>
>
> Immediate Kill Session: sess: 0x127502da0 OS pid:
> 30697
>
>
> Immediate Kill Session#: 195, Serial#: 57362
>
>
> Immediate Kill Session: sess: 0x127508430 OS pid: 27967
>
>
> Immediate Kill Session#: 196, Serial#: 2028
>
>
> Immediate Kill Session: sess: 0x124544048 OS pid: 22900
>
>
> Immediate Kill Session#: 205, Serial#: 17412
>
>
> Immediate Kill Session: sess: 0x127515c98 OS pid: 20110
>
>
> Immediate Kill Session#: 206, Serial#: 14805
>
>
> Immediate Kill Session: sess: 0x1245518b0 OS pid: 10464
>
>
> Immediate Kill Session#: 207, Serial#: 52184
>
>
> Immediate Kill Session: sess: 0x1275187e0 OS pid: 19787
>
>
> Immediate Kill Session#: 208, Serial#: 62825
>
>
> Immediate Kill Session: sess: 0x1245543f8 OS pid: 13578
>
>
> Immediate Kill Session#: 213, Serial#: 36907
>
>
> Immediate Kill Session: sess: 0x1275209b8 OS pid: 31397
>
>
> Immediate Kill Session#: 214, Serial#: 49032
>
>
> Immediate Kill Session: sess: 0x12455c5d0 OS pid: 2427
>
>
> Immediate Kill Session#: 215, Serial#: 2711
>
>
> Immediate Kill Session: sess: 0x127523500 OS
> pid: 15942
>
>
> Immediate Kill Session#: 216, Serial#: 30060
>
>
> Immediate Kill Session: sess: 0x12455f118 OS pid: 1217
>
>
> Immediate Kill Session#: 219, Serial#: 35932
>
>
> Immediate Kill Session: sess: 0x127528b90 OS pid: 27883
>
>
> Immediate Kill Session#: 222, Serial#: 26007
>
>
> Immediate Kill Session: sess: 0x1245672f0 OS pid: 1036
>
>
> Immediate Kill Session#: 223, Serial#: 42462
>
>
> Immediate Kill Session: sess: 0x12752e220 OS pid: 13726
>
>
> Immediate Kill Session#: 224, Serial#: 33323
>
>
> Immediate Kill Session: sess: 0x124569e38 OS pid: 29928
>
>
> Immediate Kill Session#: 225, Serial#: 49752
>
>
> Immediate Kill Session: sess: 0x127530d68 OS pid: 20147
>
>
> Immediate Kill Session#: 227, Serial#: 34834
>
>
> Immediate Kill Session: sess: 0x1275338b0 OS pid: 9365
>
>
> Immediate Kill Session#: 230, Serial#: 19879
>
>
> Immediate Kill Session: sess: 0x124572010 OS pid: 15791
>
>
> Immediate Kill Session#: 231, Serial#: 16554
>
>
> Immediate Kill Session: sess: 0x127538f40
> OS pid: 15490
>
>
> Immediate Kill Session#: 233, Serial#: 25251
>
>
> Immediate Kill Session: sess: 0x12753ba88 OS pid: 6972
>
>
> Immediate Kill Session#: 236, Serial#: 36970
>
>
> Immediate Kill Session: sess: 0x12457a1e8 OS pid: 12354
>
>
> Immediate Kill Session#: 244, Serial#: 37284
>
>
> Immediate Kill Session: sess: 0x124584f08 OS pid: 19290
>
>
> Immediate Kill Session#: 245, Serial#: 55792
>
>
> Immediate Kill Session: sess: 0x12754be38 OS pid: 19288
>
>
> Immediate Kill Session#: 246, Serial#: 25115
>
>
> Immediate Kill Session: sess: 0x124587a50 OS pid: 3111
>
>
> Immediate Kill Session#: 247, Serial#: 6416
>
>
> Immediate Kill Session: sess: 0x12754e980 OS pid: 19471
>
>
> Immediate Kill Session#: 251, Serial#: 19899
>
>
> Immediate Kill Session: sess: 0x127554010 OS pid: 21486
>
>
> Immediate Kill Session#: 252, Serial#: 34731
>
>
> Immediate Kill Session: sess: 0x12458fc28 OS pid: 30540
>
>
> Immediate Kill Session#: 253, Serial#: 32638
>
>
> Immediate Kill Session: sess: 0x127556b58
> OS pid: 5493
>
>
> Immediate Kill Session#: 259, Serial#: 29155
>
>
> Immediate Kill Session: sess: 0x12755ed30 OS pid: 29463
>
>
> Immediate Kill Session#: 261, Serial#: 14481
>
>
> Immediate Kill Session: sess: 0x127561878 OS pid: 31054
>
>
> Immediate Kill Session#: 265, Serial#: 37618
>
>
> Immediate Kill Session: sess: 0x127566f08 OS pid: 868
>
>
> Immediate Kill Session#: 267, Serial#: 42580
>
>
> Immediate Kill Session: sess: 0x127569a50 OS pid: 16839
>
>
> Immediate Kill Session#: 268, Serial#: 50893
>
>
> Immediate Kill Session: sess: 0x1245a5668 OS pid: 27778
>
>
> Immediate Kill Session#: 274, Serial#: 34459
>
>
> Immediate Kill Session: sess: 0x1245ad840 OS pid: 9808
>
>
> Immediate Kill Session#: 278, Serial#: 59445
>
>
> Immediate Kill Session: sess: 0x1245b2ed0 OS pid: 28434
>
>
> Immediate Kill Session#: 281, Serial#: 50119
>
>
> Immediate Kill Session: sess: 0x12757c948 OS pid: 12606
>
>
> Immediate Kill Session#: 282, Serial#: 30208
>
>
> Immediate Kill Session: sess: 0x1245b8560
> OS pid: 17944
>
>
> Immediate Kill Session#: 285, Serial#: 53580
>
>
> Immediate Kill Session: sess: 0x127581fd8 OS pid: 16670
>
>
> Immediate Kill Session#: 286, Serial#: 5929
>
>
> Immediate Kill Session: sess: 0x1245bdbf0 OS pid: 20149
>
>
> Immediate Kill Session#: 289, Serial#: 53725
>
>
> Immediate Kill Session: sess: 0x127587668 OS pid: 14697
>
>
> Immediate Kill Session#: 290, Serial#: 30378
>
>
> Immediate Kill Session: sess: 0x1245c3280 OS pid: 19757
>
>
> Immediate Kill Session#: 293, Serial#: 53710
>
>
> Immediate Kill Session: sess: 0x12758ccf8 OS pid: 11096
>
>
> Immediate Kill Session#: 296, Serial#: 34022
>
>
> Immediate Kill Session: sess: 0x1245cb458 OS pid: 10881
>
>
> Immediate Kill Session#: 299, Serial#: 53951
>
>
> Immediate Kill Session: sess: 0x127594ed0 OS pid: 1453
>
>
> Immediate Kill Session#: 304, Serial#: 15149
>
>
> Immediate Kill Session: sess: 0x1245d6178 OS pid: 22008
>
>
> Immediate Kill Session#: 308, Serial#: 34245
>
>
> Immediate Kill Session: sess:
> 0x1245db808 OS pid: 19156
>
>
> Immediate Kill Session#: 315, Serial#: 15240
>
>
> Immediate Kill Session: sess: 0x1275aa910 OS pid: 32148
>
>
> Immediate Kill Session#: 317, Serial#: 41792
>
>
> Immediate Kill Session: sess: 0x1275ad458 OS pid: 15660
>
>
> Immediate Kill Session#: 318, Serial#: 7839
>
>
> Immediate Kill Session: sess: 0x1245e9070 OS pid: 24999
>
>
> Immediate Kill Session#: 321, Serial#: 4422
>
>
> Immediate Kill Session: sess: 0x1275b2ae8 OS pid: 16028
>
>
> Immediate Kill Session#: 324, Serial#: 6833
>
>
> Immediate Kill Session: sess: 0x1245f1248 OS pid: 21909
>
>
> Immediate Kill Session#: 332, Serial#: 18018
>
>
> Immediate Kill Session: sess: 0x1245fbf68 OS pid: 15819
>
>
> Immediate Kill Session#: 333, Serial#: 37534
>
>
> Immediate Kill Session: sess: 0x1275c2e98 OS pid: 16433
>
>
> Immediate Kill Session#: 334, Serial#: 50463
>
>
> Immediate Kill Session: sess: 0x1245feab0 OS pid: 5660
>
>
> Immediate Kill Session#: 335, Serial#: 11994
>
>
> Immediate Kill Session: sess:
> 0x1275c59e0 OS pid: 29575
>
>
> Immediate Kill Session#: 336, Serial#: 26542
>
>
> Immediate Kill Session: sess: 0x1246015f8 OS pid: 31868
>
>
> Immediate Kill Session#: 345, Serial#: 46583
>
>
> Immediate Kill Session: sess: 0x1275d3248 OS pid: 25399
>
>
> Sun Apr 05 15:05:43 2009
>
>
> ARCH: Archival stopped, error occurred. Will continue
> retrying
>
>
> ORACLE Instance ifxdb2 - Archival Error
>
>
> ORA-16014: log 10 sequence# 1917 not archived, no available
> destinations
>
>
> ORA-00312: online log 10 thread 2:
> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
>
>
> Errors in file
> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
>
>
> ORA-16014: log 10 sequence# 1917 not archived, no available
> destinations
>
>
> ORA-00312: online log 10 thread 2:
> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
>
>
> Sun Apr 05 15:10:52 2009
>
>
> kcrrdmx: Successful archiving of previously failed ORL
>
>
> Archiver process freed from errors. No longer stopped
>
>
> Sun Apr 05 15:10:53
> 2009
>
>
> ALTER SYSTEM SET
> service_names='ifxdb.gointranet.com','ifxserv'
> SCOPE=MEMORY SID='ifxdb2';
> These incidences are all
> occurring during archiving (redo logs and database files
> are using ASM,
> archiving and backups are on OCFS2). Even though it usually
> only lasts a few
> minutes, it's very noticeable to the customers because
> of all the sessions that
> are terminated.
>
> What should we be looking at to resolve this problem?
> Please let me know if you have any questions.
>
> Thanks,
> Diane Petersen
> ServerCare, Inc.
>
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/a333c776/attachment-0001.html
More information about the Ocfs2-users
mailing list