[Ocfs2-users] Encountered disk I/O error 19502

Diane Petersen diane_petersen at yahoo.com
Mon Apr 6 09:42:07 PDT 2009


Hi,

We already have TAF implemented, unfortunately that doesn't help. It suppose TAF might help if the instance was terminated, but that's not happening instead it terminates these individual sessions directly.

This happens on both nodes during writes to the OCFS2 partition at random times but never at the same time. There is nothing else in the db alert log or crs logs other than what I've included below.

Thanks,
Diane Petersen
ServerCare, Inc.




________________________________
From: Karim Alkhayer <kkhayer at gmail.com>
To: Diane Petersen <diane_petersen at yahoo.com>; ocfs2-users at oss.oracle.com
Sent: Monday, April 6, 2009 9:11:06 AM
Subject: RE: [Ocfs2-users] Encountered disk I/O error 19502

 
Hello Diane,
 
I believe that implementing TAF could help a bit in this case, at
least to become transparent to the end users, unless of course, the following
points are blocking in your case:
 
1.       ALTER SESSION statements are lost:  
Statements such as "ALTER
SESSION ..." are not automatically re-issued to the server following a
failover. This can have a significant effect on application behavior. For
example: 
ALTER SESSION
SET NLS_DATE_FORMAT='YYYY-MM-DD';
select sysdate
from dual;
Result>
2009-01-31
<< Fail
over the connection >>
select sysdate
from dual;
Result>
31-JAN-09
2.       In-progress transactions must be rolled back 
3.       Continuing work on existing cursors may raise an error (eg:
ORA-25401 "cannot continue fetches") 
4.       Failed over selects may take time to re-position (when FAILOVER_TYPE=SELECT) 
5.       Client awareness of a Failover
 
Can we have an overview of the database setup, nature of
transactions, and parameters?
 
It would also help to examine the troublesome node behavior and
recovery measures.
 
Best regards,
Karim Alkhayer
 
From:ocfs2-users-bounces at oss.oracle.com [mailto:ocfs2-users-bounces at oss.oracle.com] On
Behalf Of Diane Petersen
Sent: Monday, April 06, 2009 4:06 PM
To: ocfs2-users at oss.oracle.com
Subject: [Ocfs2-users] Encountered disk I/O error 19502
 
Hi,

We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux kernel
2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on both nodes almost
ever other day. The system administrator has checked the SAN array and said
there are no issues being reported. 

Another part of the problem, it appears the instances alter the service_names
parameter not allowing new connections to the node with the reported error, but
also terminate sessions already connected using the RAC service. The errors all
start with - Encountered disk I/O error 19502 - and contain the following:
ARC2: Encountered disk I/O error 19502


 (ifxdb2)


Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:


ORA-19502: write error on file
 "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)


ORA-27072: File I/O error


Linux-x86_64 Error: 5: Input/output error


Additional information: 4


Additional information: 155649


Additional information: -1


ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)


Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:


ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)


ORA-27072: File I/O error


Linux-x86_64 Error: 5: Input/output error


Additional information: 4


Additional information: 155649


Additional information: -1


ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)


ARC2: I/O error 19502 archiving log 10 to '/u03/arch/2_1917_656008464.dbf'


ARCH: Archival stopped, error occurred. Will continue retrying


ORACLE
 Instance ifxdb2 - Archival Error


ORA-16038: log 10 sequence# 1917 cannot be archived


ORA-19502: write error on file "", block number  (block size=)


ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'


Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:


ORA-16038: log 10 sequence# 1917 cannot be archived


ORA-19502: write error on file "", block number  (block size=)


ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'


Sun Apr 05 15:05:16 2009


ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY SID='ifxdb2';


Immediate Kill Session#: 185, Serial#: 40263


Immediate Kill Session: sess: 0x1274fabc8  OS pid: 13270


Immediate Kill Session#: 187, Serial#: 41391


Immediate Kill Session: sess: 0x1274fd710  OS pid: 27697


Immediate Kill Session#: 191, Serial#: 40464


Immediate Kill Session: sess: 0x127502da0  OS pid:
 30697


Immediate Kill Session#: 195, Serial#: 57362


Immediate Kill Session: sess: 0x127508430  OS pid: 27967


Immediate Kill Session#: 196, Serial#: 2028


Immediate Kill Session: sess: 0x124544048  OS pid: 22900


Immediate Kill Session#: 205, Serial#: 17412


Immediate Kill Session: sess: 0x127515c98  OS pid: 20110


Immediate Kill Session#: 206, Serial#: 14805


Immediate Kill Session: sess: 0x1245518b0  OS pid: 10464


Immediate Kill Session#: 207, Serial#: 52184


Immediate Kill Session: sess: 0x1275187e0  OS pid: 19787


Immediate Kill Session#: 208, Serial#: 62825


Immediate Kill Session: sess: 0x1245543f8  OS pid: 13578


Immediate Kill Session#: 213, Serial#: 36907


Immediate Kill Session: sess: 0x1275209b8  OS pid: 31397


Immediate Kill Session#: 214, Serial#: 49032


Immediate Kill Session: sess: 0x12455c5d0  OS pid: 2427


Immediate Kill Session#: 215, Serial#: 2711


Immediate Kill Session: sess: 0x127523500  OS
 pid: 15942


Immediate Kill Session#: 216, Serial#: 30060


Immediate Kill Session: sess: 0x12455f118  OS pid: 1217


Immediate Kill Session#: 219, Serial#: 35932


Immediate Kill Session: sess: 0x127528b90  OS pid: 27883


Immediate Kill Session#: 222, Serial#: 26007


Immediate Kill Session: sess: 0x1245672f0  OS pid: 1036


Immediate Kill Session#: 223, Serial#: 42462


Immediate Kill Session: sess: 0x12752e220  OS pid: 13726


Immediate Kill Session#: 224, Serial#: 33323


Immediate Kill Session: sess: 0x124569e38  OS pid: 29928


Immediate Kill Session#: 225, Serial#: 49752


Immediate Kill Session: sess: 0x127530d68  OS pid: 20147


Immediate Kill Session#: 227, Serial#: 34834


Immediate Kill Session: sess: 0x1275338b0  OS pid: 9365


Immediate Kill Session#: 230, Serial#: 19879


Immediate Kill Session: sess: 0x124572010  OS pid: 15791


Immediate Kill Session#: 231, Serial#: 16554


Immediate Kill Session: sess: 0x127538f40 
 OS pid: 15490


Immediate Kill Session#: 233, Serial#: 25251


Immediate Kill Session: sess: 0x12753ba88  OS pid: 6972


Immediate Kill Session#: 236, Serial#: 36970


Immediate Kill Session: sess: 0x12457a1e8  OS pid: 12354


Immediate Kill Session#: 244, Serial#: 37284


Immediate Kill Session: sess: 0x124584f08  OS pid: 19290


Immediate Kill Session#: 245, Serial#: 55792


Immediate Kill Session: sess: 0x12754be38  OS pid: 19288


Immediate Kill Session#: 246, Serial#: 25115


Immediate Kill Session: sess: 0x124587a50  OS pid: 3111


Immediate Kill Session#: 247, Serial#: 6416


Immediate Kill Session: sess: 0x12754e980  OS pid: 19471


Immediate Kill Session#: 251, Serial#: 19899


Immediate Kill Session: sess: 0x127554010  OS pid: 21486


Immediate Kill Session#: 252, Serial#: 34731


Immediate Kill Session: sess: 0x12458fc28  OS pid: 30540


Immediate Kill Session#: 253, Serial#: 32638


Immediate Kill Session: sess: 0x127556b58
  OS pid: 5493


Immediate Kill Session#: 259, Serial#: 29155


Immediate Kill Session: sess: 0x12755ed30  OS pid: 29463


Immediate Kill Session#: 261, Serial#: 14481


Immediate Kill Session: sess: 0x127561878  OS pid: 31054


Immediate Kill Session#: 265, Serial#: 37618


Immediate Kill Session: sess: 0x127566f08  OS pid: 868


Immediate Kill Session#: 267, Serial#: 42580


Immediate Kill Session: sess: 0x127569a50  OS pid: 16839


Immediate Kill Session#: 268, Serial#: 50893


Immediate Kill Session: sess: 0x1245a5668  OS pid: 27778


Immediate Kill Session#: 274, Serial#: 34459


Immediate Kill Session: sess: 0x1245ad840  OS pid: 9808


Immediate Kill Session#: 278, Serial#: 59445


Immediate Kill Session: sess: 0x1245b2ed0  OS pid: 28434


Immediate Kill Session#: 281, Serial#: 50119


Immediate Kill Session: sess: 0x12757c948  OS pid: 12606


Immediate Kill Session#: 282, Serial#: 30208


Immediate Kill Session: sess: 0x1245b8560
  OS pid: 17944


Immediate Kill Session#: 285, Serial#: 53580


Immediate Kill Session: sess: 0x127581fd8  OS pid: 16670


Immediate Kill Session#: 286, Serial#: 5929


Immediate Kill Session: sess: 0x1245bdbf0  OS pid: 20149


Immediate Kill Session#: 289, Serial#: 53725


Immediate Kill Session: sess: 0x127587668  OS pid: 14697


Immediate Kill Session#: 290, Serial#: 30378


Immediate Kill Session: sess: 0x1245c3280  OS pid: 19757


Immediate Kill Session#: 293, Serial#: 53710


Immediate Kill Session: sess: 0x12758ccf8  OS pid: 11096


Immediate Kill Session#: 296, Serial#: 34022


Immediate Kill Session: sess: 0x1245cb458  OS pid: 10881


Immediate Kill Session#: 299, Serial#: 53951


Immediate Kill Session: sess: 0x127594ed0  OS pid: 1453


Immediate Kill Session#: 304, Serial#: 15149


Immediate Kill Session: sess: 0x1245d6178  OS pid: 22008


Immediate Kill Session#: 308, Serial#: 34245


Immediate Kill Session: sess:
 0x1245db808  OS pid: 19156


Immediate Kill Session#: 315, Serial#: 15240


Immediate Kill Session: sess: 0x1275aa910  OS pid: 32148


Immediate Kill Session#: 317, Serial#: 41792


Immediate Kill Session: sess: 0x1275ad458  OS pid: 15660


Immediate Kill Session#: 318, Serial#: 7839


Immediate Kill Session: sess: 0x1245e9070  OS pid: 24999


Immediate Kill Session#: 321, Serial#: 4422


Immediate Kill Session: sess: 0x1275b2ae8  OS pid: 16028


Immediate Kill Session#: 324, Serial#: 6833


Immediate Kill Session: sess: 0x1245f1248  OS pid: 21909


Immediate Kill Session#: 332, Serial#: 18018


Immediate Kill Session: sess: 0x1245fbf68  OS pid: 15819


Immediate Kill Session#: 333, Serial#: 37534


Immediate Kill Session: sess: 0x1275c2e98  OS pid: 16433


Immediate Kill Session#: 334, Serial#: 50463


Immediate Kill Session: sess: 0x1245feab0  OS pid: 5660


Immediate Kill Session#: 335, Serial#: 11994


Immediate Kill Session: sess:
 0x1275c59e0  OS pid: 29575


Immediate Kill Session#: 336, Serial#: 26542


Immediate Kill Session: sess: 0x1246015f8  OS pid: 31868


Immediate Kill Session#: 345, Serial#: 46583


Immediate Kill Session: sess: 0x1275d3248  OS pid: 25399


Sun Apr 05 15:05:43 2009


ARCH: Archival stopped, error occurred. Will continue retrying


ORACLE Instance ifxdb2 - Archival Error


ORA-16014: log 10 sequence# 1917 not archived, no available destinations


ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'


Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:


ORA-16014: log 10 sequence# 1917 not archived, no available destinations


ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'


Sun Apr 05 15:10:52 2009


kcrrdmx: Successful archiving of previously failed ORL


Archiver process freed from errors. No longer stopped


Sun Apr 05 15:10:53
 2009


ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY SID='ifxdb2';
These incidences are all
occurring during archiving (redo logs and database files are using ASM,
archiving and backups are on OCFS2). Even though it usually only lasts a few
minutes, it's very noticeable to the customers because of all the sessions that
are terminated. 

What should we be looking at to resolve this problem?
Please let me know if you have any questions.

Thanks,
Diane Petersen
ServerCare, Inc.


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/f335d218/attachment-0001.html 


More information about the Ocfs2-users mailing list