[Ocfs2-users] Encountered disk I/O error 19502
Karim Alkhayer
kkhayer at gmail.com
Mon Apr 6 10:57:05 PDT 2009
What if you run RMAN "backup archivelog all delete input every two hours"
this can help to reduce the number of files stored. This is SAN based backup
What is your setup in terms of backup?
From: Diane Petersen [mailto:diane_petersen at yahoo.com]
Sent: Monday, April 06, 2009 6:42 PM
To: Karim Alkhayer; ocfs2-users at oss.oracle.com
Subject: Re: [Ocfs2-users] Encountered disk I/O error 19502
Hi,
We already have TAF implemented, unfortunately that doesn't help. It suppose
TAF might help if the instance was terminated, but that's not happening
instead it terminates these individual sessions directly.
This happens on both nodes during writes to the OCFS2 partition at random
times but never at the same time. There is nothing else in the db alert log
or crs logs other than what I've included below.
Thanks,
Diane Petersen
ServerCare, Inc.
_____
From: Karim Alkhayer <kkhayer at gmail.com>
To: Diane Petersen <diane_petersen at yahoo.com>; ocfs2-users at oss.oracle.com
Sent: Monday, April 6, 2009 9:11:06 AM
Subject: RE: [Ocfs2-users] Encountered disk I/O error 19502
Hello Diane,
I believe that implementing TAF could help a bit in this case, at least to
become transparent to the end users, unless of course, the following points
are blocking in your case:
1. ALTER SESSION statements are lost:
Statements such as "ALTER SESSION ..." are not automatically re-issued to
the server following a failover. This can have a significant effect on
application behavior. For example:
ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD';
select sysdate from dual;
Result> 2009-01-31
<< Fail over the connection >>
select sysdate from dual;
Result> 31-JAN-09
2. In-progress transactions must be rolled back
3. Continuing work on existing cursors may raise an error (eg:
ORA-25401 "cannot continue fetches")
4. Failed over selects may take time to re-position (when
FAILOVER_TYPE=SELECT)
5. Client awareness of a Failover
Can we have an overview of the database setup, nature of transactions, and
parameters?
It would also help to examine the troublesome node behavior and recovery
measures.
Best regards,
Karim Alkhayer
From: ocfs2-users-bounces at oss.oracle.com
[mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of Diane Petersen
Sent: Monday, April 06, 2009 4:06 PM
To: ocfs2-users at oss.oracle.com
Subject: [Ocfs2-users] Encountered disk I/O error 19502
Hi,
We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux
kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on both
nodes almost ever other day. The system administrator has checked the SAN
array and said there are no issues being reported.
Another part of the problem, it appears the instances alter the
service_names parameter not allowing new connections to the node with the
reported error, but also terminate sessions already connected using the RAC
service. The errors all start with - Encountered disk I/O error 19502 - and
contain the following:
ARC2: Encountered disk I/O error 19502
(ifxdb2)
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file
"/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)
ARC2: I/O error 19502 archiving log 10 to '/u03/arch/2_1917_656008464.dbf'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE
Instance ifxdb2 - Archival Error
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:05:16 2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY
SID='ifxdb2';
Immediate Kill Session#: 185, Serial#: 40263
Immediate Kill Session: sess: 0x1274fabc8 OS pid: 13270
Immediate Kill Session#: 187, Serial#: 41391
Immediate Kill Session: sess: 0x1274fd710 OS pid: 27697
Immediate Kill Session#: 191, Serial#: 40464
Immediate Kill Session: sess: 0x127502da0 OS pid:
30697
Immediate Kill Session#: 195, Serial#: 57362
Immediate Kill Session: sess: 0x127508430 OS pid: 27967
Immediate Kill Session#: 196, Serial#: 2028
Immediate Kill Session: sess: 0x124544048 OS pid: 22900
Immediate Kill Session#: 205, Serial#: 17412
Immediate Kill Session: sess: 0x127515c98 OS pid: 20110
Immediate Kill Session#: 206, Serial#: 14805
Immediate Kill Session: sess: 0x1245518b0 OS pid: 10464
Immediate Kill Session#: 207, Serial#: 52184
Immediate Kill Session: sess: 0x1275187e0 OS pid: 19787
Immediate Kill Session#: 208, Serial#: 62825
Immediate Kill Session: sess: 0x1245543f8 OS pid: 13578
Immediate Kill Session#: 213, Serial#: 36907
Immediate Kill Session: sess: 0x1275209b8 OS pid: 31397
Immediate Kill Session#: 214, Serial#: 49032
Immediate Kill Session: sess: 0x12455c5d0 OS pid: 2427
Immediate Kill Session#: 215, Serial#: 2711
Immediate Kill Session: sess: 0x127523500 OS
pid: 15942
Immediate Kill Session#: 216, Serial#: 30060
Immediate Kill Session: sess: 0x12455f118 OS pid: 1217
Immediate Kill Session#: 219, Serial#: 35932
Immediate Kill Session: sess: 0x127528b90 OS pid: 27883
Immediate Kill Session#: 222, Serial#: 26007
Immediate Kill Session: sess: 0x1245672f0 OS pid: 1036
Immediate Kill Session#: 223, Serial#: 42462
Immediate Kill Session: sess: 0x12752e220 OS pid: 13726
Immediate Kill Session#: 224, Serial#: 33323
Immediate Kill Session: sess: 0x124569e38 OS pid: 29928
Immediate Kill Session#: 225, Serial#: 49752
Immediate Kill Session: sess: 0x127530d68 OS pid: 20147
Immediate Kill Session#: 227, Serial#: 34834
Immediate Kill Session: sess: 0x1275338b0 OS pid: 9365
Immediate Kill Session#: 230, Serial#: 19879
Immediate Kill Session: sess: 0x124572010 OS pid: 15791
Immediate Kill Session#: 231, Serial#: 16554
Immediate Kill Session: sess: 0x127538f40
OS pid: 15490
Immediate Kill Session#: 233, Serial#: 25251
Immediate Kill Session: sess: 0x12753ba88 OS pid: 6972
Immediate Kill Session#: 236, Serial#: 36970
Immediate Kill Session: sess: 0x12457a1e8 OS pid: 12354
Immediate Kill Session#: 244, Serial#: 37284
Immediate Kill Session: sess: 0x124584f08 OS pid: 19290
Immediate Kill Session#: 245, Serial#: 55792
Immediate Kill Session: sess: 0x12754be38 OS pid: 19288
Immediate Kill Session#: 246, Serial#: 25115
Immediate Kill Session: sess: 0x124587a50 OS pid: 3111
Immediate Kill Session#: 247, Serial#: 6416
Immediate Kill Session: sess: 0x12754e980 OS pid: 19471
Immediate Kill Session#: 251, Serial#: 19899
Immediate Kill Session: sess: 0x127554010 OS pid: 21486
Immediate Kill Session#: 252, Serial#: 34731
Immediate Kill Session: sess: 0x12458fc28 OS pid: 30540
Immediate Kill Session#: 253, Serial#: 32638
Immediate Kill Session: sess: 0x127556b58
OS pid: 5493
Immediate Kill Session#: 259, Serial#: 29155
Immediate Kill Session: sess: 0x12755ed30 OS pid: 29463
Immediate Kill Session#: 261, Serial#: 14481
Immediate Kill Session: sess: 0x127561878 OS pid: 31054
Immediate Kill Session#: 265, Serial#: 37618
Immediate Kill Session: sess: 0x127566f08 OS pid: 868
Immediate Kill Session#: 267, Serial#: 42580
Immediate Kill Session: sess: 0x127569a50 OS pid: 16839
Immediate Kill Session#: 268, Serial#: 50893
Immediate Kill Session: sess: 0x1245a5668 OS pid: 27778
Immediate Kill Session#: 274, Serial#: 34459
Immediate Kill Session: sess: 0x1245ad840 OS pid: 9808
Immediate Kill Session#: 278, Serial#: 59445
Immediate Kill Session: sess: 0x1245b2ed0 OS pid: 28434
Immediate Kill Session#: 281, Serial#: 50119
Immediate Kill Session: sess: 0x12757c948 OS pid: 12606
Immediate Kill Session#: 282, Serial#: 30208
Immediate Kill Session: sess: 0x1245b8560
OS pid: 17944
Immediate Kill Session#: 285, Serial#: 53580
Immediate Kill Session: sess: 0x127581fd8 OS pid: 16670
Immediate Kill Session#: 286, Serial#: 5929
Immediate Kill Session: sess: 0x1245bdbf0 OS pid: 20149
Immediate Kill Session#: 289, Serial#: 53725
Immediate Kill Session: sess: 0x127587668 OS pid: 14697
Immediate Kill Session#: 290, Serial#: 30378
Immediate Kill Session: sess: 0x1245c3280 OS pid: 19757
Immediate Kill Session#: 293, Serial#: 53710
Immediate Kill Session: sess: 0x12758ccf8 OS pid: 11096
Immediate Kill Session#: 296, Serial#: 34022
Immediate Kill Session: sess: 0x1245cb458 OS pid: 10881
Immediate Kill Session#: 299, Serial#: 53951
Immediate Kill Session: sess: 0x127594ed0 OS pid: 1453
Immediate Kill Session#: 304, Serial#: 15149
Immediate Kill Session: sess: 0x1245d6178 OS pid: 22008
Immediate Kill Session#: 308, Serial#: 34245
Immediate Kill Session: sess:
0x1245db808 OS pid: 19156
Immediate Kill Session#: 315, Serial#: 15240
Immediate Kill Session: sess: 0x1275aa910 OS pid: 32148
Immediate Kill Session#: 317, Serial#: 41792
Immediate Kill Session: sess: 0x1275ad458 OS pid: 15660
Immediate Kill Session#: 318, Serial#: 7839
Immediate Kill Session: sess: 0x1245e9070 OS pid: 24999
Immediate Kill Session#: 321, Serial#: 4422
Immediate Kill Session: sess: 0x1275b2ae8 OS pid: 16028
Immediate Kill Session#: 324, Serial#: 6833
Immediate Kill Session: sess: 0x1245f1248 OS pid: 21909
Immediate Kill Session#: 332, Serial#: 18018
Immediate Kill Session: sess: 0x1245fbf68 OS pid: 15819
Immediate Kill Session#: 333, Serial#: 37534
Immediate Kill Session: sess: 0x1275c2e98 OS pid: 16433
Immediate Kill Session#: 334, Serial#: 50463
Immediate Kill Session: sess: 0x1245feab0 OS pid: 5660
Immediate Kill Session#: 335, Serial#: 11994
Immediate Kill Session: sess:
0x1275c59e0 OS pid: 29575
Immediate Kill Session#: 336, Serial#: 26542
Immediate Kill Session: sess: 0x1246015f8 OS pid: 31868
Immediate Kill Session#: 345, Serial#: 46583
Immediate Kill Session: sess: 0x1275d3248 OS pid: 25399
Sun Apr 05 15:05:43 2009
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance ifxdb2 - Archival Error
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:10:52 2009
kcrrdmx: Successful archiving of previously failed ORL
Archiver process freed from errors. No longer stopped
Sun Apr 05 15:10:53
2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY
SID='ifxdb2';
These incidences are all occurring during archiving (redo logs and database
files are using ASM, archiving and backups are on OCFS2). Even though it
usually only lasts a few minutes, it's very noticeable to the customers
because of all the sessions that are terminated.
What should we be looking at to resolve this problem?
Please let me know if you have any questions.
Thanks,
Diane Petersen
ServerCare, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/3df96f97/attachment-0001.html
More information about the Ocfs2-users
mailing list