[Ocfs2-users] Encountered disk I/O error 19502

Diane Petersen diane_petersen at yahoo.com
Mon Apr 6 07:05:44 PDT 2009 

Hi,

We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on both nodes almost ever other day. The system administrator has checked the SAN array and said there are no issues being reported. 

Another part of the problem, it appears the instances alter the service_names parameter not allowing new connections to the node with the reported error, but also terminate sessions already connected using the RAC service. The errors all start with - Encountered disk I/O error 19502 - and contain the following:


ARC2: Encountered disk I/O error 19502
 (ifxdb2)
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
ARC2: I/O error 19502 archiving log 10 to '/u03/arch/2_1917_656008464.dbf'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance ifxdb2 - Archival Error
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number  (block size=)
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number  (block size=)
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:05:16 2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY SID='ifxdb2';
Immediate Kill Session#: 185, Serial#: 40263
Immediate Kill Session: sess: 0x1274fabc8  OS pid: 13270
Immediate Kill Session#: 187, Serial#: 41391
Immediate Kill Session: sess: 0x1274fd710  OS pid: 27697
Immediate Kill Session#: 191, Serial#: 40464
Immediate Kill Session: sess: 0x127502da0  OS pid: 30697
Immediate Kill Session#: 195, Serial#: 57362
Immediate Kill Session: sess: 0x127508430  OS pid: 27967
Immediate Kill Session#: 196, Serial#: 2028
Immediate Kill Session: sess: 0x124544048  OS pid: 22900
Immediate Kill Session#: 205, Serial#: 17412
Immediate Kill Session: sess: 0x127515c98  OS pid: 20110
Immediate Kill Session#: 206, Serial#: 14805
Immediate Kill Session: sess: 0x1245518b0  OS pid: 10464
Immediate Kill Session#: 207, Serial#: 52184
Immediate Kill Session: sess: 0x1275187e0  OS pid: 19787
Immediate Kill Session#: 208, Serial#: 62825
Immediate Kill Session: sess: 0x1245543f8  OS pid: 13578
Immediate Kill Session#: 213, Serial#: 36907
Immediate Kill Session: sess: 0x1275209b8  OS pid: 31397
Immediate Kill Session#: 214, Serial#: 49032
Immediate Kill Session: sess: 0x12455c5d0  OS pid: 2427
Immediate Kill Session#: 215, Serial#: 2711
Immediate Kill Session: sess: 0x127523500  OS pid: 15942
Immediate Kill Session#: 216, Serial#: 30060
Immediate Kill Session: sess: 0x12455f118  OS pid: 1217
Immediate Kill Session#: 219, Serial#: 35932
Immediate Kill Session: sess: 0x127528b90  OS pid: 27883
Immediate Kill Session#: 222, Serial#: 26007
Immediate Kill Session: sess: 0x1245672f0  OS pid: 1036
Immediate Kill Session#: 223, Serial#: 42462
Immediate Kill Session: sess: 0x12752e220  OS pid: 13726
Immediate Kill Session#: 224, Serial#: 33323
Immediate Kill Session: sess: 0x124569e38  OS pid: 29928
Immediate Kill Session#: 225, Serial#: 49752
Immediate Kill Session: sess: 0x127530d68  OS pid: 20147
Immediate Kill Session#: 227, Serial#: 34834
Immediate Kill Session: sess: 0x1275338b0  OS pid: 9365
Immediate Kill Session#: 230, Serial#: 19879
Immediate Kill Session: sess: 0x124572010  OS pid: 15791
Immediate Kill Session#: 231, Serial#: 16554
Immediate Kill Session: sess: 0x127538f40  OS pid: 15490
Immediate Kill Session#: 233, Serial#: 25251
Immediate Kill Session: sess: 0x12753ba88  OS pid: 6972
Immediate Kill Session#: 236, Serial#: 36970
Immediate Kill Session: sess: 0x12457a1e8  OS pid: 12354
Immediate Kill Session#: 244, Serial#: 37284
Immediate Kill Session: sess: 0x124584f08  OS pid: 19290
Immediate Kill Session#: 245, Serial#: 55792
Immediate Kill Session: sess: 0x12754be38  OS pid: 19288
Immediate Kill Session#: 246, Serial#: 25115
Immediate Kill Session: sess: 0x124587a50  OS pid: 3111
Immediate Kill Session#: 247, Serial#: 6416
Immediate Kill Session: sess: 0x12754e980  OS pid: 19471
Immediate Kill Session#: 251, Serial#: 19899
Immediate Kill Session: sess: 0x127554010  OS pid: 21486
Immediate Kill Session#: 252, Serial#: 34731
Immediate Kill Session: sess: 0x12458fc28  OS pid: 30540
Immediate Kill Session#: 253, Serial#: 32638
Immediate Kill Session: sess: 0x127556b58  OS pid: 5493
Immediate Kill Session#: 259, Serial#: 29155
Immediate Kill Session: sess: 0x12755ed30  OS pid: 29463
Immediate Kill Session#: 261, Serial#: 14481
Immediate Kill Session: sess: 0x127561878  OS pid: 31054
Immediate Kill Session#: 265, Serial#: 37618
Immediate Kill Session: sess: 0x127566f08  OS pid: 868
Immediate Kill Session#: 267, Serial#: 42580
Immediate Kill Session: sess: 0x127569a50  OS pid: 16839
Immediate Kill Session#: 268, Serial#: 50893
Immediate Kill Session: sess: 0x1245a5668  OS pid: 27778
Immediate Kill Session#: 274, Serial#: 34459
Immediate Kill Session: sess: 0x1245ad840  OS pid: 9808
Immediate Kill Session#: 278, Serial#: 59445
Immediate Kill Session: sess: 0x1245b2ed0  OS pid: 28434
Immediate Kill Session#: 281, Serial#: 50119
Immediate Kill Session: sess: 0x12757c948  OS pid: 12606
Immediate Kill Session#: 282, Serial#: 30208
Immediate Kill Session: sess: 0x1245b8560  OS pid: 17944
Immediate Kill Session#: 285, Serial#: 53580
Immediate Kill Session: sess: 0x127581fd8  OS pid: 16670
Immediate Kill Session#: 286, Serial#: 5929
Immediate Kill Session: sess: 0x1245bdbf0  OS pid: 20149
Immediate Kill Session#: 289, Serial#: 53725
Immediate Kill Session: sess: 0x127587668  OS pid: 14697
Immediate Kill Session#: 290, Serial#: 30378
Immediate Kill Session: sess: 0x1245c3280  OS pid: 19757
Immediate Kill Session#: 293, Serial#: 53710
Immediate Kill Session: sess: 0x12758ccf8  OS pid: 11096
Immediate Kill Session#: 296, Serial#: 34022
Immediate Kill Session: sess: 0x1245cb458  OS pid: 10881
Immediate Kill Session#: 299, Serial#: 53951
Immediate Kill Session: sess: 0x127594ed0  OS pid: 1453
Immediate Kill Session#: 304, Serial#: 15149
Immediate Kill Session: sess: 0x1245d6178  OS pid: 22008
Immediate Kill Session#: 308, Serial#: 34245
Immediate Kill Session: sess: 0x1245db808  OS pid: 19156
Immediate Kill Session#: 315, Serial#: 15240
Immediate Kill Session: sess: 0x1275aa910  OS pid: 32148
Immediate Kill Session#: 317, Serial#: 41792
Immediate Kill Session: sess: 0x1275ad458  OS pid: 15660
Immediate Kill Session#: 318, Serial#: 7839
Immediate Kill Session: sess: 0x1245e9070  OS pid: 24999
Immediate Kill Session#: 321, Serial#: 4422
Immediate Kill Session: sess: 0x1275b2ae8  OS pid: 16028
Immediate Kill Session#: 324, Serial#: 6833
Immediate Kill Session: sess: 0x1245f1248  OS pid: 21909
Immediate Kill Session#: 332, Serial#: 18018
Immediate Kill Session: sess: 0x1245fbf68  OS pid: 15819
Immediate Kill Session#: 333, Serial#: 37534
Immediate Kill Session: sess: 0x1275c2e98  OS pid: 16433
Immediate Kill Session#: 334, Serial#: 50463
Immediate Kill Session: sess: 0x1245feab0  OS pid: 5660
Immediate Kill Session#: 335, Serial#: 11994
Immediate Kill Session: sess: 0x1275c59e0  OS pid: 29575
Immediate Kill Session#: 336, Serial#: 26542
Immediate Kill Session: sess: 0x1246015f8  OS pid: 31868
Immediate Kill Session#: 345, Serial#: 46583
Immediate Kill Session: sess: 0x1275d3248  OS pid: 25399
Sun Apr 05 15:05:43 2009
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance ifxdb2 - Archival Error
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:10:52 2009
kcrrdmx: Successful archiving of previously failed ORL
Archiver process freed from errors. No longer stopped
Sun Apr 05 15:10:53 2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY SID='ifxdb2';These incidences are all occurring during archiving (redo logs and database files are using ASM, archiving and backups are on OCFS2). Even though it usually only lasts a few minutes, it's very noticeable to the customers because of all the sessions that are terminated. 

What should we be looking at to resolve this problem?
Please let me know if you have any questions.

Thanks,
Diane Petersen
ServerCare, Inc.


      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/5a872bff/attachment.html

More information about the Ocfs2-users mailing list