[Ocfs2-users] Encountered disk I/O error 19502

Mon Apr 6 08:37:59 PDT 2009

do you notice any errors in messages files during the same time as the 
error reported in alert.log?.

please forward the output of  echo "stat 
/arch/2_1917_656008464.dbf"|debugfs.ocfs2 <device>

Can you please file a tar with Oracle and forward me the tar#.

thanks,
--Srini

Diane Petersen wrote:
> Hi,
>
> We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux 
> kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on 
> both nodes almost ever other day. The system administrator has checked 
> the SAN array and said there are no issues being reported.
>
> Another part of the problem, it appears the instances alter the 
> service_names parameter not allowing new connections to the node with 
> the reported error, but also terminate sessions already connected 
> using the RAC service. The errors all start with - Encountered disk 
> I/O error 19502 - and contain the following:
>
> ARC2: Encountered disk I/O error 19502
>  (ifxdb2)
> Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-19502: write error on file
>  "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
> ORA-27072: File I/O error
> Linux-x86_64 Error: 5: Input/output error
> Additional information: 4
> Additional information: 155649
> Additional information: -1
> ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
> Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
> ORA-27072: File I/O error
> Linux-x86_64 Error: 5: Input/output error
> Additional information: 4
> Additional information: 155649
> Additional information: -1
> ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
> ARC2: I/O error 19502 archiving log 10 to '/u03/arch/2_1917_656008464.dbf'
> ARCH: Archival stopped, error occurred. Will continue retrying
> ORACLE
>  Instance ifxdb2 - Archival Error
> ORA-16038: log 10 sequence# 1917 cannot be archived
> ORA-19502: write error on file "", block number  (block size=)
> ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-16038: log 10 sequence# 1917 cannot be archived
> ORA-19502: write error on file "", block number  (block size=)
> ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Sun Apr 05 15:05:16 2009
> ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY SID='ifxdb2';
> Immediate Kill Session#: 185, Serial#: 40263
> Immediate Kill Session: sess: 0x1274fabc8  OS pid: 13270
> Immediate Kill Session#: 187, Serial#: 41391
> Immediate Kill Session: sess: 0x1274fd710  OS pid: 27697
> Immediate Kill Session#: 191, Serial#: 40464
> Immediate Kill Session: sess: 0x127502da0  OS pid:
>  30697
> Immediate Kill Session#: 195, Serial#: 57362
> Immediate Kill Session: sess: 0x127508430  OS pid: 27967
> Immediate Kill Session#: 196, Serial#: 2028
> Immediate Kill Session: sess: 0x124544048  OS pid: 22900
> Immediate Kill Session#: 205, Serial#: 17412
> Immediate Kill Session: sess: 0x127515c98  OS pid: 20110
> Immediate Kill Session#: 206, Serial#: 14805
> Immediate Kill Session: sess: 0x1245518b0  OS pid: 10464
> Immediate Kill Session#: 207, Serial#: 52184
> Immediate Kill Session: sess: 0x1275187e0  OS pid: 19787
> Immediate Kill Session#: 208, Serial#: 62825
> Immediate Kill Session: sess: 0x1245543f8  OS pid: 13578
> Immediate Kill Session#: 213, Serial#: 36907
> Immediate Kill Session: sess: 0x1275209b8  OS pid: 31397
> Immediate Kill Session#: 214, Serial#: 49032
> Immediate Kill Session: sess: 0x12455c5d0  OS pid: 2427
> Immediate Kill Session#: 215, Serial#: 2711
> Immediate Kill Session: sess: 0x127523500  OS
>  pid: 15942
> Immediate Kill Session#: 216, Serial#: 30060
> Immediate Kill Session: sess: 0x12455f118  OS pid: 1217
> Immediate Kill Session#: 219, Serial#: 35932
> Immediate Kill Session: sess: 0x127528b90  OS pid: 27883
> Immediate Kill Session#: 222, Serial#: 26007
> Immediate Kill Session: sess: 0x1245672f0  OS pid: 1036
> Immediate Kill Session#: 223, Serial#: 42462
> Immediate Kill Session: sess: 0x12752e220  OS pid: 13726
> Immediate Kill Session#: 224, Serial#: 33323
> Immediate Kill Session: sess: 0x124569e38  OS pid: 29928
> Immediate Kill Session#: 225, Serial#: 49752
> Immediate Kill Session: sess: 0x127530d68  OS pid: 20147
> Immediate Kill Session#: 227, Serial#: 34834
> Immediate Kill Session: sess: 0x1275338b0  OS pid: 9365
> Immediate Kill Session#: 230, Serial#: 19879
> Immediate Kill Session: sess: 0x124572010  OS pid: 15791
> Immediate Kill Session#: 231, Serial#: 16554
> Immediate Kill Session: sess: 0x127538f40 
>  OS pid: 15490
> Immediate Kill Session#: 233, Serial#: 25251
> Immediate Kill Session: sess: 0x12753ba88  OS pid: 6972
> Immediate Kill Session#: 236, Serial#: 36970
> Immediate Kill Session: sess: 0x12457a1e8  OS pid: 12354
> Immediate Kill Session#: 244, Serial#: 37284
> Immediate Kill Session: sess: 0x124584f08  OS pid: 19290
> Immediate Kill Session#: 245, Serial#: 55792
> Immediate Kill Session: sess: 0x12754be38  OS pid: 19288
> Immediate Kill Session#: 246, Serial#: 25115
> Immediate Kill Session: sess: 0x124587a50  OS pid: 3111
> Immediate Kill Session#: 247, Serial#: 6416
> Immediate Kill Session: sess: 0x12754e980  OS pid: 19471
> Immediate Kill Session#: 251, Serial#: 19899
> Immediate Kill Session: sess: 0x127554010  OS pid: 21486
> Immediate Kill Session#: 252, Serial#: 34731
> Immediate Kill Session: sess: 0x12458fc28  OS pid: 30540
> Immediate Kill Session#: 253, Serial#: 32638
> Immediate Kill Session: sess: 0x127556b58
>   OS pid: 5493
> Immediate Kill Session#: 259, Serial#: 29155
> Immediate Kill Session: sess: 0x12755ed30  OS pid: 29463
> Immediate Kill Session#: 261, Serial#: 14481
> Immediate Kill Session: sess: 0x127561878  OS pid: 31054
> Immediate Kill Session#: 265, Serial#: 37618
> Immediate Kill Session: sess: 0x127566f08  OS pid: 868
> Immediate Kill Session#: 267, Serial#: 42580
> Immediate Kill Session: sess: 0x127569a50  OS pid: 16839
> Immediate Kill Session#: 268, Serial#: 50893
> Immediate Kill Session: sess: 0x1245a5668  OS pid: 27778
> Immediate Kill Session#: 274, Serial#: 34459
> Immediate Kill Session: sess: 0x1245ad840  OS pid: 9808
> Immediate Kill Session#: 278, Serial#: 59445
> Immediate Kill Session: sess: 0x1245b2ed0  OS pid: 28434
> Immediate Kill Session#: 281, Serial#: 50119
> Immediate Kill Session: sess: 0x12757c948  OS pid: 12606
> Immediate Kill Session#: 282, Serial#: 30208
> Immediate Kill Session: sess: 0x1245b8560
>   OS pid: 17944
> Immediate Kill Session#: 285, Serial#: 53580
> Immediate Kill Session: sess: 0x127581fd8  OS pid: 16670
> Immediate Kill Session#: 286, Serial#: 5929
> Immediate Kill Session: sess: 0x1245bdbf0  OS pid: 20149
> Immediate Kill Session#: 289, Serial#: 53725
> Immediate Kill Session: sess: 0x127587668  OS pid: 14697
> Immediate Kill Session#: 290, Serial#: 30378
> Immediate Kill Session: sess: 0x1245c3280  OS pid: 19757
> Immediate Kill Session#: 293, Serial#: 53710
> Immediate Kill Session: sess: 0x12758ccf8  OS pid: 11096
> Immediate Kill Session#: 296, Serial#: 34022
> Immediate Kill Session: sess: 0x1245cb458  OS pid: 10881
> Immediate Kill Session#: 299, Serial#: 53951
> Immediate Kill Session: sess: 0x127594ed0  OS pid: 1453
> Immediate Kill Session#: 304, Serial#: 15149
> Immediate Kill Session: sess: 0x1245d6178  OS pid: 22008
> Immediate Kill Session#: 308, Serial#: 34245
> Immediate Kill Session: sess:
>  0x1245db808  OS pid: 19156
> Immediate Kill Session#: 315, Serial#: 15240
> Immediate Kill Session: sess: 0x1275aa910  OS pid: 32148
> Immediate Kill Session#: 317, Serial#: 41792
> Immediate Kill Session: sess: 0x1275ad458  OS pid: 15660
> Immediate Kill Session#: 318, Serial#: 7839
> Immediate Kill Session: sess: 0x1245e9070  OS pid: 24999
> Immediate Kill Session#: 321, Serial#: 4422
> Immediate Kill Session: sess: 0x1275b2ae8  OS pid: 16028
> Immediate Kill Session#: 324, Serial#: 6833
> Immediate Kill Session: sess: 0x1245f1248  OS pid: 21909
> Immediate Kill Session#: 332, Serial#: 18018
> Immediate Kill Session: sess: 0x1245fbf68  OS pid: 15819
> Immediate Kill Session#: 333, Serial#: 37534
> Immediate Kill Session: sess: 0x1275c2e98  OS pid: 16433
> Immediate Kill Session#: 334, Serial#: 50463
> Immediate Kill Session: sess: 0x1245feab0  OS pid: 5660
> Immediate Kill Session#: 335, Serial#: 11994
> Immediate Kill Session: sess:
>  0x1275c59e0  OS pid: 29575
> Immediate Kill Session#: 336, Serial#: 26542
> Immediate Kill Session: sess: 0x1246015f8  OS pid: 31868
> Immediate Kill Session#: 345, Serial#: 46583
> Immediate Kill Session: sess: 0x1275d3248  OS pid: 25399
> Sun Apr 05 15:05:43 2009
> ARCH: Archival stopped, error occurred. Will continue retrying
> ORACLE Instance ifxdb2 - Archival Error
> ORA-16014: log 10 sequence# 1917 not archived, no available destinations
> ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
> ORA-16014: log 10 sequence# 1917 not archived, no available destinations
> ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
> Sun Apr 05 15:10:52 2009
> kcrrdmx: Successful archiving of previously failed ORL
> Archiver process freed from errors. No longer stopped
> Sun Apr 05 15:10:53
>  2009
> ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY SID='ifxdb2';
> These incidences are all occurring during archiving (redo logs and 
> database files are using ASM, archiving and backups are on OCFS2). 
> Even though it usually only lasts a few minutes, it's very noticeable 
> to the customers because of all the sessions that are terminated.
>
> What should we be looking at to resolve this problem?
> Please let me know if you have any questions.
>
> Thanks,
> Diane Petersen
> ServerCare, Inc.
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/c64879ea/attachment.html 