<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman,new york,times,serif;font-size:12pt"><div>Hi,<br><br>We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on both nodes almost ever other day. The system administrator has checked the SAN array and said there are no issues being reported. <br><br>Another part of the problem, it appears the instances alter the service_names parameter not allowing new connections to the node with the reported error, but also terminate sessions already connected using the RAC service. The errors all start with - Encountered disk I/O error 19502 - and contain the following:<br><br><pre>ARC2: Encountered disk I/O error 19502<br> (ifxdb2)<br>Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>ORA-19502: write error on file
"/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)<br>ORA-27072: File I/O error<br>Linux-x86_64 Error: 5: Input/output error<br>Additional information: 4<br>Additional information: 155649<br>Additional information: -1<br>ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)<br>Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)<br>ORA-27072: File I/O error<br>Linux-x86_64 Error: 5: Input/output error<br>Additional information: 4<br>Additional information: 155649<br>Additional information: -1<br>ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)<br>ARC2: I/O error 19502 archiving log 10 to '/u03/arch/2_1917_656008464.dbf'<br>ARCH: Archival stopped, error occurred. Will continue retrying<br>ORACLE
Instance ifxdb2 - Archival Error<br>ORA-16038: log 10 sequence# 1917 cannot be archived<br>ORA-19502: write error on file "", block number (block size=)<br>ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>ORA-16038: log 10 sequence# 1917 cannot be archived<br>ORA-19502: write error on file "", block number (block size=)<br>ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>Sun Apr 05 15:05:16 2009<br>ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY SID='ifxdb2';<br>Immediate Kill Session#: 185, Serial#: 40263<br>Immediate Kill Session: sess: 0x1274fabc8 OS pid: 13270<br>Immediate Kill Session#: 187, Serial#: 41391<br>Immediate Kill Session: sess: 0x1274fd710 OS pid: 27697<br>Immediate Kill Session#: 191, Serial#: 40464<br>Immediate Kill Session: sess: 0x127502da0 OS pid:
30697<br>Immediate Kill Session#: 195, Serial#: 57362<br>Immediate Kill Session: sess: 0x127508430 OS pid: 27967<br>Immediate Kill Session#: 196, Serial#: 2028<br>Immediate Kill Session: sess: 0x124544048 OS pid: 22900<br>Immediate Kill Session#: 205, Serial#: 17412<br>Immediate Kill Session: sess: 0x127515c98 OS pid: 20110<br>Immediate Kill Session#: 206, Serial#: 14805<br>Immediate Kill Session: sess: 0x1245518b0 OS pid: 10464<br>Immediate Kill Session#: 207, Serial#: 52184<br>Immediate Kill Session: sess: 0x1275187e0 OS pid: 19787<br>Immediate Kill Session#: 208, Serial#: 62825<br>Immediate Kill Session: sess: 0x1245543f8 OS pid: 13578<br>Immediate Kill Session#: 213, Serial#: 36907<br>Immediate Kill Session: sess: 0x1275209b8 OS pid: 31397<br>Immediate Kill Session#: 214, Serial#: 49032<br>Immediate Kill Session: sess: 0x12455c5d0 OS pid: 2427<br>Immediate Kill Session#: 215, Serial#: 2711<br>Immediate Kill Session: sess: 0x127523500 OS
pid: 15942<br>Immediate Kill Session#: 216, Serial#: 30060<br>Immediate Kill Session: sess: 0x12455f118 OS pid: 1217<br>Immediate Kill Session#: 219, Serial#: 35932<br>Immediate Kill Session: sess: 0x127528b90 OS pid: 27883<br>Immediate Kill Session#: 222, Serial#: 26007<br>Immediate Kill Session: sess: 0x1245672f0 OS pid: 1036<br>Immediate Kill Session#: 223, Serial#: 42462<br>Immediate Kill Session: sess: 0x12752e220 OS pid: 13726<br>Immediate Kill Session#: 224, Serial#: 33323<br>Immediate Kill Session: sess: 0x124569e38 OS pid: 29928<br>Immediate Kill Session#: 225, Serial#: 49752<br>Immediate Kill Session: sess: 0x127530d68 OS pid: 20147<br>Immediate Kill Session#: 227, Serial#: 34834<br>Immediate Kill Session: sess: 0x1275338b0 OS pid: 9365<br>Immediate Kill Session#: 230, Serial#: 19879<br>Immediate Kill Session: sess: 0x124572010 OS pid: 15791<br>Immediate Kill Session#: 231, Serial#: 16554<br>Immediate Kill Session: sess: 0x127538f40
OS pid: 15490<br>Immediate Kill Session#: 233, Serial#: 25251<br>Immediate Kill Session: sess: 0x12753ba88 OS pid: 6972<br>Immediate Kill Session#: 236, Serial#: 36970<br>Immediate Kill Session: sess: 0x12457a1e8 OS pid: 12354<br>Immediate Kill Session#: 244, Serial#: 37284<br>Immediate Kill Session: sess: 0x124584f08 OS pid: 19290<br>Immediate Kill Session#: 245, Serial#: 55792<br>Immediate Kill Session: sess: 0x12754be38 OS pid: 19288<br>Immediate Kill Session#: 246, Serial#: 25115<br>Immediate Kill Session: sess: 0x124587a50 OS pid: 3111<br>Immediate Kill Session#: 247, Serial#: 6416<br>Immediate Kill Session: sess: 0x12754e980 OS pid: 19471<br>Immediate Kill Session#: 251, Serial#: 19899<br>Immediate Kill Session: sess: 0x127554010 OS pid: 21486<br>Immediate Kill Session#: 252, Serial#: 34731<br>Immediate Kill Session: sess: 0x12458fc28 OS pid: 30540<br>Immediate Kill Session#: 253, Serial#: 32638<br>Immediate Kill Session: sess: 0x127556b58
OS pid: 5493<br>Immediate Kill Session#: 259, Serial#: 29155<br>Immediate Kill Session: sess: 0x12755ed30 OS pid: 29463<br>Immediate Kill Session#: 261, Serial#: 14481<br>Immediate Kill Session: sess: 0x127561878 OS pid: 31054<br>Immediate Kill Session#: 265, Serial#: 37618<br>Immediate Kill Session: sess: 0x127566f08 OS pid: 868<br>Immediate Kill Session#: 267, Serial#: 42580<br>Immediate Kill Session: sess: 0x127569a50 OS pid: 16839<br>Immediate Kill Session#: 268, Serial#: 50893<br>Immediate Kill Session: sess: 0x1245a5668 OS pid: 27778<br>Immediate Kill Session#: 274, Serial#: 34459<br>Immediate Kill Session: sess: 0x1245ad840 OS pid: 9808<br>Immediate Kill Session#: 278, Serial#: 59445<br>Immediate Kill Session: sess: 0x1245b2ed0 OS pid: 28434<br>Immediate Kill Session#: 281, Serial#: 50119<br>Immediate Kill Session: sess: 0x12757c948 OS pid: 12606<br>Immediate Kill Session#: 282, Serial#: 30208<br>Immediate Kill Session: sess: 0x1245b8560
OS pid: 17944<br>Immediate Kill Session#: 285, Serial#: 53580<br>Immediate Kill Session: sess: 0x127581fd8 OS pid: 16670<br>Immediate Kill Session#: 286, Serial#: 5929<br>Immediate Kill Session: sess: 0x1245bdbf0 OS pid: 20149<br>Immediate Kill Session#: 289, Serial#: 53725<br>Immediate Kill Session: sess: 0x127587668 OS pid: 14697<br>Immediate Kill Session#: 290, Serial#: 30378<br>Immediate Kill Session: sess: 0x1245c3280 OS pid: 19757<br>Immediate Kill Session#: 293, Serial#: 53710<br>Immediate Kill Session: sess: 0x12758ccf8 OS pid: 11096<br>Immediate Kill Session#: 296, Serial#: 34022<br>Immediate Kill Session: sess: 0x1245cb458 OS pid: 10881<br>Immediate Kill Session#: 299, Serial#: 53951<br>Immediate Kill Session: sess: 0x127594ed0 OS pid: 1453<br>Immediate Kill Session#: 304, Serial#: 15149<br>Immediate Kill Session: sess: 0x1245d6178 OS pid: 22008<br>Immediate Kill Session#: 308, Serial#: 34245<br>Immediate Kill Session: sess:
0x1245db808 OS pid: 19156<br>Immediate Kill Session#: 315, Serial#: 15240<br>Immediate Kill Session: sess: 0x1275aa910 OS pid: 32148<br>Immediate Kill Session#: 317, Serial#: 41792<br>Immediate Kill Session: sess: 0x1275ad458 OS pid: 15660<br>Immediate Kill Session#: 318, Serial#: 7839<br>Immediate Kill Session: sess: 0x1245e9070 OS pid: 24999<br>Immediate Kill Session#: 321, Serial#: 4422<br>Immediate Kill Session: sess: 0x1275b2ae8 OS pid: 16028<br>Immediate Kill Session#: 324, Serial#: 6833<br>Immediate Kill Session: sess: 0x1245f1248 OS pid: 21909<br>Immediate Kill Session#: 332, Serial#: 18018<br>Immediate Kill Session: sess: 0x1245fbf68 OS pid: 15819<br>Immediate Kill Session#: 333, Serial#: 37534<br>Immediate Kill Session: sess: 0x1275c2e98 OS pid: 16433<br>Immediate Kill Session#: 334, Serial#: 50463<br>Immediate Kill Session: sess: 0x1245feab0 OS pid: 5660<br>Immediate Kill Session#: 335, Serial#: 11994<br>Immediate Kill Session: sess:
0x1275c59e0 OS pid: 29575<br>Immediate Kill Session#: 336, Serial#: 26542<br>Immediate Kill Session: sess: 0x1246015f8 OS pid: 31868<br>Immediate Kill Session#: 345, Serial#: 46583<br>Immediate Kill Session: sess: 0x1275d3248 OS pid: 25399<br>Sun Apr 05 15:05:43 2009<br>ARCH: Archival stopped, error occurred. Will continue retrying<br>ORACLE Instance ifxdb2 - Archival Error<br>ORA-16014: log 10 sequence# 1917 not archived, no available destinations<br>ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>ORA-16014: log 10 sequence# 1917 not archived, no available destinations<br>ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>Sun Apr 05 15:10:52 2009<br>kcrrdmx: Successful archiving of previously failed ORL<br>Archiver process freed from errors. No longer stopped<br>Sun Apr 05 15:10:53
2009<br>ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY SID='ifxdb2';</pre>These incidences are all occurring during archiving (redo logs and database files are using ASM, archiving and backups are on OCFS2). Even though it usually only lasts a few minutes, it's very noticeable to the customers because of all the sessions that are terminated. <br><br>What should we be looking at to resolve this problem?<br>Please let me know if you have any questions.<br><br>Thanks,<br>Diane Petersen<br>ServerCare, Inc.<br><br></div></div><br>
</body></html>