<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
<title></title>
</head>
<body bgcolor="#ffffff" text="#000000">
do you notice any errors in messages files during the same time as the
error reported in alert.log?. <br>
<br>
please forward the output of echo "stat
/arch/2_1917_656008464.dbf"|debugfs.ocfs2 <device><br>
<br>
Can you please file a tar with Oracle and forward me the tar#.<br>
<br>
thanks,<br>
--Srini<br>
<br>
Diane Petersen wrote:
<blockquote cite="mid762813.83962.qm@web54406.mail.yahoo.com"
type="cite">
<style type="text/css"><!-- DIV {margin:0px;} --></style>
<div
style="font-family: times new roman,new york,times,serif; font-size: 12pt;">
<div>Hi,<br>
<br>
We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux
kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on
both nodes almost ever other day. The system administrator has checked
the SAN array and said there are no issues being reported. <br>
<br>
Another part of the problem, it appears the instances alter the
service_names parameter not allowing new connections to the node with
the reported error, but also terminate sessions already connected using
the RAC service. The errors all start with - Encountered disk I/O error
19502 - and contain the following:<br>
<br>
<pre>ARC2: Encountered disk I/O error 19502
(ifxdb2)
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file
"/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 155649
Additional information: -1
ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)
ARC2: I/O error 19502 archiving log 10 to '/u03/arch/2_1917_656008464.dbf'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE
Instance ifxdb2 - Archival Error
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16038: log 10 sequence# 1917 cannot be archived
ORA-19502: write error on file "", block number (block size=)
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:05:16 2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY SID='ifxdb2';
Immediate Kill Session#: 185, Serial#: 40263
Immediate Kill Session: sess: 0x1274fabc8 OS pid: 13270
Immediate Kill Session#: 187, Serial#: 41391
Immediate Kill Session: sess: 0x1274fd710 OS pid: 27697
Immediate Kill Session#: 191, Serial#: 40464
Immediate Kill Session: sess: 0x127502da0 OS pid:
30697
Immediate Kill Session#: 195, Serial#: 57362
Immediate Kill Session: sess: 0x127508430 OS pid: 27967
Immediate Kill Session#: 196, Serial#: 2028
Immediate Kill Session: sess: 0x124544048 OS pid: 22900
Immediate Kill Session#: 205, Serial#: 17412
Immediate Kill Session: sess: 0x127515c98 OS pid: 20110
Immediate Kill Session#: 206, Serial#: 14805
Immediate Kill Session: sess: 0x1245518b0 OS pid: 10464
Immediate Kill Session#: 207, Serial#: 52184
Immediate Kill Session: sess: 0x1275187e0 OS pid: 19787
Immediate Kill Session#: 208, Serial#: 62825
Immediate Kill Session: sess: 0x1245543f8 OS pid: 13578
Immediate Kill Session#: 213, Serial#: 36907
Immediate Kill Session: sess: 0x1275209b8 OS pid: 31397
Immediate Kill Session#: 214, Serial#: 49032
Immediate Kill Session: sess: 0x12455c5d0 OS pid: 2427
Immediate Kill Session#: 215, Serial#: 2711
Immediate Kill Session: sess: 0x127523500 OS
pid: 15942
Immediate Kill Session#: 216, Serial#: 30060
Immediate Kill Session: sess: 0x12455f118 OS pid: 1217
Immediate Kill Session#: 219, Serial#: 35932
Immediate Kill Session: sess: 0x127528b90 OS pid: 27883
Immediate Kill Session#: 222, Serial#: 26007
Immediate Kill Session: sess: 0x1245672f0 OS pid: 1036
Immediate Kill Session#: 223, Serial#: 42462
Immediate Kill Session: sess: 0x12752e220 OS pid: 13726
Immediate Kill Session#: 224, Serial#: 33323
Immediate Kill Session: sess: 0x124569e38 OS pid: 29928
Immediate Kill Session#: 225, Serial#: 49752
Immediate Kill Session: sess: 0x127530d68 OS pid: 20147
Immediate Kill Session#: 227, Serial#: 34834
Immediate Kill Session: sess: 0x1275338b0 OS pid: 9365
Immediate Kill Session#: 230, Serial#: 19879
Immediate Kill Session: sess: 0x124572010 OS pid: 15791
Immediate Kill Session#: 231, Serial#: 16554
Immediate Kill Session: sess: 0x127538f40
OS pid: 15490
Immediate Kill Session#: 233, Serial#: 25251
Immediate Kill Session: sess: 0x12753ba88 OS pid: 6972
Immediate Kill Session#: 236, Serial#: 36970
Immediate Kill Session: sess: 0x12457a1e8 OS pid: 12354
Immediate Kill Session#: 244, Serial#: 37284
Immediate Kill Session: sess: 0x124584f08 OS pid: 19290
Immediate Kill Session#: 245, Serial#: 55792
Immediate Kill Session: sess: 0x12754be38 OS pid: 19288
Immediate Kill Session#: 246, Serial#: 25115
Immediate Kill Session: sess: 0x124587a50 OS pid: 3111
Immediate Kill Session#: 247, Serial#: 6416
Immediate Kill Session: sess: 0x12754e980 OS pid: 19471
Immediate Kill Session#: 251, Serial#: 19899
Immediate Kill Session: sess: 0x127554010 OS pid: 21486
Immediate Kill Session#: 252, Serial#: 34731
Immediate Kill Session: sess: 0x12458fc28 OS pid: 30540
Immediate Kill Session#: 253, Serial#: 32638
Immediate Kill Session: sess: 0x127556b58
OS pid: 5493
Immediate Kill Session#: 259, Serial#: 29155
Immediate Kill Session: sess: 0x12755ed30 OS pid: 29463
Immediate Kill Session#: 261, Serial#: 14481
Immediate Kill Session: sess: 0x127561878 OS pid: 31054
Immediate Kill Session#: 265, Serial#: 37618
Immediate Kill Session: sess: 0x127566f08 OS pid: 868
Immediate Kill Session#: 267, Serial#: 42580
Immediate Kill Session: sess: 0x127569a50 OS pid: 16839
Immediate Kill Session#: 268, Serial#: 50893
Immediate Kill Session: sess: 0x1245a5668 OS pid: 27778
Immediate Kill Session#: 274, Serial#: 34459
Immediate Kill Session: sess: 0x1245ad840 OS pid: 9808
Immediate Kill Session#: 278, Serial#: 59445
Immediate Kill Session: sess: 0x1245b2ed0 OS pid: 28434
Immediate Kill Session#: 281, Serial#: 50119
Immediate Kill Session: sess: 0x12757c948 OS pid: 12606
Immediate Kill Session#: 282, Serial#: 30208
Immediate Kill Session: sess: 0x1245b8560
OS pid: 17944
Immediate Kill Session#: 285, Serial#: 53580
Immediate Kill Session: sess: 0x127581fd8 OS pid: 16670
Immediate Kill Session#: 286, Serial#: 5929
Immediate Kill Session: sess: 0x1245bdbf0 OS pid: 20149
Immediate Kill Session#: 289, Serial#: 53725
Immediate Kill Session: sess: 0x127587668 OS pid: 14697
Immediate Kill Session#: 290, Serial#: 30378
Immediate Kill Session: sess: 0x1245c3280 OS pid: 19757
Immediate Kill Session#: 293, Serial#: 53710
Immediate Kill Session: sess: 0x12758ccf8 OS pid: 11096
Immediate Kill Session#: 296, Serial#: 34022
Immediate Kill Session: sess: 0x1245cb458 OS pid: 10881
Immediate Kill Session#: 299, Serial#: 53951
Immediate Kill Session: sess: 0x127594ed0 OS pid: 1453
Immediate Kill Session#: 304, Serial#: 15149
Immediate Kill Session: sess: 0x1245d6178 OS pid: 22008
Immediate Kill Session#: 308, Serial#: 34245
Immediate Kill Session: sess:
0x1245db808 OS pid: 19156
Immediate Kill Session#: 315, Serial#: 15240
Immediate Kill Session: sess: 0x1275aa910 OS pid: 32148
Immediate Kill Session#: 317, Serial#: 41792
Immediate Kill Session: sess: 0x1275ad458 OS pid: 15660
Immediate Kill Session#: 318, Serial#: 7839
Immediate Kill Session: sess: 0x1245e9070 OS pid: 24999
Immediate Kill Session#: 321, Serial#: 4422
Immediate Kill Session: sess: 0x1275b2ae8 OS pid: 16028
Immediate Kill Session#: 324, Serial#: 6833
Immediate Kill Session: sess: 0x1245f1248 OS pid: 21909
Immediate Kill Session#: 332, Serial#: 18018
Immediate Kill Session: sess: 0x1245fbf68 OS pid: 15819
Immediate Kill Session#: 333, Serial#: 37534
Immediate Kill Session: sess: 0x1275c2e98 OS pid: 16433
Immediate Kill Session#: 334, Serial#: 50463
Immediate Kill Session: sess: 0x1245feab0 OS pid: 5660
Immediate Kill Session#: 335, Serial#: 11994
Immediate Kill Session: sess:
0x1275c59e0 OS pid: 29575
Immediate Kill Session#: 336, Serial#: 26542
Immediate Kill Session: sess: 0x1246015f8 OS pid: 31868
Immediate Kill Session#: 345, Serial#: 46583
Immediate Kill Session: sess: 0x1275d3248 OS pid: 25399
Sun Apr 05 15:05:43 2009
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance ifxdb2 - Archival Error
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Errors in file /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:
ORA-16014: log 10 sequence# 1917 not archived, no available destinations
ORA-00312: online log 10 thread 2: '+REDO1/ifxdb/onlinelog/group_10.265.656605479'
Sun Apr 05 15:10:52 2009
kcrrdmx: Successful archiving of previously failed ORL
Archiver process freed from errors. No longer stopped
Sun Apr 05 15:10:53
2009
ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY SID='ifxdb2';</pre>
These incidences are all occurring during archiving (redo logs and
database files are using ASM, archiving and backups are on OCFS2). Even
though it usually only lasts a few minutes, it's very noticeable to the
customers because of all the sessions that are terminated. <br>
<br>
What should we be looking at to resolve this problem?<br>
Please let me know if you have any questions.<br>
<br>
Thanks,<br>
Diane Petersen<br>
ServerCare, Inc.<br>
<br>
</div>
</div>
<br>
<pre wrap="">
<hr size="4" width="90%">
_______________________________________________
Ocfs2-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a>
<a class="moz-txt-link-freetext" href="http://oss.oracle.com/mailman/listinfo/ocfs2-users">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a></pre>
</blockquote>
</body>
</html>