<html><head><style type="text/css"><!-- DIV {margin:0px;} --></style></head><body><div style="font-family:times new roman,new york,times,serif;font-size:12pt"><div>Hi Luis,<br><br>We are using ASM diskgroups +DATA1 and +REDO1 for datafiles and redo logs respectively. We have two separate OCFS2 partitions, /u02 is for RMAN backups and /u03 is for the archive logs for both nodes. I think what you're referring to are the redo logs in ASM which eaxh instance is attempting to write out to the OCFS2 partition during the archive process. Here's a copy of the /etc/fstab:<br><br>LABEL=/ / ext3 defaults 1
1<br>tmpfs /dev/shm tmpfs defaults 0 0<br>devpts /dev/pts devpts gid=5,mode=620 0 0<br>sysfs /sys
sysfs defaults 0 0<br>proc /proc proc defaults 0 0<br>LABEL=SWAP-sda5 swap swap defaults 0 0<br>/dev/mapper/disk1p1 /u02 ocfs2
_netdev,datavolume,nointr 0 0<br>/dev/mapper/disk4p1 /u03 ocfs2 _netdev,datavolume,nointr 0 0<br><br></div><div style="font-family: times new roman,new york,times,serif; font-size: 12pt;">I believe this is a problem writing to the OCFS2 partition not reading from ASM, but I don't know what's causing it.<br><br>Thanks,<br>Diane Petersen<br>ServerCare, Inc.<br><div style="font-family: arial,helvetica,sans-serif; font-size: 10pt;"><font size="2" face="Tahoma"><hr size="1"><b><span style="font-weight: bold;">From:</span></b> Luis Freitas <lfreitas34@yahoo.com><br><b><span style="font-weight: bold;">To:</span></b> ocfs2-users@oss.oracle.com; Diane Petersen <diane_petersen@yahoo.com><br><b><span style="font-weight:
bold;">Sent:</span></b> Monday, April 6, 2009 10:44:52 AM<br><b><span style="font-weight: bold;">Subject:</span></b> Re: [Ocfs2-users] Encountered disk I/O error 19502<br></font><br>
<br>Diane,<br><br> Are you using ASM and OCFS2? Some of the log messages point to a disk group.<br><br> Can you post a copy of your /etc/fstab with the mount options?<br><br>Regards,<br>Luis<br><br><br>--- On Mon, 4/6/09, Diane Petersen <<a ymailto="mailto:diane_petersen@yahoo.com" href="mailto:diane_petersen@yahoo.com">diane_petersen@yahoo.com</a>> wrote:<br><br>> From: Diane Petersen <<a ymailto="mailto:diane_petersen@yahoo.com" href="mailto:diane_petersen@yahoo.com">diane_petersen@yahoo.com</a>><br>> Subject: Re: [Ocfs2-users] Encountered disk I/O error 19502<br>> To: "Karim Alkhayer" <<a ymailto="mailto:kkhayer@gmail.com" href="mailto:kkhayer@gmail.com">kkhayer@gmail.com</a>>, <a ymailto="mailto:ocfs2-users@oss.oracle.com" href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br>> Date: Monday, April 6, 2009, 1:42 PM<br>> Hi,<br>> <br>> We already have TAF implemented,
unfortunately that<br>> doesn't help. It suppose TAF might help if the instance<br>> was terminated, but that's not happening instead it<br>> terminates these individual sessions directly.<br>> <br>> This happens on both nodes during writes to the OCFS2<br>> partition at random times but never at the same time. There<br>> is nothing else in the db alert log or crs logs other than<br>> what I've included below.<br>> <br>> Thanks,<br>> Diane Petersen<br>> ServerCare, Inc.<br>> <br>> <br>> <br>> <br>> ________________________________<br>> From: Karim Alkhayer <<a ymailto="mailto:kkhayer@gmail.com" href="mailto:kkhayer@gmail.com">kkhayer@gmail.com</a>><br>> To: Diane Petersen <<a ymailto="mailto:diane_petersen@yahoo.com" href="mailto:diane_petersen@yahoo.com">diane_petersen@yahoo.com</a>>;<br>> <a ymailto="mailto:ocfs2-users@oss.oracle.com"
href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br>> Sent: Monday, April 6, 2009 9:11:06 AM<br>> Subject: RE: [Ocfs2-users] Encountered disk I/O error 19502<br>> <br>> <br>> Hello Diane,<br>> <br>> I believe that implementing TAF could help a bit in this<br>> case, at<br>> least to become transparent to the end users, unless of<br>> course, the following<br>> points are blocking in your case:<br>> <br>> 1. ALTER SESSION statements are lost: <br>> Statements such as "ALTER<br>> SESSION ..." are not automatically re-issued to the<br>> server following a<br>> failover. This can have a significant effect on application<br>> behavior. For<br>> example: <br>> ALTER SESSION<br>> SET NLS_DATE_FORMAT='YYYY-MM-DD';<br>> select sysdate<br>> from dual;<br>> Result><br>> 2009-01-31<br>> << Fail<br>> over the
connection >><br>> select sysdate<br>> from dual;<br>> Result><br>> 31-JAN-09<br>> 2. In-progress transactions must be rolled back <br>> 3. Continuing work on existing cursors may raise an<br>> error (eg:<br>> ORA-25401 "cannot continue fetches") <br>> 4. Failed over selects may take time to re-position<br>> (when FAILOVER_TYPE=SELECT) <br>> 5. Client awareness of a Failover<br>> <br>> Can we have an overview of the database setup, nature of<br>> transactions, and parameters?<br>> <br>> It would also help to examine the troublesome node behavior<br>> and<br>> recovery measures.<br>> <br>> Best regards,<br>> Karim Alkhayer<br>> <br>> From:<a ymailto="mailto:ocfs2-users-bounces@oss.oracle.com"
href="mailto:ocfs2-users-bounces@oss.oracle.com">ocfs2-users-bounces@oss.oracle.com</a><br>> [mailto:<a ymailto="mailto:ocfs2-users-bounces@oss.oracle.com" href="mailto:ocfs2-users-bounces@oss.oracle.com">ocfs2-users-bounces@oss.oracle.com</a>] On<br>> Behalf Of Diane Petersen<br>> Sent: Monday, April 06, 2009 4:06 PM<br>> To: <a ymailto="mailto:ocfs2-users@oss.oracle.com" href="mailto:ocfs2-users@oss.oracle.com">ocfs2-users@oss.oracle.com</a><br>> Subject: [Ocfs2-users] Encountered disk I/O error 19502<br>> <br>> Hi,<br>> <br>> We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5<br>> with Linux kernel<br>> 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing<br>> errors on both nodes almost<br>> ever other day. The system administrator has checked the<br>> SAN array and said<br>> there are no issues being reported. <br>> <br>> Another part of the problem, it appears the instances
alter<br>> the service_names<br>> parameter not allowing new connections to the node with the<br>> reported error, but<br>> also terminate sessions already connected using the RAC<br>> service. The errors all<br>> start with - Encountered disk I/O error 19502 - and contain<br>> the following:<br>> ARC2: Encountered disk I/O error 19502<br>> <br>> <br>> (ifxdb2)<br>> <br>> <br>> Errors in file<br>> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>> <br>> <br>> ORA-19502: write error on file<br>> "/u03/arch/2_1917_656008464.dbf", block number<br>> 155649 (block size=512)<br>> <br>> <br>> ORA-27072: File I/O error<br>> <br>> <br>> Linux-x86_64 Error: 5: Input/output error<br>> <br>> <br>> Additional information: 4<br>> <br>> <br>> Additional information: 155649<br>> <br>> <br>> Additional information: -1<br>>
<br>> <br>> ORA-19502: write error on file<br>> "/u03/arch/2_1917_656008464.dbf", block number<br>> 155649 (block size=512)<br>> <br>> <br>> Errors in file<br>> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>> <br>> <br>> ORA-19502: write error on file<br>> "/u03/arch/2_1917_656008464.dbf", block number<br>> 155649 (block size=512)<br>> <br>> <br>> ORA-27072: File I/O error<br>> <br>> <br>> Linux-x86_64 Error: 5: Input/output error<br>> <br>> <br>> Additional information: 4<br>> <br>> <br>> Additional information: 155649<br>> <br>> <br>> Additional information: -1<br>> <br>> <br>> ORA-19502: write error on file<br>> "/u03/arch/2_1917_656008464.dbf", block number<br>> 155649 (block size=512)<br>> <br>> <br>> ARC2: I/O error 19502 archiving log 10 to<br>> '/u03/arch/2_1917_656008464.dbf'<br>> <br>> <br>> ARCH:
Archival stopped, error occurred. Will continue<br>> retrying<br>> <br>> <br>> ORACLE<br>> Instance ifxdb2 - Archival Error<br>> <br>> <br>> ORA-16038: log 10 sequence# 1917 cannot be archived<br>> <br>> <br>> ORA-19502: write error on file "", block number <br>> (block size=)<br>> <br>> <br>> ORA-00312: online log 10 thread 2:<br>> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>> <br>> <br>> Errors in file<br>> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>> <br>> <br>> ORA-16038: log 10 sequence# 1917 cannot be archived<br>> <br>> <br>> ORA-19502: write error on file "", block number <br>> (block size=)<br>> <br>> <br>> ORA-00312: online log 10 thread 2:<br>> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>> <br>> <br>> Sun Apr 05 15:05:16 2009<br>> <br>> <br>> ALTER SYSTEM SET<br>> service_names='<a
target="_blank" href="http://ifxdb.gointranet.com">ifxdb.gointranet.com</a>' SCOPE=MEMORY<br>> SID='ifxdb2';<br>> <br>> <br>> Immediate Kill Session#: 185, Serial#: 40263<br>> <br>> <br>> Immediate Kill Session: sess: 0x1274fabc8 OS pid: 13270<br>> <br>> <br>> Immediate Kill Session#: 187, Serial#: 41391<br>> <br>> <br>> Immediate Kill Session: sess: 0x1274fd710 OS pid: 27697<br>> <br>> <br>> Immediate Kill Session#: 191, Serial#: 40464<br>> <br>> <br>> Immediate Kill Session: sess: 0x127502da0 OS pid:<br>> 30697<br>> <br>> <br>> Immediate Kill Session#: 195, Serial#: 57362<br>> <br>> <br>> Immediate Kill Session: sess: 0x127508430 OS pid: 27967<br>> <br>> <br>> Immediate Kill Session#: 196, Serial#: 2028<br>> <br>> <br>> Immediate Kill Session: sess: 0x124544048 OS pid: 22900<br>> <br>> <br>> Immediate Kill
Session#: 205, Serial#: 17412<br>> <br>> <br>> Immediate Kill Session: sess: 0x127515c98 OS pid: 20110<br>> <br>> <br>> Immediate Kill Session#: 206, Serial#: 14805<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245518b0 OS pid: 10464<br>> <br>> <br>> Immediate Kill Session#: 207, Serial#: 52184<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275187e0 OS pid: 19787<br>> <br>> <br>> Immediate Kill Session#: 208, Serial#: 62825<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245543f8 OS pid: 13578<br>> <br>> <br>> Immediate Kill Session#: 213, Serial#: 36907<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275209b8 OS pid: 31397<br>> <br>> <br>> Immediate Kill Session#: 214, Serial#: 49032<br>> <br>> <br>> Immediate Kill Session: sess: 0x12455c5d0 OS pid: 2427<br>> <br>> <br>> Immediate Kill Session#: 215,
Serial#: 2711<br>> <br>> <br>> Immediate Kill Session: sess: 0x127523500 OS<br>> pid: 15942<br>> <br>> <br>> Immediate Kill Session#: 216, Serial#: 30060<br>> <br>> <br>> Immediate Kill Session: sess: 0x12455f118 OS pid: 1217<br>> <br>> <br>> Immediate Kill Session#: 219, Serial#: 35932<br>> <br>> <br>> Immediate Kill Session: sess: 0x127528b90 OS pid: 27883<br>> <br>> <br>> Immediate Kill Session#: 222, Serial#: 26007<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245672f0 OS pid: 1036<br>> <br>> <br>> Immediate Kill Session#: 223, Serial#: 42462<br>> <br>> <br>> Immediate Kill Session: sess: 0x12752e220 OS pid: 13726<br>> <br>> <br>> Immediate Kill Session#: 224, Serial#: 33323<br>> <br>> <br>> Immediate Kill Session: sess: 0x124569e38 OS pid: 29928<br>> <br>> <br>> Immediate Kill Session#: 225,
Serial#: 49752<br>> <br>> <br>> Immediate Kill Session: sess: 0x127530d68 OS pid: 20147<br>> <br>> <br>> Immediate Kill Session#: 227, Serial#: 34834<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275338b0 OS pid: 9365<br>> <br>> <br>> Immediate Kill Session#: 230, Serial#: 19879<br>> <br>> <br>> Immediate Kill Session: sess: 0x124572010 OS pid: 15791<br>> <br>> <br>> Immediate Kill Session#: 231, Serial#: 16554<br>> <br>> <br>> Immediate Kill Session: sess: 0x127538f40 <br>> OS pid: 15490<br>> <br>> <br>> Immediate Kill Session#: 233, Serial#: 25251<br>> <br>> <br>> Immediate Kill Session: sess: 0x12753ba88 OS pid: 6972<br>> <br>> <br>> Immediate Kill Session#: 236, Serial#: 36970<br>> <br>> <br>> Immediate Kill Session: sess: 0x12457a1e8 OS pid: 12354<br>> <br>> <br>> Immediate Kill Session#: 244,
Serial#: 37284<br>> <br>> <br>> Immediate Kill Session: sess: 0x124584f08 OS pid: 19290<br>> <br>> <br>> Immediate Kill Session#: 245, Serial#: 55792<br>> <br>> <br>> Immediate Kill Session: sess: 0x12754be38 OS pid: 19288<br>> <br>> <br>> Immediate Kill Session#: 246, Serial#: 25115<br>> <br>> <br>> Immediate Kill Session: sess: 0x124587a50 OS pid: 3111<br>> <br>> <br>> Immediate Kill Session#: 247, Serial#: 6416<br>> <br>> <br>> Immediate Kill Session: sess: 0x12754e980 OS pid: 19471<br>> <br>> <br>> Immediate Kill Session#: 251, Serial#: 19899<br>> <br>> <br>> Immediate Kill Session: sess: 0x127554010 OS pid: 21486<br>> <br>> <br>> Immediate Kill Session#: 252, Serial#: 34731<br>> <br>> <br>> Immediate Kill Session: sess: 0x12458fc28 OS pid: 30540<br>> <br>> <br>> Immediate Kill Session#: 253, Serial#:
32638<br>> <br>> <br>> Immediate Kill Session: sess: 0x127556b58<br>> OS pid: 5493<br>> <br>> <br>> Immediate Kill Session#: 259, Serial#: 29155<br>> <br>> <br>> Immediate Kill Session: sess: 0x12755ed30 OS pid: 29463<br>> <br>> <br>> Immediate Kill Session#: 261, Serial#: 14481<br>> <br>> <br>> Immediate Kill Session: sess: 0x127561878 OS pid: 31054<br>> <br>> <br>> Immediate Kill Session#: 265, Serial#: 37618<br>> <br>> <br>> Immediate Kill Session: sess: 0x127566f08 OS pid: 868<br>> <br>> <br>> Immediate Kill Session#: 267, Serial#: 42580<br>> <br>> <br>> Immediate Kill Session: sess: 0x127569a50 OS pid: 16839<br>> <br>> <br>> Immediate Kill Session#: 268, Serial#: 50893<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245a5668 OS pid: 27778<br>> <br>> <br>> Immediate Kill Session#: 274, Serial#:
34459<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245ad840 OS pid: 9808<br>> <br>> <br>> Immediate Kill Session#: 278, Serial#: 59445<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245b2ed0 OS pid: 28434<br>> <br>> <br>> Immediate Kill Session#: 281, Serial#: 50119<br>> <br>> <br>> Immediate Kill Session: sess: 0x12757c948 OS pid: 12606<br>> <br>> <br>> Immediate Kill Session#: 282, Serial#: 30208<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245b8560<br>> OS pid: 17944<br>> <br>> <br>> Immediate Kill Session#: 285, Serial#: 53580<br>> <br>> <br>> Immediate Kill Session: sess: 0x127581fd8 OS pid: 16670<br>> <br>> <br>> Immediate Kill Session#: 286, Serial#: 5929<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245bdbf0 OS pid: 20149<br>> <br>> <br>> Immediate Kill Session#: 289, Serial#:
53725<br>> <br>> <br>> Immediate Kill Session: sess: 0x127587668 OS pid: 14697<br>> <br>> <br>> Immediate Kill Session#: 290, Serial#: 30378<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245c3280 OS pid: 19757<br>> <br>> <br>> Immediate Kill Session#: 293, Serial#: 53710<br>> <br>> <br>> Immediate Kill Session: sess: 0x12758ccf8 OS pid: 11096<br>> <br>> <br>> Immediate Kill Session#: 296, Serial#: 34022<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245cb458 OS pid: 10881<br>> <br>> <br>> Immediate Kill Session#: 299, Serial#: 53951<br>> <br>> <br>> Immediate Kill Session: sess: 0x127594ed0 OS pid: 1453<br>> <br>> <br>> Immediate Kill Session#: 304, Serial#: 15149<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245d6178 OS pid: 22008<br>> <br>> <br>> Immediate Kill Session#: 308, Serial#: 34245<br>>
<br>> <br>> Immediate Kill Session: sess:<br>> 0x1245db808 OS pid: 19156<br>> <br>> <br>> Immediate Kill Session#: 315, Serial#: 15240<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275aa910 OS pid: 32148<br>> <br>> <br>> Immediate Kill Session#: 317, Serial#: 41792<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275ad458 OS pid: 15660<br>> <br>> <br>> Immediate Kill Session#: 318, Serial#: 7839<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245e9070 OS pid: 24999<br>> <br>> <br>> Immediate Kill Session#: 321, Serial#: 4422<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275b2ae8 OS pid: 16028<br>> <br>> <br>> Immediate Kill Session#: 324, Serial#: 6833<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245f1248 OS pid: 21909<br>> <br>> <br>> Immediate Kill Session#: 332, Serial#: 18018<br>>
<br>> <br>> Immediate Kill Session: sess: 0x1245fbf68 OS pid: 15819<br>> <br>> <br>> Immediate Kill Session#: 333, Serial#: 37534<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275c2e98 OS pid: 16433<br>> <br>> <br>> Immediate Kill Session#: 334, Serial#: 50463<br>> <br>> <br>> Immediate Kill Session: sess: 0x1245feab0 OS pid: 5660<br>> <br>> <br>> Immediate Kill Session#: 335, Serial#: 11994<br>> <br>> <br>> Immediate Kill Session: sess:<br>> 0x1275c59e0 OS pid: 29575<br>> <br>> <br>> Immediate Kill Session#: 336, Serial#: 26542<br>> <br>> <br>> Immediate Kill Session: sess: 0x1246015f8 OS pid: 31868<br>> <br>> <br>> Immediate Kill Session#: 345, Serial#: 46583<br>> <br>> <br>> Immediate Kill Session: sess: 0x1275d3248 OS pid: 25399<br>> <br>> <br>> Sun Apr 05 15:05:43 2009<br>> <br>> <br>>
ARCH: Archival stopped, error occurred. Will continue<br>> retrying<br>> <br>> <br>> ORACLE Instance ifxdb2 - Archival Error<br>> <br>> <br>> ORA-16014: log 10 sequence# 1917 not archived, no available<br>> destinations<br>> <br>> <br>> ORA-00312: online log 10 thread 2:<br>> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>> <br>> <br>> Errors in file<br>> /u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:<br>> <br>> <br>> ORA-16014: log 10 sequence# 1917 not archived, no available<br>> destinations<br>> <br>> <br>> ORA-00312: online log 10 thread 2:<br>> '+REDO1/ifxdb/onlinelog/group_10.265.656605479'<br>> <br>> <br>> Sun Apr 05 15:10:52 2009<br>> <br>> <br>> kcrrdmx: Successful archiving of previously failed ORL<br>> <br>> <br>> Archiver process freed from errors. No longer stopped<br>> <br>> <br>> Sun Apr 05
15:10:53<br>> 2009<br>> <br>> <br>> ALTER SYSTEM SET<br>> service_names='ifxdb.gointranet.com','ifxserv'<br>> SCOPE=MEMORY SID='ifxdb2';<br>> These incidences are all<br>> occurring during archiving (redo logs and database files<br>> are using ASM,<br>> archiving and backups are on OCFS2). Even though it usually<br>> only lasts a few<br>> minutes, it's very noticeable to the customers because<br>> of all the sessions that<br>> are terminated. <br>> <br>> What should we be looking at to resolve this problem?<br>> Please let me know if you have any questions.<br>> <br>> Thanks,<br>> Diane Petersen<br>> ServerCare, Inc.<br>> <br>> <br>> _______________________________________________<br>> Ocfs2-users mailing list<br>> <a ymailto="mailto:Ocfs2-users@oss.oracle.com" href="mailto:Ocfs2-users@oss.oracle.com">Ocfs2-users@oss.oracle.com</a><br><span>> <a
target="_blank" href="http://oss.oracle.com/mailman/listinfo/ocfs2-users">http://oss.oracle.com/mailman/listinfo/ocfs2-users</a></span><br><br><br> <br></div></div></div><br>
</body></html>