[Ocfs2-users] Encountered disk I/O error 19502

Mon Apr 6 09:11:06 PDT 2009

Hello Diane,

I believe that implementing TAF could help a bit in this case, at least to
become transparent to the end users, unless of course, the following points
are blocking in your case:

1.       ALTER SESSION statements are lost:  

Statements such as "ALTER SESSION ..." are not automatically re-issued to
the server following a failover. This can have a significant effect on
application behavior. For example: 

ALTER SESSION SET NLS_DATE_FORMAT='YYYY-MM-DD';

select sysdate from dual;

Result> 2009-01-31

<< Fail over the connection >>

select sysdate from dual;

Result> 31-JAN-09

2.       In-progress transactions must be rolled back 

3.       Continuing work on existing cursors may raise an error (eg:
ORA-25401 "cannot continue fetches") 

4.       Failed over selects may take time to re-position (when
FAILOVER_TYPE=SELECT) 

5.       Client awareness of a Failover

Can we have an overview of the database setup, nature of transactions, and
parameters?

It would also help to examine the troublesome node behavior and recovery
measures.

Best regards,

Karim Alkhayer

From: ocfs2-users-bounces at oss.oracle.com
[mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of Diane Petersen
Sent: Monday, April 06, 2009 4:06 PM
To: ocfs2-users at oss.oracle.com
Subject: [Ocfs2-users] Encountered disk I/O error 19502

Hi,

We have a 2-node 11g RAC database running OCFS2 1.4.1-1.el5 with Linux
kernel 2.6.18-92.1.17.el5 64-bit. Lately we've been seeing errors on both
nodes almost ever other day. The system administrator has checked the SAN
array and said there are no issues being reported. 

Another part of the problem, it appears the instances alter the
service_names parameter not allowing new connections to the node with the
reported error, but also terminate sessions already connected using the RAC
service. The errors all start with - Encountered disk I/O error 19502 - and
contain the following:

ARC2: Encountered disk I/O error 19502

 (ifxdb2)

Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:

ORA-19502: write error on file
 "/u03/arch/2_1917_656008464.dbf", block number 155649 (block size=512)

ORA-27072: File I/O error

Linux-x86_64 Error: 5: Input/output error

Additional information: 4

Additional information: 155649

Additional information: -1

ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)

Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:

ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)

ORA-27072: File I/O error

Linux-x86_64 Error: 5: Input/output error

Additional information: 4

Additional information: 155649

Additional information: -1

ORA-19502: write error on file "/u03/arch/2_1917_656008464.dbf", block
number 155649 (block size=512)

ARC2: I/O error 19502 archiving log 10 to '/u03/arch/2_1917_656008464.dbf'

ARCH: Archival stopped, error occurred. Will continue retrying

ORACLE
 Instance ifxdb2 - Archival Error

ORA-16038: log 10 sequence# 1917 cannot be archived

ORA-19502: write error on file "", block number  (block size=)

ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'

Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:

ORA-16038: log 10 sequence# 1917 cannot be archived

ORA-19502: write error on file "", block number  (block size=)

ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'

Sun Apr 05 15:05:16 2009

ALTER SYSTEM SET service_names='ifxdb.gointranet.com' SCOPE=MEMORY
SID='ifxdb2';

Immediate Kill Session#: 185, Serial#: 40263

Immediate Kill Session: sess: 0x1274fabc8  OS pid: 13270

Immediate Kill Session#: 187, Serial#: 41391

Immediate Kill Session: sess: 0x1274fd710  OS pid: 27697

Immediate Kill Session#: 191, Serial#: 40464

Immediate Kill Session: sess: 0x127502da0  OS pid:
 30697

Immediate Kill Session#: 195, Serial#: 57362

Immediate Kill Session: sess: 0x127508430  OS pid: 27967

Immediate Kill Session#: 196, Serial#: 2028

Immediate Kill Session: sess: 0x124544048  OS pid: 22900

Immediate Kill Session#: 205, Serial#: 17412

Immediate Kill Session: sess: 0x127515c98  OS pid: 20110

Immediate Kill Session#: 206, Serial#: 14805

Immediate Kill Session: sess: 0x1245518b0  OS pid: 10464

Immediate Kill Session#: 207, Serial#: 52184

Immediate Kill Session: sess: 0x1275187e0  OS pid: 19787

Immediate Kill Session#: 208, Serial#: 62825

Immediate Kill Session: sess: 0x1245543f8  OS pid: 13578

Immediate Kill Session#: 213, Serial#: 36907

Immediate Kill Session: sess: 0x1275209b8  OS pid: 31397

Immediate Kill Session#: 214, Serial#: 49032

Immediate Kill Session: sess: 0x12455c5d0  OS pid: 2427

Immediate Kill Session#: 215, Serial#: 2711

Immediate Kill Session: sess: 0x127523500  OS
 pid: 15942

Immediate Kill Session#: 216, Serial#: 30060

Immediate Kill Session: sess: 0x12455f118  OS pid: 1217

Immediate Kill Session#: 219, Serial#: 35932

Immediate Kill Session: sess: 0x127528b90  OS pid: 27883

Immediate Kill Session#: 222, Serial#: 26007

Immediate Kill Session: sess: 0x1245672f0  OS pid: 1036

Immediate Kill Session#: 223, Serial#: 42462

Immediate Kill Session: sess: 0x12752e220  OS pid: 13726

Immediate Kill Session#: 224, Serial#: 33323

Immediate Kill Session: sess: 0x124569e38  OS pid: 29928

Immediate Kill Session#: 225, Serial#: 49752

Immediate Kill Session: sess: 0x127530d68  OS pid: 20147

Immediate Kill Session#: 227, Serial#: 34834

Immediate Kill Session: sess: 0x1275338b0  OS pid: 9365

Immediate Kill Session#: 230, Serial#: 19879

Immediate Kill Session: sess: 0x124572010  OS pid: 15791

Immediate Kill Session#: 231, Serial#: 16554

Immediate Kill Session: sess: 0x127538f40 
 OS pid: 15490

Immediate Kill Session#: 233, Serial#: 25251

Immediate Kill Session: sess: 0x12753ba88  OS pid: 6972

Immediate Kill Session#: 236, Serial#: 36970

Immediate Kill Session: sess: 0x12457a1e8  OS pid: 12354

Immediate Kill Session#: 244, Serial#: 37284

Immediate Kill Session: sess: 0x124584f08  OS pid: 19290

Immediate Kill Session#: 245, Serial#: 55792

Immediate Kill Session: sess: 0x12754be38  OS pid: 19288

Immediate Kill Session#: 246, Serial#: 25115

Immediate Kill Session: sess: 0x124587a50  OS pid: 3111

Immediate Kill Session#: 247, Serial#: 6416

Immediate Kill Session: sess: 0x12754e980  OS pid: 19471

Immediate Kill Session#: 251, Serial#: 19899

Immediate Kill Session: sess: 0x127554010  OS pid: 21486

Immediate Kill Session#: 252, Serial#: 34731

Immediate Kill Session: sess: 0x12458fc28  OS pid: 30540

Immediate Kill Session#: 253, Serial#: 32638

Immediate Kill Session: sess: 0x127556b58
  OS pid: 5493

Immediate Kill Session#: 259, Serial#: 29155

Immediate Kill Session: sess: 0x12755ed30  OS pid: 29463

Immediate Kill Session#: 261, Serial#: 14481

Immediate Kill Session: sess: 0x127561878  OS pid: 31054

Immediate Kill Session#: 265, Serial#: 37618

Immediate Kill Session: sess: 0x127566f08  OS pid: 868

Immediate Kill Session#: 267, Serial#: 42580

Immediate Kill Session: sess: 0x127569a50  OS pid: 16839

Immediate Kill Session#: 268, Serial#: 50893

Immediate Kill Session: sess: 0x1245a5668  OS pid: 27778

Immediate Kill Session#: 274, Serial#: 34459

Immediate Kill Session: sess: 0x1245ad840  OS pid: 9808

Immediate Kill Session#: 278, Serial#: 59445

Immediate Kill Session: sess: 0x1245b2ed0  OS pid: 28434

Immediate Kill Session#: 281, Serial#: 50119

Immediate Kill Session: sess: 0x12757c948  OS pid: 12606

Immediate Kill Session#: 282, Serial#: 30208

Immediate Kill Session: sess: 0x1245b8560
  OS pid: 17944

Immediate Kill Session#: 285, Serial#: 53580

Immediate Kill Session: sess: 0x127581fd8  OS pid: 16670

Immediate Kill Session#: 286, Serial#: 5929

Immediate Kill Session: sess: 0x1245bdbf0  OS pid: 20149

Immediate Kill Session#: 289, Serial#: 53725

Immediate Kill Session: sess: 0x127587668  OS pid: 14697

Immediate Kill Session#: 290, Serial#: 30378

Immediate Kill Session: sess: 0x1245c3280  OS pid: 19757

Immediate Kill Session#: 293, Serial#: 53710

Immediate Kill Session: sess: 0x12758ccf8  OS pid: 11096

Immediate Kill Session#: 296, Serial#: 34022

Immediate Kill Session: sess: 0x1245cb458  OS pid: 10881

Immediate Kill Session#: 299, Serial#: 53951

Immediate Kill Session: sess: 0x127594ed0  OS pid: 1453

Immediate Kill Session#: 304, Serial#: 15149

Immediate Kill Session: sess: 0x1245d6178  OS pid: 22008

Immediate Kill Session#: 308, Serial#: 34245

Immediate Kill Session: sess:
 0x1245db808  OS pid: 19156

Immediate Kill Session#: 315, Serial#: 15240

Immediate Kill Session: sess: 0x1275aa910  OS pid: 32148

Immediate Kill Session#: 317, Serial#: 41792

Immediate Kill Session: sess: 0x1275ad458  OS pid: 15660

Immediate Kill Session#: 318, Serial#: 7839

Immediate Kill Session: sess: 0x1245e9070  OS pid: 24999

Immediate Kill Session#: 321, Serial#: 4422

Immediate Kill Session: sess: 0x1275b2ae8  OS pid: 16028

Immediate Kill Session#: 324, Serial#: 6833

Immediate Kill Session: sess: 0x1245f1248  OS pid: 21909

Immediate Kill Session#: 332, Serial#: 18018

Immediate Kill Session: sess: 0x1245fbf68  OS pid: 15819

Immediate Kill Session#: 333, Serial#: 37534

Immediate Kill Session: sess: 0x1275c2e98  OS pid: 16433

Immediate Kill Session#: 334, Serial#: 50463

Immediate Kill Session: sess: 0x1245feab0  OS pid: 5660

Immediate Kill Session#: 335, Serial#: 11994

Immediate Kill Session: sess:
 0x1275c59e0  OS pid: 29575

Immediate Kill Session#: 336, Serial#: 26542

Immediate Kill Session: sess: 0x1246015f8  OS pid: 31868

Immediate Kill Session#: 345, Serial#: 46583

Immediate Kill Session: sess: 0x1275d3248  OS pid: 25399

Sun Apr 05 15:05:43 2009

ARCH: Archival stopped, error occurred. Will continue retrying

ORACLE Instance ifxdb2 - Archival Error

ORA-16014: log 10 sequence# 1917 not archived, no available destinations

ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'

Errors in file
/u01/app/oracle/diag/rdbms/ifxdb/ifxdb2/trace/ifxdb2_arc2_15414.trc:

ORA-16014: log 10 sequence# 1917 not archived, no available destinations

ORA-00312: online log 10 thread 2:
'+REDO1/ifxdb/onlinelog/group_10.265.656605479'

Sun Apr 05 15:10:52 2009

kcrrdmx: Successful archiving of previously failed ORL

Archiver process freed from errors. No longer stopped

Sun Apr 05 15:10:53
 2009

ALTER SYSTEM SET service_names='ifxdb.gointranet.com','ifxserv' SCOPE=MEMORY
SID='ifxdb2';

These incidences are all occurring during archiving (redo logs and database
files are using ASM, archiving and backups are on OCFS2). Even though it
usually only lasts a few minutes, it's very noticeable to the customers
because of all the sessions that are terminated. 

What should we be looking at to resolve this problem?
Please let me know if you have any questions.

Thanks,
Diane Petersen
ServerCare, Inc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://oss.oracle.com/pipermail/ocfs2-users/attachments/20090406/c7d505a8/attachment-0001.html 