[Ocfs2-users] Lost write in archive logs: has it ever happened?

[Daniel Keisling]

Are you getting something similar the below?  If so, Oracle told us to
turn on Direct I/O on the database.  It's only been on for a week or so,
but we have not encounted the error again.

Mon Oct 20 22:30:15 2008
ARC9: Encountered disk I/O error 19502
Mon Oct 20 22:30:15 2008
ARC9: Closing local archive destination LOG_ARCHIVE_DEST_1:
'/var/opt/oracle/oradata/limsp/arch/3_575_659804930.arc' (error 19502)
 (limsp3)
Mon Oct 20 22:30:15 2008
Errors in file /opt/oracle/admin/limsp/bdump/limsp3_arc9_18776.trc:
ORA-19502: write error on file
"/var/opt/oracle/oradata/limsp/arch/3_575_659804930.arc", blockno 1
(blocksize=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1
Additional information: -1
ORA-19502: write error on file
"/var/opt/oracle/oradata/limsp/arch/3_575_659804930.arc", blockno 1
(blocksize=512)
Mon Oct 20 22:30:15 2008
Errors in file /opt/oracle/admin/limsp/bdump/limsp3_arc9_18776.trc:
ORA-19502: write error on file
"/var/opt/oracle/oradata/limsp/arch/3_575_659804930.arc", blockno 1
(blocksize=512)
ORA-27072: File I/O error
Linux-x86_64 Error: 5: Input/output error
Additional information: 4
Additional information: 1
Additional information: -1
ORA-19502: write error on file
"/var/opt/oracle/oradata/limsp/arch/3_575_659804930.arc", blockno 1
(blocksize=512)
ARC9: I/O error 19502 archiving log 11 to
'/var/opt/oracle/oradata/limsp/arch/3_575_659804930.arc'
ARCH: Archival stopped, error occurred. Will continue retrying
Mon Oct 20 22:30:15 2008
ORACLE Instance limsp3 - Archival Error
Mon Oct 20 22:30:15 2008
ORA-16038: log 11 sequence# 575 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m1.log'
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m2.log'
Mon Oct 20 22:30:15 2008
Errors in file /opt/oracle/admin/limsp/bdump/limsp3_arc9_18776.trc:
ORA-16038: log 11 sequence# 575 cannot be archived
ORA-19502: write error on file "", blockno  (blocksize=)
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m1.log'
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m2.log'
Mon Oct 20 22:30:16 2008
ALTER SYSTEM SET service_names='test','limsp.ppdi.com' SCOPE=MEMORY
SID='limsp3';
Mon Oct 20 22:30:16 2008
ALTER SYSTEM SET service_names='limsp.ppdi.com' SCOPE=MEMORY
SID='limsp3';
Mon Oct 20 22:30:16 2008
Immediate Kill Session#: 166, Serial#: 1681
Immediate Kill Session: sess: 0x1592fcff8  OS pid: 335
Mon Oct 20 22:30:29 2008
ARCH: Archival stopped, error occurred. Will continue retrying
Mon Oct 20 22:30:29 2008
ORACLE Instance limsp3 - Archival Error
Mon Oct 20 22:30:29 2008
ORA-16014: log 11 sequence# 575 not archived, no available destinations
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m1.log'
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m2.log'
Mon Oct 20 22:30:29 2008
Errors in file /opt/oracle/admin/limsp/bdump/limsp3_arca_18780.trc:
ORA-16014: log 11 sequence# 575 not archived, no available destinations
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m1.log'
ORA-00312: online log 11 thread 3:
'/var/opt/oracle/oradata/limsp/redot3g11m2.log'
Mon Oct 20 22:35:31 2008
kcrrdmx: Successful archiving of previously failed ORL
Archiver process freed from errors. No longer stopped
Mon Oct 20 22:35:32 2008 

> -----Original Message-----
> From: ocfs2-users-bounces at oss.oracle.com 
> [mailto:ocfs2-users-bounces at oss.oracle.com] On Behalf Of 
> Silviu Marin-Caea
> Sent: Wednesday, December 03, 2008 9:17 AM
> To: ocfs2-users at oss.oracle.com
> Subject: Re: [Ocfs2-users] Lost write in archive logs: has it 
> ever happened?
> 
> On Monday 22 September 2008 15:02:36 Silviu Marin-Caea wrote:
> > We have 2 nodes with OCFS2 1.2.3 (SLES9).  The archive logs 
> are generated
> > on an OCFS2 volume (mounted with nointr,datavolume).  It 
> has happened 3
> > times in one year that some archivelog had a lost write.  
> We have detected
> > this when applying the archivelogs on the standby database (with
> > dataguard).  We had to copy some datafiles from the 
> production database to
> > the standby and let it resume the recovery process.
> >
> > Has it ever occurred a data loss of this kind (lost write) 
> on an OCFS2
> > volume, version 1.2.3 x86_64?
> >
> > We had 32 bit servers before with OCFS2 that was even older 
> than 1.2.3 and
> > those servers never had such a problem with archivelogs.
> >
> > The storage is Dell/EMC Clariion CX3-40.  The storage on 
> the old servers
> > was CX300.
> >
> > We are worried that this lost writes could occur not only 
> in archivelogs
> > but in the datafiles as well...
> >
> > Not saying that OCFS2 is the cause, the problem might be 
> with something
> > else, but we must investigate everything.
> 
> OCFS2 is not the cause.  The error just occurred again and 
> this time we had 
> the archivelogs multiplexed on both OCFS2 and local storage 
> (reiserfs).  Both 
> archives have identical MD5 sums.
> 
> There was no lost write, just some bullshit that Oracle 
> support tries to feed 
> us.
> 
> There is still an unknown, but it's not related to OCFS2.  
> The unknown is that 
> archives on the standby database have different MD5 sums than 
> the ones on 
> production.  All the archives, not just the corrupt one.  
> Does dataguard 
> intervenes in some way in the archives during transmission?  
> I thought it was 
> just supposed to transfer them unchanged, then apply them on 
> the standby.
> 
> 
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
> 
> 

______________________________________________________________________
This email transmission and any documents, files or previous email
messages attached to it may contain information that is confidential or
legally privileged. If you are not the intended recipient or a person
responsible for delivering this transmission to the intended recipient,
you are hereby notified that you must not read this transmission and
that any disclosure, copying, printing, distribution or use of this
transmission is strictly prohibited. If you have received this transmission
in error, please immediately notify the sender by telephone or return email
and delete the original transmission and its attachments without reading
or saving in any manner.