[Ocfs2-users] re: how should ocfs2 react to nic hardware issue
Peter Santos
psantos at cheetahmail.com
Mon Dec 4 13:42:26 PST 2006
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Adam,
I basically have a client tnsnames.ora file like this (see below)
The entries dbinsto1,dbinsto2 and dbinsto3 are dns entries that point to the
virtual ip managed by Oracle via the racgvip script in CRS_HOME/bin dir.
ORACTAH =
(DESCRIPTION =
(LOAD_BALANCE = ON)
(FAILOVER = ON)
(ADDRESS = (PROTOCOL = TCP)(HOST = dbinsto1)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = dbinsto2)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = dbinsto3)(PORT = 1521))
(CONNECT_DATA =
(SERVER = SHARED)
(SERVICE_NAME = ORACTAH)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 20)
(DELAY = 5)
)
)
)
1. From the client I connect like this: sqlplus user/password at ORACTAH
2. I verify which instance I'm connected to like this: select sys_context('USERENV','INSTANCE_NAME') from dual;
3. Then I did an "ifdown eth0" on the node that I connected above (dbo2)
4. After that I re-tried my query from step 2 and the execution hung for about 15 minutes.
5. After 15 minutes or so, my sqlplus session received the error: ORA-12152: TNS:unable to send break message.
6. After the error message, I got my sqlplus prompt back and re-execute the query from step 2.
This query ran just fine and told me that I was now connected to instance3 on dbo3.
My ocfs2/o2cb service functioned just fine because it's setup to run on eth1 (private interconnect). Only when I
take down eth1 does the o2cb service detect a problem and evicts the node .. but this is probably the proper
behavior.
My concern is that when eth0 goes down the following bad things happened:
- connections to that instance "hang" for a long time. Can probably fixed via some configuration.
- crs_stat reported that the instance was in an "unknown" state.
- the instance unix processes remain up on the machine.
- the listener process remains up on the machine.
- SRVCTL returns the following when I try to check the status of everything on that server.
PRKO-2015 : Error in checking condition of instance on node: dbo2
ASM instance +ASM2 is running on node dbo2.
VIP is not running on node: dbo2
GSD is running on node: dbo2
Listener is not running on node: dbo2
ONS daemon is running on node: dbo2
- The crsd.log on dbo2 shows this
2006-12-04 11:01:01.341: [CRSAPP][1436752224]0CheckResource error for ora.dbo2.vip error code = 1
2006-12-04 11:01:01.343: [CRSRES][1436752224]0In stateChanged, ora.dbo2.vip target is ONLINE
2006-12-04 11:01:01.343: [CRSRES][1436752224]0ora.dbo2.vip on dbo2 went OFFLINE unexpectedly
2006-12-04 11:01:01.344: [CRSRES][1436752224]0StopResource: setting CLI values
2006-12-04 11:01:01.381: [CRSRES][1436752224]0Attempting to stop `ora.dbo2.vip` on member `dbo2`
2006-12-04 11:01:01.714: [CRSRES][1436752224]0Stop of `ora.dbo2.vip` on member `dbo2` succeeded.
2006-12-04 11:01:01.715: [CRSRES][1436752224]0ora.dbo2.vip RESTART_COUNT=0 RESTART_ATTEMPTS=0
2006-12-04 11:01:01.774: [CRSRES][1436752224]0ora.dbo2.vip failed on dbo2 relocating.
2006-12-04 11:01:01.978: [CRSRES][1436752224]0StopResource: setting CLI values
2006-12-04 11:01:02.019: [CRSRES][1436752224]0Attempting to stop `ora.dbo2.LISTENER_DBO2.lsnr` on member `dbo2`
2006-12-04 11:01:02.352: [CRSRES][1436752224]0Stop of `ora.dbo2.LISTENER_DBO2.lsnr` on member `dbo2` succeeded.
2006-12-04 11:01:02.352: [CRSRES][1436752224]0StopResource: setting CLI values
2006-12-04 11:01:02.393: [CRSRES][1436752224]0Attempting to stop `ora.ORACTAH.ORACTAH2.inst` on member `dbo2`
2006-12-04 11:03:09.483: [CRSAPP][1436752224]0StopResource error for ora.ORACTAH.ORACTAH2.inst error code = 1
2006-12-04 11:03:09.544: [CRSRES][1436752224][ALERT]0`ora.ORACTAH.ORACTAH2.inst` on member `dbo2` has
experienced an unrecoverable failure.
2006-12-04 11:03:09.544: [CRSRES][1436752224]0Human intervention required to resume its availability.
I've tried this before and when the session query hangs, if I go to the server and do a shutdown abort
on the instance, then failover takes place and everything is good .. otherwise the instance just appears to
stay up and oracle CRS appears to have no idea on how to deal with it.
- -peter
Adam Kenger wrote:
> Peter - where did your user connect to and what was the status of the
> service and the status of the instances on each node? Was it a straight
> SQL/PLUS connection or was it from an application server? Once
> connected, verify where you are connected to by >select * from
> v$instance; make sure you are disconnecting the node you think you are
> connected to. When you connect via sql/plus to a service, you may not
> always end up on the node that you think you did. My experience however
> suggests that through straight sql/plus connected to a service where the
> TAF policy is set to basic and more than one node is configured as
> "preferred" you should see no interruption in connection when you kill
> one of the nodes.
>
> srvctl status service -s "oractah"
> srvctl status instance -i "instancename1,instancename2,instancename3"
> (fill in your instance names...)
> srvctl status database -d "databasename" (fill in your database name)
>
> Adam
>
>
>
>
>
>
> On Dec 1, 2006, at 11:07 PM, Peter Santos wrote:
>
> Adam,
> thanks for the feedback.
> I wanted to quickly show you my setup because I believe that
> when I took down eth0, my connection hung even though my TAF policy is
> setup
> properly.
>
>
> eth0 - public ip of the machine
> eth0:1 - vip managed by oracle. This is also what clients use to
> connect to.
> dns has entries called dbinsto# that point to the 3 vips on
> my cluster.
>
> eth1 - private ip, used for cluster interconnect and o2cb service
> (cluster.conf)
>
> my client tnsnames.ora
> =======================================================================
> ORACTAH =
> (DESCRIPTION =
> (LOAD_BALANCE = ON)
> (FAILOVER = ON)
> (ADDRESS = (PROTOCOL = TCP)(HOST = dbinsto1)(PORT = 1521)) <-- vip
> (ADDRESS = (PROTOCOL = TCP)(HOST = dbinsto2)(PORT = 1521)) <-- vip
> (ADDRESS = (PROTOCOL = TCP)(HOST = dbinsto3)(PORT = 1521)) <-- vip
> (CONNECT_DATA =
> (SERVER = SHARED)
> (SERVICE_NAME = ORACTAH)
> (FAILOVER_MODE =
> (TYPE = SELECT)
> (METHOD = BASIC)
> (RETRIES = 20)
> (DELAY = 5)
> )
> )
> )
>
> Since the o2cb service operates via eth1 and there is no lost
> connectivity
> to the shared device, o2cb should continue to work just fine, but
> I was sure that the vip on this node did not get moved to another node.
>
> When I repeated this same process on eth1, the o2cb service evicted
> this node
> from the cluster and it was eventually rebooted.. not sure if it was
> the o2cb service that caused it to reboot or oracle's CRS daemons.
>
> I'll keep testing further.
>
> thanks
> -peter
>
>
>
> Adam Kenger wrote:
>>>> Peter - depending on how you have your RAC cluster setup, the hang on
>>>> the front end is not that unexpected. It depends on how the user was
>>>> connected and how the TAF policy was set up. Eth0 is your public
>>>> interface I assume. Was the user connecting to the IP on that interface
>>>> or to the VIP set up by RAC? When you down eth0 the VIP on that
>>>> interface should get pushed over onto one of the other 2 nodes in the
>>>> cluster. If you're connecting to a "service" versus an actual
>>>> "instance" there should be no hang on the front end. If you're actually
>>>> connected directly to the instance on the node, then you'll be out of
>>>> luck if you disconnect that instance. As an example, this is what the
>>>> corresponding tnsnames.ora file looks like :
>>>>
>>>> MYDBSERVICE =
>>>> (DESCRIPTION =
>>>> (ADDRESS_LIST =
>>>> (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))
>>>> (ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521))
>>>> (ADDRESS = (PROTOCOL = TCP)(HOST = node3-vip)(PORT = 1521))
>>>> )
>>>> (CONNECT_DATA =
>>>> (SERVICE_NAME = mydbservice.db.mydomain.com)
>>>> )
>>>> )
>>>>
>>>> MYDB1 =
>>>> (DESCRIPTION =
>>>> (ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 15
>>>> 21))
>>>> (CONNECT_DATA =
>>>> (SERVER = DEDICATED)
>>>> (SERVICE_NAME = mydb.db.mydomain.com)
>>>> (INSTANCE_NAME = mydb1)
>>>> )
>>>> )
>>>>
>>>> If you connected to the service "MYDBSERVICE" you could survive the
>>>> failure of any given node. You'd seamlessly fail-over onto one of the
>>>> other nodes. If you connect directly to the "MYDB1" instance, you'll be
>>>> out of luck if you drop the connection to it.
>>>>
>>>> As far as o2cb goes, you are right I believe. Eventually, it will be
>>>> determined that the node is no longer heartbeating and will either panic
>>>> or reboot.
>>>>
>>>> For your testing, just be careful you're not confusing the OCFS2 layer
>>>> with the Oracle CRS/RAC layers.
>>>>
>>>> Comments welcome....
>>>>
>>>> Hope that helps
>>>>
>>>> Adam
>>>>
>>>>
>>>>
>>>>
>>>> On Nov 30, 2006, at 3:30 PM, Peter Santos wrote:
>>>>
>>>> guys,
>>>> I'm trying to test how my 10gR2 oracle cluster (3 nodes) on SuSe
>>>> reacts to a network card hardware failure.
>>>> I have eth0 and eth1 as my network cards, I took down eth0 (ifdown
>>>> eth0) to see what would happen and
>>>> I didn't get any reaction from the o2cb service. This is probably
>>>> the correct behavior since my
>>>> /etc/ocfs2/cluster.conf uses eth1 as the connection channel?
>>>>
>>>> If I take down eth1 I suspect o2cb will eventually reboot the
>>>> machine right? I'm not using any bonding.
>>>>
>>>> My concern is that when I took down eth0, I had a user logged into
>>>> the instance and everything just "hung" for
>>>> that user, until I manually took down the instance with
>>>> "SRVCTL"... then the user connection failed over to
>>>> a working instance.
>>>>
>>>> Anyway, just trying to get some general knowledge of the behavior
>>>> of o2cb in order to understand
>>>> my testing.
>>>>
>>>> -peter
>>>>
>>>>
>>>>>
> _______________________________________________
> Ocfs2-users mailing list
> Ocfs2-users at oss.oracle.com
> http://oss.oracle.com/mailman/listinfo/ocfs2-users
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFFdJZBoyy5QBCjoT0RAj0UAKCXLPEwhvyrVYJE3DYc2tSSZ1Z26wCfeVAG
rhYtyG3dLAV11j+zbMbzhHE=
=IiVU
-----END PGP SIGNATURE-----
More information about the Ocfs2-users
mailing list